Unable to install couchbase server on kubernetes cluster (memcached issue)

Couchbase Server Kubernetes
1

I am trying to deploy couchbase server on my local kind cluster (podman) using the offical couchbase docker image.

This is my couchbase-cluster.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: couchbase
labels:
app: couchbase
env: dev
spec:
replicas: 1
selector:
matchLabels:
app: couchbase
env: dev
template:
metadata:
labels:
app: couchbase
env: dev
spec:
containers:
- name: couchbase
image: couchbase/server:enterprise-7.2.5
imagePullPolicy: IfNotPresent
ports:
- name: first
containerPort: 8091
- name: second
containerPort: 8092
- name: third
containerPort: 8093
- name: forth
containerPort: 8094
- name: fifth
containerPort: 8085
- name: last
containerPort: 11210

Once I apply my deployment to the cluster with the command kubectl apply -f couchbase-cluster.yaml, the pod starts.

NAME READY STATUS RESTARTS AGE
couchbase-597ddbf7b9-bhwjl 1/1 Running 0 93s

And checking my container logs I have this:
kubectl logs couchbase-597ddbf7b9-bhwjl
Starting Couchbase Server – Web UI available at http://:8091
and logs available in /opt/couchbase/var/lib/couchbase/logs

However while trying to exec into my pod to check other logs (kubectl exec -it couchbase-597ddbf7b9-bhwjl – /bin/bash), I find these errors (couchbase logs present at location /opt/couchbase/var/lib/couchbase/logs)

couchbase.log:
[couchdb:info,2024-05-18T21:28:10.418Z,couchdb_ns_1@cb.local:couch_audit<0.284.0>:couch_log:info:30]Unable to connect to memcahced: {error,{badmatch,{error,econnrefused}}}.
[couchdb:error,2024-05-18T21:28:10.418Z,couchdb_ns_1@cb.local:couch_audit<0.284.0>:couch_log:error:33]Error in sending log messsage to memcached Reason: no_socket

error.log:
[ns_server:error,2024-05-18T21:33:17.318Z,ns_1@cb.local:memcached_config_mgr<0.19772.0>:memcached_config_mgr:push_tls_config:237]Failed to push TLS config to memcached: couldnt_connect_to_memcached
[ns_server:error,2024-05-18T21:33:21.341Z,ns_1@cb.local:prometheus_cfg<0.19154.0>:prometheus_cfg:terminate:544]Terminate: shutdown

memcached.log:
2024-05-18T21:33:21.709373+00:00 INFO ---------- Opening logfile:
2024-05-18T21:33:21.710194+00:00 INFO Couchbase version 7.2.5-7596 starting.
2024-05-18T21:33:21.710212+00:00 INFO Process identifier: 4375
2024-05-18T21:33:21.710223+00:00 INFO recalculate_max_connections: {“engine_fds”:1073675759,“max_connections”:65000,“max_fds”:1073741816,“system_connections”:5000}
2024-05-18T21:33:21.710266+00:00 INFO Breakpad enabled. Minidumps will be written to ‘/opt/couchbase/var/lib/couchbase/crash’
2024-05-18T21:33:21.710302+00:00 INFO Fine clock overhead: 21ns
2024-05-18T21:33:21.710319+00:00 INFO Coarse clock overhead: 6ns)
2024-05-18T21:33:21.710330+00:00 INFO (Clock measurement period: 1ns)
2024-05-18T21:33:21.710523+00:00 INFO Using SLA configuration: {“COMPACT_DB”:{“slow”:“1800 s”},“CREATE_BUCKET”:{“slow”:“5 s”},“DELETE_BUCKET”:{“slow”:“10 s”},“SELECT_BUCKET”:{“slow”:“10 ms”},“SEQNO_PERSISTENCE”:{“slow”:“30 s”},“comment”:“Current MCBP SLA configuration”,“default”:{“slow”:“500 ms”},“version”:1}
2024-05-18T21:33:21.710535+00:00 INFO Enable standard input listener
2024-05-18T21:33:21.710584+00:00 INFO NUMA: Not available - not setting mem policy.
2024-05-18T21:33:21.710596+00:00 INFO Loading RBAC configuration from [/opt/couchbase/var/lib/couchbase/config/memcached.rbac]
2024-05-18T21:33:21.710636+00:00 INFO Loading error maps from [/opt/couchbase/etc/couchbase/kv/error_maps]
2024-05-18T21:33:21.711310+00:00 INFO Prometheus Exporter started, listening on family:inet port:11280
2024-05-18T21:33:21.711325+00:00 INFO Starting external authentication manager
2024-05-18T21:33:21.715214+00:00 WARNING Failed to create audit thread
2024-05-18T21:33:21.715239+00:00 CRITICAL FATAL: Failed to start audit daemon
2024-05-18T21:33:21.715706+00:00 INFO ---------- Closing logfile

Why is it happening? I am not able to port forward properly to setup the cluster on UI too.
I think the couchbase server has not started properly but not sure what is wrong.
Kindly help! Please let me know if any other details are needed.

2

I suspect the issue is that you only have the non-tls ports configured, and it’s trying to use tls. Couchbase Server Ports | Couchbase Docs

18T21:33:17.318Z,ns_1@cb.local:memcached_config_mgr<0.19772.0>:memcached_config_mgr:push_tls_config:237]Failed to push TLS config to memcached: couldnt_connect_to_memcached

3

Hello, Thanks a lot for your response.

I exposed 11207 port (memcached_ssl_port) but I still have these errors.
Just to note that I am not using operator, I am just trying to deploy a simple couchbase pod with 1 node.

From debug.log
Server error during processing: [“web request failed”,
{path,“/poolsStreaming/default”},
{method,‘GET’},
{type,exit},
{what,
{{{{badmatch,
{error,couldnt_connect_to_memcached}},
[{ns_audit_cfg,notify_memcached,1,
[{file,“src/ns_audit_cfg.erl”},
{line,153}]},
{ns_audit_cfg,handle_info,2,
[{file,“src/ns_audit_cfg.erl”},
{line,128}]},
{gen_server,try_dispatch,4,
[{file,“gen_server.erl”},{line,1123}]},
{gen_server,handle_msg,6,
[{file,“gen_server.erl”},{line,1200}]},
{proc_lib,init_p_do_apply,3,
[{file,“proc_lib.erl”},{line,240}]}]},
{gen_server,call,[ns_audit_cfg,get_uid]}},
{gen_server,call,
[menelaus_web_cache,
#Fun<menelaus_web_cache.2.98503217>,
infinity]}}},
{trace,
[{gen_server,call,3,
[{file,“gen_server.erl”},{line,385}]},
{menelaus_web_pools,pool_info,6,
[{file,“src/menelaus_web_pools.erl”},
{line,115}]},
{menelaus_util,
‘-handle_streaming/2-fun-0-’,3,
[{file,“src/menelaus_util.erl”},
{line,607}]},
{menelaus_util,streaming_inner,6,
[{file,“src/menelaus_util.erl”},
{line,676}]},
{menelaus_util,handle_streaming,7,
[{file,“src/menelaus_util.erl”},
{line,653}]},
{request_tracker,request,2,
[{file,“src/request_tracker.erl”},
{line,40}]},
{menelaus_util,handle_request,2,
[{file,“src/menelaus_util.erl”},
{line,234}]},
{mochiweb_http,headers,6,
[{file,
“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl”},
{line,153}]}]}]
[error_logger:info,2024-05-21T08:42:53.197Z,ns_1@cb.local:menelaus_sup<0.619.0>:ale_error_logger_handler:do_log:101]
=========================PROGRESS REPORT=========================
supervisor: {local,menelaus_sup}
started: [{pid,<0.22663.0>},
{id,menelaus_web_cache},
{mfargs,{menelaus_web_cache,start_link,}},
{restart_type,permanent},
{significant,false},
{shutdown,5000},
{child_type,worker}]

[ns_server:debug,2024-05-21T08:42:53.201Z,ns_1@cb.local:ns_audit_cfg<0.22661.0>:ns_audit_cfg:notify_memcached:152]Instruct memcached to reload audit config
[error_logger:info,2024-05-21T08:42:53.201Z,ns_1@cb.local:<0.693.0>:ale_error_logger_handler:do_log:101]
=========================PROGRESS REPORT=========================
supervisor: {<0.693.0>,suppress_max_restart_intensity}
started: [{pid,<0.22661.0>},
{id,ns_audit_cfg},
{mfargs,{ns_audit_cfg,start_link,}},
{restart_type,permanent},
{significant,false},
{shutdown,1000},
{child_type,worker}]

[ns_server:warn,2024-05-21T08:42:53.203Z,ns_1@cb.local:<0.22665.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.
[ns_server:debug,2024-05-21T08:42:53.293Z,ns_1@cb.local:ldap_auth_cache<0.427.0>:active_cache:cleanup:258]Cache ldap_auth_cache cleanup: 0/0 records deleted
[ns_server:warn,2024-05-21T08:42:53.538Z,ns_1@cb.local:<0.22555.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.
[ns_server:warn,2024-05-21T08:42:53.671Z,ns_1@cb.local:<0.22695.0>:ns_memcached:connect:1457]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}.
[ns_server:debug,2024-05-21T08:42:53.671Z,ns_1@cb.local:<0.22695.0>:memcached_refresh:do_refresh:171]Failed to connect to memcached: couldnt_connect_to_memcached
[ns_server:debug,2024-05-21T08:42:53.671Z,ns_1@cb.local:memcached_refresh<0.304.0>:memcached_refresh:update_refresh_state:137]Refresh of [rbac,isasl] failed. Retry in 1000 ms.
[ns_server:warn,2024-05-21T08:42:53.672Z,ns_1@cb.local:<0.22628.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.
[ns_server:warn,2024-05-21T08:42:53.814Z,ns_1@cb.local:<0.22652.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.
[ns_server:warn,2024-05-21T08:42:54.205Z,ns_1@cb.local:<0.22665.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.
[ns_server:warn,2024-05-21T08:42:54.541Z,ns_1@cb.local:<0.22555.0>:ns_memcached:connect:1457]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}.
[ns_server:error,2024-05-21T08:42:54.542Z,ns_1@cb.local:memcached_config_mgr<0.9597.0>:memcached_config_mgr:push_tls_config:240]Failed to push TLS config to memcached: couldnt_connect_to_memcached

Thanks for any feedback.

4

Corrected: Thats on the http port of the management server. For ssl that’s 18091. Just map all the ports.

5

Thanks again. But I don’t find 18901 port specified in the doc, could you please be kind enough to list all the required ports that I need map.

On the contrary the docker run command doesn’t seem to map all the ports(Just asking why we need to expose all ports for this case)

Do you have any example of a deployment file please

6

But I don’t find 18901 port specified in the doc

18091 sorry.

Do you have any example of a deployment file please

I do not.

7

Tried to map all the ports, below is my deplyment file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: couchbase-test
labels:
app: couchbase
env: dev
spec:
replicas: 1
selector:
matchLabels:
app: couchbase
env: dev
template:
metadata:
labels:
app: couchbase
env: dev
spec:
containers:
- name: couchbase
image: couchbase
imagePullPolicy: IfNotPresent
ports:
- name: first
containerPort: 8091
- name: second
containerPort: 8092
- name: third
containerPort: 8093
- name: forth
containerPort: 8094
- name: fifth
containerPort: 8095
- name: six
containerPort: 8096
- name: six1
containerPort: 8097
- name: seven
containerPort: 9123
- name: eight
containerPort: 11207
- name: nine
containerPort: 11210
- name: ten
containerPort: 11280
- name: eleven
containerPort: 18091
- name: twelve
containerPort: 18092
- name: thirteen
containerPort: 18093
- name: fourteen
containerPort: 18094
- name: fifteen
containerPort: 18095
- name: sixteen
containerPort: 18096
- name: seventeen
containerPort: 18097

Still facing this memcached error:

[ns_server:warn,2024-05-22T08:27:39.222Z,ns_1@cb.local:<0.2342.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.
[user:info,2024-05-22T08:27:39.335Z,ns_1@cb.local:<0.518.0>:ns_log:consume_log:76]Service ‘memcached’ exited with status 134. Restarting. Messages:
2024-05-22T08:27:39.328786+00:00 CRITICAL *** Fatal error encountered during exception handling ***
2024-05-22T08:27:39.328839+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Resource temporarily unavailable
terminate called after throwing an instance of ‘std::system_error’
what(): Resource temporarily unavailable
[ns_server:warn,2024-05-22T08:27:39.835Z,ns_1@cb.local:<0.2355.0>:ns_memcached:connect:1457]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}.
[ns_server:warn,2024-05-22T08:27:39.963Z,ns_1@cb.local:<0.2314.0>:ns_memcached:connect:1460]Unable to connect: {error,{badmatch,[{inet,{error,econnrefused}}]}}, retrying.

Thanks for any feedback.

9

But i want to install couchbase on kubernetes cluster without operator.
Please guide me on the same.