Couchbase Cluster through kubernetes deployment fails

Couchbase Server Kubernetes
1

Am trying to deploy couchbase cluster for a PoC in the following environment, used the YAML that is available in the operator by default.
OS Version :RHEL 8.4
Kuberneters Version : 1.22.12
Couchbase Operator Version : 2.3.1
Couchbase Server Version: 7.0.3

Kuberentes Setup:
NAME STATUS ROLES AGE VERSION
vmy2061.ete.ka.sw.se Ready control-plane,master 2d3h v1.22.12
vmy2062.ete.ka.sw.se Ready worker1 2d3h v1.22.12
vmy2063.ete.ka.sw.se Ready worker2 2d3h v1.22.12
vmy2064.ete.ka.sw.se Ready worker3 2d3h v1.22.12

YAML:
apiVersion: v1
kind: Secret
metadata:
name: cb-example-auth
type: Opaque
data:
username: QWRtaW5pc3RyYXRvcg== # Administrator
password: cGFzc3dvcmQ= # password

apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
name: default

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
name: cb-example
spec:
image: vmy2062.ete.ka.sw.se:5000/couchbase/server:7.0.3 “Created a local registry and pushed the image to the registry it is up and running”
security:
adminSecret: cb-example-auth
networking:
exposeAdminConsole: true
adminConsoleServices:
- data
buckets:
managed: true
servers:
- size: 4
name: all_services
services:
- data
- index
- query
- search
- eventing
- analytics
Couchbase Server Status:
NAME READY STATUS RESTARTS AGE
cb-example-0000 1/1 Running 0 6m7s
couchbase-operator-5f797df6d6-hkjc5 1/1 Running 0 15h
couchbase-operator-admission-cc4f8b989-44k8g 1/1 Running 0 15h

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cb-example-0000 1/1 Running 0 7m2s 192.168.66.173 vmy2062.ete.ka.sw.se 0/1
couchbase-operator-5f797df6d6-hkjc5 1/1 Running 0 15h 192.168.165.191 vmy2063.ete.ka.sw.se
couchbase-operator-admission-cc4f8b989-44k8g 1/1 Running 0 15h 192.168.165.190 vmy2063.ete.ka.sw.se

From the YAML couchbase cluster should deploy 4 pods one in each node, but only one pod is getting deployed and keeps on restarting.
Found the error in below operator log,

{“level”:“info”,“ts”:1660200131.37507,“logger”:“cluster”,“msg”:“Reconciliation failed”,“cluster”:“default/cb-example”,“error”:“fail to create member’s pod (cb-example-0000): dial tcp: i/o timeout”,“stack”:“github.com/couchbase/couchbase-operator/pkg/util/netutil.WaitForHostPort\n\tgithub.com/couchbase/couchbase-operator/pkg/util/netutil/netutil.go:37\ngithub.com/couchbase/couchbase-operator/pkg/util/k8sutil.WaitForPod\n\tgithub.com/couchbase/couchbase-operator/pkg/util/k8sutil/k8sutil.go:289\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).waitForCreatePod\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/pod.go:108\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).createPod\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/pod.go:41\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).createMember\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/member.go:168\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).createInitialMember\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/member.go:318\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).create\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:325\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:148\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227”}

can you please help in resolving the issue? Please revert in case if you need further information.

2

The port check is failing.

3

Is it 8091? there is no much detail in the logs, any way forward on addressing the issue?

4
5

I got the following error in goxdcr.log as well as in http_access_internal.log for goxdcr, other requests for promotheus are serving.

2022-08-12T07:20:24.579Z ERRO GOXDCR.TopoSvc: Original REST request (http://127.0.0.1:8091/pools/nodes) received “unknown pool” response. The URL may be incorrect or requested resource no longer exists
2022-08-12T07:20:34.568Z INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration:
None dcpAction: Reset processCpu: 0 idleCpu: 99

8091 Port Status in the running pod:
[couchbase@cb-example-0000 /]$ netstat -plant | grep 8091
tcp 0 0 0.0.0.0:8091 0.0.0.0:* LISTEN 217/beam.smp
tcp 0 0 0.0.0.0:18091 0.0.0.0:* LISTEN 217/beam.smp
tcp 0 0 127.0.0.1:53926 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:53934 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:53924 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:53912 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:53912 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:53936 127.0.0.1:8091 ESTABLISHED 472/prometheus
tcp 0 0 127.0.0.1:54060 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:53928 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:53958 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:53928 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:53910 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:53916 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:54060 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:53934 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:53922 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp 0 0 127.0.0.1:53910 127.0.0.1:8091 ESTABLISHED 492/saslauthd-port
tcp 0 0 127.0.0.1:8091 127.0.0.1:53916 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:53936 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:53926 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:53958 127.0.0.1:8091 ESTABLISHED 472/prometheus
tcp 0 0 127.0.0.1:8091 127.0.0.1:53922 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:53924 127.0.0.1:8091 ESTABLISHED 505/goxdcr
tcp6 0 0 :::8091 :::* LISTEN 217/beam.smp
tcp6 0 0 :::18091 :::* LISTEN 217/beam.smp

6

[couchbase@cb-example-0000 /]$ netstat -plant | grep 8091
tcp 0 0 0.0.0.0:8091 0.0.0.0:* LISTEN 217/beam.smp
tcp 0 0 0.0.0.0:18091 0.0.0.0:* LISTEN 217/beam.smp
tcp 0 0 127.0.0.1:47138 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:47138 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:47142 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:47150 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:47156 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:47136 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:47168 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:47150 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:47166 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:47168 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:47170 127.0.0.1:8091 ESTABLISHED 485/prometheus
tcp 0 0 127.0.0.1:8091 127.0.0.1:47286 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:47166 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:47286 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:47148 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:47170 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:47156 127.0.0.1:8091 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:8091 127.0.0.1:47148 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:8091 127.0.0.1:47142 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:47176 127.0.0.1:8091 ESTABLISHED 485/prometheus
tcp 0 0 127.0.0.1:9123 127.0.0.1:38091 TIME_WAIT -
tcp 0 0 127.0.0.1:47136 127.0.0.1:8091 ESTABLISHED 506/saslauthd-port
tcp 0 0 127.0.0.1:8091 127.0.0.1:47176 ESTABLISHED 217/beam.smp
tcp6 0 0 :::8091 :::* LISTEN 217/beam.smp
tcp6 0 0 :::18091 :::* LISTEN 217/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 18091
tcp 0 0 0.0.0.0:18091 0.0.0.0:* LISTEN 217/beam.smp
tcp6 0 0 :::18091 :::* LISTEN 217/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 8092
tcp 0 0 0.0.0.0:8092 0.0.0.0:* LISTEN 364/beam.smp
tcp 0 0 0.0.0.0:18092 0.0.0.0:* LISTEN 364/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 18092
tcp 0 0 0.0.0.0:18092 0.0.0.0:* LISTEN 364/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 8094
[couchbase@cb-example-0000 /]$ netstat -plant | grep 8095
[couchbase@cb-example-0000 /]$ netstat -plant | grep 8096
[couchbase@cb-example-0000 /]$ netstat -plant | grep 8097
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9100
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9101
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9102
[couchbase@cb-example-0000 /]$ netstat -plant | grep 19102
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9103
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9104
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9105
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9110
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9111
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9112
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9113
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9114
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9115
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9116
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9117
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9118
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9119
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9120
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9121
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9122
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9123
tcp 0 0 127.0.0.1:9123 0.0.0.0:* LISTEN 485/prometheus
tcp 0 0 127.0.0.1:9123 127.0.0.1:34133 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:43165 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:38845 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:37953 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:35657 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:39249 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:46657 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:33011 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:34601 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:44693 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:36193 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:44697 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:43605 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:44269 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:42279 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:44499 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:43061 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:36113 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:39833 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:33685 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:45797 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:44395 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:46625 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:40365 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:39073 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:35165 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:40641 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:34159 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:41727 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:37029 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:34631 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:45771 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:44753 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:43621 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:34997 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:35723 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:36031 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:37177 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:36975 TIME_WAIT -
tcp 0 0 127.0.0.1:9123 127.0.0.1:46785 TIME_WAIT -
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9124
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9130
tcp 0 0 127.0.0.1:11280 127.0.0.1:39130 TIME_WAIT -
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9140
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9998
tcp 0 0 127.0.0.1:9998 0.0.0.0:* LISTEN 519/goxdcr
tcp 0 0 127.0.0.1:9998 127.0.0.1:51862 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:9998 127.0.0.1:52791 ESTABLISHED 519/goxdcr
tcp 0 0 127.0.0.1:51862 127.0.0.1:9998 ESTABLISHED 485/prometheus
tcp 0 0 127.0.0.1:52791 127.0.0.1:9998 ESTABLISHED 217/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 9999
[couchbase@cb-example-0000 /]$ netstat -plant | grep 11209
tcp 0 0 0.0.0.0:11209 0.0.0.0:* LISTEN 521/memcached
tcp 0 0 127.0.0.1:11209 127.0.0.1:59049 ESTABLISHED 521/memcached
tcp 0 0 127.0.0.1:59049 127.0.0.1:11209 ESTABLISHED 364/beam.smp
tcp6 0 0 :::11209 :::* LISTEN 521/memcached
[couchbase@cb-example-0000 /]$ netstat -plant | grep 11210
tcp 0 0 0.0.0.0:11210 0.0.0.0:* LISTEN 521/memcached
tcp6 0 0 :::11210 :::* LISTEN 521/memcached
[couchbase@cb-example-0000 /]$ netstat -plant | grep 11280
tcp 0 0 127.0.0.1:11280 0.0.0.0:* LISTEN 521/memcached
tcp 0 0 127.0.0.1:11280 127.0.0.1:39256 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39216 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39278 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39234 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39258 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39300 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39320 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39236 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39298 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39276 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39318 TIME_WAIT -
tcp 0 0 127.0.0.1:11280 127.0.0.1:39214 TIME_WAIT -
[couchbase@cb-example-0000 /]$ netstat -plant | grep 21100
tcp 0 0 0.0.0.0:21100 0.0.0.0:* LISTEN 217/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 21200
tcp 0 0 127.0.0.1:21200 0.0.0.0:* LISTEN 9/beam.smp
tcp 0 0 127.0.0.1:50099 127.0.0.1:21200 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:21200 127.0.0.1:50099 ESTABLISHED 9/beam.smp
[couchbase@cb-example-0000 /]$ netstat -plant | grep 21300
tcp 0 0 127.0.0.1:21300 0.0.0.0:* LISTEN 364/beam.smp
tcp 0 0 127.0.0.1:46003 127.0.0.1:21300 ESTABLISHED 217/beam.smp
tcp 0 0 127.0.0.1:21300 127.0.0.1:46003 ESTABLISHED 364/beam.smp

These are the list of port number documented, i have verified in pod certain ports are listening and certain are not. but the port used for node to node communication none of them are listening.

7

Any update here. Installation is stuck due to the above issue.