Oprator 2.0 not creating the cluster properly in own cloud k8s

Couchbase Server Kubernetes
1

Hi , I am facing the cluster creating issue using Operator version 2. In some namespaces the cluster is running fine using the admission controller and operator. But using the same chart only changing the release version the operator trying to create the cluster pod but only one pod will come-up and it wait for unhealthy for long time and deleted it. Below is the details. And please help to find out the root cause.

Inside the pod services are running
edge-couchbase-cluster-0000:/opt/couchbase/bin# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:21300 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:18091 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:21100 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:21200 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN -
tcp6 0 0 :::21300 :::* LISTEN -
tcp6 0 0 :::18091 :::* LISTEN -
tcp6 0 0 :::21100 :::* LISTEN -
tcp6 0 0 :::21200 :::* LISTEN -
tcp6 0 0 :::4369 :::* LISTEN -

POD status
edge-couchbase-admission-controller-6c474c7c64-krtp2 1/1 Running 0 27m
edge-couchbase-cluster-0000 0/1 Running 0 27m
edge-couchbase-operator-74f867ff4c-wzwpp 1/1 Running 3

Operator Logs

{“level”:“info”,“ts”:1594381584.4862847,“logger”:“cluster”,“msg”:“Watching new cluster”,“cluster”:“cmp-edge-usage-analytics/edge-couchbase-cluster”}
{“level”:“info”,“ts”:1594381584.4864922,“logger”:“cluster”,“msg”:“Janitor starting”,“cluster”:“cmp-edge-usage-analytics/edge-couchbase-cluster”}
{“level”:“error”,“ts”:1594381584.5791261,“logger”:“cluster”,“msg”:“Cluster setup failed”,“cluster”:“cmp-edge-usage-analytics/edge-couchbase-cluster”,“error”:“unexpected cluster phase: Failed”,“stacktrace”:"github.com/couchbase/couchbase-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/couchbase/couchbase-operator/pkg/cluster

2

That log entry is far too new. Do you have the logs for the first error encountered?

If not, you can fake it by:

  • stopping the operator.
  • doing kubectl edit -n cmp-edge-usage-analytics configmap/edge-couchbase-cluster then edit the phase key to be Running.
  • restarting the operator.

This will then either work, or tell you why it’s getting into the Failed state in the first place.

3

Thanks for the help Simon. I could found the issue is solved after changing the helm values.yaml file option use the username and password from the values.yaml file entry.

I have used the secret name option in the first and I have noticed that some miss communication is happening and only one cluster pod will come up in running state. So I juts removed that from helm values file fixed the issue. Now the cluster is running successfully in different namespaces without any issues.