Internal error from using kubernetes operator

Couchbase Server Kubernetes
1

I am getting the following error when using the kubernetes operator:
Error from server (InternalError): error when creating “./scripts/couchbase-cluster.yaml”: Internal error occurred: failed calling webhook “couchbase-operator-admission.alfa.svc”: Post https://couchbase-operator-admission.alfa.svc:443/couchbaseclusters/mutate?timeout=10s: dial tcp 10.96.147.187:443: connect: connection refused
Error from server (InternalError): error when creating “./scripts/couchbase-cluster.yaml”: Internal error occurred: failed calling webhook “couchbase-operator-admission.alfa.svc”: Post https://couchbase-operator-admission.alfa.svc:443/couchbaseclusters/mutate?timeout=10s: dial tcp 10.96.147.187:443: connect: connection refused

I am not sure that I am setting up kubernetes correctly. I’ve embedded the following in my kubenetes run script:
kubectl create -f ${_SCRIPT_PATH}/crd.yaml
${_SCRIPT_PATH}/cbopcfg create admission
${_SCRIPT_PATH}/cbopcfg create operator
kubectl create -f ${_SCRIPT_PATH}/couchbase-cluster.yaml

where _SCRIPT_PATH is the directory where the scripts are located.

2

Which version are you using?

It’s also good to wait for the admission controller and operator to start up. The installation page walks you through the steps to install and check things are working: Install the Operator on Kubernetes | Couchbase Docs

3

I just downloaded it yesterday. Which step requires waiting, and how long do I need to wait?

4

It should have a version number in it, you may have downloaded an earlier release yesterday is my concern.

The docs show you what to check for, I have some scripts that just wait for the deployments to finish. The docs then show you how to do a dry run check to verify the DAC is working. I would do this first before trying to apply your full cluster.

  echo "Waiting for DAC to complete..."
  until kubectl rollout status deployment couchbase-operator-admission; do
      echo -n "."
      sleep 2
  done
  echo " done"
  echo "Waiting for operator to complete..."
  until kubectl rollout status deployment couchbase-operator; do
      echo -n "."
      sleep 2
  done
  echo " done"

Add any namespaces you are using to the commands.
Once these are done you can do the dry run: Install the Operator on Kubernetes | Couchbase Docs

Any errors have a look at the logs and/or do a kubectl describe ... to see if there is a problem with the set up - it may be a more general networking problem.

5

The version is 2.2.0-250

I will try the wait you suggest.

6

I am getting the same error…
Error from server (InternalError): error when creating “prod-tst-cluster.yaml”: Internal error occurred: failed calling webhook “cbc-couchbase-admission-controller.default.svc”: Post “https://cbc-couchbase-admission-controller.default.svc:443/couchbaseclusters/validate?timeout=10s”: EOF

admission controller log:
{“level”:“info”,“ts”:1636459219.6574273,“logger”:“main”,“msg”:“Mutating resource”,“operation”:“CREATE”,“kind”:“couchbase.com/v2, Kind=CouchbaseCluster”,“namespace”:“default”,“name”:“cb-clust
er”}
{“level”:“info”,“ts”:1636459219.7277,“logger”:“main”,“msg”:“Validating resource”,“operation”:“CREATE”,“kind”:“couchbase.com/v2, Kind=CouchbaseCluster”,“namespace”:“default”,“name”:“cb-cluste
r”}
2021/11/09 12:00:19 http: panic serving 10.0.0.50:37906: runtime error: invalid memory address or nil pointer dereference
goroutine 16 [running]:
net/http.(*conn).serve.func1(0xc00043c640)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangHc7tn/go1.16.3/src/net/http/server.go:1824 +0x153panic(0x13b7b00, 0x1f32f20) /home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangHc7tn/go1.16.3/src/runtime/panic.go:971 +0x499

kernel panic

chart = couchbase-operator-2.2.105 App vrsion = 2.2.1

7

It looks like this is an unrelated issue - the DAC is triggering a panic so the original request from the operator is failing.

We need some more details to see what is going on here, can you collect the support information and provide it?

Is this from a fresh cluster install or is there anything else happening prior to this? Both logs should show the exact version used as well so if you can provide that - it is at the start of each. What configuration are you applying? The cbopinfo tool should capture all this for us, apart from the DAC logs if it is in another namespace.

8

actually I just “solved” it - I redid the CRD installation and reinstalled operator and administrator via cbopcfg
thank you!

9

Hi @Patrick_Stephens, I am facing the same error.
CAO is 2.3.2 and K8s is 1.24.3.

I even verified the CAO deployments which are running.

root@k8s-test01:~/cao232# kubectl rollout status deployment couchbase-operator-admission
deployment “couchbase-operator-admission” successfully rolled out
root@k8s-test01:~/cao232#
root@k8s-test01:~/cao232# kubectl rollout status deployment couchbase-operator
deployment “couchbase-operator” successfully rolled out

I want to attach cbopinfo logs here, however I am not authorized to upload .gz file here!

root@k8s-test01:~/cao232/bin# ./cao collect-logs
no CouchbaseCluster resources discovered in name space default
Wrote cluster information to cbopinfo-20221012T051821+0000.tar.gz