Failed calling webhook admission controller during cluster creation

Couchbase Server Kubernetes
1

Trying to create a fresh cluster using helm chart of autonomous operator. Getting an error in operator pod

couchbase-operator {"level":"info","ts":1650599320.0278394,"logger":"main","msg":"couchbase-operator","version":"2.3.0 (build 301)","revision":"207fa7b86260bc7c75c1c1868cda753370991e64"}
couchbase-operator {"level":"info","ts":1650599321.079249,"msg":"Waited for 1.032641718s due to client-side throttling, not priority and fairness, request: GET:https://10.206.0.1:443/apis/cloud.google.com/v1?timeout=32s\n"}
couchbase-operator {"level":"info","ts":1650599322.7341886,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"0.0.0.0:8383"}
couchbase-operator {"level":"info","ts":1650599322.7356246,"msg":"attempting to acquire leader lease vish/couchbase-operator...\n"}
couchbase-operator {"level":"info","ts":1650599322.7356772,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":":8383"}
couchbase-operator {"level":"info","ts":1650599377.839182,"msg":"successfully acquired lease vish/couchbase-operator\n"}
couchbase-operator {"level":"info","ts":1650599377.8396215,"logger":"controller.couchbase-controller","msg":"Starting EventSource","source":"kind source: *v2.CouchbaseCluster"}
couchbase-operator {"level":"info","ts":1650599377.8396907,"logger":"controller.couchbase-controller","msg":"Starting Controller"}
couchbase-operator {"level":"info","ts":1650599377.9407725,"logger":"controller.couchbase-controller","msg":"Starting workers","worker count":4}
couchbase-operator {"level":"info","ts":1650599377.9412856,"logger":"cluster","msg":"Watching new cluster","cluster":"vish/couchbase-cluster"}
couchbase-operator {"level":"info","ts":1650599410.012061,"logger":"cluster","msg":"Resource updated","cluster":"vish/couchbase-cluster","diff":"  string(\n- \t\"size: 0\\n\",\n+ \t\"conditions:\\n- lastTransitionTime: \\\"2022-04-22T03:50:10Z\\\"\\n  lastUpdateTime: \\\"2022-04-22T03:50:10Z\\\"\\n  message: 'Internal error occurred: failed calling webhook \\\"vish-couchbase-admission-controller.vish.svc\\\":\\n    Post \\\"https://vish-couchbase-admission-control\"...,\n  )\n"}
couchbase-operator {"level":"info","ts":1650599420.025339,"logger":"cluster","msg":"unable to update status","cluster":"vish/couchbase-cluster","error":"Internal error occurred: failed calling webhook \"vish-couchbase-admission-controller.vish.svc\": Post \"https://vish-couchbase-admission-controller.vish.svc:443/couchbaseclusters/validate?timeout=10s\": context deadline exceeded"}
couchbase-operator {"level":"error","ts":1650599420.0254312,"logger":"controller","msg":"Failed to create Couchbase cluster","cluster":{"namespace":"vish","name":"couchbase-cluster"},"error":"Internal error occurred: failed calling webhook \"vish-couchbase-admission-controller.vish.svc\": Post \"https://vish-couchbase-admission-controller.vish.svc:443/couchbaseclusters/validate?timeout=10s\": context deadline exceeded","stacktrace":"github.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:76\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
couchbase-operator {"level":"error","ts":1650599420.025517,"logger":"controller.couchbase-controller","msg":"Reconciler error","name":"couchbase-cluster","namespace":"vish","error":"Internal error occurred: failed calling webhook \"vish-couchbase-admission-controller.vish.svc\": Post \"https://vish-couchbase-admission-controller.vish.svc:443/couchbaseclusters/validate?timeout=10s\": context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
couchbase-operator {"level":"info","ts":1650599420.0313213,"logger":"cluster","msg":"Watching new cluster","cluster":"vish/couchbase-cluster"}

Any idea why this happens? I was able to make a curl request to the admission controller pod from a different pod but not sure why operator complains

2

Hi @Vishvajit_Kher It’s possible that you already had an admission controller running elsewhere? If that’s the case the new one will not be able to register. Also, it please share logs from the Admission Operator?

3

If you look at the architecture documentation Couchbase Operator Architecture | Couchbase Docs then you’ll note that the operator has nothing directly to do with the DAC. In this case the operator is the client, and the error is coming from the Kubernetes API as it’s not able to contact the service.

Now take that information, and the fact it was a timeout, and that suggests there is a network blackhole somewhere. So, you’ll need to provide a lot more information. What platform and version are you using? What CNI provider and version are you using? Is there some funky network rules you need to mention? If so does any of Kubernetes Network Policies Using Deny-All Default | Couchbase Docs help?