Operator 2.0 on MicroK8s

Couchbase Server Kubernetes
1

Hi,

I have used operator 2.0 (and also 1.2) and deployed it on Microk8s (installed on ubuntu VM).
I have enabled storage, helm, dns and dashboard of the Microk8s add-on.
After deployed operator I have deployed a couchbase cluster with 3 servers (pods).
The 3 pods running ok.
Then I used a spring-boot-miroservices to access at the coucnhbase cluster and I have’nt problems.

When suspended the VM and the Microk8s restart I found the following situation:

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
couchbase-operator-7b75d445d9-qfz29 1/1 Running 0 20m
couchbase-operator-admission-6d45d566d4-fv4hp 1/1 Running 2 21h
rec-cb-0000 0/1 Unknown 0 18h
rec-cb-0001 0/1 Unknown 0 18h
rec-cb-0002 0/1 Unknown 0 18h

The status of server pod is Unknown.
The logs of operator reports the following error:

time=“2020-05-13T07:35:47Z” level=error msg=“failed to update members: cluster does not have any Pods that are running or recoverable” cluster-name=rec-cb module=cluster

Why, what is happening?

Thanks a lot
Luigi

2

Microk8s is an unsupported platform, so its behavior is totally untested. What I can tell you, is that on all other platforms, when the VM is brought down and back up, all the pods will be cleared up by Kubernetes. The expectation here is that the Deployment/StatefulSet controllers will bring the pods back. Likewise the Operator expects there to be no pods when recovering. Please note, in order to recover after a “lights-out” incident you need to have been using persistent volumes.

To work around your problem, stop the operator (kubectl scale --replicas=0 deployment/couchbase-operator). Delete the unknown pods (for i in {0…2}; do kubectl delete pod rec-cb-000${i}; done), then restart the Operator (kubectl scale --replicas=1 deployment/couchbase-operator).

3

Hi,
Thanks for your response.

I enabled into the cluster deployment the persistent volumes:

…
servers:
- size: 3
name: all_services
services:
- data
- index
- query
- search
volumeMounts:
default: couchbase
volumeClaimTemplates:

  • metadata:
    name: couchbase
    spec:
    storageClassName: “microk8s-hostpath”
    resources:
    requests:
    storage: 1Gi

Where the storage class configured are the following:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
microk8s-hostpath (default) microk8s.io/hostpath Delete Immediate false 8d
standard-lazy-bound microk8s.io/hostpath Delete WaitForFirstConsumer true 80m

I have used also standard-lazy-bound as reported into the your documentations

When I stop K8s and restart it the result is the same.
kubectl get pods
NAME READY STATUS RESTARTS AGE
couchbase-operator-7b75d445d9-458xz 1/1 Running 1 27m
couchbase-operator-admission-6d45d566d4-fv4hp 1/1 Running 6 30h
rec-cb-0000 0/1 Unknown 0 14m
rec-cb-0001 0/1 Unknown 0 14m
rec-cb-0002 0/1 Unknown 0 14m

Any idea?

Thanks