Couchbase Autonomous Operator with Persistent Volumes on Azure AKS

We are super excited to announce industry FIRST NoSQL Database natively integrated with kubernetes (k8s) API, via CRD, giving immense power, agility, cloud portability to the customers so that they can be free from

1. Managing or administering Couchbase Cluster
2. Worrying about node failures, k8s operator spins a pod for you
3. Worrying about performing rebalance operation post node addition, k8s operator performs for you
4. Vendor lock-in

Integrating natively with k8s gives us to define a custom controller, through which we can define workflow(s) for certain conditions that happen on the Couchbase cluster. By writing that logic in custom controller it gives us ability to manage the Couchbase Cluster’s better.

Running Couchbase Autonomous Operator on Azure AKS is currently in technical preview.

Here is what we are going to do

1. Login to Azure with CLI (az login)
2. Create a Resource Group
3. Create a k8s cluster in AKS
4. Access the k8s dashboard (Optional)
5. Deploy Couchbase Autonomous Operator
6. Deploy the Couchbase Cluster in AKS
7. Insert some(~100K) documents in the cluster
8. Scale up the cluster with one command
9. Delete a pod, simulating a node failure
10. See that k8s watches that event, and brings up the new pod, to match the cluster definition
11. References

Now let’s go through the deployment of Autonomous Operator on Azure Kubernetes Service step by step in extremely detailed fashion

Login to Azure with CLI (az login)

Create a Resource Group

Create a k8s cluster

Login to Azure portal, and search for Azure Kubernetes Service in the search icon, we should see this screen

Create K8s Service

Chose the size of the instance for k8s nodes and select the number of nodes as per the requirements

Choose the instance type

Selecting the instance types

Choose the number the k8s worker nodes

Enable RBAC for k8s cluster

Choose RBAC for k8s

Choose default networking

Pick default setting for Networking

Enable container monitoring

Pick default setting for monitoring containers

Make sure validation passes and hit create

Click next and see that validation has passed for k8s cluster

Run command to get k8s cluster running in Azure, locally (I find it personally useful as gives me ability to manage remote k8s cluster running on AKS. Other option would to be use Azure Shell)

Below screenshot gives idea how to check if AKS cluster config is set correct locally or not.

Check for k8s cluster running on AKS

Access the k8s dashboard (Optional)

All the k8s assets can be managed via kubectl and with GUI access to Couchbase cluster, it becomes more manageable. However I would like to do some administration of k8s assets via k8s Dashboard too.

[TIP] If k8s dashboard gives permission issues like

then run the following command

At this time we should have access for k8s dashboard running at URL http://127.0.0.1:8001/

Deploy Couchbase Autonomous Operator

Remember that deploying CB operator is a one time job, and it’s very straightforward

I have downloaded the operator zip files on my local laptop from the URL Download Couchbase Operator Package

Enable RBAC for couchbase cluster in k8s

Deploy the couchbase operator

It becomes ready in less than minute, we can watch the deployment as it is happening

Deploy the Couchbase Cluster in AKS

Now that we have deployed the Autonomous Operator in AKS, now lets deploy the Couchbase Cluster

With AKS, we get StorageClass by default, lets check

For our deployment we will choose managed-premium

Detailed yaml file can be found here, couchbase-persistent-cluster.yaml

It takes few minutes for couchbase cluster to come up and we will be creating 4 pods, and for the first time, it downloads the docker image from registry, perhaps good time to get coffee!

Exposing service cb-op-aks-demo-ui from NodePort(Default to LoadBalancer), gives us ability to access Couchbase Server GUI over public IP

Expose cb-op-aks-demo-ui from NodePort to LoadBalancer

Running command

should give the output like below

Now login to the GUI with default username/password i.e Administrator/password

Login to the GUI on public IP exposed by Load Balancer

Insert some(~100K) documents in the cluster

[TIP] Utility cbc-pillowfight needs to be installed on the container before running it. Its installation is beyond the scope of this blog.

We should see bucket stats chart lit up

Bucket Stats when pillowfight is running

Scale up the cluster with one command

Scaling the couchbase cluster is very simple job with kubernetes, just update the couchbase-persistent-cluster.yaml file, say we want to increase data nodes capacity from 3 to 5, so change servers:size to 5

Scaling the cluster up

We will see the new pods are getting created and pods aks couchbase nodes are getting added to cluster, rebalance is performed, simply amazing!

pods getting created and rebalance is performed

Cluster has been scaled up and new cluster definition looks like this, observer that out of total 100K items, each node has ~20K items/node

New cluster definition

[TIP] Let’s check if we have PV’s in our containers

Login to the pod, and run the command lsblk -a

In couchbase-persistent-cluster.yaml we had defined storage volume to be of size 4GiB, and in the highlighted snippet we see that we have Couchbase directories of 4GiB!

Delete a pod, simulating a node failure

Delete a pod cb-op-aks-demo-0000 from k8s dashboard, its a data service node

Delete a pod from k8s dashboard

With the autoFailoverTimeout variable timeout defined in couchbase-persistent-cluster.yaml, node gets auto-failed over and its detected by Autonomous Operator and hence by the Couchbase Cluster

Node get deleted and failed over

Couchbase custom controller is constantly watching the cluster definition and it needs to have total of 4 nodes or k8s pods, and it sees one is gone, hence it creates a new pod, joins the cluster and rebalance is performed.

New pod is created, joined the cluster and rebalance operation is performed, all of it automatically

From couchbase operator pods logs

rebalance progress in coucbase-operator-xxx logs

Let’s take a moment, to see what happened here!

A node had failed, it was detected by Custom Controller aka Couchbase Operator, it also watches couchbase cluster definition and sees that it needs to have 7 pods or 7 couchbase nodes total, sees one is gone, it waits for autoFailoverTimeout, and spins a new pod, joins the cluster and rebalance operation is performed.

Here at Couchase we believe in sustained differentiation for better serving our customer in terms of agility, performance and cutting edge technology. This is certainly game changer. We will appreciate your feedback, looking forward to making your journey towards microservices a step easy with Couchbase Autonomous Operator on kubernetes running on AKS.

References

https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv

https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster

The new Couchbase Autonomous Operator 1.0 for Kubernetes and OpenShift is now GA!

https://docs.couchbase.com/operator/1.0/overview.html