Databases on Kubernetes – How to Recover from Failures, Scale Up and Down in a Few Line Commands

A month ago, Kubernetes launched a beta for Local Persistent Volumes. In summary, it means that if a Pod using a local disk get killed, no data will be lost (let’s ignore edge cases here). The secret is that a new Pod will be rescheduled to run on the same node, leveraging the disk which already exists there.

Of course, its downside is that we are tying our Pod to a specific node, but if we consider the time and effort spent on loading a copy of the data somewhere else, being able to leverage the same disk becomes a big advantage.

Cloud-native databases, like Couchbase, are designed to handle gracefully nodes or Pods failures. Usually, those nodes are configured to have at least 3 replicas of the data. Therefore, even if you lose one, another one will take over, and the cluster manager or a DBA will trigger a rebalancing process to guarantee that it still has the same 3 copies.

When we put together the Auto-Repair pattern of Kubernetes, Local Persistent Volumes and the recovery process of cloud-native databases, we end up with a very consistent self-healing mechanism. This combination is ideal for use cases which demands high availability, that is why running databases in Kubernetes is becoming such a hot topic nowadays. I have mentioned in a previous blog post some of its advantages, but today, I would like to demonstrate it in action to show you why it is one of the next big things.

Let’s see how easy it is to deploy, recover from pod failures, and scale up and down a database for Kubernetes:

Video Transcription

Configuring your Kubernetes Cluster

Let’s start by configuring your Kubernetes cluster. For this demo, I do not recommend using MiniKube. If you don’t have a cluster for testing, you can create a quick one using tools like Stackpoint.

YAML Files

 All files used in the video are available here:

Deploying Couchbase’s Kubernetes Operator

An Operator in Kubernetes, from a 10000 foot overview, is a set of custom controllers for a given purpose. In this demo, the Operator is responsible for joining new nodes to the cluster, trigger data rebalancing, and correctly scale up and down the database on Kubernetes:

• Configuring permissions:

• Deploying the operator:

You can check the official documentation here.

Deploying a Database on Kubernetes

• Let’s create the username and password that we are going to use to log-in to the web console:

• Finally, let’s deploy our database on Kubernetes by simply executing the following command:

After a few minutes, you will notice that your database with 3 nodes is up and running:

I won’t get into too many details on how couchbase-cluster.yaml works (official doc here). But, I would like to highlight two important sessions in this file:

• The following session specifies the bucket name and the number of replicas of the data:

• The session below specifies the number of servers (3) and which services should run in each node.

Accessing your Database on Kubernetes

There are many ways in which you can expose the web console to the external world. Ingress, for instance, is one of them. However, for the purpose of this demo, let’s simply forward the port 8091 of the pod cb-example-0000 to our local machine

Now, you should be able to access Couchbase’s Web Console on your local machine at http://localhost:8091:

Notice that all three nodes are already talking with each other:

Recovering from a Database Node Failure on Kubernetes

I added some data to illustrate that nothing is lost during the whole process:

Now, we can delete a pod to see how the cluster behaves:

Couchbase will immediately notice that a node “disappeared” and the recovery process will start. As we specified at couchbase-cluster.yaml that we always want 3 servers running, Kubernetes will start a new instance called cb-example-0003:

Once cb-example-003 is up, the operator kicks in to join the newly created node to the cluster, and then triggers data rebalancing

As you can see, no data was lost during this process. Rerunning the same query results in the same number of documents:

Scaling Up a Database on Kubernetes

Let’s scale up from 3 to 6 nodes; all we have to do is change the size parameter on couchbase-cluster.yaml:

Then, we update our configuration by running:

After a few minutes you will see that all the 3 extra nodes were created:

And again, the operator will automatically rebalance the data:

Scaling Down a Database on Kubernetes

The scaling down process is very similar to the scaling up one. All we need to do is to change the size parameter from 6 to 3:

And then, we run the replace command again to update the configuration:

However, there is a small detail here as we can’t just kill 3 nodes at the same time without some risk of data loss. To avoid this problem, the operator scales down the cluster gradually, a single instance at a time, triggering rebalancing to ensure that all the 3 replicas of the data are preserved:

• Operator shutting down node cb-example-0006:

• Operator shutting down node cb-example-0005

• Shutting down node cb-example-0004

• Finally, we are back to 3 nodes again

Multidimensional Scaling

 You can also leverage multi-dimensional scaling by specifying the services you want to run in each node:

 What about other Databases in Kubernetes?

Yes! you can run some of them on Kubernetes already, like MySQL and Postgres as notable examples. They are also trying to automate most of those infrastructure operations we have discussed here. Unfortunately, they are not officially supported yet, so deploying them might not be as simple as this one.

If you want to read more about it, please refer to those two amazing talks at Kubecon:

• Running MySQL on Kubernetes
• Kube-native Postgres

Conclusion

Currently, databases are using Local Ephemeral Storages to store their data (Couchbase included). The reason for it is simple: It is the option which provides the best performance. Some databases are also offering support for Remote Persistent Storages despite the massive latency impact. We are looking forward to Local Persistent Storage going GA, as it will address most of the developers fears with this new trend.

So far, fully-managed databases were the only option you had if you want to get rid of the burden of managing your database. The price of this freedom, of course, comes in the form of some extra zeros in your bill and a very limited performance/architecture control. Leveraging Kubernetes for database scaling and self-management is emerging as a third option, lying between managing everything by yourself and relying on someone else to do it for you.

If you have any questions, please leave it in the comments or tweet me at @deniswsrosa. I will write part 2 of this article to answer all of them.