Couchbase Backup / Restore in K8s Environment

1. Introduction

Periodic backup of data is an important part of any production database deployment, which helps ensure data recovery in the event of a disaster and it also minimizes data inconsistency when a restore is required.

Couchbase provides cbbackupmgr utility that has been improved over the years to become an enterprise-grade Backup and Restore tool to backup large data sets with much higher performance therefore we recommend this tool to be used in the production. It is worth mentioning that in Couchbase Server 6.5 we have overhauled the backup-storage engine completely, and introduced higher compression ratio, which has resulted in much improved backup-restore performance and reduced storage requirements for each backup snapshot, resulting in cost saving.

---

2. Best Practice

Although cbbackupmgr exists under Couchbase_HOME, it is not recommended to run this utility from any of the active nodes in the cluster. As it would be competing for resources of active requests and could potentially hamper the performance of your database system.

It is, therefore, a best practice to provide a separate instance (for backup and restore needs) with only the Couchbase binaries installed but no Couchbase services running, so resources can be better managed for both the database cluster and the backup node.

As can be seen from the above figure, a separate backup/restore node is provisioned in addition to a five node Couchbase cluster. Another best practice is to allocate sufficient storage to hold at least 5x the Couchbase data set size so there is enough space to store required snapshots of the database to meet the Recovery Point Objective (RPO) of the business.

3. Backup Strategy

cbbackupmgr provides a suite of commands which enables DBAs to implement a backup strategy that best suite their business needs. Here are some of the commands:

• cbbackupmgr backup – Backs up data from a Couchbase cluster.
• cbbackupmgr compact – Compacts a backup
• cbbackupmgr merge – Merges backups together
• cbbackupmgr config – Creates a new backup repository
• cbbackupmgr list – Lists backups in the archive

Using these commands one can implement any of the three backup strategies as mentioned in the documentation. In the example below, we will describe Periodic Merge strategy, in context of Couchbase Cluster running within Kubernetes environment.

4. Periodic Merge

This backup strategy is intended to have the lowest database overhead as it requires the least amount of time to backup the changes and practically no resources consumption from the database cluster to consolidate the data during the compaction and merge process (as it happens on the backup node).

On a high level here is how Periodic Merge strategy works:

1. Setup backup repository using cbbackupmgr config
2. Take an incremental backup of the database (in the repository) using cbbackupmgr backup
3. Perform backup compaction using cbbackupmgr compact so that disk space can be efficiently used.
4. Merge ‘n’ oldest backups using cbbackupmgr merge so that the number of backups in the repository doesn’t grow infinitely and space requirements remain under check.

Note: The above steps are captured in the backup-with-periodic-merge.sh script, which we will later use in our Kubernetes setup to take periodic backups.

5. Backup Couchbase Data

In my last blog on Couchbase Autonomous Operator, I have described step-by-step on how to deploy self-healing, highly-available Couchbase cluster using Persistent Volumes. Assuming you have followed those steps and deployed the cluster already, steps below will describe how you can setup automatic backup capability using cronjob. It is considered best practice to regularly backup your data, and also test restoring backups to confirm the restore process before disaster recovery is actually required.

This functionality is not provided by the Operator and left to the cluster administrator to define backup policies and test data restoration. This section describes some common patterns that may be employed to perform the required functions.

5.1. Create Storage Class

The Kubernetes resource definitions below illustrate a typical arrangement for backup that saves the state of the entire cluster. We would need to first define the StorageClass which we will format using xfs for the most optimal performance.

Using above definition in backup-sc.yaml file, we can create storage class like this:

5.2. Create Persistent Volume

A persistent volume is claimed to keep data safe in the event of an outage. You will need to plan the claim size based on your expected data set size, the number of days data retention and whether incremental backups are used at all.

Save above definition in backup-pvc.yaml and create the claim:

5.3. Configure Backup repository

Before we can begin taking snapshots of our data periodically, we need to configure the backup archive location. A job is created to mount the persistent volume and initialize a backup repository. The repository is named couchbase which will map to the cluster name in later specifications.

Save above definition in config.yaml and create backup repository:

5.3. Run Backup as CronJob

Create a cronjob as described in the periodic-backup.yaml file, which takes a backup of the Couchbase cluster by a) downloading the backup script in the pod b) running the script and taking backup of the cluster data using the persistent storage volume.

In the above YAML we are running backup every 5 mins but you can change the frequency so it can meet your business RPO. As our Couchbase cluster is deployed within namespace emart so we will deploy the backup cronjob under the same namespace:

5.4 Validate Periodic Backup Job

At this point, you can begin to watch the cronjob kicking in at every 5 minutes. And once it becomes active it will execute three initContainers (wget-backup-script, chmod-script, periodic-merge) in sequential order followed by the cointainers (delete-script):

You can view the logs of each initContainers after the pod displays status Completed. The initContainers we are interested in is called periodic-merge:

Note: As can be seen from the logs above, before the merge step there were four backups available and after merge there are three backup snapshots that are referred to as RESTOREPOINTS in backup-with-periodic-merge.sh script.

This concludes the backup section.

6. Restoring

Much like a backup, we can restore data to a new Couchbase cluster with a Kubernetes Job.

If you would rather like to create a temporary backup-restore pod to see what backups are available or to troubleshoot an issue, you can mount the same persistentVolumeClaim to a new pod. Here is the definition of the pod which can be stored in backup-pod.yaml:

Run kubectl to bring up the pod temporarily:

Once backup-node is Running, we can login to that pod:

And execute cbbackupmgr list command to view existing backups:

And you can also run cbbackupmgr restore command manually:

Once you are done restoring just delete the pod:

7. Conclusion

We walked through step-by-step on how one can configure a backup cronjob, which automates the process of taking the periodic backup at a predefined interval. We used a backup-with-periodic-merge.sh script, that executes a) backup, b) compaction and c) merge within a single script. This script was then used in the periodic-backup.yaml file, which automated the process of taking backup within the Kubernetes environment. We hope you would use the best practices described in this blog and plan on taking regular backups as well as validate those backups using restore command regularly.