I’m running a CB cluster on Kubernetes. My server version is 4.1 Community.
Now, I’ve made a backup script (and tested it successfully), which I copied to the Docker container running my master instance:
#!/bin/bash
TIMESTAMP=`date +%Y\%m\%d\%H\%M`
OUTPUT=/mnt/couchbase/backups/db-backup-$TIMESTAMP
cbbackup couchbase://localhost:8091 $OUTPUT -m full -u User -p Password;
tar -cvzf $OUTPUT.tar $OUTPUT;
rm -rf $OUTPUT;
I’ve also added a cron rule to run a backup every six hours:
0 */6 * * * /mnt/couchbase/backups/backup-script.sh > /mnt/couchbase/backups/backup-date +\%Y\%m\%d\%H\%M-cron.log 2>&1
Also, my source images are couchbase:community-4.1.0
Now, running the script manually works without any problems, however, for some reason cronjobs are not being triggered (since I don-t even have a log, successful or not).
So, how can I use cron on the Docker container?
Ideally, I’d like to run the cbbackup command from an external source (maybe another server, or even the instance running the containers) but that only gives error: error: SASL auth socket error: 10.32.3.5:11210, user: user.
So, how can I get a reliable way of backing up the database that doesn’t need to run inside a container or how can I make cron work on the container?
Hi, depending on which version of Kubernetes you’re using, I find that Kubernetes Jobs are a good fit to create backups, since it’s much easier to acces Couchbase cluster (running on Kubernetes) from a container that’s running inside the same Kubernetes cluster.
I created a small Docker image with python and couchbase-cli utils in it. I run this as a Job in cluster, it does a cbbackup and uploads the backup to Azure storage, but of course you can do anything with it depending on your needs. I think it would be ideal to mount persistent storage in Job container and perform backups directly there, but that’s not possible in my case (my issue is not related to Kubernetes or Couchbase).
How to run this Job periodically? At first, I had setup a cron job on one of my master instances. The cron job role was to create Kubernetes Job, not actually perform a backup, that was done by the Job itself. The cron job was basically just one line script, something like kubectl create -f job.yaml with some simple sed to edit job name, which has to be unique. Recently I upgraded our cluster to Kubernetes 1.4, which means that now I use Scheduled Job to run my Job peridocally and don’t have to run a cron job from master node. (I wanted to provide a link to Scheduled Job, but was told that new users can only include 2 links :))
I can’t answer your original question, so I provided you with an alternative that I’m using and it’s been working fine so far. Regarding cron jobs running inside containers - I’m not even sure whether it’s possible or not, but I don’t think it’s a good idea.
@lizard has provided an excellent explanation of how you could handle back ups within Kubernetes/Docker!
p.s the link for Scheduled Jobs (from 1.5 known directly as CronJobs) is http://kubernetes.io/docs/user-guide/cron-jobs/
I wanted to explain why backups fail when running outside of a container, this is due to the networking.
By default, all docker containers on a single machine run on their own private network (shown as docker0 within the containers themselves). This means that when your cluster nodes are initialised they pick up the IP address assigned to them on this interface. This IP address is then the address that the other nodes in the cluster use to communicate with the node, this is the address used for this node in the cluster map.
When SDKs or other clients (such as backup) wish to connect to a cluster, they first connect to a single node and then use the information retrieved (the cluster map) to automatically discover and connect to all other nodes in the cluster. This process is known as bootstrapping, although newer clients use a more efficient protocol.
So in this case cbbackup will connect localhost:8091 (which has been forwarded from port 8091 in the container) and read the cluster map provided by the server. In this case it looks like the node’s address in the cluster map is 10.32.3.5, so cbbackup will try and directly connect to the memcached port (11210) for this IP. Unfortunately as the machine is not on the same network, it cannot connect to that IP directly and fails.
I am not too familiar with altering the default docker networking settings, but I imagine it should be possible to create a network which would also be fully accessible from the host machine. See the docker documentation for more details on that, perhaps somebody more knowledgable than I may be able to give you some guidance.