Hi I’ve just started to notice a quite painful limitation of doing backups in couchbase (at least in Community Edition I think) I was wondering if there were any workarounds to speed up the process of doing a backup/restore?
When migrating data in some single node cluster dev to dev scenarios we have simply copied the data directory from one machine to another, and that has been good enough.
However I now have a scenario where I’d like to move 50gb of data in a 5 node cluster onto a new 3 node cluster. The above technique doesn’t work any more.
So I’m starting to look at cbbackup
.\cbbackup -m diff http://example.com:8091 /backups/mybackup -u Administrator -p redacted
Which is quite slow (about 90-120 minutes at best)
I made a few observations:
- Running the backup on one of the machines within the cluster is a lot faster than running it on another machine on the same subnet (at least twice as fast). So it’s actually faster to backup onto a cluster machine and then copy the backup onto another networked machine.
- The diff or cumulative backup modes will still generate a full backup and take just as long as a full backup. Someone please correct me if I’m wrong, but I suspect that this is because I have a community edition server, even though there are no warnings about this in cbbackup and the resulting folder even has diff in the name. (Would be nice if this was made clearer in the application or the documentation.)
- There is an option -t to specify the number of threads, I didnt notice much difference when changing this value to 4 on a 4 core machine, is there a sweet spot to aim for here, or is the default setting good enough?
I’m a bit surprised how cumbersome this is, even in a so called free product. Have I missed something? Is there any other options I could try out to speed things up? I also wondered if I could reduce the number of nodes on my source cluster to match the target cluster, and then go with the filecopy approach.
Thanks in advance!