Dynamic IP node down

Couchbase Server
1

Hi guys,
We had to shut down our Couchbase server nodes for maintenance and “forgot” that one of the nodes has a dynamic IP. The problem is that since the IP, and DNS (provided by AWS) have changed that node will not become active again.
I know that we should have removed the node and performed a rebalance before shutting it down, but we slipped this one. I thought about failing over the node, but the data is not complete on the other nodes, so I am stuck and do not know how I can get out of this. Is there any way we can force Couchbase to change the Server Node Name of a down node so it reintegrates the cluster?

We are running Couchbase version 4.1.0-5005 Enterprise Edition

Any help will be appreciated.
Thanks

2

I’m not really aware of any voodoo to easily change node names. I think it would need to be done on the other nodes in the cluster which is kind of risky. I think the best solution would be to use cbtransfer to copy the documents from the database files on disk on the ‘dynamic’ node and into the remainder of the cluster.

You can find cbtransfer in the Couchbase bin directory:

$ export PATH=$PATH:/opt/couchbase/bin
$ cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data/ couchbase://<ip-in-rest-of-cluster>:8091 -b <bucket-name> -B <bucket-name>

Note this will also copy all of the replica data from that node. Since you were in a position to just shutdown a node without failing over or rebalancing I assume there was no traffic so this shouldn’t cause any issues as the replicas would be up to date. If that would be an issue then you can use the vbucket flag ‘-i’ to transfer each active vbucket one by one. You will need to failover that node (and rebalance) before you do this.

I recommend that you make a backup of the working nodes before you do this.

3

Good morning Will, and thanks for your answer.
We have resolved the issue with Couchbase support, it turns out that the message telling us that the data on that node was not fully available was not relevant on this time; they examined our logs and found out so. Therefore we were able to simply failover and read the server node.

I will keep the cbtransfer information on hand though because it might prove useful at some point.

Thanks