How do I move data from a problematic node to a new node in the same cluster?

Couchbase Server
1

Most of the cbtransfer examples are geared towards moving data between clusters or from a cluster to disk.

I want to move data from a faulty node to a brand new machine/node, the new node has joined the cluster but I can’t rebalance since it keeps failing.

I’m trying to move the data from the problematic node to a new node, remove the problematic node and try rebalancing again.

What would you do if rebalancing wasn’t an option?

Log messages from the failed rebalance operation:

  1. https://pastebin.com/1KcRfTGy “exited with unexpected_exit”
  2. https://pastebin.com/PJBfWdzT “mover_crashed”
2

How many nodes do you have in the cluster?

Are you able to remove any other node?

I can think of the following ways:

  1. By any chance, is this a single node Development cluster? If so, instead of adding a new node to this cluster, create a new single node cluster with the new node and configure XDCR between the 2 clusters. Data will move from old cluster to new cluster.

  2. If it is a multi-node cluster and its only 1 node that is gone bad, then do a swap-rebalance i.e. remove old node and add new node together. And then click rebalance. It will move/transfer data from the outgoing node to the incoming node. However this too requires rebalance. Rebalance will be required in all different ways that you can try for your situation.

  3. Do a cbbackup from old cluster then remove all nodes or simply destroy the cluster and recreate the cluster and do a cbrestore.

There might be more ways which we can think of once you share your env details.

HTH