Hello
*Because it translates from Japanese, expression may be wrong. I’m sorry.
We are running a 3 node cluster of couchbase community edition 3.0.1-1444.x86_64.
I restarted one of the clusters for maintenance, but after maintenance work I noticed that some data was missing.
I want to take measures to prevent the same thing happening, but I am in trouble because I do not know the cause of the problem.
I would like to know if there is a possible cause or method of investigation.
During the maintenance work I did the following work without stopping reading / writing to the couchbase cluster.
- Failover (Graceful)
- Node OS Reboot
- Node Delta Recovery
- Cluster Rebalance
When searching the file output by cbbackup with sqlite3, it seems that it was in a state that it could not be seen via couchbase cluster because data judged to be missing existed.
Data when missing was confirmed from couchbase console DataBuckets> bucket> Documents
"Error: notfound (Document does not exist)" was displayed.
I thought that data loss will not occur because Graceful Failover is running, but is there a mistake in the work procedure?
We addressed by recovering the missing data from cbbackup and overwriting the key.
Since the version we use is old, there may be problems that have been solved.
We plan to implement it preferentially if there is a chance that it will be resolved by upgrading the latest version.
Thanks