I’m having an issue where after (I believe) a power outage, my node is coming up stuck in warmup and won’t come up. Is anyone able to try to help me recover from this? I looked in the babysitter log and saw this (“aqquotes” is the name of one of our buckets)
memcached<0.140.0>: 2025-03-06T03:14:17.784387+00:00 ERROR 64: Failed to create bucket [aqquotes]: CouchKVStore::initialize: readVBState error:Error, file_name:/opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.181
If there are any other logs that would be helpful please let me know
When you create a bucket in Couchbase it creates 1024 file/shards or vBuckets.
The log is talking about vBucket #304. Let’s check to see if your data is still good. The file is a binary file so you have to use a special tool to read the file.
You can use the CLI tool in /opt/couchbase/bin/couch_dbdump
So you want to do /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.18 and hopefully you should see some data.
Looks like someone wrote a Blog about here: Recovering Couchbase data directly from vBuckets - Avira Blog
Thank you for the helpful reply!
Running couch_dbdump on the file it is complaining about ( 304.couch.181) returns an error:
Dumping "/opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.181":
Failed to dump database "/opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.181": error reading file
Total docs: 0
Is there any way to go about repairing these files?
Do you have a backup of the bucket using cbbackup ?
When you set up your bucket did you create replica? i.e. a copy of this file on another node.
No to both, unfortunately.
Yes, I know that’s a horrible setup. But here we are.
Is there any way to go about recovering the malformed files? cat ing the file shows there still data there, so I’m hoping there’s a way to fix it.
I’m okay with some data loss here but I’m hoping the solution isn’t just to start over
To get the cluster back up and running you can delete the file/shard/vBucket that the babysitter is complaining about. On the next write to the cluster to that file the cluster will re-create the file.
I would recommend you do 1 replica in a bucket and use the cbbackup