Node stuck in warmup

Couchbase Server
1

I’m having an issue where after (I believe) a power outage, my node is coming up stuck in warmup and won’t come up. Is anyone able to try to help me recover from this? I looked in the babysitter log and saw this (“aqquotes” is the name of one of our buckets)

memcached<0.140.0>: 2025-03-06T03:14:17.784387+00:00 ERROR 64: Failed to create bucket [aqquotes]: CouchKVStore::initialize: readVBState error:Error, file_name:/opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.181

If there are any other logs that would be helpful please let me know

2

When you create a bucket in Couchbase it creates 1024 file/shards or vBuckets.
The log is talking about vBucket #304. Let’s check to see if your data is still good. The file is a binary file so you have to use a special tool to read the file.

You can use the CLI tool in /opt/couchbase/bin/couch_dbdump

So you want to do /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.18 and hopefully you should see some data.

Looks like someone wrote a Blog about here: Recovering Couchbase data directly from vBuckets - Avira Blog

3

Thank you for the helpful reply!

Running couch_dbdump on the file it is complaining about ( 304.couch.181) returns an error:

Dumping "/opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.181":
Failed to dump database "/opt/couchbase/var/lib/couchbase/data/aqquotes/304.couch.181": error reading file

Total docs: 0

Is there any way to go about repairing these files?

4

Do you have a backup of the bucket using cbbackup ?

When you set up your bucket did you create replica? i.e. a copy of this file on another node.

5

No to both, unfortunately.

Yes, I know that’s a horrible setup. But here we are.

Is there any way to go about recovering the malformed files? cat ing the file shows there still data there, so I’m hoping there’s a way to fix it.

I’m okay with some data loss here but I’m hoping the solution isn’t just to start over

6

To get the cluster back up and running you can delete the file/shard/vBucket that the babysitter is complaining about. On the next write to the cluster to that file the cluster will re-create the file.

I would recommend you do 1 replica in a bucket and use the cbbackup

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.