Cbrestore error on restoreIndexMetadata

Couchbase Server
1

Hi
I get the error stated below when restoring a backup taken from a different CB server.

2018-07-31 20:50:32,704: mt “POST /restoreIndexMetadata?bucket= HTTP/1.1” 500 558
2018-07-31 20:50:32,705: mt [‘Internal server error, please retry your request’]

Due to this, it does not restore index metadata. I’m using the CB Enterprise Edition 5.1.0 build 5552.

Appreciate any feedback to resolve this.

Thanks.

2

Hi Bathige,

Can you please mention the exact commands used for backup and restore of the couchbase data? Also, can you please attach the error log messages in indexer.log file, at the time of restore error?

Thanks.

3

Hi Amit,
It is pretty easy to recreate the issue for anyone. Please follow the steps below;

  1. Load one of the sample buckets. e.g: travel-sample.
  2. Backup the travel-sample bucket using the command below;
    cbbackup http://127.0.0.1:8091 sample-backup -m full -u Administrator -p password --single-node
  3. Delete the bucket
  4. Create a bucket called ts-restored
  5. Restore travel-sample bucket using the command below;
    cbrestore sample-backup/2018-07-31T214725Z -u Administrator -p password -b travel-sample -B ts-restored http://10.1.1.1:8091

cbrestore output
[####################] 100.0% (10492/estimated 10492 msgs)
bucket: travel-sample, msgs transferred…
: total | last | per sec
batch : 30 | 30 | 35.3
byte : 11975610 | 11975610 | 14079527.8
msg : 10492 | 10492 | 12335.3
2018-08-01 15:59:46,247: mt Starting new HTTP connection (1): 10.1.41.235
2018-08-01 15:59:46,249: mt “GET /pools/default/nodeServices HTTP/1.1” 200 1085
2018-08-01 15:59:46,251: mt Starting new HTTP connection (1): 10.1.38.96
2018-08-01 15:59:46,677: mt “POST /restoreIndexMetadata?bucket=ts-restored HTTP/1.1” 500 558
2018-08-01 15:59:46,677: mt [‘Internal server error, please retry your request’]
done

indexer.log I found the error below;
2018-08-01T15:59:46.411+00:00 [Error] requestHandler.makeCreateIndexRequest(): create index request fails. Error=Bucket does not exist or temporarily unavailable for creating new index. Please retry the operation at a later time (err=Bucket not found).

Now I know the issue here. My original bucket name is “travel-sample” but I’m restoring it as “ts-restored” because this allows in cbrestore utility to restore a bucket as a different name. But then when it tries to apply the index metadata, it can not find the original bucket name, well make sense. However, cbrestore utility then should not allow to restore with a different name.

See below for the output of cbrestore with destination bucket name exactly same as original bucket name.

[####################] 100.0% (10492/estimated 10492 msgs)
bucket: travel-sample, msgs transferred…
: total | last | per sec
batch : 30 | 30 | 35.0
byte : 11975610 | 11975610 | 13961625.4
msg : 10492 | 10492 | 12232.0
2018-08-01 16:05:25,852: mt Starting new HTTP connection (1): 10.1.41.235
2018-08-01 16:05:25,854: mt “GET /pools/default/nodeServices HTTP/1.1” 200 1085
2018-08-01 16:05:25,855: mt Starting new HTTP connection (1): 10.1.38.96
2018-08-01 16:05:27,635: mt “POST /restoreIndexMetadata?bucket=travel-sample HTTP/1.1” 200 18
done

Thanks Amit for the information about the logs which needs to be looked at.

4

Hi Bathige,

Thanks for providing the commands. This is a known issue: POST /restoreIndexMetadata works fine only when the source and destination buckets have same name. With different names, it can throw an error.

You can find the details in this ticket: https://issues.couchbase.com/browse/MB-29843

Thanks.

5

Hi Amit,

Thanks for confirming that this is a known issue.
It would have been much easier and time saving, if Couchbase doc itself mention this know issue. So that whoever tries this will not spend hours to troubleshoot. Just a suggestion.

Thanks.