Cbbackupmgr restore - Error restoring cluster: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Couchbase Server Kubernetes
1

Environment details:

  • Server: Kubernetes on AWS.
  • kubectl get nodes version: v1.19
  • Couchbase version: Enterprise Edition 6.6
  • cbbackupmgr version: 6.6.1-9213
  • Backup Location: AWS S3 Bucket
  • I access the Couchbase cluster via my local machine using Kubernates and visual code studio.

I seem to be facing a issue when trying a restore on 2 buckets. So this are the steps I take:

  1. I create a new Repo for the 2 buckets:
cbbackupmgr config -a s3://my-backup-location/archive/ -r MyDevManualBackup --disable-eventing --disable-analytics --include-buckets bucket1,bucket2 --obj-staging-dir /mnt/amazs3mnt --obj-region eu-west-2 --s3-force-path-style --obj-access-key-id **************  --obj-secret-access-key **********************
  1. I manually started a backup that runs without any issues:
cbbackupmgr backup -c xxx.xxx.xxx.xxx:9999 -u abackupuser -p ********************** -a s3://my-backup-location/archive/ -r MyDevManualBackup --full-backup --obj-staging-dir /mnt/amazs3mnt --obj-region eu-west-2 --s3-force-path-style --obj-access-key-id **************  --obj-secret-access-key **********************

image
image1380×307 63.7 KB

image
image1367×147 55.5 KB

  1. I manually start the restore procedure using this command:
CB_MEMCACHED_OPERATION_TIMEOUT=12h cbbackupmgr restore -c xxx.xxx.xxx.xxx:9999 -u abackupuser -p ********************** -a s3://my-backup-location/archive/ -a s3://my-backup-location/archive/ -r MyDevManualBackup --start 2022-12-02T12_08_16.385800635Z --end 2022-12-02T12_08_16.385800635Z --include-buckets bucket1,bucket2 --force-updates --obj-staging-dir /mnt/staging --obj-region eu-west-2 --s3-force-path-style --obj-access-key-id **********************  --obj-secret-access-key ********************************************

But the backup restore fails when it starts restoring the second bucket:

image
image1389×494 97.9 KB

I have also tried without using this parameter CB_MEMCACHED_OPERATION_TIMEOUT but the result is the same. I even tried restoring just one bucket at a time, same error.

Any ideas on were to start looking in order to identify the cause of the error?

2

Ok, after many attempts I found the issue. When restoring run the command on one of the data nodes, but -c should be another data node, not the same one. So don’t run cbbackupmgr restore -c 69.96.69.96:9999 from cli on 69.96.69.96. Instead run from cli 69.96.69.96 restore on another data node like -c 69.96.69.70:9999