Cbbackup shows "mt error: unable to access REST API" in the middle of the backup

Couchbase Server
1

Hello.
I’m using Couchbase Enterprise Edition 5.5.6 build 4733 and trying to take a full backup by cbbackup.
It showed following error during the backup. Somehow it generates 2 directories in backup destination. I thought it will create 1 directory for full backup. Can somebody help us taking a healthy full backup?

Couchbase Enterprise Edition 5.5.6 build 4733

Error message by cbbackup command

  [################    ] 82.3% (376575107/estimated 457794217 msgs)
bucket: catalog, msgs transferred...
       :                total |         last |    per sec
 byte  :         531958547028 | 531958547028 |  8111948.2
2021-03-24 03:28:40,469: mt error: unable to access REST API: localhost:8091/pools/default/buckets/catalog/ddocs; please check source URL, server status, username (-u) and password (-p); response: 400; reason: provide_design
  [#############       ] 66.7% (220984314/estimated 331182283 msgs)
bucket: engage, msgs transferred...
       :                total |         last |    per sec
 byte  :         602497903617 | 602497903617 | 25697420.2
2021-03-24 09:59:26,379: mt error: unable to access REST API: localhost:8091/pools/default/buckets/engage/ddocs; please check source URL, server status, username (-u) and password (-p); response: 400; reason: provide_design
done

Backup directories

 ls -ltr
drwxr-xr-x 3 root root 28 Mar 23 23:31 2021-03-23T161543Z-diff
drwxr-xr-x 4 root root 49 Mar 24 03:28 2021-03-23T161543Z-full
2

Hi Keiichi,
Thanks for using our product. I did try your backup steps in our server and I could not reproduce your issue. So it may be your firewall or local network issue. You could run with flag -v to get more information

[root@s44016 ~]# /opt/couchbase/bin/cbbackup http://localhost:8091 /tmp/backup/ -u Administrator -p password -t 8 -v
2021-03-26 10:51:31,150: mt cbbackup…
2021-03-26 10:51:31,150: mt source : http://localhost:8091
2021-03-26 10:51:31,151: mt sink : /tmp/backup/
2021-03-26 10:51:31,151: mt opts : {‘username’: ‘’, ‘verbose’: 1, ‘extra’: {‘max_retry’: 10.0, ‘rehash’: 0.0, ‘dcp_consumer_queue_length’: 1000.0, ‘data_only’: 0.0, ‘uncompress’: 0.0, ‘nmv_retry’: 1.0, ‘conflict_resolve’: 1.0, ‘cbb_max_mb’: 100000.0, ‘report’: 5.0, ‘mcd_compatible’: 1.0, ‘try_xwm’: 1.0, ‘backoff_cap’: 10.0, ‘batch_max_bytes’: 400000.0, ‘report_full’: 2000.0, ‘flow_control’: 1.0, ‘batch_max_size’: 1000.0, ‘seqno’: 0.0, ‘design_doc_only’: 0.0, ‘allow_recovery_vb_remap’: 0.0, ‘recv_min_bytes’: 4096.0}, ‘collection’: None, ‘ssl’: False, ‘threads’: 8, ‘key’: None, ‘password’: ‘’, ‘id’: None, ‘bucket_source’: None, ‘silent’: False, ‘dry_run’: False, ‘single_node’: False, ‘vbucket_list’: None, ‘separator’: ‘::’, ‘mode’: ‘diff’}
2021-03-26 10:51:31,168: mt Starting new HTTP connection (1): localhost
2021-03-26 10:51:31,205: mt bucket: travel-sample
2021-03-26 10:51:54,310: w0 source : http://localhost:8091(travel-sample@127.0.0.1:8091)
2021-03-26 10:51:54,310: w0 sink : /tmp/backup/(travel-sample@127.0.0.1:8091)
2021-03-26 10:51:54,310: w0 : total | last | per sec
2021-03-26 10:51:54,311: w0 batch : 91 | 91 | 3.9
2021-03-26 10:51:54,311: w0 byte : 36196306 | 36196306 | 1567102.1
2021-03-26 10:51:54,311: w0 msg : 31591 | 31591 | 1367.7
[####################] 100.0% (31591/estimated 31591 msgs)
bucket: travel-sample, msgs transferred…
: total | last | per sec
batch : 91 | 91 | 3.9
byte : 36196306 | 36196306 | 1543284.7
msg : 31591 | 31591 | 1346.9
2021-03-26 10:51:54,674: mt Starting new HTTP connection (1): localhost
2021-03-26 10:51:54,681: mt Starting new HTTP connection (1): 127.0.0.1
2021-03-26 10:51:54,721: mt Starting new HTTP connection (1): localhost
2021-03-26 10:51:54,728: mt Starting new HTTP connection (1): 127.0.0.1
done

In 5.5.6 enterprise edition, we have cbbackupmgr command which is better tool to run backup and restore.
Could you try again with cbbackupmgr?
Here is example to run backup with cbbackupmgr

[root@s44016 ~]# mkdir /tmp/bk
[root@s44016 ~]# /opt/couchbase/bin/cbbackupmgr config -a /tmp/bk/ -r backup
Backup repository backup created successfully in archive /tmp/bk/
[root@s44016 ~]# /opt/couchbase/bin/cbbackupmgr backup -r backup -a /tmp/bk/ -c localhost -u Administrator -p password
Backing up to 2021-03-26T10_59_42.044066465-07_00
Copied all data in 12s (Avg. 3.05MB/Sec) 31591 items / 36.59MB
travel-sample [==============================================================================================================================================================================================================================] 100.00%
Backup successfully completed
[root@s44016 ~]#

If you have any quesion, drop me a line at thuan at couchbase dot com
Thanks
Thuan

3

Hello @ky3960,

To follow on from @thuan reply, I wanted to point out that cbbackup and cbrestore are deprecated. For Couchbase Server Enterprise Edition 5.5.6 it’s strongly recommended that cbbackupmgr is used to do backing up and restoring of the cluster.

With regards to the error reported; cbbackup is failing to get the design documents from the REST endpoint localhost:8091/pools/default/buckets/catalog/ddocs. It’s receiving a 400 error which suggests that cbbackup is making an incorrect request. I think I have managed to reproduce this by using a node in the cluster as the host for cbbackup that does not have the Data Service on it.

Can you try the command again against a node with the Data Service? It would be great if you could report back.

Here is my output,

Node 10.112.186.102 has just the Query Service.

  [####################] 100.0% (7303/estimated 7303 msgs)
bucket: beer-sample, msgs transferred...
       :                total |       last |    per sec
 byte  :              2542924 |    2542924 |   836731.5
2021-03-28 18:42:04,388: mt error: unable to access REST API: 10.112.186.102:8091/pools/default/buckets/beer-sample/ddocs; please check source URL, server status, username (-u) and password (-p); response: 400; reason: provide_design
done

When a node that has the Data Service is used, it works without issue:

[vagrant@node1-cb556-centos7 ~]$ /opt/couchbase/bin/cbbackup http://10.112.186.101:8091 ~/backup -u Administrator -p password
  [####################] 100.0% (7303/estimated 7303 msgs)
bucket: beer-sample, msgs transferred...
       :                total |       last |    per sec
 byte  :              2542924 |    2542924 |   850127.1
done

Thanks,
Patrick

4

Thank you. I was running cbbackup against index node and it was causing the error. I run same command against Data node and it finished without that error.

1 Like
5

Thanks for getting back to me and confirming that was the issue!

6

Hi, We are having a 3 nodes Couchbase cluster of the community edition. We are taking of full backup of the cluster using cbbackup CLI tool. It never completed 100% of backup. It stops at a
01couchbasemanualbackup
certain percentage of backup and shows the backup is completed.