Exception with cbrestore on mac os X - again

Couchbase Server
1

Hello,

Sorry for reposting this issue but I posted it 13 days ago and did not get any answer… And I did not find any other way to get your attention than posting this again.

I am using couchbase server 4.5.1.

I am trying to cbrestore a bucket which has been backedup with cbbackup on ubuntuu and I am getting the following exception :

/Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbrestore . http://localhost:8091 -u Administrator -p st0ryplAyr --bucket-source=storyplayer-api --bucket-destination=storyplayer-api -v -x rehash=1
2017-06-27 09:51:02,610: mt cbrestore...
2017-06-27 09:51:02,610: mt  source : .
2017-06-27 09:51:02,610: mt  sink   : http://localhost:8091
2017-06-27 09:51:02,610: mt  opts   : {'username': '<xxx>', 'verbose': 1, 'extra': {'max_retry': 10.0, 'rehash': 1.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 0.0, 'recv_min_bytes': 4096.0}, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'storyplayer-api', 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': 'storyplayer-api', 'add': False, 'vbucket_list': None}
2017-06-27 09:51:02,615: mt bucket: storyplayer-api
Exception in thread s0:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/pump_mc.py", line 91, in run
    rv, batch, need_backoff = self.scatter_gather(mconns, batch)
  File "/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/pump_cb.py", line 72, in scatter_gather
    rv, conn = self.find_conn(mconns, vbucket_id, msgs)
  File "/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/pump_cb.py", line 316, in find_conn
    host_port = serverList[vBucketMap[vbucket_id][0]]
IndexError: list index out of range

I had the same error (or comparable) previously which is why I am using the option rehash=1 but this time it does not help.

Any idea about what I could do ?

2

Hi @thomas1,

Could you post a small sample backup file? I’d like to try and reproduce, and file a bug if it is.

3

Hello Matthew,
Sorry for the delay of your response. Thank you to have a look at my issue. My problem is that I don’t know how to post a “small sample”.
I only have a full backup of my db… Do you know how I can get a small sample.
Best regards
Thomas

4

Hello again,
Trying to find a way to make a partial backup, I noticed that there is also a rehash option to cbbackup. Do you think I should use it for my backup ?
So far, I am only using the rehash option when restoring…
Thanks
Thomas

5

Hi guys. I’ve just faced with a similar situation. I am trying to copy bucket from remote cb cluster (6 nodes) to my 1 node cb server launched locally.
Both servers (remote and local) launched at linux os and use couchbase community edition 4.5.1
I’ve created a backup from cluster using command:

/opt/couchbase/bin/cbbackup couchbase://127.0.0.1:8091 ./backup/cluster -b default
[####################] 100.0% (19879/estimated 19879 msgs)
bucket: default, msgs transferred...
:                total |       last |    per sec
byte  :              5074316 |    5074316 |  1237546.5
done

Then I tried to restore it using cbrestore tool:
cbrestore -vvv -x rehash=1 --add --bucket-destination=source --username=root --password=password /opt/couchbase/backups couchbase://localhost:8091

And I got errors related to vbuckets:

018-05-11 16:19:06,182: mt cbrestore…
2018-05-11 16:19:06,182: mt source : /opt/couchbase/backups
2018-05-11 16:19:06,182: mt sink : couchbase://localhost:8091
2018-05-11 16:19:06,182: mt opts : {‘username’: ‘’, ‘verbose’: 3, ‘extra’: {‘max_retry’: 10.0, ‘rehash’: 1.0, ‘dcp_consumer_queue_length’: 1000.0, ‘data_only’: 0.0, ‘uncompress’: 0.0, ‘nmv_retry’: 1.0, ‘conflict_resolve’: 1.0, ‘cbb_max_mb’: 100000.0, ‘report’: 5.0, ‘mcd_compatible’: 1.0, ‘try_xwm’: 1.0, ‘backoff_cap’: 10.0, ‘batch_max_bytes’: 400000.0, ‘report_full’: 2000.0, ‘flow_control’: 1.0, ‘batch_max_size’: 1000.0, ‘seqno’: 0.0, ‘design_doc_only’: 0.0, ‘recv_min_bytes’: 4096.0}, ‘ssl’: False, ‘threads’: 4, ‘to_date’: None, ‘key’: None, ‘password’: ‘’, ‘id’: None, ‘bucket_source’: None, ‘silent’: False, ‘dry_run’: False, ‘from_date’: None, ‘bucket_destination’: ‘source’, ‘add’: True, ‘vbucket_list’: None}
2018-05-11 16:19:06,182: mt source_class: <class ‘pump_bfd.BFDSource’>
2018-05-11 16:19:06,182: mt sink_class: <class ‘pump_cb.CBSink’>
2018-05-11 16:19:06,183: mt rest_request: root@localhost:8091/pools/default/buckets
2018-05-11 16:19:06,185: mt source_bucket: default
2018-05-11 16:19:06,185: mt sink_bucket: source
2018-05-11 16:19:06,185: mt source_buckets: default
2018-05-11 16:19:06,185: mt bucket: default
2018-05-11 16:19:06,185: mt source_nodes: cb0.my-cluster.net:8091,cb1.my-cluster.net:8091,cb2.my-cluster.net:8091,cb3.my-cluster.net:8091,cb4.my-cluster.net:8091,cb5.my-cluster.net:8091
2018-05-11 16:19:06,186: mt enqueueing node: cb0.my-cluster.net:8091
2018-05-11 16:19:06,186: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb0.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,187: w0 node: cb0.my-cluster.net:8091
2018-05-11 16:19:06,189: mt enqueueing node: cb1.my-cluster.net:8091
2018-05-11 16:19:06,189: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb1.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,189: w1 node: cb1.my-cluster.net:8091
2018-05-11 16:19:06,191: w0 connect_db: /opt/couchbase/backups/bucket-default/node-cb0.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,191: mt enqueueing node: cb2.my-cluster.net:8091
2018-05-11 16:19:06,191: w2 node: cb2.my-cluster.net:8091
2018-05-11 16:19:06,192: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb2.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,192: w1 connect_db: /opt/couchbase/backups/bucket-default/node-cb1.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,192: w2 connect_db: /opt/couchbase/backups/bucket-default/node-cb2.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,195: mt enqueueing node: cb3.my-cluster.net:8091
2018-05-11 16:19:06,196: w3 node: cb3.my-cluster.net:8091
2018-05-11 16:19:06,196: w3 connect_db: /opt/couchbase/backups/bucket-default/node-cb3.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,197: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb3.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,199: mt enqueueing node: cb4.my-cluster.net:8091
2018-05-11 16:19:06,200: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb4.my-cluster.net%3A8091/data-0000.cbb
2018-05-11 16:19:06,203: mt enqueueing node: cb5.my-cluster.net:8091
2018-05-11 16:19:06,203: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb5.my-cluster.net%3A8091/data-0000.cbb
Exception in thread s0:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 810, in __bootstrap_inner
self.run()
File “/usr/lib/python2.7/threading.py”, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File “/opt/couchbase/lib/python/pump_mc.py”, line 91, in run
rv, batch, need_backoff = self.scatter_gather(mconns, batch)
File “/opt/couchbase/lib/python/pump_cb.py”, line 72, in scatter_gather
rv, conn = self.find_conn(mconns, vbucket_id, msgs)
File “/opt/couchbase/lib/python/pump_cb.py”, line 316, in find_conn
host_port = serverList[vBucketMap[vbucket_id][0]]
IndexError: list index out of range
Exception in thread s2:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 810, in __bootstrap_inner
self.run()
File “/usr/lib/python2.7/threading.py”, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File “/opt/couchbase/lib/python/pump_mc.py”, line 91, in run
rv, batch, need_backoff = self.scatter_gather(mconns, batch)
File “/opt/couchbase/lib/python/pump_cb.py”, line 72, in scatter_gather
rv, conn = self.find_conn(mconns, vbucket_id, msgs)
File “/opt/couchbase/lib/python/pump_cb.py”, line 316, in find_conn
host_port = serverList[vBucketMap[vbucket_id][0]]
IndexError: list index out of range
2018-05-11 16:19:06,280: s3 error: async operation: error: map missing vbucket_id: 23211; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination on sink: couchbase://localhost:8091(default@cb3.my-cluster.net:8091)
Exception in thread s1:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 810, in __bootstrap_inner
self.run()
File “/usr/lib/python2.7/threading.py”, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File “/opt/couchbase/lib/python/pump_mc.py”, line 91, in run
rv, batch, need_backoff = self.scatter_gather(mconns, batch)
File “/opt/couchbase/lib/python/pump_cb.py”, line 72, in scatter_gather
rv, conn = self.find_conn(mconns, vbucket_id, msgs)
File “/opt/couchbase/lib/python/pump_cb.py”, line 316, in find_conn
host_port = serverList[vBucketMap[vbucket_id][0]]
IndexError: list index out of range
2018-05-11 16:19:06,348: w3 pump (/opt/couchbase/backups(default@cb3.my-cluster.net:8091)->couchbase://localhost:8091(default@cb3.my-cluster.net:8091)) done.
2018-05-11 16:19:06,348: w3 source : /opt/couchbase/backups(default@cb3.my-cluster.net:8091)
2018-05-11 16:19:06,348: w3 sink : couchbase://localhost:8091(default@cb3.my-cluster.net:8091)
2018-05-11 16:19:06,348: w3 : total | last | per sec
2018-05-11 16:19:06,349: w3 node: cb3.my-cluster.net:8091, done; rv: error: map missing vbucket_id: 23211; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination
2018-05-11 16:19:06,349: w3 node: cb4.my-cluster.net:8091
2018-05-11 16:19:06,349: w3 connect_db: /opt/couchbase/backups/bucket-default/node-cb4.my-cluster.net%3A8091/data-0000.cbb
Exception in thread s3:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 810, in __bootstrap_inner
self.run()
File “/usr/lib/python2.7/threading.py”, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File “/opt/couchbase/lib/python/pump_mc.py”, line 91, in run
rv, batch, need_backoff = self.scatter_gather(mconns, batch)
File “/opt/couchbase/lib/python/pump_cb.py”, line 72, in scatter_gather
rv, conn = self.find_conn(mconns, vbucket_id, msgs)
File “/opt/couchbase/lib/python/pump_cb.py”, line 316, in find_conn
host_port = serverList[vBucketMap[vbucket_id][0]]
IndexError: list index out of range

I also tried to restore without rehash option, then I got similar errors:

2018-05-12 20:55:59,509: mt cbrestore…
2018-05-12 20:55:59,509: mt source : /opt/couchbase/backups
2018-05-12 20:55:59,509: mt sink : couchbase://127.0.0.1:8091
2018-05-12 20:55:59,509: mt opts : {‘username’: ‘’, ‘verbose’: 3, ‘extra’: {‘max_retry’: 10.0, ‘rehash’: 0.0, ‘dcp_consumer_queue_length’: 1000.0, ‘data_only’: 0.0, ‘uncompress’: 0.0, ‘nmv_retry’: 1.0, ‘conflict_resolve’: 1.0, ‘cbb_max_mb’: 100000.0, ‘report’: 5.0, ‘mcd_compatible’: 1.0, ‘try_xwm’: 1.0, ‘backoff_cap’: 10.0, ‘batch_max_bytes’: 400000.0, ‘report_full’: 2000.0, ‘flow_control’: 1.0, ‘batch_max_size’: 1000.0, ‘seqno’: 0.0, ‘design_doc_only’: 0.0, ‘recv_min_bytes’: 4096.0}, ‘ssl’: False, ‘threads’: 4, ‘to_date’: None, ‘key’: None, ‘password’: ‘’, ‘id’: None, ‘bucket_source’: None, ‘silent’: False, ‘dry_run’: False, ‘from_date’: None, ‘bucket_destination’: ‘source’, ‘add’: True, ‘vbucket_list’: None}
2018-05-12 20:55:59,509: mt source_class: <class ‘pump_bfd.BFDSource’>
2018-05-12 20:55:59,509: mt sink_class: <class ‘pump_cb.CBSink’>
2018-05-12 20:55:59,509: mt rest_request: root@127.0.0.1:8091/pools/default/buckets
2018-05-12 20:55:59,512: mt source_bucket: default
2018-05-12 20:55:59,512: mt sink_bucket: source
2018-05-12 20:55:59,512: mt source_buckets: default
2018-05-12 20:55:59,512: mt bucket: default
2018-05-12 20:55:59,512: mt source_nodes: cb0.my-cluster.net:8091,cb1.my-cluster.net:8091,cb2.my-cluster.net:8091,cb3.my-cluster.net:8091,cb4.my-cluster.net:8091,cb5.my-cluster.net:8091
2018-05-12 20:55:59,515: mt enqueueing node: cb0.my-cluster.net:8091
2018-05-12 20:55:59,515: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb0.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,515: w0 node: cb0.my-cluster.net:8091
2018-05-12 20:55:59,517: w0 connect_db: /opt/couchbase/backups/bucket-default/node-cb0.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,518: mt enqueueing node: cb1.my-cluster.net:8091
2018-05-12 20:55:59,518: w1 node: cb1.my-cluster.net:8091
2018-05-12 20:55:59,518: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb1.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,518: w1 connect_db: /opt/couchbase/backups/bucket-default/node-cb1.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,521: mt enqueueing node: cb2.my-cluster.net:8091
2018-05-12 20:55:59,521: w2 node: cb2.my-cluster.net:8091
2018-05-12 20:55:59,521: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb2.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,524: w2 connect_db: /opt/couchbase/backups/bucket-default/node-cb2.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,524: mt enqueueing node: cb3.my-cluster.net:8091
2018-05-12 20:55:59,524: w3 node: cb3.my-cluster.net:8091
2018-05-12 20:55:59,525: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb3.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,526: w3 connect_db: /opt/couchbase/backups/bucket-default/node-cb3.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,529: mt enqueueing node: cb4.my-cluster.net:8091
2018-05-12 20:55:59,529: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb4.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,532: mt enqueueing node: cb5.my-cluster.net:8091
2018-05-12 20:55:59,532: mt connect_db: /opt/couchbase/backups/bucket-default/node-cb5.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,588: s2 error: async operation: error: map missing vbucket_id: 512; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination on sink: couchbase://127.0.0.1:8091(default@cb2.my-cluster.net:8091)
2018-05-12 20:55:59,603: s1 error: async operation: error: map missing vbucket_id: 171; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination on sink: couchbase://127.0.0.1:8091(default@cb1.my-cluster.net:8091)
Exception in thread s0:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 810, in __bootstrap_inner
self.run()
File “/usr/lib/python2.7/threading.py”, line 763, in run
self.__target(*self.__args, **self.__kwargs)
File “/opt/couchbase/lib/python/pump_mc.py”, line 91, in run
rv, batch, need_backoff = self.scatter_gather(mconns, batch)
File “/opt/couchbase/lib/python/pump_cb.py”, line 72, in scatter_gather
rv, conn = self.find_conn(mconns, vbucket_id, msgs)
File “/opt/couchbase/lib/python/pump_cb.py”, line 316, in find_conn
host_port = serverList[vBucketMap[vbucket_id][0]]
IndexError: list index out of range
2018-05-12 20:55:59,617: s3 error: async operation: error: map missing vbucket_id: 513; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination on sink: couchbase://127.0.0.1:8091(default@cb3.my-cluster.net:8091)
2018-05-12 20:55:59,666: w2 pump (/opt/couchbase/backups(default@cb2.my-cluster.net:8091)->couchbase://127.0.0.1:8091(default@cb2.my-cluster.net:8091)) done.
2018-05-12 20:55:59,667: w2 source : /opt/couchbase/backups(default@cb2.my-cluster.net:8091)
2018-05-12 20:55:59,667: w2 sink : couchbase://127.0.0.1:8091(default@cb2.my-cluster.net:8091)
2018-05-12 20:55:59,667: w2 : total | last | per sec
2018-05-12 20:55:59,668: w2 node: cb2.my-cluster.net:8091, done; rv: error: map missing vbucket_id: 512; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination
2018-05-12 20:55:59,668: w2 node: cb4.my-cluster.net:8091
2018-05-12 20:55:59,669: w2 connect_db: /opt/couchbase/backups/bucket-default/node-cb4.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,686: w1 pump (/opt/couchbase/backups(default@cb1.my-cluster.net:8091)->couchbase://127.0.0.1:8091(default@cb1.my-cluster.net:8091)) done.
2018-05-12 20:55:59,686: w1 source : /opt/couchbase/backups(default@cb1.my-cluster.net:8091)
2018-05-12 20:55:59,686: w1 sink : couchbase://127.0.0.1:8091(default@cb1.my-cluster.net:8091)
2018-05-12 20:55:59,686: w1 : total | last | per sec
2018-05-12 20:55:59,687: w1 node: cb1.my-cluster.net:8091, done; rv: error: map missing vbucket_id: 171; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination
2018-05-12 20:55:59,687: w1 node: cb5.my-cluster.net:8091
2018-05-12 20:55:59,692: w1 connect_db: /opt/couchbase/backups/bucket-default/node-cb5.my-cluster.net%3A8091/data-0000.cbb
2018-05-12 20:55:59,699: w3 pump (/opt/couchbase/backups(default@cb3.my-cluster.net:8091)->couchbase://127.0.0.1:8091(default@cb3.my-cluster.net:8091)) done.
2018-05-12 20:55:59,699: w3 source : /opt/couchbase/backups(default@cb3.my-cluster.net:8091)
2018-05-12 20:55:59,699: w3 sink : couchbase://127.0.0.1:8091(default@cb3.my-cluster.net:8091)
2018-05-12 20:55:59,699: w3 : total | last | per sec
2018-05-12 20:55:59,699: w3 node: cb3.my-cluster.net:8091, done; rv: error: map missing vbucket_id: 513; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination
2018-05-12 20:55:59,720: s2 error: async operation: error: map missing vbucket_id: 768; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination on sink: couchbase://127.0.0.1:8091(default@cb4.my-cluster.net:8091)
2018-05-12 20:55:59,733: s1 error: async operation: error: map missing vbucket_id: 911; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination on sink: couchbase://127.0.0.1:8091(default@cb5.my-cluster.net:8091)
2018-05-12 20:55:59,741: w2 pump (/opt/couchbase/backups(default@cb4.my-cluster.net:8091)->couchbase://127.0.0.1:8091(default@cb4.my-cluster.net:8091)) done.
2018-05-12 20:55:59,741: w2 source : /opt/couchbase/backups(default@cb4.my-cluster.net:8091)
2018-05-12 20:55:59,742: w2 sink : couchbase://127.0.0.1:8091(default@cb4.my-cluster.net:8091)
2018-05-12 20:55:59,742: w2 : total | last | per sec
2018-05-12 20:55:59,742: w2 node: cb4.my-cluster.net:8091, done; rv: error: map missing vbucket_id: 768; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination
2018-05-12 20:55:59,747: w1 pump (/opt/couchbase/backups(default@cb5.my-cluster.net:8091)->couchbase://127.0.0.1:8091(default@cb5.my-cluster.net:8091)) done.
2018-05-12 20:55:59,747: w1 source : /opt/couchbase/backups(default@cb5.my-cluster.net:8091)
2018-05-12 20:55:59,747: w1 sink : couchbase://127.0.0.1:8091(default@cb5.my-cluster.net:8091)
2018-05-12 20:55:59,747: w1 : total | last | per sec
2018-05-12 20:55:59,748: w1 node: cb5.my-cluster.net:8091, done; rv: error: map missing vbucket_id: 911; perhaps your source does not have vbuckets; if so, try using moxi (HOST:11211) as a destination

Is it a bug? If so, which version it’s fixed?