I have 3 node couchbase cluster. I was trying to get all data from one of the buckets in CSV format.
Documentation says that cbtransfer utility can do it for me.
So, I logged into one of the servers and ran it. It completed without errors:
cbtransfer http://localhost:8091 csv:/tmp/m1.csv -b member -u Administrator -p mypwd
2017-01-18 14:54:45,802: mt skipping bucket that is not a couchbase-bucket: default
2017-01-18 14:54:45,803: mt skipping bucket that is not a couchbase-bucket: esratemp
[####################] 100.0% (91529/91519 msgs)… [# ] 5.5% (1693/30544 msgs)
bucket: member, msgs transferred…
: total | last | per sec
batch : 3376 | 3376 | 187.3
byte : 560189814 | 560189814 | 31087039.0
msg : 91529 | 91529 | 5079.3
2017-01-18 14:55:03,829: mt warning: cannot save bucket design on a CSV destination
done
The CSV file was created, however, when I am doing
wc -l m1.csv
am getting
30376 m1.csv
The first line of the file is header, so, apparently only 30375 documents were transferred out of 91529.
Does it mean, that only one node data was transferred? Documentation says that cbtransfer utility works for the whole cluster and its output did say that it processed 91529 messages.
So, where are the other 61154 records?
I did run this utility against two other nodes and got two more files that are identical to the first one.
I.e. more than 2/3 of the data vanishes.
Hi Gary,
Could you tell me what Couchbase Server version you are running with this issue?
I did try with your scenario with Couchbase Server 4.5.0 and I could not reproduce it.
Here is the steps and output of my test.
Install Couchbase Server 4.5.0 on 3 Ubuntu 12.04 servers
Create a cluster of 3 nodes
Create 2 memcached buckets (default and test-imem)
Create 1 couchbase bucket (test-2-cb)
Load 10 thousands document into bucket test-2-cb
Transfer 10k docs on bucket test-2-cb out with format csv to folder /tmp/backup
In folder /tmp/backup, there are 3 files with total 10,003 lines (1 header line for each file)
root@ubuntu:~# /opt/couchbase/bin/cbworkloadgen -n localhost:8091 -b test-2-cb -j --prefix=test_
2017-01-23 09:04:32,156: mt skipping bucket that is not a couchbase-bucket: default
2017-01-23 09:04:32,156: mt skipping bucket that is not a couchbase-bucket: test-1mem
[####################] 100.0% (10527/estimated 10526 msgs)
bucket: default, msgs transferred...
: total | last | per sec
byte : 721927 | 721927 | 497614.7
done
root@ubuntu:~#
root@ubuntu:~# rm -rf /tmp/backup/*
root@ubuntu:~# /opt/couchbase/bin/cbtransfer http://localhost:8091 csv:/tmp/backup/test-2-cb.csv -b test-2-cb -u Administrator -p password
2017-01-23 09:05:24,501: mt skipping bucket that is not a couchbase-bucket: default
2017-01-23 09:05:24,501: mt skipping bucket that is not a couchbase-bucket: test-1mem
[####################] 100.0% (10000/estimated 10000 msgs)
bucket: test-2-cb, msgs transferred...
: total | last | per sec
byte : 686888 | 686888 | 330235.7
done
root@ubuntu:~# ls /tmp/backup/
test-2-cb_test-2-cb_12.11.10.130%3A8091.csv test-2-cb_test-2-cb_12.11.10.131%3A8091.csv test-2-cb_test-2-cb_12.11.10.132%3A8091.csv
root@ubuntu:~# wc -l /tmp/backup/test*
3342 /tmp/backup/test-2-cb_test-2-cb_12.11.10.130%3A8091.csv
3340 /tmp/backup/test-2-cb_test-2-cb_12.11.10.131%3A8091.csv
3321 /tmp/backup/test-2-cb_test-2-cb_12.11.10.132%3A8091.csv
10003 total
root@ubuntu:~#
Hi Thuan,
I am working with the v.2.2
I did try that again on the same cluster with the same bucket as well as with another bucket and got similar results.
zeus@cqzsdb01:~$ cbtransfer http://cqzsdb01.sling.com:8091 csv:/tmp/m1.csv -b member -u Administrator -p mypwd
2017-01-23 11:17:32,040: mt skipping bucket that is not a couchbase-bucket: default
2017-01-23 11:17:32,041: mt skipping bucket that is not a couchbase-bucket: esratemp
[####################] 100.0% (91766/91762 msgs)… [ ] 0.1% (78/61291 msgs)
bucket: member, msgs transferred…
: total | last | per sec
batch : 3382 | 3382 | 171.5
byte : 563060295 | 563060295 | 28555197.5
msg : 91766 | 91766 | 4653.8
2017-01-23 11:17:52,204: mt warning: cannot save bucket design on a CSV destination
done
zeus@cqzsdb01:~$ cbtransfer http://cqzsdb01.sling.com:8091 csv:/tmp/m2.csv -b rubens -u Administrator -p mypwd
2017-01-23 11:18:30,011: mt skipping bucket that is not a couchbase-bucket: default
2017-01-23 11:18:30,011: mt skipping bucket that is not a couchbase-bucket: esratemp
[####################] 100.0% (1564884/1564884 msgs)… [ ] 0.0% (314/817940 msgs)
bucket: rubens, msgs transferred…
: total | last | per sec
batch : 8237 | 8237 | 42.5
byte : 2115027356 | 2115027356 | 10908801.0
msg : 1564884 | 1564884 | 8071.3
2017-01-23 11:21:43,898: mt warning: cannot save bucket design on a CSV destination
done
zeus@cqzsdb01:~$ wc -l /tmp/*.csv
30472 /tmp/m1.csv
746945 /tmp/m2.csv
777417 total
So, as before about 1/3 documents from first bucket and less then 1/2 from second.
May be there was a bug in the old version, corrected in the newer.
I, however, have another suspicion. If you can check it out, I’d appreciate.
We have few (2-3) corrupted documents in our database. I.e. they were corrupted during the testing but keys were lost and now we can’t delete them, so they just linger there. Those documents cannot be converted to valid JSON. Maybe, when cbtransfer encounters one of those it stops writing the CSV file.
Can you test this scenario, please?
If you can also tell me how to recover from that, I will be very appreciative.