Possible bug: Running cbbackupmgr with docker

Couchbase Server
1

Hello!

I have a working installation of Couchbase Server Enterprise Edition 5.5.1 build 3511 running within a Docker container. I’m able to access the dashboard from localhost:8091, and have a cluster running with two other instances in their own docker containers, all within a docker network. Now, I’m attempting to create a backup of this cluster using cbbackupmgr from the host OS (Windows). However, the backup fails due to a tcp connection timing out. The full log file is as follows:

2018-09-05T12:45:09.464-04:00 (Cmd) cbbackupmgr version 1.3.0
2018-09-05T12:45:09.467-04:00 (Cmd) backup --archive C:\tmp\backup --repo cluster --cluster'couchbase://localhost --username <ud>Administrator</ud> --password ******** --purge
2018-09-05T12:45:09.509-04:00 (Rest) GET http://localhost:8091/pools 200
2018-09-05T12:45:09.777-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:09.799-04:00 (Plan) Executing transfer plan
2018-09-05T12:45:09.799-04:00 (Plan) Checking for data movement restrictions between travel-sample and travel-sample
2018-09-05T12:45:09.805-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:09.829-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:09.865-04:00 (Rest) GET http://localhost:8091/pools/default 200
2018-09-05T12:45:09.867-04:00 Transfering from Couchbase Server 5.5.1
2018-09-05T12:45:09.924-04:00 (Plan) Transfering bucket configuration for travel-sample to travel-sample
2018-09-05T12:45:09.959-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:09.972-04:00 (Plan) Transfering views definitions for travel-sample to travel-sample
2018-09-05T12:45:10.002-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:10.012-04:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
2018-09-05T12:45:10.809-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:11.811-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:12.813-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:13.817-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:14.819-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:15.821-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:16.823-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:17.826-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:18.829-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:19.831-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:20.834-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:21.836-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:22.840-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:23.843-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:24.845-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:25.847-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:26.849-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:27.852-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:28.854-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:29.857-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:30.859-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:31.017-04:00 ERRO: dial tcp 172.18.0.2:8091: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. -- plan.(*views).execute() at views.go:58
2018-09-05T12:45:31.017-04:00 (Plan) Transfer plan failed due to error dial tcp 172.18.0.2:8091: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
2018-09-05T12:45:31.018-04:00 (Cmd) Error backing up cluster: dial tcp 172.18.0.2:8091: connectex: A' connection attempt failed because the connected party did not properly respond after a period of time, or' established connection failed because connected host has failed to respond.

172.18.0.2:8091 is the IP address of the node at localhost:8091 on the internal Docker network. It looks like cbbackupmgr is successfully connecting to the node and grabbing what buckets exist, but when it actually goes to create the archive, it’s using the self-reported IP from the server, not the IP from the CLI invocation.

I can use “docker exec -i -t /bin/bash” to login into the docker container, and use the cbbackupmgr within the Docker container to create a backup of the cluster, using a Docker mount location to transfer the archive back to the host OS filesystem. However, initiating a backup from the host OS fails. Is this a bug?

2

Hi @ryarlagadda,

You are correct, cbbackupmgr first uses the passed in address of localhost:8091 to pull information from the server before actually backing up your data. The information it pulls contains the IP addresses of each node in the cluster. In 5.5.0+ this cluster information will contain the internal address and possible alternate addresses for each node. The cbbackupmgr tool is grabbing the internal IP address (172.18.0.2) which is not routable from the host machine. Unfortunately this network setup is non-standard for Couchbase and cbbackupmgr does not yet support using alternate addresses that the Couchbase instance provides. There is a work around, however. Simple way would be to docker exec into the Couchbase container, run cbbackupmgr, zip up the backup directory, then docker copy it out of the container back to the host machine.

3

@korriganclark ,

Given that “alternate addresses” feature was released with 6.5, would you be able to confirm if cbbackupmgr should now work with version 6.5+ in a docker env?

thanks

4

Hi @pccb,

There was a bug (MB-44925) in the alternate address support when it was initially released which meant that it didn’t work as expected. So although the feature was added in 6.5.0, you’ll need to use cbbackupmgr from at least 6.6.2 for alternate addressing support to work correctly.

If you have any other questions, or this isn’t the behavior you’re seeing, please let me know.

Thanks,
James

5

Thank you @jamesl33

We are ok to use 6.6.2. Would appreciate if you can confirm if the original problem in this thread will get resolved if we use alternate addresses?

This is how our setup looks like:
Docker is running on a physical host (lets call it A).
Couchbase is setup on a container and the IP is from Docker’s internal network (lets call it B).
We run the application and cbbackupmgr/cbimport/xdxr utilities from a remote host (lets call it C).
All Couchbase ports are mapped between A and B.

Application on C is able to successfully connect to Couchbase cluster via SDK.
cbq on C is also able to successfully connect to Couchbase.

cbbackupmgr/cbimport/xdxr utilities, when run on C or even on A, are NOT able to successfully connect to Couchbase. They are required to be run on B itself.

As we understand that is because the Couchbase cluster on docker knows only the docker network IP (i.e. B). When we initiate cbbackupmgr on C, we found that it is able to reach Couchbase but then Couchbase returns the IP Address B asking cbbackupmgr to connect to B. At that point it starts failing because cbbackupmgr (on C) cannot (obviously) connect to B because it is an internal network.

Thanks

6

Hi @pccb,

From my understanding of what you’d described here, I believe it should work (providing that alternate addressing is setup correctly; considering you’re using the SDKs to connection, it should be).

Please let me know if this isn’t the case, and we can investigate further.

Thanks,
James