Hi all,
I’ve been having issues with Sync Gateway reconnecting to Couchbase node.
Currently, we’re running one Couchbase node on VM with only 4GB of RAM, as our system is not in production phase yet.
Sometimes, Couchbase node will run out of memory and restart itself, which will result in Sync Gateway losing connection to it, and it won’t be able to reconnect, thus making Sync Gateway unable to sync data from mobile application to Couchbase.
When trying to query Sync Gateway for users using /default/_users/ endpoint, I will get response:
{"error":"not_found","reason":"no such database \"default\""}
I investigated it a bit, by restarting Couchbase node manually, and looking at the Sync Gateway logs. It seems to me that Sync Gateway tries to reconnect when node comes back online, but it gives up too early, while bucket is still in warmup phase.
Bucket that we’re using on this node has about 2 million documents. I tried it on a freshly installed node on my local machine, without any data, and Sync Gateway was able to reconnect without any problem.
Is there a way to configure time between retries, and/or increase number of retries?
This is Sync Gateway log snippet, right after restarting Couchbase node (I only edited out IP addresses and changed bucket name):
> _time=2017-05-12T08:41:30.569+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 1s
> _time=2017-05-12T08:41:31.521+00:00 _level=INFO _msg=Got new configuration for bucket testbucket
> _time=2017-05-12T08:41:31.574+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 1s
> _time=2017-05-12T08:41:32.581+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 1s
> _time=2017-05-12T08:41:33.589+00:00 _level=ERROR _msg=go-couchbase: Error connecting to tap feed of XXX.XXX.XXX.XXX:11210: dial tcp XXX.XXX.XXX.XXX:11210: getsockopt: connection refused
> _time=2017-05-12T08:41:33.590+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 2s
> _time=2017-05-12T08:41:35.603+00:00 _level=ERROR _msg=go-couchbase: Error connecting to tap feed of XXX.XXX.XXX.XXX:11210: dial tcp XXX.XXX.XXX.XXX:11210: getsockopt: connection refused
> _time=2017-05-12T08:41:35.604+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 4s
> _time=2017-05-12T08:41:37.681+00:00 _level=ERROR _msg= Unable to decode response unexpected EOF
> _time=2017-05-12T08:41:37.681+00:00 _level=INFO _msg= Trying with http://XXX.XXX.XXX.XXX:8091/pools/default/bucketsStreaming/testbucket
> 2017-05-12T08:41:37.682Z WARNING: Bucket Updater for bucket testbucket returned error: Get http://XXX.XXX.XXX.XXX:8091/pools/default/bucketsStreaming/testbucket: dial tcp XXX.XXX.XXX.XXX:8091: getsockopt: connection refused -- base.GetCouchbaseBucket.func1() at bucket.go:404
> 2017-05-12T08:41:37.682Z WARNING: Lost TAP feed for bucket testbucket, with error: Get http://XXX.XXX.XXX.XXX:8091/pools/default/bucketsStreaming/testbucket: dial tcp XXX.XXX.XXX.XXX:8091: getsockopt: connection refused -- rest.(*ServerContext)._getOrAddDatabaseFromConfig.func1() at server_context.go:350
> 2017-05-12T08:41:37.682Z CRUD: Taking Database : default, offline
> 2017-05-12T08:41:37.682Z CRUD: Waiting for all active calls to complete on Database : default
> 2017-05-12T08:41:37.682Z CRUD: Database : default, is offline
> 2017-05-12T08:41:37.683Z Bucket: Refresh() [451.884µs]
> 2017-05-12T08:41:37.688Z Bucket: Refresh() [485.151µs]
> 2017-05-12T08:41:37.699Z Bucket: Refresh() [607.833µs]
> 2017-05-12T08:41:37.720Z Bucket: Refresh() [618.451µs]
> 2017-05-12T08:41:37.761Z Bucket: Refresh() [618.335µs]
> 2017-05-12T08:41:37.842Z Bucket: Refresh() [647.681µs]
> 2017-05-12T08:41:38.003Z Bucket: Refresh() [696.431µs]
> 2017-05-12T08:41:38.323Z Bucket: Refresh() [730.359µs]
> 2017-05-12T08:41:38.965Z Bucket: Refresh() [775.791µs]
> _time=2017-05-12T08:41:39.667+00:00 _level=ERROR _msg=go-couchbase: Error connecting to tap feed of XXX.XXX.XXX.XXX:11210: dial tcp XXX.XXX.XXX.XXX:11210: getsockopt: connection refused
> _time=2017-05-12T08:41:39.667+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 8s
> 2017-05-12T08:41:40.246Z Bucket: Refresh() [709.234µs]
> 2017-05-12T08:41:42.807Z Bucket: Refresh() [748.918µs]
> _time=2017-05-12T08:41:47.668+00:00 _level=INFO _msg=go-couchbase: TAP connection lost; reconnecting to bucket "testbucket" in 16s
> 2017-05-12T08:41:47.931Z Bucket: Refresh() [3.703865ms]
> 2017-05-12T08:41:47.932Z CRUD: Connection to TAP feed for default re-established, bringing DB back online
> 2017-05-12T08:41:57.932Z Closing db /default (bucket "testbucket")
> 2017-05-12T08:41:57.932Z Bucket: Close() [5.107µs]
> 2017-05-12T08:41:57.932Z Opening db /default as bucket "testbucket", pool "default", server <http://XXX.XXX.XXX.XXX:8091>
> 2017-05-12T08:41:57.944Z Opening Couchbase database testbucket on <http://XXX.XXX.XXX.XXX:8091> as user "testbucket"
> _time=2017-05-12T08:41:58.033+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.033+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.034Z Bucket: PutDDoc("sync_gateway", ...) [213.742µs]
> 2017-05-12T08:41:58.034Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.034+00:00 _level=INFO _msg= Trying with http://XXX.XXX.XXX.XXX:8091/pools/default/bucketsStreaming/testbucket
> _time=2017-05-12T08:41:58.039+00:00 _level=INFO _msg=Got new configuration for bucket testbucket
> _time=2017-05-12T08:41:58.042+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.042+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.042Z Bucket: PutDDoc("sync_gateway", ...) [101.625µs]
> 2017-05-12T08:41:58.042Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.052+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.052+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.052Z Bucket: PutDDoc("sync_gateway", ...) [93.147µs]
> 2017-05-12T08:41:58.052Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.072+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.072+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.072Z Bucket: PutDDoc("sync_gateway", ...) [112.332µs]
> 2017-05-12T08:41:58.072Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.113+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.113+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.113Z Bucket: PutDDoc("sync_gateway", ...) [109.255µs]
> 2017-05-12T08:41:58.113Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.193+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.193+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.193Z Bucket: PutDDoc("sync_gateway", ...) [148.197µs]
> 2017-05-12T08:41:58.193Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.353+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.354+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.354Z Bucket: PutDDoc("sync_gateway", ...) [194.331µs]
> 2017-05-12T08:41:58.354Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:58.674+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:58.674+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:58.674Z Bucket: PutDDoc("sync_gateway", ...) [241.676µs]
> 2017-05-12T08:41:58.674Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:41:59.315+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:41:59.315+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:41:59.315Z Bucket: PutDDoc("sync_gateway", ...) [239.081µs]
> 2017-05-12T08:41:59.315Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:42:00.595+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:42:00.595+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:42:00.595Z Bucket: PutDDoc("sync_gateway", ...) [252.036µs]
> 2017-05-12T08:42:00.595Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:42:03.156+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:42:03.156+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:42:03.156Z Bucket: PutDDoc("sync_gateway", ...) [231.187µs]
> 2017-05-12T08:42:03.156Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> _time=2017-05-12T08:42:08.276+00:00 _level=INFO _msg=Non-healthy node; node details:
> _time=2017-05-12T08:42:08.276+00:00 _level=INFO _msg=Hostname=XXX.XXX.XXX.XXX:8091, Status=warmup, CouchAPIBase=http://XXX.XXX.XXX.XXX:8092/testbucket%2Be728bb55ee23432db07e8a1a156b2c22, ThisNode=true
> 2017-05-12T08:42:08.277Z Bucket: PutDDoc("sync_gateway", ...) [212.053µs]
> 2017-05-12T08:42:08.277Z WARNING: Error installing Couchbase design doc: no available couch rest URLs -- db.installViews.func1() at database.go:609
> 2017-05-12T08:42:08.277Z WARNING: RetryLoop for Attempt to install Couchbase design doc bucket : sync_gateway giving up after 12 attempts -- base.RetryLoop() at util.go:301
> 2017-05-12T08:42:08.277Z ERROR: no available couch rest URLs -- rest.(*ServerContext).TakeDbOnline() at server_context.go:577
> _time=2017-05-12T08:42:08.277+00:00 _level=ERROR _msg= Bucket Updater exited with err Get http://XXX.XXX.XXX.XXX:8091/pools/default/bucketsStreaming/testbucket: dial tcp XXX.XXX.XXX.XXX:8091: getsockopt: connection refused
> _time=2017-05-12T08:42:18.938+00:00 _level=INFO _msg=Got new configuration for bucket testbucket