Hi,
I have a cluster of 6 server nodes and I want add 1 server node to cluster.
In my cluster I have 5 bucket with 13 milon of entry and about 100 map-reduce for each bucket.
I try to add new server node, after 2-3 hours is failed becauase always on node goes down.
In the log see :
Rebalance exited with reason {{nodedown,‘ns_1@couchbase02’},
{gen_server,call,
[{‘janitor_agent-bucket_logger’,
‘ns_1@couchbase02’},
{get_dcp_docs_estimate,367,
[‘ns_1@couchbase01’,
‘ns_1@couchbase07’]},
infinity]}}
and :
Node ‘ns_1@couchbase04’ saw that node ‘ns_1@couchbase02’ went down. Details: [{nodedown_reason,
net_tick_timeout}]
another error after:
Service ‘goxdcr’ exited with status 1. Restarting. Messages: MetadataService 2016-12-14T09:19:39.613+01:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale. Was never updated yet., num_of_retry=2
MetadataService 2016-12-14T09:19:39.613+01:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale. Was never updated yet., num_of_retry=3
MetadataService 2016-12-14T09:19:39.613+01:00 [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale. Was never updated yet., num_of_retry=4
RemoteClusterService 2016-12-14T09:19:39.613+01:00 [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
[goport] 2016/12/14 09:19:39 /opt/couchbase/bin/goxdcr terminated: exit status 1
Service ‘projector’ exited with status 1. Restarting. Messages: github.com/couchbase/indexing/secondary/common.ExitOnStdinClose()
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:316 +0x99
created by main.main
/home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/projector/main.go:93 +0x90d
[goport] 2016/12/14 09:19:38 /opt/couchbase/bin/projector terminated: exit status 2
Service ‘query’ exited with status 1. Restarting. Messages: _time=2016-12-14T09:19:34+01:00 _level=ERROR _msg= Unable to initialize cbauth. Error CBAuth database is stale. Was never updated yet.
_time=2016-12-14T09:19:34+01:00 _level=WARN _msg=Unable to intialize cbAuth, access to couchbase buckets may be restricted
_time=2016-12-14T09:19:36+01:00 _level=ERROR _msg=Cannot connect url http://127.0.0.1:8091 - cause: HTTP error 401 Unauthorized getting “http://127.0.0.1:8091/pools”:
[goport] 2016/12/14 09:19:36 /opt/couchbase/bin/cbq-engine terminated: exit status 1