Rebalance exited with reason badmatch

Couchbase Server
1

I’m trying to add a new node (d-gp2-couch1-1.*****.com) to a current cluster but getting the following when trying to rebalance that new node into the cluster. any ideas on what would cause this?

Blockquote
Rebalance exited with reason {badmatch,
{leader_activities_error,
{default,{ns_janitor,“cfg”,apply_config}},
{no_quorum,
[{required_quorum,
{all,
{set,4,16,16,8,80,48,
{,,,,,,,,,,,,,
,,},
{{,,,,,,,,,,,,,
[‘ns_1@d-gp2-couch1-8.…com’,
'ns_1@d-gp2-couch1-7.…com’,
‘ns_1@d-gp2-couch1-3.…com’,
'ns_1@d-gp2-couch1-1.…com’],
,}}}}},
{leases,
[‘ns_1@d-gp2-couch1-3.…com’,
'ns_1@d-gp2-couch1-8.…com’,
‘ns_1@d-gp2-couch1-7.*****…com’]}]}}}

2

Blockquote
Node ('ns_1@d-gp2-couch1-1..com’) was automatically failed over. Reason: The data service did not respond for the duration of the auto-failover threshold. Either none of the buckets have warmed up or there is an issue with the data service.
auto_failover 001
ns_1@d-gp2-couch1-3.…com
2:14:51 AM Thu Nov 25, 2021
Deactivating failed over nodes ['ns_1@d-gp2-couch1-1.…com’]
ns_rebalancer 000
ns_1@d-gp2-couch1-3.…com
2:14:51 AM Thu Nov 25, 2021
Failed over ['ns_1@d-gp2-couch1-1.…com’]: ok
ns_rebalancer 000
ns_1@d-gp2-couch1-3.…com
2:14:51 AM Thu Nov 25, 2021
Shutting down bucket “cfg” on 'ns_1@d-gp2-couch1-1.…com’ for deletion
ns_memcached 000
ns_1@d-gp2-couch1-1.…com
2:14:50 AM Thu Nov 25, 2021
Starting failing over ['ns_1@d-gp2-couch1-1.…com’]
ns_rebalancer 000
ns_1@d-gp2-couch1-3.…com
2:14:50 AM Thu Nov 25, 2021
Rebalance exited with reason {badmatch,
{leader_activities_error,
{default,{ns_janitor,“cfg”,apply_config}},
{no_quorum,
[{required_quorum,
{all,
{set,4,16,16,8,80,48,
{,,,,,,,,,,,,,
,,},
{{,,,,,,,,,,,,,
[‘ns_1@d-gp2-couch1-8.…com’,
'ns_1@d-gp2-couch1-7.…com’,
‘ns_1@d-gp2-couch1-3.…com’,
'ns_1@d-gp2-couch1-1.…com’],
,}}}}},
{leases,
[‘ns_1@d-gp2-couch1-3.…com’,
'ns_1@d-gp2-couch1-8.…com’,
'ns_1@d-gp2-couch1-7.…com’]}]}}} hide
ns_orchestrator 000
ns_1@d-gp2-couch1-3.…com
2:14:42 AM Thu Nov 25, 2021
Started rebalancing bucket cfg
ns_rebalancer 000
ns_1@d-gp2-couch1-3.*****…com
2:14:27 AM Thu Nov 25, 2021

3

also see this when trying to rebalance…

[ns_server:warn,2021-11-25T03:53:01.281Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

[ns_server:info,2021-11-25T03:53:01.359Z,ns_1@d-gp2-couch1-1.imovetv.com:<0.8565.0>:diag_handler:log_all_dcp_stats:195]logging dcp stats

[ns_server:info,2021-11-25T03:53:01.360Z,ns_1@d-gp2-couch1-1.imovetv.com:<0.8565.0>:diag_handler:log_all_dcp_stats:199]end of logging dcp stats

[ns_server:warn,2021-11-25T03:53:02.282Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

[ns_server:warn,2021-11-25T03:53:03.283Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

[ns_server:warn,2021-11-25T03:53:04.285Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

[ns_server:warn,2021-11-25T03:53:05.286Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

[ns_server:warn,2021-11-25T03:53:06.288Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

[ns_server:warn,2021-11-25T03:53:07.289Z,ns_1@d-gp2-couch1-1.imovetv.com:kv_monitor<0.2597.0>:kv_monitor:get_buckets:275]The following buckets are not ready: [“cfg”]

4

the problem ended up being that node 3 was not communicating properly. I ended up having to bounce node 3 and that solved the problem.