Can't drop node on couchbase 6.0

Couchbase Server
1

I have three node Couchbase 6.0 cluster and I am trying to remove one node and it keeps failing on rebalance.

Log file
Rebalance exited with reason {service_rebalance_failed,index, {linked_process_died,<13621.8164.1>, {no_connection,“index-service_api”}}}ns_orchestrator 000ns_1@172.31.59.2025:58:25 PM Tue Mar 19, 2019

Starting rebalance, KeepNodes = [‘ns_1@172.31.59.204’,‘ns_1@172.31.62.190’], EjectNodes = [‘ns_1@172.31.59.202’], Failed over and being ejected nodes = ; no delta recovery nodes ns_orchestrator 004ns_1@172.31.59.2025:58:05 PM Tue Mar 19, 2019

Rebalance exited with reason {service_rebalance_failed,index, {linked_process_died,<13621.5554.1>, {no_connection,“index-service_api”}}}ns_orchestrator 000ns_1@172.31.59.2025:57:25 PM Tue Mar 19, 2019

2

I am waiting for the response.

3

@Sarbamangal_Choudhur, we’ll need to take a look at the logs to see what’s wrong here. The index service is being reported as the source of failure - {linked_process_died,<13621.8164.1>, {no_connection,“index-service_api”}

You can collect the logs by following the instructions here cbcollect_info | Couchbase Docs.

You may want to check indexer.log to see if it is crashing.

4

It is fixed now. It was port issue. We need to open port 9999 to fix the issue. I don’t know why Couchbase uses so many different ports for the different purposes.

5

Again rebalance is facing issue :
slight_smile:tarting rebalance, KeepNodes = [‘ns_1@172.31.59.202’,‘ns_1@172.31.59.204’,
‘ns_1@172.31.62.190’], EjectNodes = , Failed over and being ejected nodes = ; no delta recovery nodes

It just stuck on 43.3%.

6

Still waiting for the response

7

Hi,
Where are the logs you collect in cluster after it failed in rebalance? I don’t see any where.
Thanks

8

It shows this error:
Starting rebalance, KeepNodes = [‘ns_1@172.31.59.202’,‘ns_1@172.31.59.204’,
‘ns_1@172.31.62.190’], EjectNodes = , Failed over and being ejected nodes = ; no delta recovery nodes
ns_orchestrator 004ns_1@172.31.59.2023:05:04 PM Fri Apr 5, 2019

9

I am waiting for reply for a week.

10

@Sarbamangal_Choudhur - we will have to look at the logs to determine the cause. Could you please collect the logs using the instructions shared by @deepkaran.salooja and then upload the logs by opening a JIRA ticket?

11

I have attached the log file.

12

Can’t open Jira ticket,

13

Open Jira ticket and load the log file:
Jira Ticket : https://issues.couchbase.com/browse/MB-33692

14

@Sarbamangal_Choudhur The above bug has been updated with our findings