Service 'indexer' exited with status 2. Restarting. Messages:

Couchbase Server
1

Hi ,

Rebalencing is happening for last 22 hours and in the logs seeing below errors

Service ‘indexer’ exited with status 2. Restarting. Messages:
[github.com/couchbase/indexing/secondary/indexer.(*indexer).sendStreamUpdateForBuildIndex(0xc4203ea000](http://github.com/couchbase/indexing/secondary/indexer.(*indexer).sendStreamUpdateForBuildIndex(0xc4203ea000), 0xc437661380, 0x1, 0x1, 0x3, 0xc42402dc76, 0x6, 0xc42fa1e000, 0x400, 0x400, …)

Thanks,
Prasanna

2

@prasanpd,

Which version of couchbase-server is this. Looks like there is a panic in indexer. Can you kindly share the indexer.log for further analysis.

Thanks,
Varun

3

Hi Varun ,

it is 6.5.1-6299 version.

Will provide you the logfile shortly.

4

Hi Varun,

It’s not allowing me to upload the log file saying i am a new user to this forum.

Is there a way i can share the indexer.log file.

Thanks,
Prasanna

5

@prasanpd,

You can mail the log file at varun [dot] velamuri [at] couchbase [dot] com

Thanks,
Varun

6

@prasanpd,

From the logs shared, this seems to be a recurring panic causing because of the index: CB_INDEX_GSI_001_006. I am yet to root-cause the issue. In the mean time, can you drop the index CB_INDEX_GSI_001_006 and check the progress.

Thanks,
Varun

7

Hi Varun ,

Thank you for the update.

I am not sure if i can drop the index upfront and not sure if there are any implications of dropping it.

However will check on this.

Thanks,
Prasanna

8

@prasanpd,

From the logs, I see that the index is getting built. So, it won’t be serving any scans. Hence I believe it is safe to drop the index. After dropping the index, I believe indexer should recover from the recurring panic. Then, we can re-create the index again.

Thanks,
Varun

9

Hi Varun,

Thank you so much for the update. will recreate the index to see if it mitigates the issue, will keep you posted.

Thanks,
Prasanna

10

@prasanpd,

From the logs I see that dataport server on port 9103 fails to start up. This is because, some other service is using this port.

2020-08-27T08:12:12.390-04:00 [Error] DATP[->dataport “:9103”] failed starting! listen tcp :9103: bind: address already in use

This port is required for indexer to listen to mutations from KV node. Please check https://docs.couchbase.com/server/6.5/install/install-ports.html for all the ports required by indexer process. Please change the port of the other service for indexer to get un-blocked.

Thanks,
Varun