Indexer on one server continuously crashing

Couchbase Server
,
1

The indexer service keeps crashing (core dumped) and restarting on a single server in our cluster.
I’d like to remove the indexes that are created on this one and re-create them elsewhere, but that fails too ("DROP INDEX..." fails with “GSI Drop() - cause: Terminate Request due to server termination”).

In the GUI it shows as “Bootstrap” all time time for all the indexes that are on this particular server.

In the indexer.log, it complains about not being able to open an index DB file:
2017-03-21T22:51:34.503-04:00 [Error] ForestDBSnapshot::Open Unexpected Error Opening Main DB Snapshot (/couchdata/data/@2i/[INDEX-NAME-REMOVED]_2235194985276477961_0.index) SeqNum 16015538 FDB_RESULT_NO_DB_INSTANCE panic: Unable to open snapshot -FDB_RESULT_NO_DB_INSTANCE

Any help would be appreciated.

2

@uris, Looks like you have run into a storage issue unfortunately. You can stop couchbase service on the server, remove the contents of /couchdata/data/@2i and start the service again. This will force cleanup all the indexes on this server. and you can recreate them.

If you want the system to rebuild all the indexes on this server automatically, just leave the file MetadataStore in /couchdata/data/@2i and remove everything else.

3

That worked! thanks

btw (hijacking my own thread): is there a way to systematically get all the index definitions? I know I can run SELECT * FROM system:indexes but that doesn’t return where the indexes are created (it says 127.0.0.1 for all). Is there a more complete way to get the full definition of indexes so they can be backed-up and re-created if needed?

thanks again

4

@uris, you can use the backup/restore utility:
https://developer.couchbase.com/documentation/server/4.6/backup-restore/backup-restore.html

5

That would be the full backup restore? i wanted to get the list of defined indexes, their definitions, and where they were created (on which Nodes)

6

The index backup only has the metadata information (definitions, placement etc).

If you do not want the bucket data to be backed up, you can use --disable-data option to backup only the indexes. https://developer.couchbase.com/documentation/server/4.6/backup-restore/cbbackupmgr-config.html