Indexer restarts frequently on on 7.2 Community Edition

Couchbase Server
1

Indexer keeps crashing on our single node cluster with the following error message:

[Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 13216018413211800430 PartitionId: 0 SliceId: 0 Crc64: 10951100166320911113 (SnapshotInfo: seqnos: 26, 26, 2468 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 77.526577ms SnapOpenDur 85.854287ms
SIGBUS: bus error
PC=0x7dd8e3700a92 m=30 sigcode=128
signal arrived during cgo execution

goroutine 9978682 [syscall]:
runtime.cgocall(0xf3701e, 0xc00ff20668)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/cgocall.go:157 +0x4b fp=0xc00ff20640 sp=0xc00ff20608 pc=0x40ae2b
github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_kvs_close(0x7dd6f9e4d0c0)
        _cgo_gotypes.go:966 +0x47 fp=0xc00ff20668 sp=0xc00ff20640 pc=0xbb5ec7
github.com/couchbase/indexing/secondary/fdb.(*KVStore).Close.func1(0x7dd6f9e4d0c0?)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/forestdb.go:57 +0x3d fp=0xc00ff206a0 sp=0xc00ff20668 pc=0xbbcd5d
github.com/couchbase/indexing/secondary/fdb.(*KVStore).Close(0xc01b1106c0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/forestdb.go:57 +0x145 fp=0xc00ff20738 sp=0xc00ff206a0 pc=0xbbcba5
github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Destroy(0xc01de1d240)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot.go:157 +0x85 fp=0xc00ff207c8 sp=0xc00ff20738 pc=0xdc3745
github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Close.func1()
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot.go:145 +0x25 fp=0xc00ff207e0 sp=0xc00ff207c8 pc=0xdc3685
runtime.goexit()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00ff207e8 sp=0xc00ff207e0 pc=0x473c61
created by github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Close in goroutine 128
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot.go:145 +0x67

goroutine 1 [select]:
runtime.gopark(0xc00692d600?, 0x4?, 0xc0?, 0x87?, 0xc00692d4e0?)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/proc.go:398 +0xce fp=0xc00692d320 sp=0xc00692d300 pc=0x44136e
runtime.selectgo(0xc00692d600, 0xc00692d4d8, 0x1372ce0?, 0x0, 0x119f906?, 0x1)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/select.go:327 +0x725 fp=0xc00692d440 sp=0xc00692d320 pc=0x4517a5
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc00149aa80)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1070 +0x1e6 fp=0xc00692d650 sp=0xc00692d440 pc=0xdcf806
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x1cffcb0?)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:516 +0x254b fp=0xc00692dca8 sp=0xc00692d650 pc=0xdcacab
main.main()
        goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:182 +0xf92 fp=0xc00692df40 sp=0xc00692dca8 pc=0xf35a32
runtime.main()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/proc.go:267 +0x2bb fp=0xc00692dfe0 sp=0xc00692df40 pc=0x440efb
runtime.goexit()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00692dfe8 sp=0xc00692dfe0 pc=0x473c61

goroutine 2 [force gc (idle), 72 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/proc.go:398 +0xce fp=0xc000070fa8 sp=0xc000070f88 pc=0x44136e
runtime.goparkunlock(...)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/proc.go:404
runtime.forcegchelper()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/proc.go:322 +0xb3 fp=0xc000070fe0 sp=0xc000070fa8 pc=0x4411d3
runtime.goexit()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000070fe8 sp=0xc000070fe0 pc=0x473c61
created by runtime.init.6 in goroutine 1
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/proc.go:310 +0x1a

This usually happens when I’m trying to upsert documents in a loop. And when indexer goes down, the upsert fails.

2

We recommend dropping all indexes and recreating them. In some cases, the index service may crash because the index has a strange structure.

1 Like
3

Thank you for you response @myoungsig.youn
I did further investigation and found this error in the indexer.log:

2024-12-27T05:57:47.932+00:00 [ERRO][FDB] Error in reading a doc block with id 1784653502932882 from a database file '/opt/couchbase/var/lib/couchbase/data/@2i/UserBucket_idx_user_address_index_1234.index/data.fdb.0'
2024-12-27T05:57:47.932+00:00 [ERRO][FDB] Error in reading a doc length from offset 7309940748013084790 in block id 1784653502932882 from a database file '/opt/couchbase/var/lib/couchbase/data/@2i/UserBucket_idx_user_address_index_1234.index/data.fdb.0'
2024-12-27T05:57:47.932+00:00 [ERRO][FDB] Error in reading the doc length metadata with offset 7309940748013084790 from a database file '/opt/couchbase/var/lib/couchbase/data/@2i/UserBucket_idx_user_address_index_1234.index/data.fdb.0'
2024-12-27T05:57:47.932+00:00 [ERRO][FDB] docio_read_doc_key error: read failure on offset 7309940748013084790 in a database file '/opt/couchbase/var/lib/couchbase/data/@2i/UserBucket_idx_user_address_index_1234.index/data.fdb.0' : FDB status -5, lastbid 0xffffffffffffffff, curblock 0xffffffffffffffff, curpos 0x0

Indexer goes down after outputing this error followed by the SIGBUS: bus error error that is mentioned in the question above. I have the index created like this:

CREATE INDEX `idx_user_address_index_1234` ON `UserBucket`(`userId`, `full_name`, `address_description`, `contact_details`, `profile_summary`, `description_with_profile`, `objectId`) WHERE (`userId` = "1234");

We have about 3000 GSI under the default collection in our production environment. If I drop all indexes, how do I go about recreating them? The only way I can think of is retrieving all indexes from system:indexes and constructing the index queries from the index metadata. Is there a better way to achieve this?

4

I think the file where you saved idx_user_address_index_1234 is weird, try dropping that index and recreating it. This seems to be the best way to do it in the community version. If you’re using it in production, you might want to consider the enterprise version, where you can get support for issues like this.

5

Try drop only index in question. @amit.kulkarni

To Get Indexes on cluster

curl -s Administrator:password@127.0.0.1:9102/getIndexStatus |jq '.status[].definition' | sed 's/^"//g' | sed 's/"$/;/g' | sed 's/\\"/"/g'

1 Like
6

Do you mean the filename is weird, or its contents? Could you elaborate?

7

Thank you for your response @vsr1
I tried dropping and recreating that particular index (idx_user_address_index_1234) but it didn’t help. The indexer is still crashing with the same error.

8

Drop the index (don’t create it) And see if indexer is stable.
Then remove few index keys from trailing (Which are not part of query predicate and see which one causing issue, if ok use non covered index).

9

Hmmm, maybe you could try deleting the idx_user_address_index_1234 index and creating an index with WHERE (userId = ‘1234’) minus the idx_user_address_index_1234 index to include all users. If that doesn’t work, consider upgrading to the latest version, 7.6.2