Couchbase 5.1.1 CE Indexer service crashes (exited with status 134)

Couchbase Server
1

Having an issue with the indexer service crashing on a node. The indexer continually restarts and is not able to recover.
indexer.log

[Warn] StreamState::updateHWT Received Partial Last Snapshot in HWT Bucket logs StreamId MAINT_STREAM vbucket 604 Snapshot 0-99 Seqno 99 Vbuuid 193060474569261 lastSnap 0-95 lastSnapSeqno 33
panic: runtime error: slice bounds out of range

goroutine 278 [running]:
panic(0xcde720, 0xc4200180b0)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc42019af78 sp=0xc42019aee8
runtime.panicslice()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:34 +0x6d fp=0xc42019afa8 sp=0xc42019af78
github.com/couchbase/indexing/secondary/collatejson.(*Codec).extractEncodedField(0xc42011a8c0, 0xc4264ea0c2, 0x0, 0x3840, 0x0, 0xc4264ea0b3, 0xf, 0x384f, 0xc4264ea0c2, 0xe54, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/desc.go:201 +0xf64 fp=0xc42019b0e0 sp=0xc42019afa8
github.com/couchbase/indexing/secondary/collatejson.(*Codec).extractEncodedField(0xc42011a8c0, 0xc4264ea031, 0xe63, 0x38d1, 0x0, 0xc4230c2d80, 0x1e, 0x40, 0xc4264ea031, 0xee5, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/desc.go:178 +0x1309 fp=0xc42019b218 sp=0xc42019b0e0
github.com/couchbase/indexing/secondary/collatejson.(*Codec).extractEncodedField(0xc42011a8c0, 0xc4264ea000, 0xee5, 0x3902, 0xb, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d, 0x4e99d3, 0x159812a, 0x2, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/desc.go:137 +0x137 fp=0xc42019b350 sp=0xc42019b218
github.com/couchbase/indexing/secondary/collatejson.(*Codec).ReverseCollate(0xc42011a8c0, 0xc4264ea000, 0xf16, 0x3902, 0xc42011d4c0, 0xb, 0xd, 0x1, 0x3a, 0x7f0fdca9e250)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/desc.go:12 +0x99 fp=0xc42019b3d8 sp=0xc42019b350
github.com/couchbase/indexing/secondary/indexer.NewSecondaryIndexEntry2(0xc423956000, 0xf16, 0x1000, 0xc42393cf60, 0x2a, 0x2a, 0xc4264ea000, 0x1, 0xc42011d4c0, 0xb, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/index_entry.go:171 +0x379 fp=0xc42019b4d8 sp=0xc42019b3d8
github.com/couchbase/indexing/secondary/indexer.NewSecondaryIndexEntry(0xc423956000, 0xf16, 0x1000, 0xc42393cf60, 0x2a, 0x2a, 0x0, 0x1, 0xc42011d4c0, 0xb, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/index_entry.go:123 +0xfa fp=0xc42019b590 sp=0xc42019b4d8
github.com/couchbase/indexing/secondary/indexer.GetIndexEntryBytes2(0xc423956000, 0xf16, 0x1000, 0xc42393cf60, 0x2a, 0x2a, 0x312d2c305b2c0000, 0x1, 0xc42011d4c0, 0xb, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/index_entry.go:505 +0x17b fp=0xc42019b668 sp=0xc42019b590
github.com/couchbase/indexing/secondary/indexer.GetIndexEntryBytes(0xc423956000, 0xf16, 0x1000, 0xc42393cf60, 0x2a, 0x2a, 0xc426510000, 0x1, 0xc42011d4c0, 0xb, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/index_entry.go:542 +0x1a8 fp=0xc42019b748 sp=0xc42019b668
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).Insert(0xc42442f340, 0xc423956000, 0xf16, 0x1000, 0xc42393cf60, 0x2a, 0x2a, 0xc423951620, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:277 +0xdd fp=0xc42019b810 sp=0xc42019b748
github.com/couchbase/indexing/secondary/indexer.(*flusher).processUpsert(0xc426473240, 0xc4239548a0, 0xc42393cf60, 0x2a, 0x2a, 0xc423951620)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:436 +0x1dd fp=0xc42019bb30 sp=0xc42019b810
github.com/couchbase/indexing/secondary/indexer.(*flusher).flush(0xc426473240, 0xc4231c4880, 0xc420160001)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:386 +0x445 fp=0xc42019bdc8 sp=0xc42019bb30
github.com/couchbase/indexing/secondary/indexer.(*flusher).flushSingleMutation(0xc426473240, 0xc4231c4880, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:335 +0x49 fp=0xc42019be18 sp=0xc42019bdc8
github.com/couchbase/indexing/secondary/indexer.(*flusher).flushSingleVbucketUptoSeqno(0xc426473240, 0x15b0000, 0xc42001da40, 0xc4261a0001, 0xc4246633f8, 0x4, 0xc40000023c, 0x66, 0x40e401, 0xc426481c80, ...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:314 +0x241 fp=0xc42019bf40 sp=0xc42019be18
runtime.goexit()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42019bf48 sp=0xc42019bf40
created by github.com/couchbase/indexing/secondary/indexer.(*flusher).flushQueue
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:195 +0x279

UI Log:

Service 'indexer' exited with status 134. Restarting. Messages:
0xc4253c835c)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/sema.go:47 +0x30 fp=0xc4253e56f8 sp=0xc4253e56d8
sync.(*WaitGroup).Wait(0xc4253c8350)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/sync/waitgroup.go:131 +0x97 fp=0xc4253e5748 sp=0xc4253e56f8
github.com/couchbase/indexing/secondary/indexer.(*flusher).flushQueue.func1(0xc4253c0001, 0xc4253c8350, 0xc4253cbd40)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:205 +0xb9 fp=0xc4253e57a8 sp=0xc4253e5748
runtime.goexit()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4253e57b0 sp=0xc4253e57a8
created by github.com/couchbase/indexing/secondary/indexer.(*flusher).flushQueue
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/flusher.go:209 +0x5a1
[goport(/opt/couchbase/bin/indexer)] 2019/03/07 21:25:55 child process exited with status 134

The only way to recover from this issue is to backup the content of the buckets and restore them to a newly provisioned couchbase node but there has been an instance where the content of a bucket being restored will cause this issue directly after restore.

2

This Issues seems to have been caused by a DESC called on a column improperly. I.E.
(...`columnA` DESC)
vs
(...`columnA DESC`)

The former was causing a DESC ordering on every Column in the index and causing the indexer to crash.

1 Like
3

@jcooper
(… columnA DESC) is the right syntax for creating descending ordered index. example:
CREATE INDEX def_city_desc ON travel-sample(city DESC);

Can you please share your full create index statement that caused the crash as well as a sample document?

Could you please share indexer.log as well?

Thanks,
Prathibha

1 Like
4

your example is fine when you are only indexing one column but becomes a problem when doing it one multiple columns:

CREATE INDEX def_city_desc ON `travel-sample` ( `state`, `city` DESC);

This will run a DESC order on all columns. If you only want city to be ordered in the index it needs to be inside like so:

CREATE INDEX def_city_desc ON `travel-sample` ( `state`, `city DESC`);

5

Hi @jcooper
If you want DESC ordering for more than one, you need to specify it explicitly for each key. Ex:
CREATE INDEX def_city_desc ON travel-sample ( state DESC, city DESC);

Also, the second example in your previous response is in correct. It treats “city DESC” as one field and tries to index it.

To summarize, if you want DESC on one key, use syntax:
CREATE INDEX def_city_desc ON travel-sample ( state, city DESC);
If you want in more than one, use below:
CREATE INDEX def_city_desc ON travel-sample ( state DESC, city DESC, country DESC);

Can you please share the create index statement for which you observed the crash?

Thanks,
Prathibha

6

This is the best example I can provide atm but my problem was resolved by changing from
...(`state`, `city` DESC)... to ...(`state`, `city DESC`)... on my index creation. So if there is no difference in these two styles there is some mystical magic at work here.