Every 2-3 days, in some of my FTS nodes, the FTS service is restarted randomly (at times outside working hours with no use) and in web console logs appear this:
Service 'fts' exited with status 2. Restarting. Messages: /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/net.go:176 +0x6d golang.org/x/net/netutil.(*limitListenerConn).Read(0xc460b6c4b0,
Apparently everything works fine and there is no downtime. Should I worry or is it normal?
They are Scorch indexes (6.0.0 Community)
FTS servers only have this service (no Data, Analitic or Events services) and it is oversized in RAM and CPU.
FTS usage is minimal (< 1FTS query/sec and 0-100 Index Scanned/Sec).
Service 'fts' exited with status 2. Restarting. Messages:
crypto/tls.(*Conn).Read(0xc4bc2b0380, 0xc47ce8c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/crypto/tls/conn.go:1134 +0x110
net/http.(*persistConn).Read(0xc463990240, 0xc47ce8c000, 0x1000, 0x1000, 0x0, 0x0, 0xc421804b68)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/http/transport.go:1391 +0x140
bufio.(*Reader).fill(0xc4764d0c00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).Peek(0xc4764d0c00, 0x1, 0xc4962b2360, 0xc420964480, 0x0, 0xc4c0bd2ed0, 0xc4b7d00c00)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/bufio/bufio.go:129 +0x3a
net/http.(*persistConn).readLoop(0xc463990240)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/http/transport.go:1539 +0x185
created by net/http.(*Transport).dialConn
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/http/transport.go:1186 +0xa2e
[goport(/opt/couchbase/bin/cbft)] 2019/02/19 18:26:02 child process exited with status 2
Service 'fts' exited with status 2. Restarting. Messages:
godeps/src/github.com/couchbase/vellum/decoder_v1.go:240 +0x2c9
github.com/couchbase/vellum.(*FSTIterator).next(0xc422f52b60, 0xffffffffffffffff, 0x9, 0xc4249375a0)
godeps/src/github.com/couchbase/vellum/fst_iterator.go:216 +0x1bc
github.com/couchbase/vellum.(*FSTIterator).Next(0xc422f52b60, 0x1, 0xc42114acc0)
godeps/src/github.com/couchbase/vellum/fst_iterator.go:185 +0x34
github.com/blevesearch/bleve/index/scorch/segment/zap.(*DictionaryIterator).Next(0xc42031c5a0, 0x0, 0x0, 0x0)
godeps/src/github.com/blevesearch/bleve/index/scorch/segment/zap/dict.go:254 +0x12d
github.com/blevesearch/bleve/index/scorch.(*cachedFieldDocs).prepareField(0xc42114a840, 0xc42287c401, 0x1a, 0xc4212aac40)
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_segment.go:161 +0x4a1
created by github.com/blevesearch/bleve/index/scorch.(*cachedDocs).prepareFields
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_segment.go:191 +0x250
[goport(/opt/couchbase/bin/cbft)] 2019/02/15 09:46:18 child process exited with status 2
Service 'fts' exited with status 2. Restarting. Messages:
0xc4440f4db0, 0xc43c48f7a1, 0x8, 0xc4496fc4c0)
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_segment.go:161 +0x4a1
created by github.com/blevesearch/bleve/index/scorch.(*cachedDocs).prepareFields
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_segment.go:191 +0x250
goroutine 7151 [chan receive]:
github.com/blevesearch/bleve/index/scorch.(*cachedDocs).prepareFields(0xc4203726a0, 0xc427cc0800, 0x1, 0x1, 0xc4496fc4c0, 0x125d780, 0x435648)
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_segment.go:198 +0x356
github.com/blevesearch/bleve/index/scorch.(*IndexSnapshot).documentVisitFieldTermsOnSegment.func1(0xc4496fc4c0, 0xc427cc0800, 0x1, 0x1, 0xc4236e6c00)
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_index.go:564 +0x56
created by github.com/blevesearch/bleve/index/scorch.(*IndexSnapshot).documentVisitFieldTermsOnSegment
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_index.go:563 +0x503
[goport(/opt/couchbase/bin/cbft)] 2019/02/15 09:46:09 child process exited with status 2
Service 'fts' exited with status 2. Restarting. Messages:
github.com/blevesearch/bleve.MultiSearch.func1(0x1273620, 0xc4262a8510, 0xc42363e770)
godeps/src/github.com/blevesearch/bleve/index_alias_impl.go:458 +0x152
created by github.com/blevesearch/bleve.MultiSearch
godeps/src/github.com/blevesearch/bleve/index_alias_impl.go:465 +0x19d
goroutine 2269 [chan receive]:
github.com/blevesearch/bleve/index/scorch.(*cachedDocs).prepareFields(0xc420644f00, 0xc4204eee80, 0x3, 0x4, 0xc421a28800, 0x0, 0x0)
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_segment.go:198 +0x356
github.com/blevesearch/bleve/index/scorch.(*IndexSnapshot).documentVisitFieldTermsOnSegment.func1(0xc421a28800, 0xc4204eee80, 0x3, 0x4, 0xc43032c000)
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_index.go:564 +0x56
created by github.com/blevesearch/bleve/index/scorch.(*IndexSnapshot).documentVisitFieldTermsOnSegment
godeps/src/github.com/blevesearch/bleve/index/scorch/snapshot_index.go:563 +0x503
[goport(/opt/couchbase/bin/cbft)] 2019/02/15 09:45:40 child process exited with status 2
Service 'fts' exited with status 2. Restarting. Messages: /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/net.go:176 +0x6d golang.org/x/net/netutil.(*limitListenerConn).Read(0xc460b6c4b0, 0xc44bd41000, 0x1000, 0x1000, 0x1453f40, 0x72, 0x0) <autogenerated>:1 +0x5a net/http.(*connReader).Read(0xc460b6c510, 0xc44bd41000, 0x1000, 0x1000, 0x401bca, 0x44dd68, 0x4321aa997b3cc) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc44e3feea0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).Peek(0xc44e3feea0, 0x4, 0xcad79ca52a87, 0x1453f40, 0x0, 0x0, 0x1453f40) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/bufio/bufio.go:129 +0x3a net/http.(*conn).serve(0xc4d48335e0, 0x1269280, 0xc490f8ee80) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/http/server.go:1826 +0x88f created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/x86_64/go-1.9.6/go/src/net/http/server.go:2720 +0x288 [goport(/opt/couchbase/bin/cbft)] 2019/02/15 09:45:02 child process exited with status 2