Couchbase Full Text Search Crashes

Full Text Search
1

===========================================

  • On Premise
  • Couchbase Server Enterprise Edition 7.0.2 build 6703
  • 300k documents in FTS index
  • Index Replicas: 0
  • Index Partitions: 1
  • Index Type: Version 6.0 Scorch

===========================================

Hi,

We are about to start using Couchbase Full Text Search to handle full text search queries and product catalog queries for our ecommerce website. Our site currently gets about 2.5 catalog/fts requests per second.

Our new FTS solution uses ASP.NET Core Web Api and the .NET SDK to talk to Couchbase.
To prepare for launch, I did a load test from 1am to 7 am. The load test was set to 24 requests per second for the full duration.

Over those 6 hours, either Bleve or Couchbase Full Text Server crashed 3 times. Below is each crash.

02:30

panic: runtime error: integer divide by zero

goroutine 7489118 [running]:
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc01bc26c40, 0x0, 0x0, 0x4a5200, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:578 +0x402
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc01bc26c40, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:476 +0x4d
github.com/blevesearch/zapx/v15.(*PostingsIterator).Next(0xc01bc26c40, 0x0, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:465 +0x34
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotTermFieldReader).Next(0xc014e09bc0, 0xc015d8ac28, 0x10, 0x0, 0xc0150f2ce8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index/scorch/snapshot_index_tfr.go:79 +0x7b
github.com/blevesearch/bleve/v2/search/searcher.(*TermSearcher).Next(0xc015d8ac00, 0xc0161fddd0, 0xc015fb6230, 0x1, 0x2)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_term.go:88 +0xbc
github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).Next(0xc015ceeb40, 0xc0161fddd0, 0xc015fe0100, 0x3, 0x3)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_disjunction_slice.go:209 +0xfb
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc015d8ad80, 0xc0161fddd0, 0xc019bee608, 0x1, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc015d8ae00, 0xc0161fddd0, 0xc019bee610, 0x1, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc015d8ae80, 0xc0161fddd0, 0x1, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*BooleanSearcher).advanceNextMust(0xc015f67550, 0xc0161fddd0, 0xc0146701d0, 0x1, 0x2)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_boolean.go:164 +0x6b
github.com/blevesearch/bleve/v2/search/searcher.(*BooleanSearcher).Next(0xc015f67550, 0xc0161fddd0, 0x0, 0xc015693c00, 0xc0146718f8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_boolean.go:287 +0x21f
github.com/blevesearch/bleve/v2/search/collector.(*TopNCollector).Collect(0xc0129cf680, 0x1394e60, 0xc015eaad20, 0x13a3380, 0xc015f67550, 0x13a8820, 0xc015693c00, 0x0, 0x13a3380)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/collector/topn.go:226 +0x33f
github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0xc00015e070, 0x1394e60, 0xc015eaad20, 0xc0182949c0, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index_impl.go:531 +0x155d
github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc015eaacc0, 0x1394e60, 0xc015eaad20, 0xc0182949c0, 0xc012c41760, 0xc0150f36d0, 0xe6819f)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/cache_bleve.go:111 +0x37c
github.com/blevesearch/bleve/v2.(*indexAliasImpl).SearchInContext(0xc016e76050, 0x1394e60, 0xc015eaad20, 0xc0182949c0, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index_alias_impl.go:166 +0xfa
github.com/couchbase/cbft.(*SearchService).Search(0xc00046a3d8, 0xc015824360, 0x139fae0, 0xc015d27320, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/grpc_server.go:258 +0xd6a
github.com/couchbase/cbft/protobuf._SearchService_Search_Handler(0x107be00, 0xc00046a3d8, 0x139cde0, 0xc015e23bc0, 0xc01302a020, 0x12)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/protobuf/search.pb.go:949 +0x109
github.com/couchbase/cbft.serverInterceptor(0x107be00, 0xc00046a3d8, 0x139d3e0, 0xc018294900, 0xc015e23ba0, 0x122a680, 0x1394e60, 0xc015eaa3c0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/grpc_server.go:356 +0x116
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0002e3ce0, 0x13a3800, 0xc012576900, 0xc01575ac00, 0xc000412fc0, 0x1d55180, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:1206 +0x45a
google.golang.org/grpc.(*Server).handleStream(0xc0002e3ce0, 0x13a3800, 0xc012576900, 0xc01575ac00, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc012e955c0, 0xc0002e3ce0, 0x13a3800, 0xc012576900, 0xc01575ac00)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:708 +0xa1

04:00 (didn’t crash)

2023/03/04 04:00:20 http: panic serving 172.30.21.96:54450: runtime error: integer divide by zero
goroutine 7635352 [running]:
net/http.(*conn).serve.func1(0xc0128acaa0)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1767 +0x139
panic(0x1035500, 0x1d42390)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
net/http.(*timeoutHandler).ServeHTTP(0xc014282300, 0x138ffe0, 0xc01c5608c0, 0xc0141e8300)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:3237 +0x96a
main.wrapTimeoutHandler.func1(0x138ffe0, 0xc01c5608c0, 0xc0141e8300)
        cbft/cmd/cbft/init_http.go:344 +0x108
net/http.HandlerFunc.ServeHTTP(0xc00056e7a0, 0x138ffe0, 0xc01c5608c0, 0xc0141e8300)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2007 +0x44
main.exportMuxRoutesToHttprouter.func1.1(0x138ffe0, 0xc01c5608c0, 0xc0141e8200, 0xc0142822c0, 0x1, 0x2)
        cbft/cmd/cbft/main.go:761 +0x9a
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000544680, 0x138ffe0, 0xc01c5608c0, 0xc0141e8200)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/julienschmidt/httprouter@v1.1.1-0.20170430222011-975b5c4c7c21/router.go:344 +0x932
net/http.serverHandler.ServeHTTP(0xc00085a7e0, 0x138ffe0, 0xc01c5608c0, 0xc0141e8200)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0128acaa0, 0x1394da0, 0xc01a969040)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2928 +0x384
panic: runtime error: index out of range [1155] with length 1155

05:53

panic: runtime error: index out of range [4038] with length 4038

goroutine 17383439 [running]:
github.com/RoaringBitmap/roaring.(*shortIterator).next(0xc006e9b2a0, 0x4500)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.23/shortiterator.go:24 +0x42
github.com/RoaringBitmap/roaring.(*intIterator).Next(0xc0195246c0, 0xc000004533)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.23/roaring.go:267 +0x34
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0151e08c0, 0x0, 0x8, 0x4a5288, 0x3, 0xc01bc7fb10)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:595 +0x125
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0151e08c0, 0x0, 0x1398ea0, 0xc0147d99a8, 0xc012508ba8, 0xc01c2962f8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:476 +0x4d
github.com/blevesearch/zapx/v15.(*PostingsIterator).Next(0xc0151e08c0, 0x0, 0x0, 0xc018c3c548, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:465 +0x34
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotTermFieldReader).Next(0xc01809b980, 0xc012508ea8, 0x1398ea0, 0xc0147d99a8, 0xc00086ace8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index/scorch/snapshot_index_tfr.go:79 +0x7b
github.com/blevesearch/bleve/v2/search/searcher.(*TermSearcher).Next(0xc012508e80, 0xc01a7609c0, 0xc018c3c540, 0x1, 0x2)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_term.go:88 +0xbc
github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).Next(0xc0126b2960, 0xc01a7609c0, 0xc0007af0e0, 0x3, 0x3)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_disjunction_slice.go:209 +0xfb
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc012508f00, 0xc01a7609c0, 0xc0277ce2e8, 0x1, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc012508f80, 0xc01a7609c0, 0xc0277ce2f0, 0x1, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc012509000, 0xc01a7609c0, 0x1, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*BooleanSearcher).advanceNextMust(0xc01270f760, 0xc01a7609c0, 0xc01bc7dfc0, 0x1, 0x2)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_boolean.go:164 +0x6b
github.com/blevesearch/bleve/v2/search/searcher.(*BooleanSearcher).Next(0xc01270f760, 0xc01a7609c0, 0x0, 0xc017226d00, 0xc01bc7e058)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_boolean.go:315 +0x5ca
github.com/blevesearch/bleve/v2/search/collector.(*TopNCollector).Collect(0xc000341590, 0x1394e60, 0xc01bb5b2c0, 0x13a3380, 0xc01270f760, 0x13a8820, 0xc017226d00, 0x0, 0x13a3380)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/collector/topn.go:226 +0x33f
github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0xc0004d40e0, 0x1394e60, 0xc01bb5b2c0, 0xc01ee8fc80, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index_impl.go:531 +0x155d
github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc01bb5b260, 0x1394e60, 0xc01bb5b2c0, 0xc01ee8fc80, 0xc000332b00, 0xc00086b6d0, 0xe6819f)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/cache_bleve.go:111 +0x37c
github.com/blevesearch/bleve/v2.(*indexAliasImpl).SearchInContext(0xc00208b4a0, 0x1394e60, 0xc01bb5b2c0, 0xc01ee8fc80, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index_alias_impl.go:166 +0xfa
github.com/couchbase/cbft.(*SearchService).Search(0xc0000be698, 0xc012393830, 0x139fae0, 0xc024b7b6d0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/grpc_server.go:258 +0xd6a
github.com/couchbase/cbft/protobuf._SearchService_Search_Handler(0x107be00, 0xc0000be698, 0x139cde0, 0xc00056fe20, 0xc014635720, 0x12)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/protobuf/search.pb.go:949 +0x109
github.com/couchbase/cbft.serverInterceptor(0x107be00, 0xc0000be698, 0x139d3e0, 0xc01ee8fbc0, 0xc00056fe00, 0x122a680, 0x1394e60, 0xc01bb5a960)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/grpc_server.go:356 +0x116
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00015cb00, 0x13a3800, 0xc002c94480, 0xc011c5e900, 0xc0004236b0, 0x1d55180, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:1206 +0x45a
google.golang.org/grpc.(*Server).handleStream(0xc00015cb00, 0x13a3800, 0xc002c94480, 0xc011c5e900, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc002ec5150, 0xc00015cb00, 0x13a3800, 0xc002c94480, 0xc011c5e900)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:708 +0xa1

6:21

panic: runtime error: index out of range [4038] with length 4038

goroutine 2488723 [running]:
github.com/RoaringBitmap/roaring.(*shortIterator).next(0xc01a232a40, 0xc0124a4200)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.23/shortiterator.go:24 +0x42
github.com/RoaringBitmap/roaring.(*intIterator).Next(0xc0124b0c00, 0x42b1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.23/roaring.go:267 +0x34
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc01c2eb180, 0x2bea, 0x573eb2750e132cd, 0xc012cead20, 0x50000c0034ccb18, 0x68b0fe)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:595 +0x125
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc01c2eb180, 0x2bea, 0x5, 0x693ac0, 0xc012d4d080, 0x5250f)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:476 +0x4d
github.com/blevesearch/zapx/v15.(*PostingsIterator).Advance(0xc01c2eb180, 0x2bea, 0x4, 0x2bea, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/zapx/v15@v15.2.1-0.20210809172947-0534019802b1/posting.go:471 +0x35
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotTermFieldReader).Advance(0xc016f3afc0, 0xc01b512af8, 0x8, 0x8, 0xc01212c4a8, 0xc01212c3a8, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index/scorch/snapshot_index_tfr.go:152 +0x200
github.com/blevesearch/bleve/v2/search/searcher.(*TermSearcher).Advance(0xc01212c480, 0xc0123f6990, 0xc01b512af8, 0x8, 0x8, 0x8, 0xffffffffffffffff, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_term.go:105 +0xe3
github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).Advance(0xc0043995e0, 0xc0123f6990, 0xc01b512af8, 0x8, 0x8, 0xc012407d18, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_disjunction_slice.go:240 +0xd1
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).advanceChild(0xc01212c500, 0xc0123f6990, 0x1, 0xc01b512af8, 0x8, 0x8, 0x1, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:251 +0x9f
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc01212c500, 0xc0123f6990, 0xc01cbcf498, 0x1, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:198 +0x132
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc01212c580, 0xc0123f6990, 0xc01cbcf4a0, 0x1, 0x1)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*ConjunctionSearcher).Next(0xc01212c600, 0xc0123f6990, 0x1, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_conjunction.go:216 +0x3b6
github.com/blevesearch/bleve/v2/search/searcher.(*BooleanSearcher).advanceNextMust(0xc0121d8c60, 0xc0123f6990, 0xc012407db0, 0x2, 0x2)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_boolean.go:164 +0x6b
github.com/blevesearch/bleve/v2/search/searcher.(*BooleanSearcher).Next(0xc0121d8c60, 0xc0123f6990, 0x0, 0xc012d4d080, 0xc012408f80)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/searcher/search_boolean.go:305 +0x6e0
github.com/blevesearch/bleve/v2/search/collector.(*TopNCollector).Collect(0xc0010154a0, 0x1394e60, 0xc0121c4330, 0x13a3380, 0xc0121d8c60, 0x13a8820, 0xc012d4d080, 0x0, 0x13a3380)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/search/collector/topn.go:226 +0x33f
github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0xc0003a4380, 0x1394e60, 0xc0121c4330, 0xc01620ee40, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index_impl.go:531 +0x155d
github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc0121c42d0, 0x1394e60, 0xc0121c4330, 0xc01620ee40, 0xc000a8c6e0, 0xc0034cd6d0, 0xe6819f)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/cache_bleve.go:111 +0x37c
github.com/blevesearch/bleve/v2.(*indexAliasImpl).SearchInContext(0xc002b47720, 0x1394e60, 0xc0121c4330, 0xc01620ee40, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/blevesearch/bleve/v2@v2.0.4-0.20210810162943-2b21ae8f266f/index_alias_impl.go:166 +0xfa
github.com/couchbase/cbft.(*SearchService).Search(0xc000010198, 0xc01658db00, 0x139fae0, 0xc001aded30, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/grpc_server.go:258 +0xd6a
github.com/couchbase/cbft/protobuf._SearchService_Search_Handler(0x107be00, 0xc000010198, 0x139cde0, 0xc003b48d80, 0xc016b39bc0, 0x12)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/protobuf/search.pb.go:949 +0x109
github.com/couchbase/cbft.serverInterceptor(0x107be00, 0xc000010198, 0x139d3e0, 0xc01620ed80, 0xc003b48d60, 0x122a680, 0x1394e60, 0xc0044019b0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/grpc_server.go:356 +0x116
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00040d1e0, 0x13a3800, 0xc000c13380, 0xc00235de00, 0xc0003e4d50, 0x1d55180, 0x0, 0x0, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:1206 +0x45a
google.golang.org/grpc.(*Server).handleStream(0xc00040d1e0, 0x13a3800, 0xc000c13380, 0xc00235de00, 0x0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0029e05c0, 0xc00040d1e0, 0x13a3800, 0xc000c13380, 0xc00235de00)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/google.golang.org/grpc@v1.24.0/server.go:708 +0xa1
2

HI,

can anyone help?

3

Hi @jg2

It would be helpful if you can you provide

  • a sample document
  • the JSON index definition
  • the actual query

Next what happens when you test the same query in the UI?

In addition if you have a support contract you can do a cbcollect and open a ticket.

Best

Jon Strabala
Principal Product Manager - Server‌

4

Well, when there is only 1 document in the index, the crash doesn’t occurr. The lowest number of documents that I have found to cause the crash is 10k. So this bug seems to be related to the number of documents in the index, but I’m not 100% sure.

Also, there is no one query that causes the crash. Any query can cause the crash, and if you resend a crashing query, it won’t necessarily crash again.

Anyway, I will probably just give you the 10k documents and my index def, and then see if you can re-create the problem. Be aware that you may have to run the load test for 8+ hours to see one crash. I think the more documents in the index, the more quickly the crash occurs, so I might give you more than 10k docs if you don’t want to wait 8+ hours to see bleve crash.

5

@jg2 Sounds good, of course I can randomly generate my own too, note, there is good chance a later version will solve your problem you seem to be on 7.0.2 you might want to try 7.1.3 the current GA.

Perhaps you can also sent the log file /opt/couchbase/var/lib/couchbase/logs/fts.log there might be a bit more info than what you posted.

Finally what is your memory quota set to for Search in the UI (Select Settings on the left)

6

Perhaps it makes more sense to continue this discussion in the enterprise support ticket? Ticket # 50613.

  • I have posted the contents of fts.log in the support ticket.
  • Yes , I am going to try upgrading couchbase today.
  • Memory quota in UI is 2241 MIB.
7

I’ve upload our data to the support ticket 50613.

8

Hello @jg2, the types of crashes you’ve mentioned above were caught and addressed in panics and segmentation violations · Issue #1606 · blevesearch/bleve · GitHub. So, recommend upgrading to the latest on 7.1.x (7.1.4) - which should carry fixes for the issues you’ve identified.
Thanks!

1 Like
9

I upgraded my dev version of couchbase to the latest version and have been running the tests for over 12 hours, and so far no crash, so I think it’s fixed.

Thanks

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.