Hello and happy new year!
We have a system with 1 node with total quota (4.49 GB) unallocated 950MB with 6 buckets.
Two of them have about 60 writes/min and the he other ones collecting data every hour.
On our server we run some agents to collect the status of the data (counts lasts updates) which might have 100 or more queries every one hour.
Bucket sizes are 4m and 2m for the heavy load buckets and 2k for the others.
The server crashed and restarted several time during this week with the following messages.
-
Control connection to memcached on ‘ns_1@127.0.0.1’ disconnected: {badmatch,
{error,
timeout}} (repeated 6 times) -
Service ‘query’ exited with status 137. Restarting. Messages:
net.(*netFD).accept(0xc4207d4380, 0x0, 0x188d6c0, 0xc422ce87e0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/net/fd_unix.go:430 +0x1e5 fp=0xc42003ce48 sp=0xc42003cda0
net.(*TCPListener).accept(0xc420086380, 0xc42003ce90, 0x7162be, 0x4596f0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/net/tcpsock_posix.go:136 +0x2e fp=0xc42003ce78 sp=0xc42003ce48
net.(*TCPListener).Accept(0xc420086380, 0xfdce30, 0xc420753720, 0x1896f80, 0xc4206c2510)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/net/tcpsock.go:228 +0x49 fp=0xc42003cec8 sp=0xc42003ce78
net/http.(*Server).Serve(0xc42009f4a0, 0x1895500, 0xc420086380, 0x0, 0x0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/net/http/server.go:2643 +0x228 fp=0xc42003cf70 sp=0xc42003cec8
net/http.Serve(0x1895500, 0xc420086380, 0x188c940, 0xc4202b93b0, 0x0, 0x0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/net/http/server.go:2307 +0x78 fp=0xc42003cfb0 sp=0xc42003cf70
[goport(/opt/couchbase/bin/cbq-engine)] 2019/01/05 06:38:36 child process exited with status 137
We are about to move the server into a better Hardware but for sure we will need some advice about how to avoid such issues in the future.