Error Service 'query' exited with status 134. Restarting

Couchbase Server
1

Hi

I got a lot of errors similar to the one below in our Couchbase web console(under Logs). Other than this error , there is no other errors from the log. Clients(java client) get “connection refused” error when this happens on Couchbase. Couchbase will restart the query service and then client will be back to normal. I checked memory, cpu, etc. Everything is OK. In fact, CPU(6%) and memory(18%) usage are low. This is repeatable. We try to resolve this issue by completely re-install couchbase and this problem still exists. We are running Cocubase community version 5.1 on Ubuntu 16.04 on AWS EC2.

Service ‘query’ exited with status 134. Restarting. Messages:
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/sync/once.go:44 +0xbe fp=0xc420af5f28 sp=0xc420af5ef0
github.com/couchbase/query/execution.(*base).runConsumer(0xc4203a0500, 0x169c660, 0xc4203a0500, 0xc420161a40, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:287 +0xad fp=0xc420af5f80 sp=0xc420af5f28
github.com/couchbase/query/execution.(*InitialProject).RunOnce(0xc4203a0500, 0xc420161a40, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:45 +0x5c fp=0xc420af5fc0 sp=0xc420af5f80
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420af5fc8 sp=0xc420af5fc0
created by github.com/couchbase/query/execution.(*base).runConsumer.func1
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:252 +0x487
[goport(/opt/couchbase/bin/cbq-engine)] 2018/02/28 16:49:22 child process exited with status 134

Service ‘query’ exited with status 134. Restarting. Messages:
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/net/fd_poll_runtime.go:80 +0x34 fp=0xc42026aea0 sp=0xc42026ae70
net.(*netFD).Read(0xc420578070, 0xc42c7aded1, 0x1, 0x1, 0x0, 0x1694320, 0x168a650)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/net/fd_unix.go:250 +0x1b7 fp=0xc42026af08 sp=0xc42026aea0
net.(*conn).Read(0xc42bc321b0, 0xc42c7aded1, 0x1, 0x1, 0x0, 0x0, 0x0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/net/net.go:181 +0x70 fp=0xc42026af78 sp=0xc42026af08
net/http.(*connReader).backgroundRead(0xc42c7adec0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/net/http/server.go:656 +0x58 fp=0xc42026afd8 sp=0xc42026af78
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42026afe0 sp=0xc42026afd8
created by net/http.(*connReader).startBackgroundRead
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/net/http/server.go:652 +0xdf
[goport(/opt/couchbase/bin/cbq-engine)] 2018/02/28 13:46:50 child process exited with status 134

Service ‘query’ exited with status 134. Restarting. Messages:
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/sync/once.go:44 +0xbe fp=0xc420601f28 sp=0xc420601ef0
github.com/couchbase/query/execution.(*base).runConsumer(0xc425c0e000, 0x169c460, 0xc425c0e000, 0xc420392140, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:287 +0xad fp=0xc420601f80 sp=0xc420601f28
github.com/couchbase/query/execution.(*Fetch).RunOnce(0xc425c0e000, 0xc420392140, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/fetch.go:49 +0x5c fp=0xc420601fc0 sp=0xc420601f80
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420601fc8 sp=0xc420601fc0
created by github.com/couchbase/query/execution.(*base).runConsumer.func1
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:252 +0x487
[goport(/opt/couchbase/bin/cbq-engine)] 2018/02/28 13:09:49 child process exited with status 134

Service ‘query’ exited with status 134. Restarting. Messages:
github.com/couchbase/query/execution.(*Authorize).RunOnce.func1()
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/authorize.go:91 +0x473 fp=0xc4207a1f40 sp=0xc4207a1e20
sync.(*Once).Do(0xc42041bd90, 0xc42011af88)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/sync/once.go:44 +0xbe fp=0xc4207a1f78 sp=0xc4207a1f40
github.com/couchbase/query/execution.(*Authorize).RunOnce(0xc42041bd40, 0xc420080140, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/authorize.go:101 +0x9b fp=0xc4207a1fc0 sp=0xc4207a1f78
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4207a1fc8 sp=0xc4207a1fc0
created by github.com/couchbase/query/execution.(*base).runConsumer.func1
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:252 +0x487
[goport(/opt/couchbase/bin/cbq-engine)] 2018/02/28 11:43:29 child process exited with status 134

2

You might have run out of memory and query service might restarted. Check /var/log/messages around time query service restarted.

3

Hi

Thx for the quick rely.
/var/log/messages does not exist. I assume it means no system-level error?
However, I see error in /opt/couchbase/var/lib/couchbase/logs/error.log. Please see attached file.errors.zip (6.8 KB)

4

Based on your OS check system OS log message. Looks like it is /var/log/syslog

5

System log is all clean. Please see attached syslogsyslog.zip (29.2 KB)

6

check out of memory in /var/log as described here https://askubuntu.com/questions/709336/how-to-find-out-why-process-was-killed-on-server

7

I don’t see how it can be helpful on solving this issue. I see Couchbase keep restarting it’s query service. As I mentioned earlier, I do see error on couchbase log and I posted the log. Are you able to tell what is wrong from the couchbase log? You mentioned that there is a memory issue. This server is only running couchbase. If there is a memory issue, it is caused by Couchbase. Also, I do not see any memory issue from Couchbase console or the system.

ubuntu@ip-172-31-24-79:~$ free -m
total used free shared buff/cache available
Mem: 7982 1048 5812 8 1121 6615
Swap: 0 0 0

8

Here are the screen shots from Couchbase consoleUntitledUntitled2

9

Hi @mikehcumail
Could you upload the error.log, the babysitter.log , the debug.log and the query.log.
It will give us a better idea of what is happening.
(If you could upload the entire logs directory that would be best.)

Thanks
Isha

10

Could you please confirm the version? The reason I am asking is because that stack does not look from 5.1, but 5.0.x instead.
This said, having a look at the stack it looks like your execution tree is malformed, in that a producer (InitialProject, Fetch) is trying to initiate a consumer, but the consumer address is not set, which seems to indicate that the instantiation phase did not allocate the operators correctly.
As @vsr1 points out, this is very likely down to lack of memory.

11

Hi all

Thx for replying. I really appreciate it.
I’m get the entire logs directory. It is big. It will take a while.
@Marco_Greco, you are correct. I’m using 5.01 community version, not 5.1.

12

Hi,

The file is too big to be uploaded. Could you please download it from
https://couchbase-error-logs.s3.amazonaws.com/log.zip

13

Hi

For your information, we did a test on Enterprise version 5.1 and we do see the same error happening.

14

query.log has following fatal error. That is reason query service is restarted. Opened MB-28443
Could you please post the exact sql statement you are running.

fatal error: concurrent map read and map write

goroutine 74091 [running]:
runtime.throw(0xe80b7a, 0x21)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:596 +0x95 fp=0xc420408cf8 sp=0xc420408cd8
runtime.mapaccess1_faststr(0xd2e800, 0xc421df47b0, 0xe6cdcd, 0xa, 0xc417dc11bb)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/hashmap_fast.go:217 +0x4cf fp=0xc420408d58 sp=0xc420408cf8

15

Hi vsr1

Thx for your detail explanation. Please see the code below. Am I correct that this issue only happen if $ids contains duplicated keys? We examined out code and we confirmed there are cases it may contains duplicated keys. However, it should never have more than 2 keys with the same value. For example: [k1, k1, k2, k3, k3].

StringBuffer statement = new StringBuffer(“select meta().id as id, url from " + bucket.name() + " use keys $ids”);
JsonObject params = JsonObject.create();
params.put(“ids”, ids);
N1qlQueryResult result = bucket.query(N1qlQuery.parameterized(statement.toString(), params));
if (!result.finalSuccess())
throw new CouchbaseException("Query returned with errors: " + result.errors());
for (N1qlQueryRow row : result)
out.put(row.value().getString(“id”), row.value().getString(“url”));

16

This happens only when duplicates. You can remove duplicates in the application or try this
USE EYS ARRAY_DISTINCT($ids)