Service 'query' exited with status 1 and 2

Couchbase Server
1

Hi all

I have a recurring issue on my Couchbase server when querying via the .NET SDK’s IBucket.Query method. Some queries seems to work while others don’t. The exception thrown is generally related to “The underlying connection was closed” and the Couchbase logs says either “Service ‘query’ exited with status 1” or “Service ‘query’ exited with status 2” (examples below). I have searched through several docs and forum posts but haven’t been able to find a definite answer for what this message means and what might be causing it. Does anyone maybe have an idea of what could be causing this?

Any pointers would be appreciated.

Best,
Norval

Setup:

  • CB CE 6.0.0
  • .NET SDK 2.7.2

Exception examples:

System.Net.Http.HttpRequestException: ‘Error while copying content to a stream.’

  • IOException: The read operation failed, see inner exception.
  • WebException: The request was aborted: The connection was closed unexpectedly.

System.Net.Http.HttpRequestException: ‘An error occurred while sending the request.’

  • WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
  • IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
  • SocketException: An existing connection was forcibly closed by the remote host

System.Net.Http.HttpRequestException: ‘An error occurred while sending the request.’

  • Inner Exception: WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

Corresponding Couchbase logs:
Service 'query' exited with status 1. Restarting. Messages: _time=2019-05-23T10:51:29.693+00:00 _level=INFO _msg= Initialization of cbauth succeeded _time=2019-05-23T10:51:29.707+00:00 _level=INFO _msg=New store created with url http://127.0.0.1:8091 _time=2019-05-23T10:51:29.707+00:00 _level=INFO _msg=pollEOF: About to start stdin polling _time=2019-05-23T10:51:29.712+00:00 _level=INFO _msg=cbq-engine started request-size-cap=67108864 max-concurrency=4 pipeline-batch=16 max-index-api=3 timeout=0s scan-cap=512 servicers=16 pipeline-cap=512 n1ql_feat_cntrl=3 datastore=http://127.0.0.1:8091 loglevel=INFO plus-servicers=64 request-cap=256 version=2.0.0-N1QL _time=2019-05-23T10:51:29.713+00:00 _level=ERROR _msg=cbq-engine exiting with error error=listen tcp :8093: bind: address already in use HTTP_ADDR=:8093 [goport(/opt/couchbase/bin/cbq-engine)] 2019/05/23 10:51:29 child process exited with status 1

Service 'query' exited with status 2. Restarting. Messages: /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/syscall/syscall_unix.go:162 +0x49 os.(*File).read(0xc420098000, 0xc4200b2000, 0x1000, 0x1000, 0x4ac9d5, 0xc420012180, 0x5166c3) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/os/file_unix.go:165 +0x4f os.(*File).Read(0xc420098000, 0xc4200b2000, 0x1000, 0x1000, 0x0, 0x0, 0xc420012180) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/os/file.go:101 +0x76 bufio.(*Reader).fill(0xc4200762a0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/bufio/bufio.go:97 +0x117 bufio.(*Reader).ReadSlice(0xc4200762a0, 0x3a, 0x5166c3, 0x4, 0xc4200a1f48, 0x1, 0x1) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/bufio/bufio.go:338 +0xbb main.(*NetStringReader).ReadPacket(0xc420098058, 0xc4200a1fb8, 0x4d5f63, 0xc4203d4cc0, 0xc4203d4cc0, 0xc4200a1fa8) ns_server/deps/gocode/src/goport/utils.go:46 +0x48 main.(*opsReader).loop.func1(0xc4203c5c00, 0xc42007e480, 0xc4203d4ea0) ns_server/deps/gocode/src/goport/main.go:226 +0x37 created by main.(*opsReader).loop ns_server/deps/gocode/src/goport/main.go:230 +0xf9

1 Like
2

I am facing a similar issue in my system:
Error: {“msg”: “Error performing bulk get operation - cause: unable to complete action after 4 attemps”, “retry”: true, “code”: 12008}

Corresponding log:

Service 'query' exited with status 2. Restarting. Messages:
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_index3.go:131 +0x4d3
github.com/couchbase/query/util.(*Once).Do(0xc00222b788, 0xc001903f88)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:54 +0x4a
github.com/couchbase/query/execution.(*IndexScan3).RunOnce(0xc00222b680, 0xc0058d8f00, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_index3.go:52 +0x82
created by github.com/couchbase/query/execution.(*base).runConsumer.func1
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:708 +0x2d6

goroutine 1098577 [select]:
net/http.(*persistConn).writeLoop(0xc002c50fc0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/transport.go:1339 +0x966
3

Any update on this question?
I am seeing the same error message in the logs. Recently I just added a new prepared statement - might be related since the error is in the query service.