After a VM restart the Couchbase Gateway is no longer starting

Mobile Sync Gateway
1

After a VM restart the Couchbase Gateway is no longer starting,
In the logs below there are two errors:
c:#807 db:wisk-couchbase-db #807: 503 DB is currently under maintenance – rest.(*handler).writeError() at handler.go:1306
and
http: panic serving 172.19.0.4:34672: runtime error: invalid memory address or nil pointer dereference

Any tips on how to troubleshoot these errors?

I had to get a backup to get the Couchbase gateway to start, the data was accessible on the Couchbase Server.
I waited 1 hour and the DB was still in maintenance with no other information on how long it will take.

We are using couchbase/sync-gateway:3.1.0-community and couchbase:community-7.2.0

Here are the logs:

couchbase-sync-gateway_1 | 2023-06-06T21:17:44.963Z [ERR] c:#807 db:wisk-couchbase-db #807: 503 DB is currently under maintenance -- rest.(*handler).writeError() at handler.go:1306
couchbase-sync-gateway_1 | 2023/06/13 19:42:02 http: panic serving 172.19.0.4:34672: runtime error: invalid memory address or nil pointer dereference
couchbase-sync-gateway_1 | goroutine 35702591 [running]:
couchbase-sync-gateway_1 | net/http.(*conn).serve.func1()
couchbase-sync-gateway_1 | /home/couchbase/cbdeps/go1.19.5/src/net/http/server.go:1850 +0xbf
couchbase-sync-gateway_1 | panic({0x10f0140, 0x1e81250})
couchbase-sync-gateway_1 | /home/couchbase/cbdeps/go1.19.5/src/runtime/panic.go:890 +0x262
couchbase-sync-gateway_1 | github.com/couchbase/sync_gateway/db.(*DatabaseCollection).GetDocumentWithRaw(0xc00803ca80, {0x15e97c8, 0xc059aa99b0}, {0xc036de4870, 0x2f}, 0x0?)
couchbase-sync-gateway_1 | /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/community/sync_gateway/db/crud.go:81 +0x1e9
couchbase-sync-gateway_1 | github.com/couchbase/sync_gateway/db.(*DatabaseCollection).GetDocument(0xc0164964f0?, {0x15e97c8?, 0xc059aa99b0?}, {0xc036de4870?, 0xc0164964f0?}, 0x91?)
couchbase-sync-gateway_1 | /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/community/sync_gateway/db/crud.go:52 +0x28
2

Sync Gateway will not automatically come out of offline mode. That is a manual action that needs to be performed with a /db/_online request to the Admin port.

For the panic: Do you have a wider range of logs?
In particular: The HTTP request being made (#807) that is failing. I can’t see what the path was from the response log line. Only the request log line.
And the 2nd half of the stack trace. That doesn’t look like the full thing.

3

@bbrks I have called /db/_online as well and it was not working either
I even tried to delete the db and create another one (since the data was in the Couchbase Server) that didn’t help either
Here are the full logs (the Couchbase forum was giving me 503 every time I was pasting the full logs)
https://justpaste.it/a5c9s
Thanks for you help on this issue.

4

@bbrks I think my problem was caused by this: Sync gateway database can't go online
What logs can I send to find an answer to it?
This caused massive downtime for our business.