Sync Gateway does an "Unexpected Panic" on startup after upgrade to 3.1.0 or 3.1.1 from 3.0.8?

Mobile
1

I upgraded a working 3.0.8 sync gateway on Ubuntu 20.04.6 LTS to 3.1.1 (I’ve tried 3.1.0 too).
On startup, I get a Panic due to a nil pointer (log is below).

Looking at the trace - the failure is in migrateV30Configs routine. According to the documents here:

Users should be able to upgrade to 3.1.x from 3.0.x without manual intervention

which implies that the config is backward compatible, but perhaps its not? The actual failure is in computeMetadataID.

Any ideas appreciated.
Cheers.
Paul.

2023-08-23T02:16:30.002Z ==== Couchbase Sync Gateway/3.1.1(20;d26ad31) CE ====
2023-08-23T02:16:30.191Z [ERR] Unexpected panic: runtime error: invalid memory address or nil pointer dereference - stopping process
goroutine 1 [running]:
runtime/debug.Stack()
        /home/couchbase/cbdeps/go1.19.5/src/runtime/debug/stack.go:24 +0x65
github.com/couchbase/sync_gateway/base.FatalPanicHandler()
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/base/util.go:1534 +0x49
panic({0x10fd4e0, 0x1e9a1f0})
        /home/couchbase/cbdeps/go1.19.5/src/runtime/panic.go:884 +0x212
github.com/couchbase/sync_gateway/rest.(*bootstrapContext).computeMetadataID(0xc0004d68a0, {0x15fa348, 0xc00074a690}, 0xc00003b8e0?, 0xc000486038)
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/config_manager.go:741 +0x186
github.com/couchbase/sync_gateway/rest.(*bootstrapContext).InsertConfig(0xc0004d68a0, {0x15fa2d8?, 0xc00003a060?}, {0xc00003b8e0, 0x6}, {0x127d1c4, 0x7}, 0xc000486000)
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/config_manager.go:80 +0x22d
github.com/couchbase/sync_gateway/rest.(*ServerContext).migrateV30Configs(0xc000038460, {0x15fa2d8, 0xc00003a060})
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/config.go:1380 +0x259
github.com/couchbase/sync_gateway/rest.(*ServerContext).initializeCouchbaseServerConnections(0xc000038460, {0x15fa2d8?, 0xc00003a060})
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/server_context.go:1956 +0x23a
github.com/couchbase/sync_gateway/rest.SetupServerContext({0x15fa2d8, 0xc00003a060}, 0xc000450d80, 0x3f?)
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/config.go:1270 +0x399
github.com/couchbase/sync_gateway/rest.serverMainPersistentConfig({0x15fa2d8, 0xc00003a060}, 0xc000419380, 0xc000450900)
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/main.go:145 +0x589
github.com/couchbase/sync_gateway/rest.serverMain({0x15fa2d8, 0xc00003a060}, {0xc000036080, 0x4, 0x4})
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/main.go:58 +0x145
github.com/couchbase/sync_gateway/rest.ServerMain()
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/rest/main.go:32 +0x45
main.main()
        /home/couchbase/jenkins/workspace/sgw-unix-build/3.1.1/community/sync_gateway/main.go:24 +0x17
 -- base.FatalPanicHandler() at util.go:1534
2

Can you provide a copy of the documents in all buckets starting with the name _sync:dbconfig ? This function could be reading a config document from a bucket that is not the one your database is configured with since Sync Gateway will poll all buckets for a config file to find databases.

3

Hi @torcolvin
I only have a single bucket
(confirmed using SELECT r.name FROM system:keyspaces r; which only returns my bucket).
I have not switched to collections yet, so everything is in _default.

To fetch the requested document, I used

SELECT meta().id, * from `_default` where meta().id like "_sync:dbconfig%"

The query returned the one single record

 {
    "_default": {
      "allow_conflicts": false,
      "enable_shared_bucket_access": true,
      "name": "testdb",
      "num_index_replicas": 0,
      "sync": " <this a long function to validate all incoming data>",
      "version": "2-601117a63b4bc9041829d29aab62a55a"
    },
    "id": "_sync:dbconfig:default"
  }

I can provide the sync function separately if required. It's pretty long and the output from the query is a large blob of text (i.e. unreadable). The function hasn't changed in a couple of years.
4

The short term workaround until we release 3.1.2 would be to add a key bucket with value of the name of the bucket to the _sync:dbconfig:default document and it will remove the panic. This is a hack fix for a situation that we did not anticipate before. This should allow you to run 3.1.1.

The fix will be in CBG-3353.

5

Fix installed, and it works. Thx.

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.