I’ve got some software that communicates with couchbase and the sync gateway, relying on the latter’s _changes API. This software has a suite of automated tests, and between each test I’d like to reset both couchbase and the sync gateway. Couchbase is pretty easy, I can just DELETE FROM <bucket>, but I’m having some trouble getting the sync gateway reset. I’ve tried deleting and re-creating the database, but surprisingly that doesn’t appear to clear things out. The _changes API still gives me a bunch of deleted documents, and the seq continues to increase. Is there an API I can hit (or maybe something I can do in couchbase) to actually get the sync gateway back to a fresh state? I define “fresh state” to be “gives me nothing on the _changes API, and has a seq of 0”.
Ah ha, looks like deleting and re-creating the couchbase bucket does it.
When a document is removed from Couchbase Server, for example via N1QL delete, it becomes a Server Tombstone, which is still able to be interacted via Sync Gateway. These have to be ejected manually or done via automatic compaction as per Tombstones | Couchbase Docs
Given that you also want the the full state of your Sync Gateway database to be fresh, deleting the bucket is definitely the easiest way to go about this.
@torcolvin the sync gateway doesn’t appear to be handling this very well. As soon as I blow away the couchbase bucket the sync gateway log immediately starts filling with this over and over:
2024-05-20T16:08:36.163Z [INF] CBGoUtilsLogger: Using plain authentication for user <ud>sync_gateway</ud>
2024-05-20T16:08:36.163Z [INF] CBGoUtilsLogger: Using plain authentication for user <ud>sync_gateway</ud>
2024-05-20T16:08:36.165Z [WRN] c:sync_test-SGI Error processing DCP stream - will attempt to restart/reconnect if appropriate: UPROpen receive, err: MCResponse status=EACCESS, opcode=UPR_OPEN, opaque=4027388468, msg: {"error":{"context":"Authorization failure: can't execute DCP_OPEN operation without the DcpProducer privilege","ref":"4e5881f8-5dda-4a1e-7379-d24f79d969ec"}}. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:61
2024-05-20T16:08:36.165Z [WRN] c:sync_test-SG Error processing DCP stream - will attempt to restart/reconnect if appropriate: UPROpen receive, err: MCResponse status=EACCESS, opcode=UPR_OPEN, opaque=4027388468, msg: {"error":{"context":"Authorization failure: can't execute DCP_OPEN operation without the DcpProducer privilege","ref":"316814e6-0abc-41db-4d33-317bdf76469b"}}. -- base.(*DCPReceiver).OnError() at dcp_receiver.go:61
I don’t see any obvious errors in the log on the couchbase side. Is there something else I’m supposed to do to get the sync gateway happy after I pull the rug out from under it? I tried taking the database offline before blowing the bucket away, but that didn’t seem to change anything. If I blow the database away as well, it fails to re-authenticate to couchbase, which is very confusing.
You will want to remove the database from Sync Gateway before deleting the bucket.
If a bucket is deleted from under Sync Gateway, the behavior is not defined and as you can see it will try to reconnect repeatedly.
I was running into auth issues when trying to do that, but I just found out: the _config endpoint doesn’t give me valid credentials (looks like it gives me a password that is asterisks, haha), so trying to re-create the database using the returned config doesn’t actually have valid creds to reach couchbase again. I just needed to re-use my initial config instead of using that strategy and this appears to work, although it was a little more involved. For completeness and for future readers, the order of operations for my test teardown is:
- Take the SG database offline
- Delete the SG database
- Delete the CB bucket
- Re-create the CB bucket
- Reset the CB user for SG to have the same roles (otherwise it looks like that was wiped out by step 3)
- Re-create the SG database
- Take the SG database online
I’m afraid there is still some instability with this solution. The above works most of the time, but sometimes the sync gateway gets upset again saying this over and over:
2024-05-20T21:24:24.008Z [INF] Error waiting for index "sg_access_x1" to be ready for bucket "test_test" - retrying...
2024-05-20T21:24:25.645Z [WRN] Error when querying index using statement: [SELECT meta(`sync_test`).xattrs._sync.access.foo as val FROM `sync_test` USE INDEX (sg_access_x1) WHERE ANY op in OBJECT_PAIRS(meta(`sync_test`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1] parameters: [<nil>] error:[4000] No index available on keyspace `default`:`sync_test` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`sync_test` to create a primary index, or check that your expected index is online. -- base.(*CouchbaseBucketGoCB).Query() at bucket_n1ql.go:75
I haven’t figured out how to programatically kick it out of that state at this point. @torcolvin any thoughts?
Some more logs might help. Here is a normal run, right where I re-create the SG database (you can see the PUT):
2024-05-20T21:43:28.825Z [INF] HTTP: #039: PUT /sync_test/ (as ADMIN)
2024-05-20T21:43:28.826Z [INF] Opening db /sync_test as bucket "sync_test", pool "default", server <couchbase://couchbase>
2024-05-20T21:43:28.826Z [INF] GoCBCustomSGTranscoder Opening Couchbase database sync_test on <couchbase://couchbase> as user "sync_gateway"
2024-05-20T21:43:28.901Z [INF] Auth: Attempting credential authentication against bucket sync_test on couchbase://couchbase
2024-05-20T21:43:28.916Z [INF] Successfully opened bucket sync_test
2024-05-20T21:43:28.940Z [INF] Set query timeouts for bucket sync_test to cluster:1m15s, bucket:1m15s
2024-05-20T21:43:28.940Z [INF] Initializing indexes with numReplicas: 0...
2024-05-20T21:43:29.035Z [INF] Query: Index sg_syncDocs_x1 doesn't exist, creating...
2024-05-20T21:43:29.129Z [INF] Query: Index sg_syncDocs_x1 created successfully
2024-05-20T21:43:29.164Z [INF] Query: Index sg_access_x1 doesn't exist, creating...
2024-05-20T21:43:29.268Z [INF] Query: Index sg_access_x1 created successfully
2024-05-20T21:43:29.297Z [INF] Query: Index sg_roleAccess_x1 doesn't exist, creating...
2024-05-20T21:43:29.386Z [INF] Query: Index sg_roleAccess_x1 created successfully
2024-05-20T21:43:29.387Z [INF] Query: Index sg_channels_x1 doesn't exist, creating...
2024-05-20T21:43:29.540Z [INF] Query: Index sg_channels_x1 created successfully
2024-05-20T21:43:29.620Z [INF] Query: Index sg_allDocs_x1 doesn't exist, creating...
2024-05-20T21:43:29.763Z [INF] Query: Index sg_allDocs_x1 created successfully
2024-05-20T21:43:29.817Z [INF] Query: Index sg_tombstones_x1 doesn't exist, creating...
2024-05-20T21:43:29.961Z [INF] Query: Index sg_tombstones_x1 created successfully
2024-05-20T21:43:30.017Z [INF] Query: Building deferred indexes: [sg_syncDocs_x1 sg_roleAccess_x1 sg_allDocs_x1 sg_channels_x1 sg_access_x1 sg_tombstones_x1]
2024-05-20T21:43:31.893Z [INF] Verifying index availability for bucket sync_test...
2024-05-20T21:43:31.900Z [INF] Indexes ready for bucket sync_test.
Note these two lines in particular:
2024-05-20T21:43:29.164Z [INF] Query: Index sg_access_x1 doesn't exist, creating...
2024-05-20T21:43:29.268Z [INF] Query: Index sg_access_x1 created successfully
Now look at the one that fails:
2024-05-20T21:24:19.416Z [INF] HTTP: #060: PUT /sync_test/ (as ADMIN)
2024-05-20T21:24:19.416Z [INF] Opening db /sync_test as bucket "sync_test", pool "default", server <couchbase://couchbase>
2024-05-20T21:24:19.416Z [INF] GoCBCustomSGTranscoder Opening Couchbase database sync_test on <couchbase://couchbase> as user "sync_gateway"
2024-05-20T21:24:19.488Z [INF] Auth: Attempting credential authentication against bucket sync_test on couchbase://couchbase
2024-05-20T21:24:19.491Z [INF] Successfully opened bucket sync_test
2024-05-20T21:24:19.497Z [INF] Set query timeouts for bucket sync_test to cluster:1m15s, bucket:1m15s
2024-05-20T21:24:19.497Z [INF] Initializing indexes with numReplicas: 0...
2024-05-20T21:24:19.524Z [INF] Query: Index sg_roleAccess_x1 doesn't exist, creating...
2024-05-20T21:24:19.957Z [INF] Query: Index sg_roleAccess_x1 created successfully
2024-05-20T21:24:20.030Z [INF] Query: Index sg_channels_x1 doesn't exist, creating...
2024-05-20T21:24:20.162Z [INF] Query: Index sg_channels_x1 created successfully
2024-05-20T21:24:20.176Z [INF] Query: Index sg_allDocs_x1 doesn't exist, creating...
2024-05-20T21:24:20.271Z [INF] Query: Index sg_allDocs_x1 created successfully
2024-05-20T21:24:20.272Z [INF] Query: Index sg_tombstones_x1 doesn't exist, creating...
2024-05-20T21:24:20.409Z [INF] Query: Index sg_tombstones_x1 created successfully
2024-05-20T21:24:20.473Z [INF] Query: Index sg_syncDocs_x1 doesn't exist, creating...
2024-05-20T21:24:20.580Z [INF] Query: Index sg_syncDocs_x1 created successfully
2024-05-20T21:24:20.599Z [INF] Query: Building deferred indexes: [sg_roleAccess_x1 sg_tombstones_x1 sg_syncDocs_x1 sg_channels_x1 sg_allDocs_x1]
2024-05-20T21:24:22.495Z [INF] Verifying index availability for bucket sync_test...
2024-05-20T21:24:22.498Z [WRN] Error when querying index using statement: [SELECT meta(`sync_test`).xattrs._sync.access.foo as val FROM `sync_test` USE INDEX (sg_access_x1) WHERE ANY op in OBJECT_PAIRS(meta(`sync_test`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1] parameters: [<nil>] error:[4000] No index available on keyspace `default`:`sync_test` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`sync_test` to create a primary index, or check that your expected index is online. -- base.(*CouchbaseBucketGoCB).Query() at bucket_n1ql.go:75
2024-05-20T21:24:22.498Z [INF] Error waiting for index "sg_access_x1" to be ready for bucket "sync_test" - retrying...
2024-05-20T21:24:22.600Z [WRN] Error when querying index using statement: [SELECT meta(`sync_test`).xattrs._sync.access.foo as val FROM `sync_test` USE INDEX (sg_access_x1) WHERE ANY op in OBJECT_PAIRS(meta(`sync_test`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1] parameters: [<nil>] error:[4000] No index available on keyspace `default`:`sync_test` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`sync_test` to create a primary index, or check that your expected index is online. -- base.(*CouchbaseBucketGoCB).Query() at bucket_n1ql.go:75
Note how it’s lacking the lines saying that it’s trying to create the sg_access_x1 index. It never comes back from this, and ultimately needs to be completely restarted.