Documents not syncing to mobile clients with reserved internal property _rev after upgrade to 3.1.0

[fifteen_renditions]

Hello,

since upgrading to 3.1.0, we have problems where the sync between sync gateway and mobile clients hangs indefinitely while couchbase lite is stuck on busy.
We have created an issue for the cb lite package as well (Pull replicator get stuck at busy state with SGW 3.1 · Issue #493 · cbl-dart/cbl-dart · GitHub), but we are not sure where this error comes from.

Since the upgrade to 3.1.0, some documents are created containing the _rev key as normal body field.
Right now, we believe this happens to documents synced by cb lite via blip. cb lite shows no knowledge of a _rev key inside the json body, so it is unclear if this really originates from the mobile client or if the sync gateway creates this key.
The _rev key has the same value as the meta field, so this could indicate that the problem lies within sync gateway.

This are logs from cb lite, the id cedad0b6-53a8-45d4-ab5a-c125b6fbc256 has the rogue _rev field set:
cbl_info_1689116934346.cbllog.zip (54.6 KB)
Couchbase server shows the _rev field as normal body key:

The sync gateway has the following debug log for this document:

2023-07-12T01:23:53.086+02:00 [DBG] SyncMsg+: #22: Type:rev Id:<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud> Rev:1-c47ef57b5bec43157c4d6a61bf43992a24585932 Sequence:1584 
2023-07-12T01:23:53.086+02:00 [DBG] CRUD+: Invoking sync on doc "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" rev 1-c47ef57b5bec43157c4d6a61bf43992a24585932
2023-07-12T01:23:53.086+02:00 [INF] CRUD:       Doc "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" / "1-c47ef57b5bec43157c4d6a61bf43992a24585932" in channels "{<ud>C_DOCUMENT_56273db2-244a-40f1-95da-8df732085b74</ud>, <ud>c_iss_984b04e6-0fcb-40e5-8d7e-853ff80374df</ud>, <ud>c_b_a0ccf2f7-bb35-403f-8669-6511364eedbd</ud>, <ud>c_f_984b04e6-0fcb-40e5-8d7e-853ff80374df</ud>, <ud>c_st_a0ccf2f7-bb35-403f-8669-6511364eedbd</ud>, <ud>c_c_1ba2e93e-c98c-4b20-9c25-97041eea5d90</ud>, <ud>dn_1ba2e93e-c98c-4b20-9c25-97041eea5d90</ud>, <ud>asset_verifier</ud>, <ud>dn_cc_17e95970-324a-4641-9a81-33a9f8a19af5</ud>, <ud>si_inc_273abad9-a9fa-45a9-899b-ad053f99c3af</ud>, <ud>c_si_inc_a0ccf2f7-bb35-403f-8669-6511364eedbd</ud>, <ud>c_sf_984b04e6-0fcb-40e5-8d7e-853ff80374df</ud>, <ud>dn_a0ccf2f7-bb35-403f-8669-6511364eedbd</ud>, <ud>dn_pr_273abad9-a9fa-45a9-899b-ad053f99c3af</ud>, <ud>ar_inc_17e95970-324a-4641-9a81-33a9f8a19af5</ud>, <ud>c_ar_inc_a0ccf2f7-bb35-403f-8669-6511364eedbd</ud>, <ud>c_se_984b04e6-0fcb-40e5-8d7e-853ff80374df</ud>, <ud>dn_984b04e6-0fcb-40e5-8d7e-853ff80374df</ud>}"
2023-07-12T01:23:53.086+02:00 [INF] Access: Doc "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" grants channel access: <ud>map[login.vestigas.com%2Fauth%2Frealms%2Fvestigas_830acde4-ef26-4411-a9b1-150957cb3a83:C_DOCUMENT_56273db2-244a-40f1-95da-8df732085b74:13706197]</ud>
2023-07-12T01:23:53.087+02:00 [DBG] CRUD+: Saving doc (seq: #13706197, id: <ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud> rev: 1-c47ef57b5bec43157c4d6a61bf43992a24585932)
2023-07-12T01:23:53.087+02:00 [DBG] CRUD+: Stored doc "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" / "1-c47ef57b5bec43157c4d6a61bf43992a24585932" as #13706197
2023-07-12T01:23:53.087+02:00 [INF] Access: Rev "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" / "1-c47ef57b5bec43157c4d6a61bf43992a24585932" invalidates channels of [login.vestigas.com%2Fauth%2Frealms%2Fvestigas_830acde4-ef26-4411-a9b1-150957cb3a83]
2023-07-12T01:23:53.087+02:00 [DBG] DCP+: db:db Received #13706197 after   0ms ("<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" / "1-c47ef57b5bec43157c4d6a61bf43992a24585932")
2023-07-12T01:23:53.088+02:00 [DBG] Changes+: c:#799298 db:db MultiChangesFeed sending <ud>{Seq:13706197, ID:cedad0b6-53a8-45d4-ab5a-c125b6fbc256, Changes:[map[rev:1-c47ef57b5bec43157c4d6a61bf43992a24585932]]}</ud> <ud>  (to asset-verifier)</ud>
2023-07-12T01:23:53.091+02:00 [DBG] Changes+: MultiChangesFeed sending <ud>{Seq:13706197, ID:cedad0b6-53a8-45d4-ab5a-c125b6fbc256, Changes:[map[rev:1-c47ef57b5bec43157c4d6a61bf43992a24585932]]}</ud> <ud>  (to login.vestigas.com%2Fauth%2Frealms%2Fvestigas_830acde4-ef26-4411-a9b1-150957cb3a83)</ud>
2023-07-12T01:23:53.795+02:00 [DBG] Changes+: c:#845437 db:db MultiChangesFeed sending <ud>{Seq:13706197, ID:cedad0b6-53a8-45d4-ab5a-c125b6fbc256, Changes:[map[rev:1-c47ef57b5bec43157c4d6a61bf43992a24585932]]}</ud> <ud>  (to login.vestigas.com%2Fauth%2Frealms%2Fvestigas_830acde4-ef26-4411-a9b1-150957cb3a83)</ud>
2023-07-12T01:23:54.406+02:00 [DBG] Import+: db:db Did not import doc "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" - external update will not be accessible via Sync Gateway.  Reason: 400 top-level property '_rev' is a reserved internal property therefore cannot be imported
2023-07-12T01:23:58.863+02:00 [DBG] Import+: db:db Did not import doc "<ud>cedad0b6-53a8-45d4-ab5a-c125b6fbc256</ud>" - external update will not be accessible via Sync Gateway.  Reason: 400 top-level property '_rev' is a reserved internal property therefore cannot be imported

This leads to the following error when querying sync gateway REST API:

{
    "error": "Bad Request",
    "reason": "top-level property '_rev' is a reserved internal property therefore cannot be imported"
}

When cb lite tries to sync documents, we find the following error in sg_warn.log:

2023-07-11T20:38:29.026+02:00 [WRN] c:#744868 db:db Changes feed: error getting revision body for "<ud>d95ed1b9-8866-455c-bc91-0decf6da18dd</ud>" (1-46504989366ec3e4fe1863d5eecb2609f0ba16fa): 400 top-level property '_rev' is a reserved internal property therefore cannot be imported -- db.(*DatabaseCollectionWithUser).addDocToChangeEntry() at changes.go:133
2023-07-11T22:45:23.660+02:00 [WRN] Error from bh.handleChangesResponse: failed to GetRev for doc <ud>d95ed1b9-8866-455c-bc91-0decf6da18dd</ud> with rev 1-46504989366ec3e4fe1863d5eecb2609f0ba16fa: 400 top-level property '_rev' is a reserved internal property therefore cannot be imported -- db.(*blipHandler).sendBatchOfChanges.func1() at blip_handler.go:558

When deleting the _rev key from the documents, the cb lite sync starts to work again, but sometimes only after force-closing the app, i.e. completely disconnecting and reconnecting the sync.

Sadly this bug breaks the mobile sync, one of the main benefits of couchbase and sync gateway.

This is one of more breaking bugs we have encountered since upgrading to 3.1.0, I’ll link the other (unresolved) issuer here, maybe they are somewhat related:

Every REST API call with oidc Bearer Token increases sequence number since sync gateway 3.1, leading to spamming changes feed

Hi, since upgrading to sync gateway 3.1, we experience issues with API calls authenticated using Bearer token. I am testing with Couchbase Sync Gateway/3.1.0(592;2a9837d) EE. Any API call with a Bearer Token will increase the sequence number. When using the _changes feed, this will increase the sequence number every second and spams changes. When calling changes feed with /db/_changes?include_docs=true&since=10147607 we immediately get the result back: { "results": [ { …

Http: panic serving <IP>: runtime error: invalid memory address or nil pointer dereference after sync gateway upgrade to 3.1

I’ve now tried it with a fresh install, still the same problem. These are the sync gateway console logs with log_level: trace, when making an api request: couchbase-sgw-sync-gateway-1 | 2023-06-12T23:09:23.448Z [TRC] gocb+: CCCPPOLL: Got Block: {"rev":70,"revEpoch":1,"name":"my-bucket","nodeLocator":"vbucket","bucketType":"membase","uuid":"f4f927413f0fcab1bb16cdf726e2a42f","uri":"/pools/default/buckets/my-bucket?bucket_uuid=f4f927413f0fcab1bb16cdf726e2a42f","streamingUri":"/pools/default/buck…

Sync gateway database can't go online

I recently upgraded from couchbase/sync-gateway:3.0.4-community to couchbase/sync-gateway:3.1.0-community (using these images for my docker container). I have one database, and after the upgrade it is not possible to get this db online. When starting sync gateway, I get the following error: Database has collections that require resync before it can go online: [_default._default] -- rest.(*ServerContext).TakeDbOnline() at server_context.go:1069 I have done a resync of the db. It seems it is n…

We have the same config as in our other issues, e.g. see this one: Http: panic serving <IP>: runtime error: invalid memory address or nil pointer dereference after sync gateway upgrade to 3.1 - #5 by fifteen_renditions

We would highly appreciate help here!

[perry]

Hi @fifteen_renditions - just to let you know that I’ve reached out to the maintainer of the Dart implementation of couchbase lite and ask them to look into this issue. You may also want to consider coming into our Discord server to discuss live if you like.

Looks like the other issues have all been resolved, and Sync Gateway 3.1.1 was just released: Release Notes | Couchbase Docs

[fifteen_renditions]

After some more debugging and discussion, we came to a conclusion:

• The _rev key was added by some couchbase unrelated code and was a mistake of the surrounding code (documents were received using sync gateway changes feed where they contain the _id and _rev key, and then updated via couchbase server, without removing the _rev key.)
• I think starting with sync gateway 3.1.0 the (current) flutter cb lite sync can hang if documents contain the _rev key.

[gabriel]

@fifteen_renditions Thanks for the update!