We’ve encountered a synchronisation issue in our edge-device application that’s impacting data consistency. Our approach involves aggregating data at the day level, storing it in a single document. This means any transaction within a day is saved to the same document, leading to multiple mutations of the same document within a day.
Our edge app uses Sync Gateway to replicate data to edge devices. However, we’ve received reports from customers about missing data for specific days. Our initial investigation revealed that certain documents are not being synchronised to the Sync Gateway.
A notable observation is that all missing documents lack specific Sync Gateway attributes in their metadata, as seen in the UI. We also tried manually updating these docs which changes the revision number but looks SG rejects these docs because the attributes are never populated for these docs
We seek guidance on the following:
Diagnosing the Issue: How can we identify the root cause of why these specific documents are not syncing with the Sync Gateway?
Forced Synchronisation: Is there a possibility to force-sync these missing documents to the Sync Gateway using the REST API, considering that they differ only in data content but include other required fields?
Any insights or recommendations on resolving this synchronization issue would be greatly appreciated.
To confirm my understanding, this is an issue where documents in Couchbase Server are not being imported by Sync Gateway? And thus not being synced to your edge devices?
Can you clarify which revision number you’re talking about when you observe that changing?
I suspect this is a case of Couchbase Server-side modifications not being imported by Sync Gateway. This could be for a number of reasons but it’s worth checking the most obvious ones first:
Is import_docs enabled on the SG database?
Do you observe any errors in the logs during import? You should be able to search for specific document IDs
Is your Sync Function rejecting documents being written by the import process?
Yes, documents in Couchbase server are not being imported by Sync Gateway and thus not being synced to edge devices.
I’m referring to the revision number that gets updated whenever we mutate a document. This revision number is shown in the metadata of the document.
Yes, import_docs in enabled and works for 99% docs, however, a very few times it doesn’t replicate data to edge(Couchbase lite).
Hence wanted to know if there is a REST endpoint to see
force push items so that they are replicated to Couchbase lite
I think you may be hitting a bug we’ve previously fixed. The result is that a subset of documents aren’t imported until SG restarts and restarts its connections to Couchbase Server.
Unfortunately there are no APIs or even even Sync Gateway logging to see unobserved mutations that haven’t been imported, as SG is never notified about the document update in the first place to know to run import.
The issue still persists even after updating to SG 3.1
We suspect this happens whenever there is a rebalance on the data nodes, Is there a correlation?
The workaround which I created was intended to be used sporadically, however, it appears like the wrapper is being used for more than 10% of the docs where sync to edge devices fail.
I also found in these cases the revision-id(highlighted in the screenshot) shown in the meta-section of the Couchbase UI doesn’t get updated, though there is change in data.
Hello,
Make sure your import_docs is enabled in Sync Gateway’s configuration to facilitate document import from Couchbase Server. In most cases, occasional failures may occur, necessitating further action. While there isn’t a direct REST endpoint to force-push items to Couchbase Lite, updating documents on the server side can trigger Sync Gateway to initiate replication.
