Hi all,
we have an app that we update with content from couchbase through a sync gateway.
Now, at the moment it grows revisions hence the sync gets slower and slower.
Is it feasible to just set revs_limit in the sync config file
to let’s say 3 (or even 1?), as we are not interested
in keeping versioning documents? We dont use db shadowing.
Do I have to consider anything else to keep the sync sleak?
Significantly lowering the revs_limit increases the risk of false conflicts appearing during push replication. If the client is pushing a revision with an ancestor that has been pruned away on the server, that revision will end up being treated as a conflict.
How low you can set your revs_limit depends on how your documents are being updated, and how frequently.
If you’ve got multiple authors updating documents frequently (every few seconds), then you’ll want to keep revs_limit more conservatively scoped - I’d say at least 20.
If you’ve only got a single author updating documents relatively infrequently (every minute or more), you might be fine with a revs_limit as low as 3.
I wouldn’t recommend going with revs_limit=1 - at that point you might end up with unpredictable results during both push and pull replicaiton.
thanks. yea that makes sense. We keep 3 revisions, as we only update documents server-side (no distributed updates by others)
If the server doc has updated more than three times while a particular client was offline, then when the client reconnects and pulls the document, it won’t find a common ancestor with the server, so it’ll think there’s a conflict.
In other words the revs_limit should take into account the rate at which server docs update vs. the expected time between client pulls.
Hi @jens,
thanks for clarifying. Just to be even more clear:
If I have a local document on my app on a revision that is not existing on the server anymore after not using the app for a while. I don’t change the document locally.
Now I open the app start pulling from the server all new docs.
Shouldn’t that work fine just updating to most recent docs? Even if I dont have a common revision?
Yes, the replicator will pull the current revision from the server. But if it can’t find the common ancestor, it will treat this as a conflict — it’ll still have its local revision as well as the one from the server.
Usually this isn’t a problem. The server revision will have a higher generation number (deeper tree) so it’ll be used as the default. But if your app checks for conflicts, it’ll se that there is a conflict.
Cool, thanks. that would have been my next question. How to deal with conflicts when only pulling. But if the default is the server’s revision when no ancestor is found, it will give me some peace :).
following up on this:
I am not sure if the revs_limit=10 kicks in (I saw related posts, but unfortunately they weren’t conclusive).
What is the best way of debugging? Does the limit work immediate-effectively? Or is there some later pruning going on?
I created manually 63 revisions by repeatedly clicking save in the /_admin view on a specific document.
When I look at
http://localhost:4985/zf2016/?revs=true&revs_info=true
I still see 63 revisions in {"_revisions": “_ids”:[…]}
Setting revs_limit in the Sync Gateway config is correctly limiting the revisions for me, using the same Admin UI-based test you’re using.
What version of Sync Gateway are you using? Any chance you’ve got revs_limit defined in the wrong place in your config?
Here’s my config, which is working against the master branch of SG:
{
"interface":":4984",
"adminInterface":":4985",
"log":["*"],
"databases": {
"default": {
"server": "http://localhost:8091",
"bucket": "default",
"revs_limit": 10
}
}
}
@adamf
hmm, i put it there into the db document.
We have SG 1.2 - “version”: "Couchbase Sync Gateway/HEAD(nobranch)(3aa3c25)
{
“log”: [“CRUD”, “REST+”, “Access”],
“facebook”: {“register”: true},
“pretty”: true,
“databases”: {
“zf2016”: {
“server”: “http://localhost:8091”,
“bucket”: “zf2016”,
“revs_limit”: 10,
“unsupported”: {
“user_views”: {
“enabled”:true
}
},
a lot of other stuff coming: sync function, etc.
the revisions id array is still increasing beyong 10.
@adamf actually I looked into the DB now again and there were only 10 revisions in _raw as well as
/?revs=true&revs_info=true and _raw/
However, when I click 30 times on it again (within a few seconds). It has 40 entries again.
Is that due to some caching or late garbage collection?
As for now it doesnt seem to be immediate.
it works now immediately. sorry about the confusion, somehow some black magic happened 
now it’s not workign anymore, after restarting couchbase, and sync-gateway. This system is starting to cause a major headache. It’s hard to debug, or to see if the configuration is valid. also once you have 100.000 documents, the sync gateway admin just becomes useless.
If you’ve got any additional information on the failure scenario, pass it along and we can try to reproduce.
You can use the /db/_config endpoint on the Admin REST API to see what the currently running config has defined as the revs_limit. If you’ve got a scenario where that’s showing a revs_limit that’s not being respected when updating docs, please share the details.
The Sync Gateway Admin UI is really just a development time tool at this point. We’re in the middle of work to significantly improve the Admin UI - it’s scheduled for an upcoming Sync Gateway release.
Perfect - I didn’t see the _config endpoint…
So, we had to scrape the couchbase behind, as there have been panic errors (JSON not parsable) and we believe the system just went completely bananas. Then it worked again.
Ok, since we are in production it worked fine for great 12h with 45.000 users. Then we restarted sync_gateway due to changes to the logger.
Then revs_limit has been not functional anymore and we have no clue how to diagnose or fix it.
For new documents it still works, for old it doesnt, and it is completely opaque why, and how that even happenend.
No conflicts, only one service updating documents regularly - how much can go wrong there!?
I am sorry to say this, but cb+sg is certainly not 1.x-ready and we’ve made the call to abandon it and to switch to a hosted alternative. SG is too generic that complexity and pitfalls outweigh its benefits.
I’m sorry to hear that you’ve ended up making that decision. I haven’t actually heard of other cases where users ran into these problems with revs_limit, and it’s unfortunate we haven’t been able to get to the bottom of it. I’ll review what kind of additional diagnostics we could include to help debug this issue in gurue.
On the off-chance you tried any of these before giving up, a few questions to help us get to the bottom of this:
Thanks.
@adamf we have been facing the same issue. I have not been able to get to the http://1.1.1.1:6666/_admin/custombucket/db/_config endpoint. It just shows me a blank page with the following text and a textbox to enter docID.
0 documents, highlighted documents have access control output with the current sync function.
I have set the revs_limit to 50, but currently seeing 1679 revisions in the “revs” array of the document!!! It’s got a very erratic behaviour. I tested with creating docs before and after changing revs_limit to 10 and updating them and they both stuck to the limit. I then reduced the limit to lesser than the number of revisions the docs already had(reduced it to 5) and the next update showed only 5 revisions. If this is working, why do I have a document in the same bucket with 1679 revisions!!? The change to revs_limit=10 and 5 didnt seem to affect it one bit. There are numerous docs facing the same problem.
This has slowed down replications very significantly. Eagerly waiting for your response, Thank you.
A few comments:
Ok, so the revs_limit has been set correctly. Thanks for pointing it out!
I updated the doc again, and now it has 1680 revisions.
The documents are created and updated only by the server. It is a single process so I doubt there will be any conflicts from there.
The only difference I found between the docs that stuck correctly to the revs_limit and the ones that did not, was that the latter were getting synced to around 30 users while the other docs were not getting synced to any user. Could it be that the syncing is somehow causing conflicts? There are no updates happening from the users though, only reads.
Can there be any other reason for the revs_limit to be exceeded?