Missing Document are coming in _change feeds

Mobile
1

Hi All,

We have found that there are documents which when we make a get call the response is
{
“error”: “not_found”,
“reason”: “missing”
}

but when we fire the _change request
_changes?filter=sync_gateway/bychannel&channels=GuestStatus&include_docs=true

The complete document is available. After that if I restart the sync gateway then the document disappers from the _changes call.

I think it looks like cache issue. Hint :- May be the documents are purged and the sync gateway in not notified of the purging. (Just a guess)

I am using CBS 5.1.1 , SG 2.0. and

following is the configuration of the bucket in sync gateway :-

{
“server”: “http://...,...:8091”,
“pool”: “default”,
“bucket”: “bucketname”,
“username”: “bucketname”,
“password”: “password”,
“name”: “username”,
“sync”: “function(doc, oldDoc)\n{ \n if(doc.type)\n {\n if(doc.type == "type1" || doc.type == "type2" || doc.type == "type3")\n {\n if(doc.lastModifiedDate)\n {\n var dttimeStamp = new Date(doc.lastModifiedDate).getTime();\n dttimeStamp=dttimeStamp+(15 * 24 * 3600 * 1000);\n expiry(new Date(dttimeStamp).toISOString());\n }\n else\n {\n expiry("3600");\n }\n }\n }\n if(doc.channels) \n channel(doc.channels);\n}”,
“users”: {
“GUEST”: {
“name”: “”,
“all_channels”: null,
“disabled”: true
},
“Xuser”: {
“name”: “Xuser”,
“admin_channels”: [
“"
],
“all_channels”: null,
“password”: ""
},
“Yuser”: {
“name”: “Yuser”,
“admin_channels”: [
""
],
“all_channels”: null,
“password”: "***”
}
},
“import_docs”: “continuous”,
“cache”: {
“enable_star_channel”: null,
“channel_cache_max_length”: 100000,
“channel_cache_min_length”: 50,
“channel_cache_expiry”: 600
},
“rev_cache_size”: 200000,
“unsupported”: {
“user_views”: {},
“oidc_test_provider”: {}
},
“enable_shared_bucket_access”: true,
“session_cookie_name”: “”
}

CC @househippo, @adamf, @bbrks, @traun

Regards
Pankaj Sharma

2

Purge doesn’t remove versions of the document that may be cached by Sync Gateway node(s) - it’s strictly data storage removal.

Additional details here: https://github.com/couchbase/sync_gateway/issues/1658

3

So according to what I understood if I reduce or remove “channel_cache_expiry”: 600. Will it remove from the cache of sync gateway. Because restarting the sync gateway surely removed it from change request.

Secondly is it possible to create a temporary document in couch-base to notify syncgateways that this document which was part of these channels have been purged, and please remove from the cache.

And this document should live for max amount of cache expiry time + some grace period.

This do create issues when there are high number of documents and those documents are not required after some days in the app. If we soft delete them then tombstone is sent to device. And there is no way to remove tombstones from the app.

The point can be said that the tombstones are not heavy and wont have much effect but what if in an app running for 6 months have 1 million tombstone documents. Then it does make an impact.

if I set TTL or Use expiry_timestamp in the app for documents then I am getting crash in the app because of

If I try to emit deleted document in a mapper then the document body is null, means the document.getproperties returns null. And I am not sure whether we can emit the deleted documents or not.

Please help me out of this. Because I dont want to go in SQL directly and run queries. Because that may not be supported in Enterprise support.

Please do note that in every 7 days each app will get 50000 new documents, and 50000 documents will be deleted after some time of new documents arrive.

The best thing which can work for me is if the crash can be removed. Then everything is set up for the release.