Don’t know if this is the right category, maybe mobile/syncgateway would be better, but this is mostly a server related issue.
Sometime Syncgateway indexes become inconsistent with their replication (see attached picture).
Does anyone else have this problem? Why does it happen?
@losapevo, Apologies for the delayed response.
Can you please share your couchbase-server version? Also, can you grep for “CollateJSONEncode” in the path /opt/couchbase/var/lib/couchbase/logs/ on all nodes and share the output. If there are no matches, we may need cbcollect_logs to debug this issue further.
Thanks,
Varun
Couchbase v6.6.2
No matches found for “CollateJSONEncode”
We tried to move the sg indexes to other nodes via ALTER INDEX. The indexes have been recalculated and now are consistent with each other.
sg_channels_x1 went from 18mln to 10mln
also sg_tombstones_x1 went from 18mln to 600k (see Compacting bucket does not purge tombstoned document - #4 by losapevo)
I saw that sg indexes have this option "retain_deleted_xattr":true. Because of that option, could it be that when tombstones are purged, indexes are not updated and increase forever?
@losapevo I am glad to hear that the issue got resolved after moving to different nodes. Will it be possible for you to share the logs so that we can root cause the issue and take it to a proper end? As far as I know, there are no known issues in 6.6.2 version that can cause initial mismatch of documents you noticed.
With regard to retain_deleted_xattr, as I understand, sync gateway requires this field when creating the indexes to support mobile replication of documents that have been deleted. On a periodic basis (once a day), sync gateway will remove the documents that are purged in the data service.
@adamf Please correct me on the behaviour of sync gateway indexes.
Thanks,
Varun
I think this is might be related to this issue: https://issues.couchbase.com/browse/CBG-2226