Sync-gateway index sg_channels_x1 growing big

[hse]

Hello,

We have a couchbase-community 7.0.2 cluster with 3 nodes with more or less 2.5mio documents, 1 sync-gateway 2.8 running with import_docs=true and 3 sg running with without import_docs.

The sync function creates a channel per document (maybe not the best approach).
What troubles me now is the size of the sg_channel_x1 index:

71.6M items and 31.6GiB

I noticed by accident while doing restoring a backup to a new cluster, that the same index, with the same data, was much much smaller:

6.9M items and 2.85GiB

Is there any way to reduce its size?
And could it be that the reason is “retain_deleted_xattr”:true ? I could not find any info about that.

Thanks in advance for your help
Henry

[bbrks]

It’s likely that the large number of items are coming from tombstones (deleted documents). If you’re doing a lot of deletions in a short space of time (e.g. in a dev/test environment) these could accumulate fairly quickly.

We keep these around for ~30 days by default (based on the server’s metadata purge interval setting) so that deletions can be replicated to clients.

These should be cleaned up by a compaction process that runs once per day on SG import nodes to check for items older than the metadata purge interval.

Tombstones | Couchbase Docs

[hse]

Thanks bbrks for the quick answer and insight, we indeed delete a lot of documents. I’ll try to figure this out and come back here if needed.

[priya.rajagopal]

And while you are at it,

@hse

The sync function creates a channel per document (maybe not the best approach).

I would certainly rethink your access control model and determine how you want to group and segregate data. Who are your users and what documents belong to each users and what is shared