I am using couchbase server 4.5, and am looking to purge a large number of documents, the process for which will be like running a query to retrieve these documents based on a field value, and then delete them using their document id.
The number of documents involved are very high, in order of 10^5. Currently, this process takes a huge amount of time as it is performed synchronously, to keep the load at a minimal.
My query is if this process can be made faster using touch(), which will give the documents a TTL of 1 sec.
I have gone through this discussion in the forum, touch vs remove . Do share your views.
If you’re performing synchronous operations, touch vs. remove will likely be similar costs - actually touch is probably more expensive as it will have to first update the TTL to the new (1s) expiry, and then later delete them.
You say you’re performing synchronous to minimise load - is that load on the cluster or on the client? The cluster should be fine to handle a bunch of removes at once - I suggest you batch up the removes - say of the order of 100 or 1000 in one go.
Does couchbase have an option to schedule queries, so that auto-deletion can happen ?
in the way that I do not have to trigger the query to run everytime, but that it runs on a cron, or something similar ?