What method would you recommend as being the most efficient to update the expiry values for every single document in a bucket?
I have found a bug in our system which is persisting data to couchbase with the wrong expiry time and I’d like to update all of the resident documents with new expiry values.
(We are using couchbase 4.0)
Hi, are you talking about the document TTL or a document field holding an expiryValue?
I am trying to update the document TTL values.
The most basic, straight forward way to to this would be to have a view or n1ql query that gives you every document key that need to be updated and then use the getAndTouch method.
As of now it’s not possible to set the TTL directly with N1QL. Maybe @geraldss can tell us more on this.
Correct. Not yet possible to set TTL with N1QL.
It should be pretty straightforward to issue a N1QL query and then update each of these documents through an .upsert() with the new TTL. I don’t know what SDK you’re using @dzagieboylo, but this could be as simple as creating a job queue for all of the keys/id’s returned.
If you want to get more sophisticated or it’s a large dataset you want to transform quickly, a little RxJava or the Spark connector with a N1QL query may work well. I have a blog on doing data transformation that may be worth reading.
Yeah it’s about 200M documents so I’ll definitely use some of the RxJava to implement it.
Just wanted to make sure there wasn’t a simpler way (like using N1QL directly) that I was missing!
Thanks for the detailed responses.
I’ve written a blog post about this describing a simple approach here: https://blog.markvincze.com/bulk-updating-document-expiry-in-couchbase/
(For me it took ~10 hours to update the expiry of 17M documents, so it would be a bit too slow for 200M, but I think the code I posted there can still be optimized.)