Way to get failover/remaining mutation events streams from client if dcp is offline for sometime during mutations

Java SDK
1

I am doing some experiment with dcp client. So far the examples are working great. I can see the options during initializeState that, we can specify two parameter from which it tries to listen dcp stream, like StremFrom.NOW to StreamTo.INFINITY. Which is great.

But i am wondering what if my dcp client goes offline for a while and i have inserted/mutated few documents in that period of time, lets say like 40 documents. how can i get the mutation event for only those 40 documents if my dcp client goes online again. Is it possible ? does dcp client keeps track of the mutated document it processed in the bucket ?

Looking for some options like StreamFrom.NOW_REMAINING to StreamTo.Infinity !

2

Hi Fahim,

You’ll need to save and restore the session state. See the StatePersistence example in the GitHub repo for a starting point.

Thanks,
David

3

Thank you for pointing that example. It helps. The example for the dcp are using a JSON file to preserve the state.

Is there anyway to preserve the state in couchbase bucket ? I can manually write it but, it would be nice to see if the dcp client handles it automatically.

The persistence file is getting very big. I have tried this for 2k document and its size is 133k . What about millions of document. and how do i scale it ?

4

Is there anyway to preserve the state in couchbase bucket ? I can manually write it but, it would be nice to see if the dcp client handles it automatically.

There’s some code in the Elasticsearch connector’s CouchbaseCheckpointDao you could borrow. At some point I hope to extract all the generic bits of the Elasticsearch connector into a new DCP application framework, but that’s a long-term goal.

The persistence file is getting very big. I have tried this for 2k document and its size is 133k . What about millions of document. and how do i scale it ?

The size of the session state document should be roughly constant; it stores a a few bits of information for each vbucket regardless of the number of documents in the bucket. It should not get much bigger as more documents are added to Couchbase.