I have been trying to use DCP to be able to receive all document mutations.
I’m using java 10 and the java dcp-client 0.20.0
I log immediately after receiving an event and only receive approximately 40% of all expected mutations.
I’ve been logging the partition number while upserting a document and this helped me finding out that there are several partitions for which I never receive any of the expected DCP mutation events, even after hours and also after restarting the streaming from my saved sessionState.
I’ve been trying to isolate the issue by starting the stream only for that partition number, and indeed it missed logging the expected mutations.
Is this a known issue or can it be linked to any common configuration mistake ?
Let me describe here my use-case:
For each document mutation, I’d like to read the updated document and upon certain values, publish a business event of my own via google cloud PubSub.
By combining the Eventing Service and the cURL function, I could call an endpoint that would send my PubSub business event. However I couldn’t find information whether it’s possible to configure the Eventing Service to keep re-executing functions until it’s successful (in case the service called by cURL occasionally fails) and therefore guarantee that my business event would always be sent.
You could put curl call in a simple ‘for’ loop that retries on failure. This would be easiest option but it’s probably not good idea if you expect the calls to take more than a few seconds to succeed finally.
If you expect calls to be slow or many retries needed before it finally succeeds, you can create a timer when a curl call fails, and retry the failing call a bit later.