DCP client in K8s cluster

Couchbase Server Kubernetes
1

We are developing a RabbitMQ connector for Couchbase based on Couchbase Java DCP Client. Its purpose is to detect data mutation and deletions in a Couchbase Bucket and publish them in a RabbitMQ exchange.

During development we tested it against a single server cluster Community Edition, now we switched to a 3 nodes cluster EE hosted in K8s and managed by CB Operator 1.2. While testing in dev environment (single node), it never missed a mutation. Once switched to multi node setup I noticed that sometimes the mutation I expected caused some controlEvent but no dataEvents, so they where missed.

My concern was that the client was connecting to a single node and so managing data events only from the v-buckets on that node. Is this correct? Does DCP works on a per-node basis?

The client used as connection host the “headless service” created by the operator, that should list all nodes of the cluster. Now, to be sure tu have them listed, I programmatically resolve the service name in a list of addresses to use an host list in DCP client, something like that:

final InetAddress SW[] = InetAddress.getAllByName(System.getenv(Constants.COUCHBASE_CLUSTER_SERVICE));
for (int i = 0; i < SW.length; i++){
    dynamicHostList.add(SW[i].getHostAddress());
}
this.client = Client.builder()
				.connectionNameGenerator(DefaultConnectionNameGenerator.forProduct("rabbit-connector", "0.0.1"))
				.connectTimeout(connectionTimeout)
				.hostnames(dynamicHostList)
				.networkResolution(networkResolution)
				.bucket(bucket)
				.credentials(bucketUsername, bucketPassword)
				.controlParam(DcpControl.Names.ENABLE_NOOP, "true")
				.compression(compressionMode)
				.mitigateRollbacks(persistencePollingIntervalMillis, TimeUnit.MILLISECONDS)
				.flowControl(flowControlBufferBytes)
				.bufferAckWatermark(60)
				.sslEnabled(false)
				.build();
		;

Now it seems to be realiable as in the past.

Is this a good strategy to manage client connection?

2

Hello @paolo.morgano,

While testing in dev environment (single node), it never missed a mutation. Once switched to multi node setup I noticed that sometimes the mutation I expected caused some controlEvent but no dataEvents, so they where missed.

DCP guarantees consistency not history. This mean that DCP will not send every mutation Couchbase Server receives. In a number of cases Couchbase Server will deduplicate operations on the same document. For example if the same document is updated twice in quick succession, DCP will only send the latest one.

This might be good enough for your solution but I just want make you aware of what DCP guarantees.

Cheers,
Patrick

3

Hi @pvarley,

thanks a lot for this information, this concept is really important to understand DCP behavior. However what I have observed is that all document’s mutation were lost, since its creation.

Do you mean that my solution (use all IP’s instead of service name) is correct or that consistency can explain some event loss?

Thanks.
Paolo