Can two Kafka connector instances write to the same destination topic?

Kafka Connector
1

Hi team,

We have two Kafka Connector sources listening for mutations in two separate Couchbase clusters(different message structure and buckets) but both are publishing the messages to the same destination Kafka topic

In Kafka terms, this means two producers writing to the same topic which is an acceptable pattern but I am just wondering if there are any special considerations for the Couchbase Connector we should make

Thank you

2

Is this related to your other post? Because that one shows topic1 and topic2.

3

It is in the same environment, but a different dicussion.

This case is about two couchbase connectors writing to the same topic; I am just trying to educate myself and learn about the subject. I figured that a new discussion was better given that it is unrelated to the other post

4

Hi @yeikel ,

The Couchbase connector doesn’t care what topic you publish to.

Just be aware that two documents in different Couchbase clusters could have the same document ID. (The same is true for different Couchbase collections – a Couchbase document ID is only guaranteed to be unique within a collection.)

As long as the document IDs are globally unique, and the consumer expects records from the different connectors, everything should be fine.

Thanks,
David

1 Like
5

Thank you @david.nault. We generate the document ID and the generation algorithm between the two topics is different. We expect no duplicates so it should be fine

Thanks!