Kafka Sink Connector - Write to multiple bucket with single connector

Kafka Connector
1

Hi, I have one connector which has below configuration fields.

"topics": "topic.a, topic.b",
"couchbase.bucket": "a-b",

You can see that topic.a and topic.b is listened and written to a-b bucket.

However I don’t want to store data of two different topics into a single bucket. I want to store topic.a data to bucket a and topic.b data to bucket b. Is there a way to do this with one connector?

2

Hi, and welcome to the forum.

A single instance of the sink connector can write to only one bucket.

I filed KAFKAC-300 to track this feature request.

In the meantime: if you’re using Couchbase 7 or later, you could write data from different Kafka topics to different collections in the same bucket (see the couchbase.topic.to.collection connector config property). Then, if you’re using Couchbase Enterprise Edition, you could use the Eventing service to copy the data from one of the collections to a different bucket. At least, I think that’s something Eventing can do. What do you think, @jon.strabala ?

Thanks,
David

3

Hi @sinkconnector,

Would using Couchbase 7 and storing the documents in different collections solve your issue, or do you definitely want to store them in separate buckets?

Thanks,
David

4

^^ If this suffices it will be the best practice.

^^ Yes Eventing could indeed be used to move any document from one bucket to another based upon some internal field (inspecting the JSON document via a JavaScript lambda) in the document itself (actually a copy and then a delete to accomplish the move).

Best

Jon Strabala
Principal Product Manager - Server‌