Can I route documents to different topics using couchbase kafka source connector

Kafka Connector
1

Kafka connector is configured to route events to one topic. I’d like to route to different topics depending on the document type.

I’d like if possible to do this on the source connection side of things rather than on the other side of the broker using streams.

I’d also like to avoid setting up per topic source connector.

I’m already using a custom source filter and also have played around with the single message transform thank you for the examples!

2

Hi Naftali,

Sure, you can do that with a custom SourceHandler.

Alternatively…

If you’re using Couchbase 7 or later, you can derive the topic name from the Couchbase collection where the document is stored. For example, if you put all your Widget documents in the “widget” collection, and your Invoice documents in the “invoice” collection, then just configure the connector like this:

couchbase.topic=${collection}

Your widgets will get published to the “widget” topic, and invoices will get publised to the “invoice” topic.

See couchbase.topic docs for more info.

Thanks,
David

1 Like
3

Thank you @david.nault I have it working (and we would love to use collections but right now the migration cost is enormous).

In any case, logging tells me that the filter is run before source handler. but I also see that there is a passesFilter method in the SourceHandler. Is there any advantage to having an isolated filter class or should I be doing the filter in the SourceHandler class?

1 Like
4

Hi Naftali,

Is there any advantage to having an isolated filter class or should I be doing the filter in the SourceHandler class?

There is no advantage. Feel free to do whichever is more convenient.

Thanks,
David