Hello!
Is it any way how to push documents into Kafka topics as json (avro) object/document, not as single string?
Hi Irek,
Yes, you can do this by writing a custom SourceHandler that defines the Avro schema(s) for your Couchbase documents.
The Kafka Connect Couchbase distribution includes an example project showing how to build and use a custom SourceHandler.
The example code is for publishing documents as JSON. To use Avro with a custom schema instead, start by copying DefaultSchemaSourceHandler.java into the example project. The default SourceHandler uses the same schema for every document, and (as you observed) puts the whole document in a single byte array. Modify the code to define a schema that matches your Couchbase documents. Use this schema to create a Struct that will hold the document contents.
The connector receives the Couchbase document content as a byte array. Parse the byte array into JSON using the whatever JSON library you prefer. Then copy the fields of the JSON into the Struct.
Let me know if you have any questions along the way.
Thanks,
David
1 Like