Couchbase connector to elastic search - multiple indexes, pipeline problems

Hi Ehsan,

we aim to have incoming documents from a bucket to be indexed into more than one index

I don’t have experience with Logstash, so I don’t know if that would work. The typical recommendation is to bring up two connectors.

when we change the pipeline to remove some fields, document is moved to cbes-rejected.

In general, pipelines with Painless scripts should work. If a document ends up in cbes-rejects index, it’s because Elasticsearch reported an non-transient error when the connector tried to index the document.

The first place I’d look would be the error field of the cbes-rejects entry associated with the failed document. The next place I’d look would be the Elasticsearch log.

As for possible causes…

Is it possible that the rejected documents do not have the field the Painless script is attempting to remove?

See if you can index one of the rejected documents manually using the Elasticsearch REST API (don’t forget the pipeline query parameter :slight_smile: ). This will help verify the pipeline script is working correctly.

Thanks,
David