I wrote a server (referencing the elastic serach connectors) that listen to new documents and forward that as notification to the clients. In our prototype, the time taken to publish into couchbase including round-trip is 3-5ms but the time taken for couchbase to update the custom server thru xdcr took 40-50ms and it is incredibly consistent regardless of the size of payload (within a range of 1k-10k)
Presuming that XDCR is http-based, it is likely to be batch than streaming. What is the time interval for this publishing? and if it is possible to lower this latency?
@wai.kwang.mak,
(IMAGE BELOW). I’ve found that the only real setting that makes a big difference as far as speed with ES is:
- Changing the XDCR Optimistic Replication Threshold FROM: 256 TO: 20971520(20MB).
- The next thing is simple get more ES nodes/JVMs
- Play around with custom tokenization and indexing in ES.
Also Couchbase now has FTS built-in as of CB 5.0 GA. Check it out.
https://developer.couchbase.com/documentation/server/current/fts/full-text-intro.html
@househippo I did the same with the threshold and got to a point of ~30ms for latency but I would like to lower it to 10ms if not 5ms.
@wai.kwang.mak,
In CB XDCR:
-Version 1 is Couchbase’s older replication method to another cluster. Its REST based and feed off the Couchbase’s Cluster internal replication queue via TAP.
TAP: Couchbase SDKs
-Version 2 is Couchbase’s newer replication method to another cluster. It’s uses memcached TCP connection and feeds off the Couchbase’s Cluster internal replication queue via DCP
DCP: Couchbase SDKs
The current CB to ES transport uses Version 1 because ES uses HTTP/REST to do CRUD.
Indeed, I am happy to write custom solution using DCP (anyway, I wrote one referencing version 1, and of course network stiffer 
). Unfortunately, there isn’t enough material to do the same for DCP.
