Document not being replicated

Couchbase Server
1

Hello,

We are running Couchbase 4.5.1-2844 community edition in 3 datacenter with 5 nodes in each. We are observing some strange behavior. If I create a document in datacenter A, it is not being replicated to the other 2 data centers. However, if I create a document in data center B or C, it is replicated to A without any issues. This behavior seems to be consistent and reproducible. Any ideas here? I tried looking at the logs but I am not exactly sure what to look for. Any help is appreciated.

Thanks,

-K

2

Hi @k_reid,

How is your XDCR configured? XDCR is directional, so it’s possible that you are missing an XDCR setup from A to B/C.

3

Hi @matthew.groves,

I’ve verified and all the configurations and everything looks ok. Maybe if you can provide some insight into what I should look for in the logs.

4

Can you give us more details like what topology, have you set up XDCR with datacenters A,B, C in bidirectional ring/ mesh topology? Are they are all bidirectional active-active, where each of the DCs are configured for taking writes along with reads?

5

Hi Matthew,

Our XDCR is set up as follows:

A replicates to B and C
B replicates to A and C

C replicates to A and B

Each replication goes to a different node in the target cluster. For example, when A replicates to B, it uses a different B node than when C replicates to B.

Thanks,

K

6

Hi @k_reid,

It should not matter to which node in the target cluster you set up the XDCR too. From your setup, looks like you have bidirectional active-active XDCR setup between all nodes. Can you check /opt/couchbase/var/lib/couchbase/logs/goxdcr.log and grep to see if there are any errors? Are you creating new documents or updating existing documents? If you create a document in B, does it replicate fine to both A & C, or just to A? Similarly, from C does it replicate to both B & A, or just to A?

7

Hi @aruns1987,

This problem occurs when I am creating a new document. Things replicate fine if I create a document in datacenter B and C but things are not replicated from datacenter A. I don’t see any errors in the goxdcr.log file other than this (which is actually a debug log):

[xdcr:debug,2018-05-11T16:24:11.089Z,ns_1@<<host>>:<0.1863.5>:goxdcr_rest:stats:220]Unable to obtain stats for bucket "subscriptions" from goxdcr:
{error,
    {badmatch,
        {error,
            {eaddrinuse,
                [{lhttpc_client,send_request,1,
                     [{file,
                          "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
                      {line,220}]},
                 {lhttpc_client,execute,9,
                     [{file,
                          "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
                      {line,169}]},
                 {lhttpc_client,request,9,
                     [{file,
                          "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
                      {line,92}]}]}}},
    [{goxdcr_rest,send_with_timeout,5,
         [{file,"src/goxdcr_rest.erl"},{line,61}]},
     {goxdcr_rest,query_goxdcr,4,[{file,"src/goxdcr_rest.erl"},{line,141}]},
     {goxdcr_rest,get_from_goxdcr,3,[{file,"src/goxdcr_rest.erl"},{line,161}]},
     {goxdcr_rest,stats,1,[{file,"src/goxdcr_rest.erl"},{line,216}]},
     {goxdcr_stats_collector,grab_stats,1,
         [{file,"src/goxdcr_stats_collector.erl"},{line,37}]},
     {base_stats_collector,handle_info,2,
         [{file,"src/base_stats_collector.erl"},{line,89}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,604}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}