Couchbase clusters sites need an VPN between them to use XDCR?

Couchbase Server
1

Hello,

I have the following topology:

Cluster 1 - Los Angeles - node1 - IP 10.5.0.4
Cluster 1 - Los Angeles - node2 - IP 10.5.0.6

Cluster 2 - Frankfurt - node1 - IP 10.6.0.4
Cluster 2 - Frankfurt - node2 - IP 10.6.0.5
  • The clusters cannot reach each other by those IP addresses. I mean, I cannot ping from Los Angeles node1 10.5.0.4 to Frankfurt 10.6.0.4. The nodes are behind an NAT, if I “ifconfig -a” I can only see the internal IPs.
    ** We are using https://www.citycloud.com/ as cloud provider.

For each node I have one external IP:

Cluster 1 - Los Angeles - node1 - IP External 89.42.XX.1
Cluster 1 - Los Angeles - node2 - IP External 89.42.XX.2

Cluster 2 - Frankfurt - node1 - IP External 89.46.XXX.1
Cluster 2 - Frankfurt - node2 - IP External 89.46.XXX.2

  • The servers can reach each other by the external IP addresses.
    I created the clusters between the nodes using the internal IP, for each region. The “Hostname” config is pointing to 127.0.0.1.

    Cluster 1 = 10.5.0.4 <-> 10.5.0.6
    Cluster 2 = 10.6.0.4 <-> 10.6.0.5

So, in the “Server Nodes” webpage link I see the internal IPs. The cluster is working fine.
Ok. Now in the Los Angeles region XDCR tab I created the following “Cluster Reference” pointing to one Frankfurt node:

89.46.XXX.1:8091

I believe it does not check anything here, it’s just a reference maybe?
Then, I created the Reference using the “default” bucket.

  • I already created the bucket on Frankurt cluster.

    Cluster: this cluster to Cluster: Frankfurt
    Bucket: default to Bucket: default

Then I started to see these errors below:

Attention - 2017-06-07 16:11:34 10.5.0.4:ToplogyChangeDetector:Post http://10.6.0.4:8092/_pre_replicate: dial tcp 
10.5.0.4:8092: getsockopt: no route to host
Attention - 2017-06-07 16:35:22 10.5.0.4:ToplogyChangeDetector:Post http://10.6.0.4:8092/_pre_replicate: dial tcp 
10.6.0.4:8092: i/o timeout

I believe because the “Cluster Reference” is returning the cluster members internal IPs from Frankfurt. Same ref. IPs that I have in the “Server Nodes” tab.
The questions are:
The idea is to have the cluster created using the internal IPs and XDCR using the external IPs. Is it possible?
Should I create the cluster nodes using the external IPs(Which does not makes much sense)?
The internal IPs from both sites need to see each other using an VPN?
It’s possible to use the same configuration that I have but when the cluster informs an IP, it will return the external IP and not the internal IP(behind the NAT)?
Did I do something wrong? :´(

2

Hello,

I figured out by myself. It was not clear by reading the CB docs and neither in my head.
I changed the /etc/hosts file on each VM and created 4 aliases, one for each VM. Pay attention to the internal and external markers.
LA - Site1

< internal-IP>  lanode1.site1
< internal-IP>  lanode2.site1
< external-IP> frnode1.site2
< external-IP> frnode2.site2

FR - Site2

< external-IP>  lanode1.site1
< external-IP>  lanode2.site1
< internal-IP> frnode1.site2
< internal-IP> frnode2.site2

In the setup webpage for each node, in the “Hostname” field I added the hostname alias I created on /etc/hosts. This will make the node to self resolve and bind to the < internal-IP>.
Now here’s the catch. When Site2 asks for the node topology, Site1 will respond back with the hostname aliases instead of the < internal-IP>. Site2 has Site1 aliases pointing out to the < external-IP>, this will make Site1 reachable by Site2 and the XDCR replication will work.

We are rich people, so we can afford external IPs for the each node. But of course, this would be expensive on an bigger environment. I believe the solution on a bigger env would to be to create an VPN between the two regions networks(even creating a new network using VPN would work). Still, you also would need to do this /etc/hosts thing so the cluster would be clustering using the same network and not the VPN.

Thx!