Setting Up Cross-VPC XDCR Between Amazon EKS Clusters

In modern distributed systems, the ability to replicate data between separate environments is crucial for ensuring high availability, disaster recovery, and performance optimization. Couchbase’s XDCR (Cross Data Center Replication) feature allows seamless replication of data between clusters, enabling robust data sharing across geographically or logically isolated environments.

This guide will walk you through setting up XDCR between two Couchbase clusters hosted in separate Amazon EKS (Elastic Kubernetes Service) clusters within different VPCs. We’ll dive into each step, from infrastructure setup to configuring DNS for cross-cluster communication and deploying Couchbase for real-time replication. By the end of this walkthrough, you’ll have a production-ready setup with the skills to replicate this in your environment.

Prerequisites

To follow this guide, ensure you have:

• • AWS CLI installed and configured
  • An AWS account with permissions for creating VPCs, EKS clusters, and security groups
  • Familiarity with Kubernetes and tools like kubectl and Helm
  • Helm installed to deploy Couchbase
  • Basic knowledge of networking concepts, including CIDR blocks, routing tables, and DNS

Step 1: Deploy EKS clusters in separate VPCs

What are we doing?

We will create two Kubernetes clusters, Cluster1 and Cluster2, in separate VPCs using eksctl. Each cluster will operate independently and have its own CIDR block to avoid IP conflicts.

Why is this important?

This separation ensures:

1. 1. Isolation for better security and management
   2. Scalability and flexibility for handling workloads
   3. Clear routing rules between clusters

Commands to create clusters

Deploy cluster1

Deploy cluster2

Expected outcome

• • Cluster1 resides in VPC 10.0.0.0/16
  • Cluster2 resides in VPC 10.1.0.0/16

Step 2: Peer the VPCs for Inter-Cluster Communication

What are we doing?

We are creating a VPC Peering connection between the two VPCs and configuring routing and security rules to enable inter-cluster communication.

Steps

2.1 Create a peering connection

• • Go to the AWS Console > VPC > Peering Connections
  • Click Create Peering Connection
  • Select the Requester VPC (Cluster1 VPC) and Accepter VPC (Cluster2 VPC)
  • Name the connection eks-peer
  • Click Create Peering Connection

2.2 Accept the peering request

• • Select the peering connection
  • Click Actions > Accept Request

2.3 Update route tables

• • For Cluster1 VPC, add a route for 10.1.0.0/16, targeting the peering connection
  • For Cluster2 VPC, add a route for 10.0.0.0/16, targeting the peering connection

2.4 Modify security groups

Why is this necessary?

Security groups act as firewalls, and we must allow traffic between the clusters explicitly.

How to modify

1. 1. Navigate to EC2 > Security Groups in the AWS Console
   2. Identify the security groups associated with Cluster1 and Cluster2
   3. For Cluster1’s security group:
      • Click Edit Inbound Rules
      • Add a rule:
        • Type: All traffic
        • Source: Security group ID of Cluster2
   4. Repeat for Cluster2, allowing traffic from Cluster1’s security group

Step 3: Test Connectivity by Deploying NGINX in Cluster2

What are we doing?

We are deploying an NGINX pod in Cluster2 to verify that Cluster1 can communicate with it.

Why is this important?

This step ensures the networking between clusters is functional before deploying Couchbase.

Steps

3.1 Create a Namespace in Cluster1 and Cluster2

3.2 Deploy NGINX in Cluster1 and Cluster2

• • Create nginx.yaml:
    
    
    YAML

    apiVersion: apps/v1 kind: Deployment metadata:   name: nginx spec:   replicas: 1   selector:     matchLabels:       app: nginx   template:     metadata:       labels:         app: nginx     spec:       containers:       - name: nginx         image: nginx         ports:         - containerPort: 80 --- apiVersion: v1 kind: Service metadata:   name: nginx spec:   clusterIP: None   ports:   - port: 80     targetPort: 80   selector:     app: nginx

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

    apiVersion: apps/v1 kind: Deployment metadata:   name: nginx spec:   replicas: 1   selector:     matchLabels:       app: nginx   template:     metadata:       labels:         app: nginx     spec:       containers:       - name: nginx         image: nginx         ports:         - containerPort: 80 --- apiVersion: v1 kind: Service metadata:   name: nginx spec:   clusterIP: None   ports:   - port: 80     targetPort: 80   selector:     app: nginx

3.3 Apply the YAML

3.4 Verify Connectivity from Cluster1

• • Exec into the pod in Cluster1:
    
    
    Shell

    kubectl exec -it -n dev <pod-name> -- /bin/bash

    1	kubectl exec -it -n dev <pod-name> -- /bin/bash

3.5 Test connectivity to Cluster2

Expected outcome

The curl command will fail without DNS forwarding, highlighting the need for further DNS configuration.

Step 4: Configuring DNS forwarding

What are we doing?

We’ll configure DNS forwarding so that services in Cluster2 can be resolved by Cluster1. This is critical for allowing applications in Cluster1 to interact with services in Cluster2 using their DNS names.

Why is this important?

Kubernetes service discovery relies on DNS, and by default, DNS queries for one cluster’s services cannot resolve in another cluster. CoreDNS in Cluster1 must forward queries to Cluster2’s DNS resolver.

Steps

4.1 Retrieve Cluster2’s DNS Service Endpoint

• • Run the following command in Cluster2 to get the DNS service endpoint:
    
    
    Shell

    kubectl get endpoints -n kube-system

    1	kubectl get endpoints -n kube-system

  • Look for the kube-dns or coredns service and note its IP address. For example:
    
    
    ---------------------------------------- NAME       ENDPOINTS          AGE kube-dns   10.1.20.116:53     3h ----------------------------------------

    1 2 3 4

    ---------------------------------------- NAME       ENDPOINTS          AGE kube-dns   10.1.20.116:53     3h ----------------------------------------

    
    

4.2 Edit the CoreDNS ConfigMap in Cluster1

• • Open the CoreDNS ConfigMap for editing:
    
    
    Shell

    kubectl edit cm coredns -n kube-system

    1	kubectl edit cm coredns -n kube-system

    
     

4.3 Add the following block to the Corefile section

Replace 10.1.20.116 with the actual DNS endpoint IP from Cluster2.

Note: We only need to use one of the CoreDNS endpoints for this ConfigMap. The IP of CoreDNS pods rarely change but can if the Node goes down. The kube-dns ClusterIP service can be used but will require the IP and Port to be open on the EKS Nodes.

4.4 Restart CoreDNS in Cluster1

• • Apply the changes by restarting CoreDNS:
    
    
    Shell

    kubectl rollout restart deployment coredns -n kube-system

    1	kubectl rollout restart deployment coredns -n kube-system

    
    

4.5 Verify DNS Forwarding

• • Exec into any pod in Cluster1:
    
    
    Shell

    kubectl exec -it -n default <pod-name> -- /bin/bash

    1	kubectl exec -it -n default <pod-name> -- /bin/bash

  • Test DNS resolution for an NGINX service in Cluster2:
    
    
    Shell

    curl nginx.prod.svc.cluster.local

    1	curl nginx.prod.svc.cluster.local

  • You should see a response from the NGINX service
    

Expected outcome

DNS queries from Cluster1 to Cluster2 should resolve successfully.

Step 5: Deploying Couchbase

What are we doing?

We’ll deploy Couchbase clusters in both Kubernetes environments using Helm. Each cluster will manage its own data independently before being connected through XDCR.

Why is this important?

Couchbase clusters form the foundation of the XDCR setup, providing a robust and scalable NoSQL database platform.

Steps

5.1 Add the Couchbase Helm Chart Repository

5.2 Deploy Couchbase in Cluster1

• • Switch to Cluster1:
    
    
    Shell

    kubectl config use-context <cluster1-context>

    1	kubectl config use-context <cluster1-context>

  • Deploy Couchbase:
    
    
    Shell

    helm install couchbase couchbase/couchbase-operator --namespace dev

    1	helm install couchbase couchbase/couchbase-operator --namespace dev

5.3 Deploy Couchbase in Cluster2

• • Switch to Cluster2:
    
    
    Shell

    kubectl config use-context <cluster2-context>

    1	kubectl config use-context <cluster2-context>

  • Deploy Couchbase:
    
    
    Shell

    helm install couchbase couchbase-operator --namespace prod

    1	helm install couchbase couchbase-operator --namespace prod

5.4 Verify Deployment

• • Check the Couchbase pods:
    
    
    Shell

    kubectl get pods -n dev  # For Cluster1 kubectl get pods -n prod # For Cluster2

    1 2	kubectl get pods -n dev  # For Cluster1 kubectl get pods -n prod # For Cluster2

    
    
  • Ensure all pods are running.

Note: If you encounter a deployment error, edit the CouchbaseCluster CRD to use a compatible image version:

Change:

To:

Expected outcome

Couchbase clusters should be running and accessible via their respective UIs.

Step 6: Setting Up XDCR

What are we doing?

We’ll configure XDCR to enable data replication between the two Couchbase clusters.

Why is this important?

XDCR ensures data consistency across clusters, supporting high availability and disaster recovery scenarios.

Steps

6.1 Get service name from Cluster2

• • In cluster2 run the following command to retrieve the service name of one of the pods so we can port-forward to it.
    
    
    Shell

    kubectl get services -n prod

    1	kubectl get services -n prod

6.2 Access the Couchbase UI for Cluster2

• • • Port-forward the Couchbase UI:
      
      
      YAML

      kubectl port-forward -n prod cluster2-0000 8091:8091

      1	kubectl port-forward -n prod cluster2-0000 8091:8091

• • Open a browser and navigate to:
    http://localhost:8091
    

• • Log in using the credentials configured during the deployment.

6.3 View Documents in Cluster2

• • In the Couchbase UI, go to Buckets
  • Note that no Documents exist in the default bucket

6.4 Access the Couchbase UI for Cluster1

• • Port-forward the Couchbase UI:
    
    
    Shell

    kubectl port-forward -n dev cluster1-0000 8091:8091

    1	kubectl port-forward -n dev cluster1-0000 8091:8091

    
    
  • Open a browser and navigate to:
    http://localhost:8091
  • Log in using the credentials configured during the deployment

6.5 Add a Remote Cluster

• • In the Couchbase UI, go to XDCR > Add Remote Cluster
  • Configure the remote cluster:
    • Cluster Name: Cluster2
    • IP/Hostname: <Cluster2 Service Name>.prod.svc.cluster.local
    • Username: Admin username for Cluster2
    • Password: Admin password for Cluster2
  • Click Save
    

6.6 Set Up Replication

• • In the Couchbase UI for Cluster1, go to XDCR > Add Replication
  • Configure the replication:
    • Replicate From Bucket: Default bucket in Cluster1
    • Replicate To Bucket: Default bucket in Cluster2
    • Remote Cluster: Select Cluster2
  • Click Save
    
    

6.7 Test Replication

• • Add sample documents to the default bucket in Cluster1:
    • In the Couchbase UI, navigate to Buckets > Documents > Add Document
    • Give the Document a unique ID and some data in JSON format
      
    • Verify that the documents appear in the default bucket of Cluster2:
      • Port-forward to Cluster2’s UI and log in:
        
        
        Shell

        kubectl port-forward -n prod cluster2-0000 8091:8091

        1	kubectl port-forward -n prod cluster2-0000 8091:8091

    • Navigate to: http://localhost:8091

Expected outcome

Data added to Cluster1 should replicate to Cluster2 in real time.

Step 7: Cleanup

What are we doing?

We’ll cleanup our AWS environment and delete all the resources we deployed.

Why is this important?

This will prevent you from incurring unnecessary charges.

Steps

7.1 Access the AWS Console

• • Go to the AWS Console > VPC > Peering Connections
  • Select and delete the peering connection
  • Go to the AWS Console > CloudFormation > Stacks
  • Select and delete the two nodegroup stacks
  • Once the two nodegroup stacks have finished deleting select and delete the cluster stacks

Expected outcome

All resources created for this tutorial are deleted from account.

Conclusion

Through this guide, we’ve successfully established XDCR between Couchbase clusters running in separate EKS clusters across AWS VPCs. This setup highlights the power of combining AWS networking with Kubernetes for robust, scalable solutions. With cross-cluster replication in place, your applications gain enhanced resilience, reduced latency for distributed users, and a solid disaster recovery mechanism.

By understanding and implementing the steps outlined here, you’re equipped to tackle real-world challenges involving multi-cluster setups, expanding your expertise in both cloud networking and distributed database management.

• • Learn more about Couchbase Cross Data Center Replication (XDCR)
  • Read the XDCR documentation
  • Read how XDCR is essential for globally distributed data, disaster recovery and high availability