Couchbase AO 2.0 XDCR on Azure Kubernetes Service, within VNET and VNET Peer using NodePort without CoreDNS Forwarding

Couchbase Server Kubernetes
1

Hello Everyone,

We have two AKS Clusters and both are in VNET peering setup. Both the clusters have Couchbase AO 2.0 running with below networking config.

networking:
exposeAdminConsole: true
adminConsoleServices:
- data
- index
- query
- search
adminConsoleServiceType: NodePort
exposedFeatures:
- admin
- client
- xdcr
exposedFeatureServiceType: NodePort
exposedFeatureTrafficPolicy: Local
serviceAnnotations:
service.beta.kubernetes.io/azure-load-balancer-internal: “true”

Now we want to setup XDCR between these two clusters. We don’t want to customize CoreDNS config (DNS forwarding) as the remote DNS IP might get changed when K8s DNS pods get restarted.

Is there any other way we can configure XDCR using K8s DNS over VNET? So that both Couchbase cluster can talk to each other using K8s DNS name.

Note: We do have internal nginx ingress controller in both the clusters. And Couchbase clusters are in unique K8s namespace.

Thanks,
Nilesh Gadhiya

2

First up, because you are using VNET peering (congratulations!) you don’t need to use exposeAdminConsole and exposedFeatures, as that adds a DNAT unnecessarily and makes things a lot harder, less flexible and ultimately less reliable.

Simply modify your firewall rules so pods from one cluster can ping pods in another. Then all you need to do is to set up a forwarding DNS server so the XDCR initiator can see the DNS names of the pods in the XDCR target, and have the initiator pods use that.

Here’s how to setup the DNS server https://docs.couchbase.com/operator/2.0/tutorial-remote-dns.html
Here’s how to configure XDCR https://docs.couchbase.com/operator/2.0/howto-xdcr.html#dns-based-addressing

3

Hello Simon,

Thank you for your prompt response.

I already have CoreDNS Forwarding set and it is working fine. I’m able to setup SDCR between two AKS clusters.
But in CoreDNS we have to specify remote ip of local dns cluster, which might get changed when k8s dns pods get deleted and recreated.

Below is the sample Core DNS Forwarding config.

kube.cluster1.movere.io:53 {
    errors
    log
    cache 30
    forward kube.cluster1.movere.io 10.10.10.10
}

This 10.10.10.10 IP can be changed right?
How do we make sure K8s DNS pods will have static IPs and not change?

Thanks,
Nilesh Gadhiya

4

It’s at the bottom of that tutorial link, you should be able to create an internal load-balancer service for the kube-dns deployment with a stable/static IP of some sort. Check with Microsoft and see what they say. I’d be interested to know what the response is!

5

I created the internal load balancer service on top of coredns pods.

apiVersion: v1
kind: Service
metadata:
name: coredns-svc
namespace: kube-system
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: “true”
spec:
selector:
k8s-app: kube-dns
type: LoadBalancer
loadBalancerIP: 0.0.0.0
sessionAffinity: None
ports:
- name: dns
protocol: UDP
port: 53

and it is working now. along with above coredns custom config map change.

Thank you very much for you help.

Thanks,
Nilesh Gadhiya

1 Like
6

I love it when a plan comes together! Thanks for sharing your experience for others to follow in future.