Couchbase AO 2.0 XDCR on Azure Kubernetes Service, within VNET and VNET Peer using NodePort

Couchbase Server Kubernetes
1

Hello Everyone,

I was able to setup the Couchbase cluster on AKS. Now this cluster will communicate with other Couchbase cluster in Azure over VNET Peering for data replication.

I need help on AKS NodePort external feature type, Where I would like to allows the cluster to be addressed with DNS over private networks (outside AKS Cluster), but withing same VNET or VNET Peer.

spec.networking.dns

The user is required to use these annotations to actually create the DNS entries as the services are created, updated and modified. The recommended solution is the Kubernetes External DNS integration. As such services will be annotated with external-dns.alpha.kubernetes.io/hostname.

The DNS configuration can also be used with the NodePort external feature type. This allows the cluster to be addressed with DNS over private networks e.g. through a VPN. In this mode of operation the Operator does not enforce the use of TLS. To create A records for your individual Couchbase server nodes, map the service DNS name annotation to the associated pod’s node IP address.

The individual DNS per-pod exposed service A records should be dynamically collated into an SRV record. This provides a stable name for the cluster as nodes are added and removed during the cluster lifecycle, and service discovery. The resource name for the given examples should be _couchbases._tcp.couchbase-0.us-east-1.example. com, with equal priority and weight and a port value of 11207.

Questions:

  1. How do i expose Couchbase endpoint DNS outside AKS cluser? Via External DNS?
  2. Should i use Node port or LoadBalancer type?
2

Actually that’s the wrong way of doing it (just recently updated!) The load-balancer networking is intended for putting clusters on the public internet only, and for nothing else. Since you are using Azure–which has a sensible networking model–you can just forward DNS from your source to your target cluster.

Have a read of https://docs.couchbase.com/operator/2.0/concept-couchbase-networking.html#inter-kubernetes-networking-with-forwarded-dns. We even have a short tutorial giving you ideas on how to configure DNS https://docs.couchbase.com/operator/2.0/tutorial-remote-dns.html.

3

So in this case, i don’t need to use LoadBalancer and External DNS right?

Also i’m trying to configure XDCR from client Couchbase cluster which is running on Azure VMs to remote Couchbase cluster running inside AKS. (VNET Peering)

To configure forward DNS on client side in Azure, do i need to create Azure Private DNS Zone?
And on the remote Couchbase cluster on AKS, i just have to make sure bi-directional traffic is allowed so that client can ping remote.

4

Correct, no public DNS, no DDNS replication (other than the built in Kubernetes stuff). One cluster need to be able to route to the other, and any firewall ports are open between the two.

You don’t need to do anything clever with the DNS, you are simply installing a DNS server with the XDCR source (or any SDK for that matter). Anything that matches the remote cluster’s domain gets forwarded to the remote Kubernetes DNS server. Simple!

5

Thank you for the quick response. I was able to resolve AKS cluster DNS from outside.
Also now able to add remote cluster (AKS) from my VMs Cluster (Source).
But when i try to add replication, it throws an error.
“:8092: getsockopt: connection refused”

I’m using NodePort type. And while adding remote cluster i did not specify any port.

6

You shouldn’t need to use “exposed features” at all with routed networking. It will cause Couchbase advertise the wrong addresses and confuse the client. Try removing spec.exposedFeatures and try again.

7

On which couchbase service should i create ingress to access couchbase dashboard?
We have 4 different couchbase services.
https://docs.couchbase.com/operator/2.0/reference-couchbasecluster-services.html

  1. couchbase : ingress does not work for dashboard
  2. couchbase-srv : ingress does not work for dashboard
  3. couchbase-ui : ingress does not work for dashboard
  4. couchbase- 0000 - Only if i create ingress on this service, i’m able to login to dashboard. But in this case i have to create multiple ingress for each couchbase node

Note: I’m using NodePort and want to access dashboard internally withing VNET

8

Port 8091 on the couchbase service is what you want. Note that it will not work unless you have client session affinity enabled (the ingress has to route all requests from that client to the same node all the time). Secondly, the services displayed via the UI may change based on what services are enabled on the node.

For precisely these reasons, we recommend you using Kubernetes port forwarding instead.

9

Yep, that worked after i added below config to couchbase ingress
annotations:
nginx.ingress.kubernetes.io/affinity: “cookie”

Also is there any way i can integrate Azure Active Directory with Couchbase?
I would like to authenticate AAD users with couchbase.

10

You can look at this for starters https://docs.couchbase.com/operator/2.0/concept-user-rbac.html. The caveat being that LDAP support in Couchbase Server has only been specified to work with OpenLDAP, so it may work, or it may not, depending on how badly Microsoft have strayed from the specification. That said it’s been a long time since I looked at the design documents, so the story may have changed!

Best I can advise it try it out and see what happens! Be good to know if it just works, so let me know your experience.

11

Thank you for the quick response. I was able to resolve AKS cluster DNS from outside.

12

CB_Error

We are getting this error suddenly in our AKS Couchbase Cluster

Rebalance exited with reason {mover_crashed,
{unexpected_exit,
{‘EXIT’,<0.4190.27>,
{{nodedown,
‘ns_1@couchbase-0000.couchbase.couchbase.svc’},
{gen_server,call,
[{‘janitor_agent-Geo’,
‘ns_1@couchbase-0000.couchbase.couchbase.svc’},
{if_rebalance,<0.23860.25>,
{update_vbucket_state,7,active,undefined,
undefined,
[[‘ns_1@couchbase-0000.couchbase.couchbase.svc’,
undefined],
[‘ns_1@couchbase-0000.couchbase.couchbase.svc’,
‘ns_1@couchbase-0007.couchbase.couchbase.svc’]]}},
infinity]}}}}}.
Rebalance Operation Id = bc475a4cebdcfc6207b415a05565a204

Worker <0.4142.27> (for action {move,{7,
[‘ns_1@couchbase-0000.couchbase.couchbase.svc’,
undefined],
[‘ns_1@couchbase-0000.couchbase.couchbase.svc’,
‘ns_1@couchbase-0007.couchbase.couchbase.svc’],
}}) exited with reason {unexpected_exit,
{‘EXIT’,
<0.4190.27>,
{{nodedown,
‘ns_1@couchbase-0000.couchbase.couchbase.svc’},
{gen_server,
call,
[{‘janitor_agent-Geo’,
‘ns_1@couchbase-0000.couchbase.couchbase.svc’},
{if_rebalance,
<0.23860.25>,
{update_vbucket_state,
7,
active,
undefined,
undefined,
[[‘ns_1@couchbase-0000.couchbase.couchbase.svc’,
undefined],
[‘ns_1@couchbase-0000.couchbase.couchbase.svc’,
‘ns_1@couchbase-0007.couchbase.couchbase.svc’]]}},
infinity]}}}}