Can anyone help me with configuring External-DNS for my Kubernetes deployment on Azure AKS ? I’m still new to many aspects of k8s.
My requirement is to give clients access to Sync Gateway and use Couchbase Connector from outside the Azure k8s cluster.
Has anyone actually accomplished this or is there a better way?
I am able to deploy a Couchbase Operator and server cluster with LoadBalancer but it gets stuck after adding the pods/members- probably waiting for the DNS names to be available, which never happens, so the operator never finishes configuring buckets and adding nodes to the cluster.
After it times out I get this error: time="2020-04-06T17:25:06Z" level=error msg="failed to reconcile: context deadline exceeded: Connection error - dial tcp: lookup cb-cluster-eus2-0002.cb-cluster-eus2.xx-couchbase.com on 15.0.0.10:53: no such host" cluster-name=cb-cluster-eus2 module=cluster
Seemingly a DNS issue and hinting at my failed External-DNS deplyment? (no clue). “xx-couchbase.com” is a made up domain at the moment as I only need internal Vnet access from the connector initially.
I found a tiny mention in the docs that the user is responsible for setting up External-dns but doesnt give any example of an applicable deployment that works with couchbase and all of its ports and services.
I tried following the generic docs on Github here for deploying external-dns and, uselessly, nginx ingress on port 80. The external-dns deployment failed to create the service with this error: Warning FailedCreate 9m39s (x618 over 6d19h) replicaset-controller Error creating: pods "external-dns-6c84ccc6bf-" is forbidden: error looking up service account default/external-dns: serviceaccount "external-dns" not found
Any ideas? Can anyone share a YAML and maybe some steps?
My setup - Couchbase 6.5, Autonomous Operator 1.2, Azure AKS on 1.15.10
In general anything to do with setting up external-dns should be delegated to the authors, they can make changes at any time and will make our doucmentation go out of date very fast. Also I’m not an expert on it where they are so should get you up and running much faster.
Hi Simon,
Thanks for the info. the newer docs did help and I managed to find a better YAML file.
I think i was missing the role binding and some rbac settings. External-DNS seems to be up and running but still not working and no A records are created in the Dns Zone.
I configured debug logging and it actually generates endpoints for the exposed services and specifies A records, however, it shows this in the external-dns pod log:
time="2020-04-07T23:04:06Z" level=debug msg="Retrieving Azure DNS zones for resource group:
ClusterEUS2."
time="2020-04-07T23:04:06Z" level=debug msg="Found 0 Azure DNS zone(s)."
So, its not finding my DNS Zone to add the records. I confirmed that the zone is in that group. I checked the user principle and it seems to be there and correct - i see the user in Azure DNS zone as a “contributor” and “reader”
So close to resolving this!
Any ideas?
Just to chime in here, I don’t think this will work with a made up domain so I suggest setting up external dns to use a registrar as mentioned in the networking docs that Simon has shared.
In the past I’ve had success setting this up by following the azure doc you’ve referenced so looks like you’re on track. I suggest making sure the nginx example works for you and then give this another try.
Also if you’re using helm to install external-dns I remember that --publish-service was not enabled by default, which may be worth checking.
Doesn’t seem like an issue from the operator perspective since it’s just creating LoadBalancer services which are annotated for the external dns controller to detect and add the Address records to your domain.
@tommie , thanks
I received a reply from a couchbase support person who noticed that in my external-dns YAML I had a typo in --domain-filter This was simple fix and now I finally have lots of A and TXT records in the DNS Zone.
But you are right, the domain I am specifying is fake so it seems to be still preventing the operator from resolving the name. I should work with my admins and work with them to set me up with service principle access to the prod DNS Zone and make a new subdomain but I was trying to avoid that - maybe no choice now. Then i’ll have to redo my TLS stuff for the correct *.domains and maybe the real cert
Its a long shot but is there a way to get the operator to resolve the names using the azure name server so I can at least have a working cluster to access from the vnet?
My primary goal is to have a working CB cluster that is accessible from the Couchbase connector running outside the kubernetes cluster. Also, I should be able to reach sync gateway by external IP address for development too right? (though the operator probably will manage that too i’m guessing)
To wrap this up, I am posting what I did to get things working temporarily with the fake domain so the operator could complete the cluster and I could access by the IP address. Couchbase support pointed me to the idea of setting an alternate DNS. I will be reconfiguring for a real development domain, certificates and TLS once we have that.
Configure CoreDNS to use Azure DNS for the fake domain in Azure DNS Zone
Edit the existing configmap that deployed with AKS (I guess you could apply YAML too) kubectl edit configmaps coredns-custom -n kube-system
Add this data block in the file
(the .server part in “mycustdns.server” is important)
The IP address is one of the azure name servers.
wait a few seconds (you can watch in one of coredns pod logs as it completes).
Verify
for validation, run this to verify that the name resolves within the k8s cluster:
kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
If you dont see a command prompt, try pressing enter.
dnstools# host cb-cluster-eus2-0000.cb-cluster.myfakedomain.com
cb-cluster-eus2-0000.cb-cluster.myfakedomain.com has address 10.62.71.48
Wait for operator or redeploy
Wait for the couchbase operator to retry (its probably stuck waiting to timeout reaching the node)
Done.
