Couchbase Helm chart and Hashicorp Consul service mesh

Couchbase Server Kubernetes
1

I am new to couchbase deployments and have recently deployed Hashicorp Consul for service mesh. I have gotten a test instance of couchbase up and running and a cluster deployed using the helm chart… However, I would like to enable connect-inject on the cluster deployment but have not been having any luck. I did see that Istio is the preferred service mesh according to documentation but wanted to be able enable connect-inject and add the sidecar to my server cluster pods.

Does anyone have experience using consul with cocuhbase and may have some pointers to enable the connect-inject annotation using the helm chart?
I have tried multiple variations for adding the annotations to the pod deployment . Any help would be greatly appreciated

example:
servers:

# -- Name for the server configuration. It must be unique.

default:



  # -- AutoscaledEnabled defines whether the autoscaling feature is enabled

  # for this class. When true, the Operator will create a

  # CouchbaseAutoscaler resource for this server class.  The

  # CouchbaseAutoscaler implements the Kubernetes scale API and can be

  # controlled by the Kubernetes horizontal pod autoscaler (HPA).

  autoscaleEnabled: false

  # -- Env allows the setting of environment variables in the Couchbase

  # server container.

  env: null

  # -- EnvFrom allows the setting of environment variables in the Couchbase

  # server container.

  envFrom: null

  # -- Pod defines a template used to create pod for each Couchbase server

  # instance.  Modifying pod metadata such as labels and annotations will

  # update the pod in-place.  Any other modification will result in a

  # cluster upgrade in order to fulfill the request. The Operator reserves

  # the right to modify or replace any field.  More info:

  # https://kubernetes.io/docs/reference/generated/kubernetes-

  # api/v1.19/#pod-v1-core

  pod:           

    template:

      metadata:

        annotations:

          consul.hashicorp.com/connect-inject: true    

          consul.hashicorp.com/connect-service: couchbase-test  

    spec: 

     serviceAccountName: couchbase-test                    

  services:

  - data

  - index

  - query

  - search

  - analytics

  - eventing

  size: 3
2

We’ve not tested it with Consul as far as I’m aware so what is the specific issue you need it to do? Ensure we have the annotation you want?

I presume you’re on the latest version (2.2.1) of the operator if you’re using the latest chart? There was an issue with annotations being removed (which led to a race condition in Istio actually) with 2.2.0 that I resolved so just wanted to check.

We have to do some extra fiddling even with Istio so it might be a bit of effort or require some updates to the operator to support.

3

Yes, I am using the latest version and just being able to use those annotations would be all that is needed to enable the sidecar and register the service with my service mesh.

I did try and adjust the server cluster pod.yaml and add the annotations manually after deployment but that did not produce the desired affect either. I was not aware if this was something that had to be done via an update or if perhaps I was making an error with my placement of annotations.

4

OK, I’ll see if I can figure out the Helm magic to get those annotations on for you.

1 Like
5

I also had a chat with product management to verify if we’ve had anyone ask for Consul support previously but nothing concrete has ever been asked. It would be good to capture your need on our public JIRA board, or if you prefer I can raise it for you? That way we can make sure we consider it when adding new features, testing, etc.

6

Somewhat surprised this hasn’t been asked about before. A lot of traffic routing options and security as well. If your team can take it into consideration to add the usage of the annotations that would be great! I looked at the jira task board but do not think I have access to add anything to it. I would definitely be available to collaborate or contribute to getting this option working.

Thank you again for your help!

7

Yeah this is why it’s vital to get it captured really so feel free to drop me an email (replace the underscore in my username with a dot and add @couchbase.com) and/or your nominated support contact so we can make sure it is. It would be good to know who this is for from a requesting perspective and include the contact for your company as well.

I will raise one to capture it anyway and also to track any possible updates we may need to helm or the operator. Apologies for not resolving it yet, I will aim to do so asap.

8

Right, you were almost there but just a little tweak - the easier way to do it is to instantiate it with Helm defaults, edit the CRD YAML directly and then reverse that into the Helm chart (fun!). You need to nest the settings under the right bit as well.

helm upgrade --install test couchbase/couchbase-operator --values ./values.yaml

Where values.yaml is:

cluster:
  servers:
    default:
      pod:
        metadata:
          labels:
            test: me
          annotations:
            anothertest: passed
            consul.hashicorp.com/connect-inject: "true"   
            consul.hashicorp.com/connect-service: couchbase-test

Note that booleans have to be quoted (the values must be strings). I’ve got another couple of others just as an example too.

This then shows the following:

kubectl describe pod test-couchbase-cluster-0000
Name:         test-couchbase-cluster-0000
...
Labels:       app=couchbase
              couchbase_cluster=test-couchbase-cluster
              couchbase_node=test-couchbase-cluster-0000
              couchbase_node_conf=default
              couchbase_server=true
              couchbase_service_analytics=enabled
              couchbase_service_data=enabled
              couchbase_service_eventing=enabled
              couchbase_service_index=enabled
              couchbase_service_query=enabled
              couchbase_service_search=enabled
              test=me
Annotations:  anothertest: passed
              consul.hashicorp.com/connect-inject: true
              consul.hashicorp.com/connect-service: couchbase-test
              operator.couchbase.com/version: 2.2.1
              pod.couchbase.com/initialized: true
9

To be clear though: CAO does not officially support Consul service mesh - we’ve not tested it and cannot offer anything more than best efforts here (it may “work” out of the box).

https://issues.couchbase.com/browse/K8S-2418 has been raised to track this request as well.

10

Giving it a try today, Thank you very much for your help!

11

For those returning, it looks like you can get it running with three annotations - see the JIRA for details: https://issues.couchbase.com/browse/K8S-2418

This is not unique to helm so will also work with the non-helm deployments of CAO.

Again though: currently Consul is not supported so whilst it may “work” to some degree there will be no official support from Couchbase for any issues until we support it officially.