Brant Burnett is a Couchbase Expert, systems architect, and .NET developer experienced in desktop and web full-stack development. For the last 12 years he has been working with CenterEdge Software, a family entertainment software company based in Roxboro, NC. Brant is experienced in developing applications for all segments of their software suite. Over the last 4 years he has worked to transition the company’s cloud infrastructure from a Microsoft SQL platform to a pure Couchbase NoSQL Data Platform. Through his work at CenterEdge Brant has been able to focus on creating serious software solutions for fun businesses.

Brant_Burnett_Headshot

Couchbase makes an excellent data store for scalable cloud applications, including applications built using a microservices architecture. Kubernetes is an incredibly powerful platform for running containerized microservices, easing DevOps management and reducing developer friction. But how can we use them together, reducing friction and management overhead throughout the entire system?

One option is to run Couchbase within Kubernetes as a StatefulSet. However, StatefulSets are still in beta. Additionally, a production Couchbase cluster creates some serious load, so running in dedicated nodes is strongly recommended. This could be accomplished in Kubernetes using taints, but at CenterEdge Software we’ve decided to keep it simple and run the Couchbase cluster in standalone EC2 instances.

This post is intended as a tutorial on configuring Kubernetes in Amazon Web Services (AWS) to seamlessly connect to a Couchbase cluster running in EC2 on AWS. Additionally, the approach is flexible enough to allow cluster swapping, or running smaller clusters within Kubernetes for development or QA environments.

TL;DR

This approach to configuring access to Couchbase in Kubernetes has several key advantages:

  1. Easier to configure your microservices with a single, simple URL
  2. Make changes in your infrastructure without needing to reconfigure every microservice
  3. Support for swapping out the cluster transparently during Couchbase upgrades
  4. Easily point the same URL in a development or QA Kubernetes cluster at a different Couchbase cluster

Steps:

  1. Make sure the Couchbase and Kubernetes clusters can reach each other over the network
  2. Create an Application Load Balancer (ALB) for bootstrapping using a single URL
  3. Create a Kubernetes Service of the ExternalName type to resolve the cluster from your microservices

Prerequisites

This post assumes that the following prerequisites have been completed:

  1. You have a Kubernetes cluster running in AWS. We deployed our cluster using the officially supported kops utility. There is a good guide here.
  2. You have a Couchbase cluster running in AWS on EC2 instances. We used this userdata script to configure the instances on startup, using i3 instance types with NVMe SSD instance storage. Disclaimer: This userdata script has not been fully vetted for production usage.
  3. Both of these clusters are in the same AWS region.
  4. The VPC where Couchbase resides must have subnets in at least two availability zones (even if Couchbase is only using one).

Setting up VPC Peering

VPC Peering will allow instances in one VPC to talk to instances in the other VPC in the same AWS region transparently as if they were on the same LAN.

Note: This step is only required if the Couchbase VPC is separate from your Kubernetes VPC. If you’re using the same VPC for both, you can skip this step.

Note: The VPCs must be using different private subnets to prevent IP address conflicts.

  1. Open Peering Connections within VPC on the AWS Console
  2. Create Peering Connection
    1. For the Requester, select your Kubernetes VPC
    2. For the Accepter, select your Couchbase VPC
  3. Once the connection is in the “Pending-acceptance” state, right-click and select Accept Request

Now that the Peering Connection is created, we need to configure the VPCs to route traffic to each other. This must be done by updating the Route Tables for both VPCs to route traffic to the other for the correct subnets.

For this example, we’ll assume that Kubernetes is running in 10.100.0.0/16, and Couchbase is running in 10.200.0.0/16.

  1. For each Route Table in the Kubernetes VPC:
    1. Go to the Routes tab and click Edit
    2. Click Add Another Route
    3. Enter the Couchbase subnet (10.200.0.0/16)
    4. Select the Peering Connection as the Target (starts with pcx-)
    5. Click Save
  2. Now repeat in the opposite direction for each Route Table in the Couchbase VPC:
    1. Go to the Routes tab and click Edit
    2. Click Add Another Route
    3. Enter the Kubernetes subnet (10.100.0.0/16)
    4. Select the Peering Connection as the Target (starts with pcx-)
    5. Click Save
  3. Additionally, ensure that the EC2 Security Group for your Couchbase instances is allowing the correct ports from the Kubernetes subnet. The required ports are the Node-to-client ports listed on this document.

Creating an Application Load Balancer for bootstrapping

The Couchbase client SDKs are capable of bootstrapping their connection to the server through multiple different protocols. In order to make the cluster easy to use from Kubernetes, we’re going to focus on bootstrapping via the HTTP protocol on port 8091. If you want to use TLS to secure the bootstrapping process, this could be done using port 18091 instead.

To support this kind of bootstrap, we’ll create an Application Load Balancer (ALB) in AWS. This will provide a single endpoint that can be reused regardless of any changes to the Couchbase cluster. It can also assist in swapping from one cluster to another during cluster upgrades.

  1. Within EC2, go to Load Balancers, and click Create Load Balancer.
  2. Select Application Load Balancer and click Continue.
  3. For “Step 1: Configure Load Balancer”:
    1. Enter a name for your load balancer, i.e., “couchbase-primary”.
    2. Scheme should be “internal”. Note: Selecting internal is very important for security.
    3. Change the Port from 80 to 8091.
    4. Select the VPC and subnets where Couchbase is running. If Couchbase is only in a single availability zone, then select one additional AZ to meet the ALB requirement.
    5. Click “Next: Configure Security Settings”.
  4. There are no settings on Step 2. Click “Next: Configure Security Groups”.
  5. Create a new security group for the ALB, with TCP port 8091 open for incoming traffic, and click “Next: Configure Routing”.
    1. You may need to edit your Couchbase EC2 Security group to open port 8091 to traffic from the new EC2 Security Group created for the ALB.
  6. Create a new Target Group with these values:
    1. Name: “couchbase-primary”
    2. Protocol: “HTTP”
    3. Port: “8091”
    4. Health Check Protocol: “HTTP”
    5. Health Check Path: “/ui/index.html”
    6. Click “Next: Register Targets”
  7. For “Step 5: Register Targets”:
    1. Select all of the Couchbase instances in the cluster.
    2. Click “Add To Registered” to add them to the list at the top.
    3. Click “Next: Review”.
  8. Click Create.
  9. Once the ALB is created, we can get its “DNS Name”, which we’ll use later.

But what about cost and performance with ALBs?

The fact of the matter is, the cost of this configuration is negligible in terms of both expense and performance. The load balancer is only used for communicating cluster configuration to the client. This configuration includes information about all of the nodes in the cluster and how to connect to them.

After bootstrapping is complete, the client is connected directly to the cluster nodes. All key/value and query operations will pass directly to the nodes, bypassing the load balancer. This means there is neither a financial nor performance cost for these operations.

Making bootstrapping from Kubernetes even easier

At this point, we can now bootstrap our microservices using a single server URL in the configuration. This is much easier than listing several servers in our configuration, each of which may change over time as the cluster mutates.

But the URL we need to use is still cumbersome, something like: internal-couchbase-123456789.us-east-1.elb.amazonaws.com. Is there a way to simplify this even further? The answer is yes!

Kubernetes can also assist with service discovery using DNS. In this case, we’ll add a Service of the ExternalName type. This effectively creates a CNAME record which forwards the traffic to an external DNS name. This record only exists for Pods within the Kubernetes cluster, so it won’t affect anything outside of our cluster.

  1. Create a file named “couchbase-primary.yaml” with this content:kind: Service
    apiVersion: v1
    metadata:
    # this will be the DNS name to access the cluster
    name: couchbase-primary
    spec:
    type: ExternalName
    # substitute the DNS name of the ALB below
    externalName: internal-couchbase-123456789.us-east-1.elb.amazonaws.com
  2. Create the Service from the yaml file:kubectl apply -f couchbase-primary.yaml

All done! Now, we can bootstrap any microservice running in our cluster with a single, simple URL:
http://couchbase-primary:8091/. If we need to access it from a pod in a different namespace, just append the namespace, for example http://couchbase-primary.default:8091/. You may also register as many clusters as you need using different domain names.

Couchbase Expert

Author

Posted by Laura Czajkowski, Developer Community Manager, Couchbase

Laura Czajkowski is the Snr. Developer Community Manager at Couchbase overseeing the community. She’s responsible for our monthly developer newsletter.

All Posts

Leave a reply