How Scopes & Collections Simplify Multi-Tenant App Deployments on Couchbase

Now it’s easier than ever to build and deploy microservices and multi-tenant applications on Couchbase. The 7.0 release introduces a new data organization feature called Scopes and Collections.

Scopes and Collections allow logical isolation of different types of data, independent lifecycle management and security control at multiple levels of granularity. Application developers use them to organize and isolate their data. DevOps and admins find the multiple levels of role-based access control (RBAC) – at the Bucket, Scope and Collection level – a powerful option to host microservices and tenants at scale.

The full functionality of this feature is now available in Couchbase Server 7.0. A limited functionality Developer Preview was already available in release 6.5, and some of you may have already kicked the tires on that.

What Are Couchbase Scopes and Collections

Scopes and Collections are logical containers within a Couchbase Bucket.

It’s helpful to think of Collections as tables within a relational database but without the data rigidity. Scopes are a set of related Collections, making them similar to an RDBMS schema. But in both cases, Scopes and Collections are more flexible since they store JSON documents.

The table below shows the mapping of familiar RDBMS constructs to Couchbase:

Mapping RDBMS concepts to Couchbase

Here is an example of organizing the Couchbase Travel App data with Scopes and Collections:

An example of Scopes and Collections with an example dataset

How to Create and Use Scopes & Collections

You can create Scopes and Collections from any of the Couchbase SDKs, couchbase-cli, REST APIs, N1QL or from the UI.

In particular, N1QLN1QL queries become much simpler and syntactically briefer as you no longer need to qualify different document types using where type = xxx. Instead you simply refer to the Collection directly.

In the example shown below, we use N1QL (I executed the N1QL statements using cbq shell) to create a Scope, a couple of Collections and some indexes. Then we run a simple query that joins two Collections.

A N1QL query to create a Scope, Collections and indexes

The Scale of Scopes & Collections

Couchbase currently allows you to have 30 Buckets in a single cluster. Scopes and Collections allow you to create orders-of-magnitude higher numbers of these entities.

Since Collections actually contain the data and Scopes organize related Collections, you typically have fewer Scopes than Collections. Global Secondary Indexes (GSIs) are created on Collections, and a Collection commonly has multiple indexes.

Keeping these requirements in mind, here is the scale of these entities that you can create in Couchbase 7.0 (a lot of them!):

• • Number of Collections allowed per cluster: 1,000
  • Number of Scopes allowed per cluster: 1,000
  • Number of Global Secondary Indexes allowed per cluster: 10,000

Enabling Multi-Tenancy at Scale

Modern applications are often written as microservices, with each application consisting of several (and sometimes hundreds) of microservices.

With the advent of SaaS and all its advantages, many of these applications are multi-tenant. For enterprise teams, it’s important to keep the cost low for hosting multiple applications and tenants (i.e., TCO), while providing the required isolation and flexibility.

Couchbase Buckets, Scopes and Collections now provide a three-level containment hierarchy to help you map your applications, tenants and microservices.

Key Features for Multi-Tenancy & Microservice Consolidation

Let’s dive into the capabilities offered by Scopes and Collections that make it possible to consolidate tenants and microservices.

Logical Isolation & Indexing

Collections allow you to isolate your data by type while enjoying the flexibility of a JSON document database model that evolves with your application.

As shown in the Travel example further above, the Airline documents go into the Airline Collection, the Hotel documents go into the Hotel Collection and so forth.

Each Collection can also be indexed individually. Different types of documents typically have different indexing needs, and Collections enable you to define different indexes for each type.

Lifecycle Management at Multiple Levels

Scopes and Collections allow you to manage and monitor your data at two new levels in addition to the existing Bucket level.

The provided functionality includes:

• • Create and drop individual Collections or Scopes
  • Set expiration (maximum time to live, a.k.a., Max TTL) at the Collection level
  • Monitor statistics at the Scope and Collection level. Although a full set of statistics are available only at Bucket level, a subset of statistics is available at Collection and Scope levels (e.g., item count, memory used, disk space used, operations per sec).

Fine-Grained Access Control with RBAC

One of the most powerful aspects of Scopes and Collections is the ability to control security at multiple levels – Bucket, Scope, Collection – using role-based access control (RBAC). This is the key to consolidating hundreds of microservices and/or tenants in a single Couchbase cluster.

Now, you can give a user access to a whole Bucket or to a Scope or to a Collection. Several out-of-box user roles are available, and more details are available in the RBAC documentation).

Fine-Grained Replication Control with XDCR

Cross Data Center Replication (XDCR) allows you to control replication at the Bucket, Scope or Collection level.

You can choose to replicate a whole Bucket, a Scope or just a Collection. You can map to a similarly named entity on the destination or remap to a different entity. You also have the flexibility of advanced filtering, which was already introduced for XDCR in release 6.5 of Couchbase.

The flexibility provided by XDCR is crucial to mapping microservices and/or tenants to Collections and Scopes as it allows you to control each microservice and tenant individually.

Backup/Restore at Multiple Levels

Continuing the theme of lifecycle management at multiple levels, Couchbase 7.0 allows you to control backup and restore options at each of these levels as well.

Hence, you can backup (and similarly restore) an individual Bucket, Scope or Collection, along with filtering the data and remapping it during restore.

Watch out for more detailed how-to blogs on each of these above themes (RBAC, XDCR and Backup/Restore) in the coming weeks!

Microservices Deployment Using Collections

A typical microservices architecture recommends that each microservice be simple (ideally single-function), loosely coupled from other microservices (which means have its own data store), and scale independently.

Prior to Couchbase Server 7.0, you could either host each microservice in a separate cluster or a separate Bucket. These deployment options have inherent limits on application density and may not always result in full hardware utilization. Now with Couchbase 7.0, you can map each microservice to one or more Collections. And since you can have up to a thousand Collections, you can have a thousand microservices in a single cluster(!).

Let’s look at the example microservice deployment shown below:

Microservices consolidation using Couchbase Scopes and Collections

The above example shows two CRM applications (Sales and Customers) deployed in a single cluster. Each of the two applications is mapped to a Bucket allowing you to control the overall resource allocation to each application.

The Sales application has two microservices: Orders and Shipping, and the Customers application has two microservices: Individuals and Accounts. We map each of the microservices to their own Collection. In this example, we didn’t use Scopes. You don’t have to define Scopes if you don’t need the extra level of organization (the default Scope is used instead).

Multi-Tenant Deployment Using Scopes

Multi-tenant applications require varying levels of isolation between tenants and varying levels of resource sharing of the underlying infrastructure. The chosen deployment architecture is a trade-off between isolation and total cost of ownership (TCO).

Some tenants may require complete physical isolation in which case separate clusters for each tenant may be the right architecture, albeit at a high TCO. Couchbase Buckets provide partial physical isolation and full security and logical isolation and can be used for separate tenants though with limits of scale and overhead-per-Bucket.

Logical and security isolation across tenants is often sufficient. With Scopes, you get security and logical isolation at more granular levels within a Bucket. You can have thousands of Scopes in a single Bucket, enabling you to host thousands of co-operative tenants (those not requiring physical isolation) in a single cluster.

The trade-offs between TCO and isolation levels between different multi-tenant architectures are captured in the table below.

Multi-tenant architecture choices using Couchbase Scopes and Collections

Imagine that the CRM applications in the example above need to become multi-tenant. You can easily do so by mapping each tenant to a separate Scope as shown below. (Cust1 and Cust2 are the two different tenants in this example.)

Multi-tenancy with Scopes in Couchbase

Achieving multi-tenancy with Scopes is fairly straightforward (without requiring application changes as you can pass a Scope context around). It’s also a scalable and cost-effective solution. Collection names need to be unique only within a Scope. Hence, you can easily deploy multiple Scopes with the same Collection names as shown above.

Next Steps

I hope you enjoy the new Scopes and Collections functionality in Couchbase. Below is a list of resources for you to get started and we look forward to your feedback on the Couchbase Forums.

Documentation & Articles

• • What’s New in Couchbase 7.0
  • Couchbase 7.0 release notes
  • The N1QL Query Language Now Supports Distributed ACID Transactions