Scopes and Collections Support in Couchbase Mobile for Edge Applications

We’re pleased to announce the general availability of Couchbase Mobile 3.1, which includes Couchbase Lite and Sync Gateway for mobile and edge computing solutions. This is a significant release that introduces cloud-to-edge support for scopes and collections in Couchbase Mobile for self managed deployments. Launched in Couchbase Service 7.0, Scopes and Collections brings the ability to isolate data logically and to enforce independent app lifecycle management at multiple levels of granularity.

With the 3.1 release of Couchbase Lite and Sync Gateway, developers can avail the benefits of scopes and collections within their mobile and edge apps. In this post, I’ll discuss the benefits of using scopes and collections in the context of some typical use cases. While the post focuses on a key feature of the 3.1 release, to learn about other enhancements and bug fixes, you can read our platform-specific release notes.

This post assumes that you are familiar with the fundamentals of Couchbase Mobile. If you are new to Couchbase Mobile, check out the overview.

Download Today!

Here’s how you can start evaluating the new features in Couchbase Lite and Sync Gateway 3.1:

• • Download Couchbase Lite 3.1
  • Download Sync Gateway 3.1
  • Get the Sync Gateway 3.1 Docker image

Use Cases for Scopes and Collections

It’s probably best to discuss the benefits of scopes and collections in the context of some real world use cases. For each of the use cases discussed, we explore bucket/database level solutions and then introduce a solution with scopes and collections.

Simplified Data Organization

The ability for developers to structure data inside buckets similar to tables inside a relational database such that related data is grouped together in separate containers, which is especially useful for developers who are migrating from a relational world.

Here are two examples to illustrate the problem:

Example

Consider a field order app with data related to products and data related to orders. There are two Couchbase Lite enabled mobile apps, one for Products and one for Orders. Products related documents are synced to the Products mobile app, while orders related documents are synced to the Orders mobile app. An app user on the products mobile app can only read products related data, while an app user on the orders mobile app can read and write orders related data.

Solution: Bucket based

• • Products and orders data are in a single bucket.
  • The bucket has a dozen document types. A doc_type field is used to logically group products related and orders related documents.
  • App users are defined at the bucket level.
  • A Sync function is defined at bucket level and is complicated. Access control rules are based on doc_type and channels defined for doc_type control which user can access which subset of documents.
  • The Products app has a pull only replication for products documents, while the Orders app has push-pull replication for orders related documents.

Solution: Scope/collections based

Use scopes and collections to organize data and to simplify access control policies. 

• Segregate products and orders related documents into two scopes within a single bucket.
• Instead of a doc_type field, logically group documents of specific type into its own collection.
• App users are then segregated by scope, so users in the products scope can only access products documents and users in the orders scope can access orders docs.
• The sync function becomes simplified by being set at the collection level. 
• The Products app has a pull only replication for products scope and the Orders app has push-pull replication for orders scope. 

Example

Consider a health care app with 30 categories of data, including patients, visits, procedures, insurers, diseases, diagnosis etc.

Patients have access to their medical records via a Couchbase Lite mobile app. Some of the documents (eg. insurers) must not be synced down to the mobile app. Some of the documents (eg. profile) can be updated by the patients via the mobile app while some (like diagnosis) are read only.

Solution: Bucket based

• • Data corresponding to the health care app is maintained in a corresponding bucket.
  • The bucket has over 30 document types. The doc_type field is used to logically group related documents.
  • A Sync function is defined at the bucket level and is complicated. Access control rules are based on doc_type and channels defined for doc_type control which user can access which subset of documents.

Import filters are at the bucket level and complicated. Use doc_type conditions to filter out documents that don’t have to be synced.

Solution: Scope/collections based

Use scopes and collections to organize data and to simplify access control policies.

• Instead of a doc_type field, logically group each document of a specific type into its own collection.
• The Sync function is simplified and defined at the collection level. Every collection is implicitly associated with a channel and as such it is much simpler to specify read/write access to collections.
• Import filter function is simplified and defined at the collection level. Filter out collections of documents that do not have to be imported.

Scalable Multi-tenant applications

The ability to support multi-tenant applications that do not need strict resource isolation but still require independent data lifecycle management.

Example

Consider an ISV hosting an events application for 20 enterprise tenants. Every tenant has a Couchbase Lite app that can sync event data belonging to the tenant. Every tenant has a distinct set of app users who can access tenant specific data. Authenticated app users can sync data via the corresponding Couchbase Lite app for their tenant. The tenants do not require strict resource isolation.

Solution: Bucket based

• Every tenant’s data is in its own bucket.
• Each bucket has dozens of document types. The doc_type field is used to logically group related documents.
• As the solution scales to more tenants, more buckets and subsequently, more clusters are added. This is expensive and is not scalable.
• Every tenant has a distinct set of app users registered at the bucket level.
• Couchbase Lite client replications are set up to sync data from the corresponding tenant’s bucket.

Solution: Scope/collections based

Scopes and collections offers a scalable alternative to supporting multi-tenant apps with independent data lifecycle management.

• Each tenant has its own scope, contained in a single events bucket.
• Each tenant’s client application syncs data associated with the tenant’s scope.
• Each tenant has a distinct set of app users registered at the scope level.
• When tenants are added or removed, a scope is added to or removed from the bucket. A tenant can be taken offline without impacting other tenants.

High Application Density at the Edge

This is the ability to support large scale cloud-to-edge deployments with a number of applications at the edge. Each edge is served using a microservices based application architecture. Microservices need loose coupling so that they can be developed, deployed, and scaled independently. Data specific to a microservice is private to the microservice.

Example

Consider a ship-to-shore app deployment, with 20 ships each running Couchbase Mobile clusters, and a Couchbase Mobile cluster in the cloud. The Couchbase Mobile cluster on each ship powers 50 applications. Each of the applications is built using a microservices architecture. Applications running on each ship sync voyage and ship related documents with shore clusters using inter-Sync Gateway replication. Each app has a distinct set of users.

Solution: Bucket based

• Each of the microservices corresponding to the 50 edge applications running on each ship is in its own bucket. The number of applications served by a cluster is limited by the number of buckets in the cluster.
• Inter-Sync Gateway replications between the buckets in the on-ship cluster and corresponding cloud cluster keeps the ships and shore in-sync. The result is a minimum of one ship-to-shore replication for each microservice.
• Setup on each ship is expensive. The cost is multiplied by the number of ships and not very scalable.

Solution: Scopes/collections based

Scopes and collections allow bucket consolidation and a more scalable and cost-effective alternative to bucket based solution.

• Each app is associated with a scope and each microservice is associated with its own collection. The result is the ability to host a large number of applications on a few clusters.

• Inter-Sync Gateway replication between the scopes in the on-ship cluster and corresponding cloud cluster keeps the ships and shore in-sync. Instead of a replication for each microservice, replication can be set up at app level.

Data Organization and Isolation on Couchbase Lite

The ability for developers to organize and logically isolate data within a single Couchbase Lite database instance.

Example

Consider a Couchbase Lite field app used by employees of a utility company to manage their tasks when in the field. The field app regularly syncs data such as projects, infrastructure maps, etc. via Sync Gateway. The field app has a “scratchpad” like functionality that allows employees to take quick notes that they can erase without affecting other documents. These notes are temporary and local-only must not be synced to a remote server.

Solution: Bucket Based

• The bucket based solution requires two separate Couchbase Lite databases within the app; one for local-only data and one for data that is synced. 
• A replicator is set up with the Couchbase Lite database that holds data to be synced.
• The database level of isolation means that queries across local-only and sync data are not possible.

Solution: Scope/collections based

Scopes and collections allow logical data isolation within a single Couchbase Lite database.

• Instead of  two Couchbase Lite databases, the use case can be supported by scopes within the same Couchbase Lite database, such that the local-only data is in one scope and data that can be synced is in another scope.
• Data is logically isolated from each other and applications can query across the local and sync data.
• Queries are simplified as they can be applied to a scope, avoiding need for filters. It is also easy to purge all documents in a local scope as needed.
• The replicator is configured to sync data associated with the sync scope via Sync Gateway.

Sync Gateway Metadata Isolation

The ability to segregate application data in buckets from system metadata maintained by Sync Gateway for its internal operations.

Example

Consider an inventory management store application which allows store clerks to pull store inventory data from their Couchbase Lite embedded mobile app. Store managers use a backend web application based on Couchbase Server SDK to interface with Couchbase Server for managing inventory data.

Any updates made via the web app are synced down to the Couchbase Lite mobile app.

Solution: Bucket based

• Sync metadata cannot be isolated from application data using the bucket-based approach.
• Sync gateway persists sync metadata and other database configurations in the bucket alongside application data. This data is intended to be used exclusively by Sync Gateway for its operations.
• There are no RBAC controls that restrict access to sync metadata which is a risk. This increases the burden on the SDK-based app to ensure that it does not modify sync metadata.
• Queries for app data need to be appropriately filtered to remove sync metadata, thereby increasing the complexity of the SDK app.

Solution: Scope/collections based

Scopes and collections are the only option for isolating sync gateway metadata.

• System data used by Sync Gateway is maintained in _default scope/collection, where it is logically isolated from application data in user defined scopes/collections.
• Eliminates the risk of the SDK-based app inadvertently deleting sync metadata by using scope RBAC controls to restrict access to the _default scope containing sync metadata.
• The SDK-based app is simplified. Separation of system metadata into its own scope simplifies implementation – queries don’t need filters, and the lifecycle of app data can be managed more easily.

Cloud-to-Edge support for Scopes and Collections

With this release, scopes and collections are available end-to-end across the entire Couchbase Mobile stack. For details, please refer to the documentation. 

The figure below is a summary:

1. 1. 1. Data in Couchbase Server is organized within scopes and collections.
      2. A Sync Gateway database is associated with a single scope. The database can be associated with one or more collections within the scope, including all collections within the scope.
      3. A Couchbase Lite client can set up one or more replications to one or more Sync Gateway database endpoints. Each replication can sync one or all collections associated with that scope.
      4. The Couchbase Lite client can also locally persist data in a scope that is not synced to remote Sync Gateway.
      5. A Sync Gateway cluster can sync one or more scopes to another Sync Gateway cluster via inter-Sync Gateway replication protocol.
      6. A Couchbase Lite client can sync one or more scopes directly to other Couchbase Lite clients over a local network via Couchbase Lite Peer-to-Peer protocol. This can be done without the need for a cloud based control point.

Compatibility

Couchbase Mobile deployments upgrading to v3.1 can continue to use the bucket based model until they are ready to upgrade to scopes and collections. This also allows deployments to continue to use a bucket based model for existing apps but can take advantage of the benefits of scopes and collections for new apps.

Resources

But that’s not all. There are other enhancements and API changes that are discussed in the release notes. Here are direct links to a few helpful resources. You can learn more on our What’s New page.

• • Scopes and Collections in Sync Gateway
  • Scopes and Collections in Couchbase Lite

Whitepapers

• • Choosing the right mobile database
  • Couchbase Mobile Use cases

Release Notes

• • Couchbase Lite 3.1
  • Sync Gateway 3.1

Hear Wayne Carter, VP of Engineer at Couchbase, talk about Scopes and Collections in Couchbase Mobile:

If you have questions or feedback, please leave a comment below or feel free to reach out to me via Twitter or email me.

Acknowledgement

Special thanks to Adam Fraser and Mark Gamble for their review of the blog post.