Couchbase Server 4.5 Memory Optimized Indexes

Introduction

Couchbase Server 4.5 has just been released. A complete overview be found here.

Let’s explore the new Memory Optimized Indexes.

To understand these enhanced indexes let’s first review the Global Secondary Index (GSI) that was introduced as part of Couchbase 4.0. GSI’s are built to support the Couchbase query language N1QL (pronounced ‘nickel’). These queries are great for supporting interactive applications that require fast response times. Prior to the introduction of GSI’s the only query Couchbase offered was in the form of map/reduce Views primarily focused on reporting with a pre-calculated result but were limited in performance and scale. With GSI’s Couchbase can not only scale data services linearly but query as well.

For example we could create an index called airlines on the travel-sample bucket shipped with the product:

Which would allow us to query all airlines quickly.

What’s New

Memory optimized global secondary indexes is an additional storage setting for Couchbase Server clusters. Memory optimized global secondary indexes (MOI) can keep up with data mutations and index maintenance at in-memory speeds. Aside from the performance characteristics, the mechanics of creating, placing, load balancing, partitioning and availability is identical in both GSI and MOI.

While leveraging MOI provides performance gains it’s imperative to monitor memory usage. MOI are restricted to reside in the managed cache of the Couchbase node. These types of indexes will stop processing data mutations if the node exhausts its available memory. This can be dynamically tuned without downtime based on needs by adjusting the Index RAM quota. This is also an area where Multi-Dimensional Scale (MDS) provides the flexibility to run nodes with larger memory footprints than other nodes in the cluster. Once more these changes can be done while your applications remain online and traffic is served.

In production deployments there are two important metrics that will provide the ability to proactively monitor for issues. These are exposed in the Couchbase administration console as well as our REST API:

• MAX Index RAM Used %: Reports the max ram quota used in percent (%)
• Remaining Index RAM: Reports the free index RAM quota for the cluster

Use Case

Couchbase provides the ability to index array elements in a JSON document. Often times documents that contain array attributes will create increased load during data mutation. Customer that need interactive, real time access to data are often times left with maintaining complex data structures to meet application SLA’s. Memory optimized global secondary index (MOI) is designed to meet the challenges of the most demanding applications.

• Travel Booking
• Leader Boards
• Fraud Detection
• Interactive Recommendations

MOI provides improved latency and throughput under demanding data mutations when compared to standard GSI indexes. To do so MOI provides:

• Lock-free processing for indexing allows massive concurrency when maintaining the incoming mutations.
• Skiplist structure in-memory storage, as opposed to B+Tree.
• Forcing the index into memory storage, means MOI does not run at disk speeds for storing the index.
• Regular snapshots persisted to disk for recovery.

Summary

The new memory optimized index is one of the new features of Couchbase 4.5. The main advantage is that it improves query performance and reduces the time it takes for indexes to be maintained.

Download Couchbase

Visit www.couchbase.com/download/ to get Couchbase 4.5 and use Memory Optimized Indexes.