What I’m learning about Couchbase is that while it aims to provide sensible defaults, it also provides options to make trade-offs. The trade-off I’m going to focus on in this blog post is Scan Consistency for N1QL queries.
Couchbase 4.5 introduces a new Scan Consistency option: AtPlus. This joins the existing options to make a total of three:
-
NotBounded (default)
-
AtPlus (new to Couchbase Server 4.5)
-
RequestPlus
This blog post is going to review NotBounded and RequestPlus, as well as introduce AtPlus. I’ll also provide a code sample that you can use to try out all three yourself.
Not Bounded
This is the default behavior.
A N1QL query that is using “Not Bounded” Scan Consistency means that it will not wait for any indexes to finish updating before running the query and returning results. Let’s say you have documents A,B,C in a bucket. At the time of the query, only A and B are indexed. With Not Bounded, only documents A and B will be returned.
Since the query is not waiting on any indexing, this is the best option for performance. Document C will show up once it has been indexed. This shouldn’t take long, but if you have just created C, for instance, and then immediately query for a full list of documents, C may not show up. To demonstrate with the travel-sample bucket:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
private static void NotBoundedExample() { Console.WriteLine("========= NonBounded (default)"); // get the current count var result1 = _bucket.Query("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") .Rows.First(); Console.WriteLine($"Initial count: {result1.airportCount}"); // insert a new airport var doc = new Document { Id = "ScanConsistency::airport::" + _random.Next(10000), Content = new { type = "airport" } }; _bucket.Insert(doc); // get the count again var result2 = _bucket.Query("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") .Rows.First(); Console.WriteLine($"Count after insert: {result2.airportCount}"); // wait a few seconds and get the count again Console.Write("Waiting for 5 seconds..."); Thread.Sleep(5000); var result3 = _bucket.Query("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") .Rows.First(); Console.WriteLine($"Count after waiting: {result3.airportCount}"); } |
What I would expect to see when running this code is:
-
Initial count: N -
Count after insert: N(still) -
Waiting
-
Count after waiting: N+1
I set the wait time at 5 seconds, which is probably overkill. I use Thread.Sleep just for demonstration purposes, but obviously that’s a clunky tactic for a real app. That brings us to RequestPlus.
RequestPlus
This scan consistency option provides nearly the opposite of Not Bounded. It will wait until all document changes and index updates (up until the query was run) are processed before it runs the query.
Here’s a simplified example sequence of events when using RequestPlus
-
Document C is created
-
N1QL query to get all documents is executed (A,B,C all exist in the bucket)
-
N1QL query is put on hold because at least one document needs to be indexed (document C).
-
Indexing process is completed. Document C is now indexed.
-
N1QL query executes, returning A,B,C.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
private static void RequestPlusExample() { Console.WriteLine("========= RequestPlus"); // get the current count var result1 = _bucket.Query("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") .Rows.First(); Console.WriteLine($"Initial count: {result1.airportCount}"); // insert a new airport var doc = new Document { Id = "ScanConsistency::airport::" + _random.Next(10000), Content = new { type = "airport" } }; _bucket.Insert(doc); // get the count again var request = QueryRequest.Create("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'"); request.ScanConsistency(ScanConsistency.RequestPlus); var result2 = _bucket.Query(request).Rows.First(); Console.WriteLine($"Count after insert with RequestPlus: {result2.airportCount}"); } |
This gives us completeness in the query results, at the expense of performance. In some cases, it doesn’t make sense to wait for everything to be indexed. And that brings us to AtPlus.
AtPlus (new to Couchbase 4.5)
This scan consistency option provides something of a middle ground between RequestPlus and Not Bounded. This is also a brand new scan consistency option for Couchbase 4.5.
With AtPlus, you’ll have to do a little more work in your code, but in return you’ll get better performance than using RequestPlus. Instead of waiting for an entire index to complete (which could be multiple documents), it will just wait for the documents that you specify to be indexed before running the query. This is sometimes known as “read your own write” or RYOW.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
private static void AtPlusExample() { Console.WriteLine("========= AtPlus"); // get the current count var result1 = _bucket.Query("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'") .Rows.First(); Console.WriteLine($"Initial count: {result1.airportCount}"); // insert a new airport var doc = new Document { Id = "ScanConsistency::airport::" + _random.Next(10000), Content = new { type = "airport" } }; var insertResult = _bucket.Insert(doc); // get the count again var state = MutationState.From(insertResult.Document); var request = new QueryRequest("SELECT COUNT(1) as airportCount FROM `travel-sample` WHERE type='airport'"); var t = request.ConsistentWith(state); var result2 = _bucket.Query(t).Rows.First(); Console.WriteLine($"Count after insert with AtPlus: {result2.airportCount}"); } |
With this example, the N1QL query will wait only for the new document to be indexed. It will not wait for anything else. In a high-volume system, this could provide a good balance of increased performance and completeness of results.
Note that at the time of writing this post, you’ll need to make sure that you explicitly set UseEnhancedDurability to true when setting up your ClientConfiguration:
|
1 2 3 4 5 6 7 8 |
config.BucketConfigs = new Dictionary { { "travel-sample", new BucketConfiguration { UseEnhancedDurability = true } } }; |
Conclusion
With N1QL, there’s a spectrum of speed and completeness. With Couchbase Server 4.5, you now have three options. NotBounded for raw speed, RequestPlus for up-to-now completeness, and AtPlus sits in between.
Please leave a comment, ping me on Twitter, or email me (matthew.groves AT couchbase DOT com).
