A frequently asked question by the Couchbase developer community is “how can I perform bulk type operations?” The expectation is that a high performance data platform will support batching to improve performance. All Couchbase SDK clients support batching operations. Through batching, requests are pipelined over the network. Increasing network throughput, reducing latency and better utilization of network resources is what batching operations are all about. The Couchbase Developer Guide contains more indepth information and architectural considerations specific to batching. If you’re not familiar with our dev guide, it’s a great place to learn the ins and outs of development with Couchbase. All of the code examples referenced in our dev guide including the one used in this blog all live in one convenient github repo.
Go Bulk Operation Walkthrough
To build a simple application with Couchbase, first include “gocb” in the import packages for the application. As a reminder, connections from the application to Couchbase are instantiated once and re-used as a lightweight persistent connection to the cluster. This allows for dynamic topology changes to happen without any disruption to the application and bucket level operations are executed efficiently.
To perform CRUD operations on documents in Couchbase, we use the bucket API. The bucket API in golang includes a type called BulkOp that is an interface for performing bulk operations. First we need to open a connection to the cluster, and then use the bucket reference we defined for use in the application
The goal of the application is to insert 10 documents into Couchbase. Couchbase is a document database and speaks JSON by default. An easy way to create the documents to use with the bulk operations is to first create a JSON type struct with a single field called “Item”. To make things simpler two variables are instantiated to assist in naming the key, and for the value of each document.
In the example below, two arrays of gocb.BulkOp are created. One for performing the bulk insert, and the other for doing the bulk get.
Through the wonderful JSON marshalling in golang, 10 items using a reference to the InsertOp type are added to the array of items that will be bulk inserted
The BulkOp type has a method called “Do” which actually performs the pipelining of requests in a batch operation on the cluster. The Do method will return an error if there is one, or nil.
Retrieving the items using a bulk get works the same way. Using the second array “itemsGet” a grouping of items using a reference to the GetOp type is created.
This short example shows the simplicity of using bulk operations with Couchbase and the power of the go sdk. Other bulk operations are supported including prepend, remove, replace, getandtouch,upsert, touch. API docs listing all of the methods are available in godocs.
