Looking for feedback on possible couchbase use case on AWS

Couchbase Server
1

Architecting some new server setups and couchbase might be a good fit for initial usage and graceful growing of service/cluster.

Our server setup has current following assumptions:

-Currently we are CPU bound, targeting to use C1-xl AWS instances (8 core, 7GB mem) with ~3GB dedicated to couchbase for hot/in-memory data
-start with single server, no replication, localhost requests only

-On growth to > 1 server, looking for a setup where we have N couchbase servers where N is equal to number of application servers, with N replicas each having the full data set on them.

-~99% of data stored will be key/value (rest will be some book keeping and stats)

-keys are UUID’s

    -Avg value size will be an ~20kb compressed blob

-Looking to esue single AWS dedicated EBS volume for persistence on each machine

-Starter setup will dedicate ~3GB to couchbase, keep latest/most popular in memory, we can handle occasional slowness if server needs to retrieve few from disk for few keys

For each request to our service we have the following Couchbase interaction(s):

-Each request to server will spawn 4 simultaneous processing steps retrieving ~1100 objects on avg from couchbase

  1. Retrieve ~500 of the 20kb objects at time
  2. Retrieve ~500 of the 20kb objects at time
  3. Retrieve ~50 of the 20kb objects at time
  4. Retrieve ~50 of the 20kb objects at time

Questions I have and feedback looking for based on above info:

-First, does this sound like sane use case for initial usage and graceful scaling with the N replica targeting?

-Assumption that multi-get for retrieving the blocks of key/values will be optimal to shave off some ms for the ~1100 retrievals needed per request?

What is timeline for latest community edition release? We will be starting some tests on latest and would like to move up to 2.x as soon as its available to test performance

Any gotchas or specific configuration(s) we should look at?

Thanks in advance for feedback…, tried to include as much info as possible as it currently stands, continuing to research so will update if other details/assumptions pop up

2

I think this is probably just misunderstanding on terminology, but normally you wouldn’t increase the number of replicas, but add more nodes to the cluster to further distribute the data.

In case if you are more familiar with RDBMS replication as a way to have copies of your data on multiple nodes in order to get more read throughput, here is a quick summary of how Couchbase does it and is commonly used on EC2:

In Couchbase, each node holds some active data and some replica data (so it is not master-slave replication). The data is equally distributed amongst all nodes. When you add nodes, a bit of data from all existing nodes is moved to the new nodes, all without app changes or downtime.

The number of replicas (you can choose 0-3) determines how many additional copies of the data are available in the cluster, to deal with nodes failing and keeping data available.

The nice thing about distributing the data amongst nodes and having the replicas spread out, means that reads get spread out amongst all servers and of course the data set can exceed capabilities of a single node and in a failure mode you get high availability with each node taking a little bit of the additional load.

So especially if you have lots of reads, you will be able to get higher throughput by having the reads being served in parallel from multiple machines.

Whilst you can read from replicas with a dedicated API, you normally should only read the active document, in order to get read consistency. So as you add more app servers, you wouldn’t have a second copy of the entire data set on the second node, instead you would end up with portions on the data on each node.

Co-locating the app and couchbase server is possible, though as you go higher throughput, most commonly you’d have the app server tier and the DB tier on seperate nodes. Nice thing of that architecture is that you can scale both tiers seperately as you need (you’d get the reads and network IO be served in parallel).

If you do indeed need multiple copies of the entire data set XDCR allows you to replicate an entire cluster to another cluster (e.g. between EAST and WEST or to EMEA).

Regarding your storage approach. EBS is recommended for storing the data, you can increase performance by striping across EBS volumes, if you find your disk persistence isn’t keeping up (or you can add more nodes to get more overall IO bandwidth). BTW, the manual has a bunch of EC2 specific best practices that are important: http://www.couchbase.com/docs/couchbase-manual-2.1.0/couchbase-bestpractice-cloud.html

3

Didn’t realize the replica cap was at 3, but that will do for some time.

The workload is bit unique in that its pretty much entirely read. We have write load in the realm of ~50-100/sec max…, at least for some period of time since its batch job based. On read side, we need scale in the realm of ~1100 20kb reads per request…, so supporting 10 simultaneous requests would result in ~11k+ reads x20kb for each object.

Other unique thing with the workload is that even though the data set will be grow large over time size/storage wize…, we only need to accommodate ~3-4GB in memory at any given time. This will grow down the road but will just move casually by adding larger memory machines into rotation and migrating traffic over and deprecating the smaller machine(s).

If we have 3 servers and replica config of 3 will couchbase be smart to put a replica on each server? For needs of this project 3 servers of 8 CPU/7GB will take it quite far. Was hoping to have entire replica locally and avoid any network latency, shave off some ms, and avoid cost of maintaining a dedicated couchbase cluster when the use case is fine to have it co-located.

4

the maximum number of replicas you can have with 3 servers would be two: The primary copy of data lives on one server and the other two can have a copy each. So each server will hold 1/3 of the primary dataset and 1/3 of all replicas (which for 2 replicas would be 2x the primary data amount). 10’s of thousands of reads per second won’t overload couchbase,as long as the working set is in RAM.

5

Thanks for feedback,

Will setup a test instance or two and see how things perform