Couchbase 4.0 CE benchmark (continue)

Couchbase Server
1

Hi…
I’ve continued experimenting with couchbase 4.0 and noticed another issue.

Setup:
Three server nodes:
3 * EC2 rx.xlarge with mounted 5000GB SSD volume (not EBS optimised) (Total size of 15T)
DataRam Quota 22407 MB (Total Ram ~66GB)
IndexRam Quota 2048 MB

  • Generated 2 indexes - one on node2 one the other on node3
    cqb> create index field1_idx on test (field1); // on node2
    cqb> create index field7_idx on test (field7); on node3

Execution:
As previously I run the ycsb client and load the server with the following command:

ycsb load couchbase -s -P workloads/workloada -p recordcount=100000000 -p core_workload_insertion_retry_limit=3 -p couchbase.url=http://node1:8091/pools -p couchbase.bucket=test -threads 20

Results:
When trying to insert 100 million docs after about ~16 million the system crashes and the exceptions I see on the client side are:

Error inserting, not retrying any more. number of attempts: 4Insertion Retry Limit: 3
2016-01-21 11:56:56,896 1336057 [Thread-2] ERROR com.yahoo.ycsb.db.CouchbaseClient - Could not insert value for key usertable:user8661638711032907145
java.lang.RuntimeException: Timed out waiting for operation
** at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:135) ~[spymemcached-2.9.1.jar:2.9.1]**
** at net.spy.memcached.internal.OperationFuture.getStatus(OperationFuture.java:236) ~[spymemcached-2.9.1.jar:2.9.1]**
** at com.yahoo.ycsb.db.CouchbaseClient.checkFutureStatus(CouchbaseClient.java:276) ~[couchbase-binding-0.7.0-SNAPSHOT.jar:na]**
** at com.yahoo.ycsb.db.CouchbaseClient.insert(CouchbaseClient.java:233) ~[couchbase-binding-0.7.0-SNAPSHOT.jar:na]**
** at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208) [core-0.7.0-SNAPSHOT.jar:na]**
** at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:578) [core-0.7.0-SNAPSHOT.jar:na]**
** at com.yahoo.ycsb.ClientThread.run(Client.java:346) [core-0.7.0-SNAPSHOT.jar:na]**
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: ec2-52-35-187-243.us-west-2.compute.amazonaws.com/172.31.28.162:11210
** at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160) ~[spymemcached-2.9.1.jar:2.9.1]**
** at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) ~[spymemcached-2.9.1.jar:2.9.1]**
** … 6 common frames omitted**

p.s
Logs from all three nodes are here

Thanks in advance, Eli.

2

Hi @eli_golin, for YCSB which couchbase repo are you using?

3

Sorry, but didn’t really get the question.

4

I am just pulling the latest from ycsb and working against it.
I think the version now is 0.7.0-SNAPSHOT

5

Hi… we ran the benchmark again yesterday.
We have 10 fields in each doc and I’ve created GSI indexes for 8 fields out of 10.
Put 4 of them on node2 and the remaining 4 on node3.
As before ycsb is working against node1.
We are running a single client instance with 20 threads.

We can see that the insertion speed during the first ~20M docs is around 11k ops/sec, but afterwards it is slowly rising towards 17k ops/sec (Wondering what is the cause of such behaviour)

Approximately after inserting 40M documents we start seeing “Temporary failures” from couchbase.
This is happening during the same time that couchbase’s memory is rising above the hight water mark(55.9GB) to approximately ~59GB.
So this is the first question, why isn’t the memory consumption being limited by the “hight watermark”?

Afterwards we can see that the insertions continue but occasionally with “Temporary failure” responses returned by the server. The client continues jumping above the “hight watermark” line another few times during its run (you can see it from the screenshots)

At around ~50M documents the client crashes with an exception related to timeout from memcached (the same exception as I posted before).

After the client crash I’ve noticed that one of the nodes is in the “pending” state and after awhile the other two nodes became "pending"as well.

The total number of documents inserted is ~60M (couchbase flushed the in memory documents after the client died).

p.s
I’m adding the client log + server logs.
Here are some monitoring snapshots that I’ve made.

Thanks, Eli.

6

Hi @eli_golin,Looks like your writes can still not keep up with the insertion rate. Disk write queue is just building up in the graphs. Couchbase does a memory write and can ingest fast but if the disk isn’t keeping up, we will eventually run out of space in memory to put new incoming updates.
You need better write throughput - to get that you can either use a better IO subsystem or get more nodes for the data service. I believe you are running this on AWS - which SKU is the VM for the nodes?
Other option is you can slow the inserts down and allow it to catch up with the disk writes.
thanks
-cihan