Couchbase 3.0.1 Performance Issue in full metadata ejection mode

Couchbase Server
1

Hey there,

At the moment I’m doing some tests with couchbase 3.0.1 community edition and the java sdk 2.0.1. In full metadata ejection mode the performance drops after approximately 100 million inserts. In contrast, the value ejection mode performs 200 million inserts without problems. Could it be a bug?

I use 2 Clients with 32 Threads for inserting and the Cluster has 2 Nodes.
The hardware specs for clients and cluster nodes are identical:
Dell PowerEdge C62202 with

  • 2 Intel Xeon E5-2609 (each 4 Cores)
  • 32 GB RAM
  • 4 x 1 TB SATA 7.2 k

I may as a new user upload any files, sorry :frowning:

Thanks

2

Interesting, I’d suspect you are hitting some resource exhaustion at some point.

Are you performing only inserts, or are you also retrieving the data back… At this 100m doc mark, can you see something odd happening? For example the disk write queue is growing too large and data keeps backing off…

Can you share the client code so we can try to reproduce it? Also, it would be great if you could run a cbcollect_info so we can take a look at server characteristics.

3

I tried only to insert 200 Mio documents. My code is based on the thumbtack ycsb version, but i added new DB adapter (based on java sdk 2.0.1).

Why should the full ejection modus get a resource problem and the value ejection modus not?

Where can i upload the source and the cbcollect_info files?

screenshots which I made yesterday:

ops per second.png896×352 26.1 KB

summary.png763×505 43.4 KB

4

Interesting. You can raise an MB ticket and attach it there, we can take it from there then (http://www.couchbase.com/issues/browse/MB).

To rule out anything odd, do you see different client behaviour when the modes are changed? For example garbage collection taking out the box or OOM issues? Something in the logs on the client can be found?

Also, can you share the code somewhere to look at (your changed adapter)?

5

One thing our support fellas pointed out is that once you hit the non 100% residency mark we need to check for every add and replace (I checked the original thumbtack code) it needs to check if the doc is already on disk or not which will slow it down.

With value ejection only this does not need to happen.

One thing, I think if you use upserts instead of insert/replace, there is an optimization on the server since we can just write. Try to change the ops to upsert instead of insert/replace and see what behavior you get.

6

I have found another screenshot on which we can see that the resident is 100%.

resident.png763×510 29.7 KB

Only one client has the following message in log:

Nov 23, 2014 11:31:26 AM com.couchbase.client.deps.io.netty.util.ResourceLeakDetector reportLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dcom.couchbase.client.deps.io.netty.leakDetectionLevel=advanced' or call ResourceLeakDetector.setLevel()

I raised an ticket (http://www.couchbase.com/issues/browse/MB-12752) and uploaded the cbcollect_info files.

7

Could someone from Couchbase re-open my ticket (http://www.couchbase.com/issues/browse/MB-12752) please?

8

Yes, done. Thanks for the additional data.