Performance degradation with later java client

Our company use Couchbase server 3.1.3 with java client version 1.3.1.
I was asked to upgrade to Couchbase server 6.5 with java client 2.7.13. The code changes was simple.
However, the load test(Jmeter) result showed the response time was about 10% to 15% slower compared with our old version.

I created some simple java programs for both 1.3.1 and 2.7.13. They bypassed the application server and went to couchbase server directly. The code is very simple. It initialized the couchbase environment and then call the “get” method to retrieve document. It run multi threads concurrently with a given duration. At the end of execution, it print out how many “get” method was called. The code snapshot is like:

For java client 1.3.1

main function —

List nodeList = new ArrayList<>();
nodeList.add(URI.create("–url–"));
CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
connectionFactory = builder.buildCouchbaseConnection(nodeList, “–bucket name–”, “–password–”);
couchbaseClient = new CouchbaseClient(connectionFactory);

ExecutorService executor = Executors.newFixedThreadPool(threads);
for (int i = 0; i < threads; i++) {
list.add(new TestRun());
}

List<Future> futures = executor.invokeAll(list);
long total = 0;
long failed = 0;
for (Future f : futures) {
total += f.get();
}

public static class TestRun implements Callable {
@Override
public Count call() throws Exception {
int counts = 0;
long start = System.currentTimeMillis();
String doc = “”;
while (true) {
long end = System.currentTimeMillis();
if (end - start > duration) {
break;
}

		String key = getKey(end);
		doc = (String) couchbaseClient.get(key);

		counts++;
	}
	return counts;
}

}

For java client 2.7.13

main function —

Builder builder = DefaultCouchbaseEnvironment.builder();
env = builder.build();
List nodeList = new ArrayList<>();
nodeList.add("-- url --");
cluster = CouchbaseCluster.create(env, nodeList);
cluster.authenticate("-- user name --", “-- password --”);
bucket = cluster.openBucket(“bucket name”);

ExecutorService executor = Executors.newFixedThreadPool(threads);
for (int i = 0; i < threads; i++) {
list.add(new TestRun());
}

List<Future> futures = executor.invokeAll(list);
long total = 0;
long failed = 0;
for (Future f : futures) {
total += f.get();
}

public static class TestRun implements Callable {
@Override
public Integer call() throws Exception {
int counts;
long start = System.currentTimeMillis();
RawJsonDocument doc = null;
while (true) {
long end = System.currentTimeMillis();
if (end - start > duration) {
break;
}

		String key = getKey(end);
		doc = bucket.get(RawJsonDocument.create(key));

		counts++;
	}
	return counts;
}

}

I tried the following couchbase servers
couchbase-server-community-4.5.1-centos7.x86_64.rpm
couchbase-server-community-5.1.1-centos7.x86_64.rpm
couchbase-server-community-6.0.0-centos7.x86_64.rpm

Java client 1.3.1 version of the test program’s performance is better than my Java client 2.7.13 program. Can anyone tell if I missed any important steps. By the way, I only had one node during the test.

The problem is identified. Java client SDK 2 and 3 is worse than java client SDK 1 in case of small document which size is about or less than 10k. Version 2 and 3 are slower to generate and retrieve smaller documents.

@wyang can you please also post the sdk3 code you tested?

Also, can you please post the actual numbers you got? Note that this is a microbenchmark and will not reflect reality when you write your actual program (unless you are accessing one document in a hot loop that doesn’t actually do anything?)

Thanks for your reply.
I tried Java SKD 2 and 3 in our developer cluster. Both showed slow response compared with SDK 1 very consistently.
Our documents size are about less than 10K. Most of them are 100 bytes. In order to dig the root cause, I created a Java program to use SDK 1 and 2 to access my local couchbase version 6. I tried to read and write different file sizes. The result is very consistent. The following are a small sample from the test. I used 5 minutes for write and 1.25 minutes to read. I will try SDK version 3 later.

document size: 20K
sdk 1.3.1: 431347 documents generated
sdk 2.7.13: 370905 documents generated
performance -14%
sdk 1.3.1: 135604 documents read
sdk 2.7.13: 142844 documents read
performance 5%

document size 10K
dk 1.3.1: 573422 documents generated
sdk 2.7.13: 427838 documents generated
performance -25%
sdk 1.3.1: 158579 documents read
sdk 2.7.13: 110387 documents read
performance -30%

document size 0.2K
dk 1.3.1: 1093663 documents generated
sdk 2.7.13: 716626 documents generated
performance -34%
sdk 1.3.1: 208337 documents read
sdk 2.7.13: 167958 documents read
performance -20%

@wyang ok please also test sdk 3. again note that I do not think that your synthetic benchmark is representative of real-world workloads, but we have tuning options available that will get sdk 2 closer to sdk 1. I would rather test SDK 2 / 3 in your real world application and then start to measure if it performs to what you need and then start tweaking from there.

We will do a load test with more machines soon. I did another test which showed 1.3.1 and 2.7.13 no significant difference. However, 3.0.1 is better than both 1.3.1.and 2.7.13. The following is a test result for 10 minutes write resource files (file size 3k), 10 minutes write tracking file (file size 0.2k), 2.5 minutes read resource files, end with another 2.5 minutes read tracking files. The test run 80 threads
SDK 1.3.1
7,642,389 resource generated
11,352,917 tracking generated
180,751 resource read
300,862 tracking read

SDK 2.7.13
7,535,380 resource generated
10,938,058 tracking generated
131,113 resource read
415,687 tracking read

SDK 3.0.1
10,563,826 resource generated
13,265,710 tracking generated
738,702 resource read
514,339 tracking read