Couchbase Performance issue/question

Hi All,

Before putting my actual question on performance I want to put below the server configuration and some details regarding our setup:

Couchbase server version: 3.0.1 Community Edition (build-1444)
Couchbase Java SDK version: 2.1

I have setup a Couchabse cluster of three nodes, with each node having the following or better configuration:

Cluster 1, Server 1,2,3:

Total Processor: Output from nproc is 8 and each having 2 cores, so in total its is a 16 core machine.
Total RAM: ~23 GB (shared with three other apps running in the same servers taking a total of less than 10GB)
No of Buckets: 1
Replication factor: 1
Couchbase bucket size: 1 GB
Number of documents in Couchbase: 191778 or less
RAM/Quota Usage: 197MB or less
Disk Usage: 169MB or less

All three nodes are in the same network in the same data center.

From this configuration my opinion is that even if I request a unique document everytime I should find it in the RAM all the time, I may be wrong here so please correct me.

For my testing purposes I setup a Jmeter on a 4th server and request a URL in our app deployed over mule with 64 threads configured to cater to HTTP requests in the same VM as Couchbase. The service on this URL has the only purpose to request a unique document everytime from Couchbse. And in the same service using JAVA System.currentTimeInMillis() method I record the difference in time before and after the couchbase call.

Jmeter starts 400 threads at a time and shoots them simultaneusly then waits for 2 minutes and again shoots those 400 threads with a loop count set to 10 i.e. total 4000 requests.

What I have noticed is that out of the total 4000 requests ~300 requests took more than 50ms ranging from 50ms to 260ms.

Please note that my intention was to find out how many requests are taking more than 50 ms as our application needs to respond within 500 ms of a client request where each request may have upto 6-8 couchbase requests.

So now my questions are:

What are Couchbase typical get times?
Is a single couchbase client(bucket) (we are using JAVA sdk) able to comprehand with 1000s of requests to Couchabse at a given time?
Is there anything simple that I am missing in my setup?
Are my expectaions from Couchabse too much for it to respond within 50ms everytime?

Following is my reference implementation:

public class CouchbaseTest implements Callable {

private static final Logger LOG = Logger.getLogger(CouchbaseTest.class);
private static final CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder().connectTimeout(ApplicationConstants.COUCHBASE_CLUSTER_CONNECTION_TIMEOUT).build();
private static final Cluster CLUSTER = CouchbaseCluster.create(env,Arrays.asList(ApplicationConstants.COUCHBASE_CLUSTER_IPS.split(",")));
public static final Bucket syncBucket = CLUSTER.openBucket(ApplicationConstants.COUCHBASE_BUCKET_NAME,ApplicationConstants.COUCHBASE_BUCKET_PASSWORD);
private static final AtomicLong impl1PrefixCount = new AtomicLong(1);

@Override
public Object onCall(final MuleEventContext context) throws Exception {
    couchbaseGetImpl1();
    return "success";
}

private void couchbaseGetImpl1() {
    final String eventId = "impl1_" + impl1PrefixCount.getAndIncrement();
    long methodCallStartTimeInMillis = System.currentTimeMillis();
    getImpl1(eventId);
    long timeTaken = System.currentTimeMillis() - methodCallStartTimeInMillis;
    if(timeTaken > 50){
        LOG.error("Implementation: 1,Thread Name: " + Thread.currentThread().getName()+"," + timeTaken);
    }
}
 
private JsonDocument getImpl1(String documentId){
    return syncBucket.get(documentId);
}

}

Any input would be much appreciated, please let me know if you need any more details regarding the same.

Anyone has any input here? And I have another questions which is, is there any place in Couchbase Forum or on Couchbase website where we can see the Couchbasae server performance comparison between different Couchbase versions for various Couchbase services? How to believe that Couchbase has improved performance with any new upgraded version. Or one has to do this Couchbase performance test before taking every upgrade ?

Hi, amit,

Thanks for your questions in the forum. Sorry I missed this thread.

To answer your questions:

  1. If data is all in memory, internally, we observed that 95% get/set latency is below 1ms. (assuming 10g network)
  2. Based on your descriptions, here are my thoughts:
    1). Do you have enough RAM for all the data? Is RAM quota 1GB or 197MB?
    2). I’m not sure how JMeter schedules client thread internally. Is there a possible there’s a delay from the client side because of thread scheduling? What happens if you reduce # of threads to be 100?
  3. Are you using 10g or 1g network?

To answer the second part:
Before every release, we go through extensive performance testing in house. We make sure there’s no regression in existing functionalities and we make sure performance improves in planned areas. We published perf numbers in blog posts and external benchmarks.
http://blog.couchbase.com/facet/Topic/Couchbase+Server

Thanks,
Qi

Hi Qi,

Thank you very much for your response. Here are the answers to the doubts that you had:

  • Yes the RAM size is 1 GB, I tried increasing that too and can increase that a lot but I did not see any corresponding improvement as 1 GB was enough too.
  • JMeter was setup to wait until all the 400 threads are created and then shoot them. As depicted in my sample code I was not relying on the JMeter time and was logging the time taken by the Couchbase at server side application.
  • The Network speed I tried the following command from one of the node connecting to the other node:

iperf -c otherhostname -P 10 -t 30

It gave me the following output

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.0 sec 2.76 GBytes 791 Mbits/sec
[ 4] 0.0-30.0 sec 4.27 GBytes 1.22 Gbits/sec
[ 6] 0.0-30.0 sec 3.67 GBytes 1.05 Gbits/sec
[ 5] 0.0-30.0 sec 3.63 GBytes 1.04 Gbits/sec
[ 12] 0.0-30.0 sec 3.90 GBytes 1.12 Gbits/sec
[ 7] 0.0-30.0 sec 3.96 GBytes 1.13 Gbits/sec
[ 10] 0.0-30.0 sec 4.17 GBytes 1.19 Gbits/sec
[ 8] 0.0-30.0 sec 1.87 GBytes 534 Mbits/sec
[ 9] 0.0-30.0 sec 5.04 GBytes 1.44 Gbits/sec
[ 11] 0.0-30.0 sec 3.98 GBytes 1.14 Gbits/sec
[SUM] 0.0-30.0 sec 37.2 GBytes 10.7 Gbits/sec

If there is no obvious reason then I think only reason I see could be the use of VMs rather than physical machines.

I have one more additional question. I see Couchbase do a lot of logging. So what are the mandatory loggins. How to shut off all other loggins. And can we disable statistic collection? Is that recommended? Because we may not require any statistics.

Thanks

  • Amit

Hi, Amit,

Thanks for more information. Unfortunately, I’m not familiar with logging setting in Couchbase. Let me get my colleague to help out here.

Thanks,
QI

Hi Amit,

Here are a few documentation links about the different log files in Couchbase and how you can set the logging levels -

  1. http://docs.couchbase.com/admin/admin/Misc/Trbl-logs.html
  2. http://developer.couchbase.com/documentation/server/current/troubleshooting/troubleshooting-logs.html

Thanks,
Don Pinto
Couchbase Server PM