Couchbase Get operaion slows down with increased thread

Couchbase Server
1

We are having a spring boot Couchbase Rest application(spring data couchbase 2.1.2 ,Spring boot 1.4). The Application get data based on Id . The size of the each document is 2MB . 2 machines with 24 core and 32 GB ram holds the couchbase clusted nodes. The problem is that when the number of threads increases the response time increases .The threads are fired using Jmeter. when the number of threads is 10 The application responds in 350 ms .When the number of threads is 50 application responds in 900 ms average. When the number of threads is 100 the application errors and the response time is 1.5 seconds average .

When we tried the same with document size of few KB we get a good Throughput. How can we tune to provide a good throughput with a 2 MB document . please help

2

@justmailphilip how good is your network connection? Keep in mind that when you have 2MB documents and you are loading in 100 in parallel (imagine 100 threads asking for it) that will be 200mb/s which is way more than a 1gbit/s link can handle.

Now in terms of tuning options, did you consider cranking up the kvEndpoints setting to a higher value than 1 on the CouchbaseEnvironment? Start setting it to 2, 5, or 10 and see if it makes a difference.

Also, did you do some profiling to make sure there are no other bottlenecks in your system like encoding and decoding? JSON marshalling can take some time as well if you are doing it on big bulks.

3

@daschl Thanks a lot for the response.We had tried cranking up the endpoints and please find the response below

Also we tried the following option to increase the reader threads based on https://developer.couchbase.com/documentation/server/current/cli/cbepctl/set-flush_param.html .

But there was no much changes in performance.

The data was serialized and stored using @cachable of Spring.So No Json Marshaling issue.

Also we tried to split the 2.5 MB documents into smaller docs (around 1 KB) *2000 docs. Now we encounter back pressure exceptions with high threads. Will increasing the Couchbase instances solve his .We use java RX as advised by couchbase

Bucket bucket = repo.getCouchbaseOperations().getCouchbaseBucket();

    return Observable.from(couchbaseRefs).flatMap(new Func1<String, Observable<JsonDocument>>() {
        @Override
        public Observable<JsonDocument> call(String id) {
            returnbucket.async().get(id);
        }
    }).toList().toBlocking().single();

Note : we are checking with network team for congestions.