.NET SDK Performance

xulane · September 9, 2018, 11:34pm

To give a little background, we’re upgrading from a very old version of Couchbase (2.1.X) to Couchbase 5.5 (.Net Client 2.6.0), and we’ve noticed a significant performance impact to our application servers following the upgrade to use the latest .net client. Our service handle batches of sets and gets (mostly gets) that range from anywhere from 3 to 600+ on average in a batch. We used a memcache bucket in the past, but we’re trying to move to an Ephemeral bucket on the new cluster. When upgrading to the latest .NET client, we used the GetDocumentAsync and UpsertAsync methods to handle these batch get/set requests using something along the lines of the following:

var result = Task.WhenAll(fileKeys.Select(key => getBucket(iEncryptor).GetDocumentAsync<Entity>(key)));
result.Wait(); // In this case their used immediately after, and according to what I read the async methods are the way to batch send requests over the network

&

    var result = Task.WhenAll(
        writeDictionary
        .Select(pair => getBucket(iEncryptor).UpsertAsync<Entity>(pair.Key, pair.Value))
    );
    result.Wait();

We’ve run several small performance tests, which end up being approx 1500 gets and 350 sets per second to our couchbase cluster, and the results are as follows:

Pre-Upgrade:
35% CPU on our .NET application server, and an average response time of 183ms for the test.

Post-Upgrade:
65% CPU on our .NET application server, and an average response time of 310ms for the test.

We’ve checked to ensure we’re re-using the same bucket, tried to increased the MaxSize on the pool (non-ssl -> so shared mux pool by default), and switched back from an Ephemeral to a memcache bucket, but all had no affect.

Based on the research I did, it looks like the async/await overhead might the culprit since each individual task is small (<1 ms typically to our cluster), and we’re dumping large batches into the generated task state-machine. I was planning to try using sync methods but across parallel threads when I found the ‘obsolete’ methods in the .net client that basically implements this already:
getBucket().Get<Entity>(fileKeys);
getBucket().Upsert(writeDictionary);

Using these methods did show a significant improvement, but it still wasn’t close to our pre-upgrade performance (I don’t think the sync method batch requests over the network which might contribute to the extra cost):
Post-Upgrade with Obsolete methods:
45% CPU on our .NET application server, and an average response time of 245ms for the test.

We added additional logging to time the batch set/gets, and ONLY changed the async methods (GetDocumentAsync/UpsertAsync) vs obsolete batch get/set methods (collection Get/Upsert), and recorded the values below:

Obsolete Methods:
45% CPU

0.782ms per each document in a ‘bulk’ get from couchbase
0.302ms per each document ‘bulk’ write to couchbase

Async based Methods:
65% CPU

1.2ms per each document in a ‘bulk’ get from couchbase
0.71ms per each document ‘bulk’ write to couchbase

*Notes:

The times are based on (total time for batch / number of items in the batch)
The performance test was the exact same
The only difference was the change in sdk method usage

I’m looking to see if anyone has come across these performance issues when dealing with batch requests to Couchbase using the latest .NET clients (or in general), or if any settings can be tweaked to improve performance.

Ideally, we’d like to get close to our pre-upgrade performance. Any input would be greatly appreciated!

jmorris · September 10, 2018, 2:58am

@xulane

You are using the async methods synchronously. Use them like this:

var result = await Task.WhenAll(fileKeys.Select(key =&gt; getBucket(iEncryptor).GetDocumentAsync&lt;Entity&gt;(key)));

And like this:

var result = await Task.WhenAll(
    writeDictionary
    .Select(pair => getBucket(iEncryptor).UpsertAsync<Entity>(pair.Key, pair.Value))
);

You should get the best performance using the SDK straight “out out of the box” without adding changes to your config with the exception of increasing the MaxSize (possibly - depends on your environment).

xulane · September 10, 2018, 1:04pm

That works in the ideal case where a service has been built around async from the start, but this is a legacy system that has existing interfaces and contracts it must adhere to.

I realize it’s never the first choice to use async via sync, but I don’t see another option (other than a major rewrite to our application) given that the sync API doesn’t have a bulk get (other than the obsolete ones), and that API doesn’t batch send those requests over the network (from what I’ve read).

I also don’t see how making the system async is going to improve the performance degradation we saw to the CPU (35% (pre-upgrade) or 45%(using obselete bulk sync) to 65% for the async option), or the response time of the service (this would require useful work that could be completed while waiting for the results, which isn’t the case for these requests – the data is required for processing to begin).

Going async would make the system more scalable, but it doesn’t mean it would gain this performance back.

MikeGoldsmith · September 10, 2018, 2:28pm

Hi @xulane

Executing an async call in a synchronous environment does have some performance penalty’s because it will block the current thread, where an await operator allows the thread to be reused within the thread pool. I do agree you should not be seeing a degradation of performance like you have indicated.

The bulk methods that you are using on the Async API are just utility methods within the API that distributes the operations over as many connections are available; it does not affect how the TCP packets are sent over the network and is comparable to doing them from your application code.

I can see you’re using Couchbase server 5.5 and .NET SDK 2.6.0; please ca you share the following:

Are you providing any custom SDK configuration?
Is the new cluster comparable in size / memory / enabled services?

Also, MeepMeep is a .NET SDK performance tool if you’d like to try that. It needs updating to the latest .NET SDK but should help give you some idea of performance availability.

xulane · September 10, 2018, 2:50pm

The new cluster is a mirror image of the old cluster resources wise - same number of virtual CPUs, RAM, etc. The major differences is we updated to Windows Server 2016, and updated to the latest version of Couchbase.

As for enabled services in CouchBase 5.5, we’re only running the data service since we’re only doing the most basic document gets/sets against it.

Configuration:
<couchbase operationTracingEnabled=“false” operationTracingServerDurationEnabled=“false”
enableOperationTiming=“false” enableQueryTiming=“false” xdt:Transform=“Replace”>
<servers>
<add uri=“http://…-1…:8091” />
<add uri=“http://…-2…:8091” />
<add uri=“http://…-3…:8091” />
<add uri=“http://…-4…:8091” />
<add uri=“http://…-5…:8091” />
<add uri=“http://…-6…:8091” />
</servers>
<buckets>
<add name=“bucketName” operationLifespan=“120000”>
<connectionPool name=“custom” maxSize=“100” minSize=“10” connectTimeout=“20000” sendTimeout=“120000” />
</add>
</buckets>
</couchbase>

Note:
The trace/timing were disabled to deal with the issue mentioned here:

operationTracingEnabled=“false”
operationTracingServerDurationEnabled=“false”
enableOperationTiming=“false”
enableQueryTiming=“false”

I’ll try MeepMeep today, and post the results.

MikeGoldsmith · September 10, 2018, 3:03pm

A maxSize of 100 is very high; the default is 2. From .NET SDK 2.5.6 we switched to multiplexing connections by default, which allow data to be send and received concurrently. These are more performant than using a traditional connection pool that sends/receives then returns to the pool. Also, all connections are built up during setup, which with 100 could affect start-up times.

Please can you try lowing this to say 5 and test again?

Otherwise the config looks fine.

PS We released 2.6.2 of the .NET SDK earlier today that should resolve the issue with operation timing.

xulane · September 10, 2018, 3:06pm

Will do. We’ll run another set of tests with the max lowered to 5, and then run MeepMeep afterwards, and post the results.

Thanks!

MikeGoldsmith · September 10, 2018, 3:07pm

I’ll look to update MeepMeep with the latest SDK version too.

xulane · September 10, 2018, 6:28pm

I re-ran our performance/load test using MaxSize=5, and I didn’t see much of a difference using the sync bulk get/set wrapper methods. There was a slight difference in response time, but I also didn’t run the perf test for an extended period of time, so I’d just consider this ‘within error’.

Post-Upgrade with Obsolete methods:

45% CPU (didn’t change)
Response Time: 256ms (vs 245ms for large pool)

jmorris · September 10, 2018, 6:56pm

@xulane -

Is this a web application or a console application? If its a non-web application it may make sense to use the older obsolete TAP based methods or at least roll your own extensions based on the concepts so that your forwards compatible esp since your app is synchronous from the start.

xulane · September 10, 2018, 8:19pm

@MikeGoldsmith

I ran the latest version of MeepMeep with the following results. I ran over a variety of workload sizes for sync and not sync.

Notes:

Per operation between sync and async matches what I’m seeing.
Async Avg operation time (ms): 2.85966974 vs Sync Avg operation time (ms): 0.921244880000001, but the operations/sec are much lower.*
*At 100k workload, Sync has a CPU of 20%(ish), and Async has a CPU of 75%+ (Is the sync option doing threaded sync requests or truly sync?)

– 500 –

Running with options:
Nodes=“http://…-1…:8091 http://…-2…:8091 http://…-3…:8091 http://…-4…:8091 http://…-5…:8091 http://…-6…:8091”
Bucket=“…”
BucketPassword=“…”
NumOfClients=1
WorkloadSize=500
DocSamplePath=“”
DocKeyPrefix=“mm”
DocKeySeed=0
DocKeyRange=1000
ClusterUsername=“…”
ClusterPassword=“…”
WarmupMs=1000
Verbose=False
MutationPercentage=0.33
WorkloadType=MutationPercentage
EnableOperationTiming=True
UseJson=False
UseSync=False