New Client seems to be 2x Slower... was - Couchbase .NET Client not writing to all nodes

.NET SDK
1

I’ve written a really awful (bad code, I’m not proud, but it was quick) client that I’m using to compare performance of CouchbaseNetClient v1.3.12 and v2.1.1.

We have a 2 node cluster for testing.

The v1-based client is working fine (both nodes in cluster), but the v2-based client is failing, consistently, for requests to the 2nd node, operations (SET / UPSERT and DELETE / REMOVE) with certain keys. I get back “NodeUnavailable”. I have verified where the documents would be written basked on responses of cbc-hash.

During the setup process, I do notice:
A first chance exception of type ‘System.OutOfMemoryException’ occurred in Couchbase.NetClient.dll
A first chance exception of type ‘Couchbase.IO.ConnectionUnavailableException’ occurred in Couchbase.NetClient.dll

Here’s the blocks of code, I can package up these projects if need be.

v1.3.12:

        public void runThread()
        {
            var stopWatch = new Stopwatch();
            stopWatch.Start();

            for (int i = 0; i < _iterations; i++)
            {
                stopWatch.Restart();
                var item = "test-" + _threadId + "-" + i;
                bool result = _form._cbClient.Store(StoreMode.Set, item, "{'data':'value'}");
                Console.Out.WriteLine(String.Format("SET '{0}' Got Response: {1}",
                    item, result.ToString()));
                _form.iterationCompleted(stopWatch.ElapsedMilliseconds, result);
            }
            stopWatch.Stop();

            // attempt to cleanup if selected
            if (_cleanup)
            {
                for (int i = 0; i < _iterations; i++)
                {
                    var item = "test-" + _threadId + "-" + i;
                    var result = _form._cbClient.Remove(item);
                    Console.Out.WriteLine(String.Format("REMOVE '{0}' Got Response: {1}",
                        item, result.ToString()));
                }
            }

            _form.finishedThread(this);
        }
    }

v2.1.1:

        public void runThread()
        {
            var stopWatch = new Stopwatch();
            stopWatch.Start();

            for (int i = 0; i < _iterations; i++)
            {
                stopWatch.Restart();
                var bucket = ClusterHelper.GetBucket(_bucketName);
                var item = "test-" + _threadId + "-" + i;
                IOperationResult result;
                result = bucket.Upsert(item, "{'data':'value'}");
                Console.Out.WriteLine(String.Format("UPSERT '{0}' Got Response: {1} with CAS: {2}",
                    item, result.Status.ToString(), result.Cas.ToString()));
                _form.iterationCompleted(stopWatch.ElapsedMilliseconds, result.Success);
            }
            stopWatch.Stop();

            // attempt to cleanup if selected
            if (_cleanup)
            {
                for (int i = 0; i < _iterations; i++)
                {
                    var bucket = ClusterHelper.GetBucket(_bucketName);
                    var item = "test-" + _threadId + "-" + i;
                    var result = bucket.Remove(item);
                    Console.Out.WriteLine(String.Format("DELETE '{0}' Got Response: {1} with CAS: {2}",
                        item, result.Status.ToString(), result.Cas.ToString()));
                }
            }

            _form.finishedThread(this);
        }

Any ideas?

-H

2

@unhuman -

Thanks for posting, would you mind showing the initialization code? I’d like to see the ClientConfiguration, alternatively you could create an NCBC and attach an example project and I can see what is going on.

-Jeff

3

CouchbaseClientsComparison.zip.ico (21.7 KB)
CouchbaseClientsComparison1x.zip.ico (21.5 KB)

Edit: whoops - I whacked the 1x version’s packages.config. Updated.

Also, doesn’t matter, but looks like I used 1.3.10, not 1.3.12, as I stated…

2 files attached. I had to add ico as an extension to the zip files to get them to upload.

one thing to note, the 1x variant uses config from App.config (and thus has some text boxes greyed). The other version will just use what’s placed in the text boxes to generate a config.

LMK if there’s anything else.

I’ve kept things simple, only giving one server in the cluster. Giving the other server results in the same behavior, for both versions.

4

@Unhuman -

bucketConfiguration.PoolConfiguration = new PoolConfiguration
{
    MinSize = Convert.ToInt32(textboxDataSize.Text),
    MaxSize = Convert.ToInt32(textboxDataSize.Text)
};

Your default value in the textBoxDataSize is 2048, so you are creating a connection pool of 2048 connections! Your likely to need no more than 10 max or if you create a massive thread count, make it some % of the total threads.

-Jeff

5

Yeah, I was onto something when I tried using an App.config setup and then the problem went away. Anyway, thanks for looking…

6

Also… Now that this (terrible) tool works. It looks like the new client is 100% slower than the old one. Running 5 threads with 100 repetitions… 33841ms (2.1.1) vs. 16286ms (1.3.10).

Seems to be consistent across # of threads - always about double. Interesting, though, that changing the Min / Max values for # of connections in the pool (both versions) makes very little difference, unless that’s explained by the tiny contents of the messages.

Finally, it appears that as the iterations get larger (1000), the times come together, perhaps the issue is the connection pool setup / management?

7

@unhuman -

At 100 iterations, you are barely warming up the client; most all of the time would be spent during initial setup, creating TCP connections, etc. The threads will likely just cause overhead here as well for the same reasons.

~34s is a long time for the client to warm up, though, any chance you configure logging and see what’s going on? I am guessing that CCCP is failing and it’s defaulting to HTTP streaming for configuration.

-Jeff

8

I’ve added timing on individual requests and it does indeed seem to be warmup. Initial requests take much longer, then drop pretty quickly… It does, however, still seem to be a bit slower… Excluding the GetBucket() time helps, but not nearly enough.

I’ll package up my updates later and post them.

Attached is a log file (zipped with fake .ico extension):
Couchbase211SlowClient.20150527.log.zip.ico (68.8 KB)

-H

9

Attached are my updated projects…

CouchbaseClientsComparison1x.zip.ico (24.3 KB) CouchbaseClientsComparison2x.zip.ico (25.2 KB)

Note the 2x version of this EXCLUDES the time to get the bucket, so in theory, hides the cost of warmup from whats displayed (but the data is logged).

This thus shows that, after warmup, the 2x client is approximately 50% slower.

10

Related: I’ve just allowed adding .zip files so you can post those without the .ico renaming.

1 Like
11

@jmorris Great meeting you at Couchbase Connect! Really enjoyed myself there! Next year, hopefully, in Vegas!

As promised, I’ve updated and attached my tool to support Async (checkbox).

Async seems to be pretty flat, maybe slightly slower, compared to sync requests (and thus still slower than 1.3.x). Please take a look to see if I’m doing anything that’s not “best practice.”

I also added a Warmup Iterations to discount for any warmup latency.

<< Removed bad upload >>

-H

12

@unhuman –

For some reason I cannot download the most recent zip, can you try reposting?

-Jeff

13

I’ve tried multiple times. I’ve tried re-packing the file. It works fine in the preview window, but does not work once the post is saved.

@ingenthr Maybe you can help?

I also noticed that you guys seem to key off the hash of a file, so if I upload the same file (with different names), twice, the original file name shows both times. Makes sense to prevent duplicate uploads, but was interesting. So, uploading aaaa.zip and bbbb.zip shows aaaa.zip twice.

-H

14

That’s interesting. Seems like a bug of some sort. I’ll have it looked into.

I’ve also upped your membership level ever so slightly. Can you see if you see the same behavior with this new level?

15

Created: https://issues.couchbase.com/browse/NCBC-914

16

Sorry for the trouble. Seems like something is wrong. We’ll get this looked into. Please leave the posting for now.

In the interim, maybe it’s best to post an issue and put the attachment there.

17

Created: https://issues.couchbase.com/browse/NCBC-914

Which references:
https://issues.couchbase.com/secure/attachment/24899/CouchbaseClientsComparison2x.zip