lcb_create as a Singleton instance

C SDK
1

Hey there,

We are using Couchbase cluster for a high-load ad-serving system.

thousands of HTTP requests are coming to our load balancer in a second.
Every HTTP request is handled by one of our application server (which is web server with couchbase C client)

We have written a C code that loaded into our HTTP web server, and every request is handled by the Web server, being filtered with the relevant parameters and headers, and then it will go with the Couchbase C client to the cluster to get a specific documnet and to increase it’s counter document.

For every single HTTP request we are using “lcb_create” to create the instance and then “lcb_connect” if the instance was created with no errors.

I’m experiencing an hectic problem with executing this code thousands of times in a second.
every ~400-500 requests we are getting a connection error:

“ERROR: Connection failure (0x18), Could not connect to server within allotted time”

seems like the couchbase cluster can not handle this amount of connections per second.
This is the current code we are using for connecting the cluster:

err = lcb_create(&instance, &create_options);
if (err != LCB_SUCCESS) {
syslog(LOG_DEBUG,“Failed to create libcouchbase instance: %s\n”, lcb_strerror(NULL, err));
return 1;
}
(void)lcb_set_error_callback(instance, error_callback);
/* Initiate the connect sequence in libcouchbase */
if ((err = lcb_connect(instance)) != LCB_SUCCESS) {
syslog(LOG_DEBUG,“Failed to initiate connect: %s\n”, lcb_strerror(NULL, err));
lcb_destroy(instance);
return 1;
}

As per my understanding, “lcb_create” and “lcb_connect” are two methods that require large amount of resources from the cluster , and therefore the couchbase is having diffucults to respond, and thats why we are getting this connections errors.
Maybe we should somehow gather our connections into one instance and do not execute “lcb_create” and “lcb_connect” for every single request.

Is there anyway to implement it in C code ?
Is it possible to check if there is already an instance that was created, and to use it instead of creating an instance every time ?

Maybe “lcb_create_io_ops” can help here ?

  • Just to clarify
    the cluster is not cpu-loaded. the servers are fine in terms of IOPS\ CPU RAM and SWAP.

we are really frustrated,
we would like to hear any opinion from you.

Thank you very much.

2

Hey,

This is the very common problem of couchbase . In my company also we are tired of this issue. We seek information from the couchbase team (US) but they suggested to increase the servers node & hardware capacity . We also got the same problem with JAVA & PHP client. Still we are searching for some new things.If your company is using Couchbase then please share the idea so that I can analyze and share some new things.

Sashi
Mumbai

3

Hi

Obviously you shouldn’t create new handle each time and connect it during the each HTTP request to your application.

  • lcb_create just allocates and initialize memory for structs
  • lcb_connect is doing actual connection to the cluster and fetch cluster config and topology using HTTP protocol
  • next operation like lcb_increment will also establish new connection to the data node (only first call)

I’ve done something similar what you are describing. You probably heard about nginx webserver, and I’ve implemented module for that, you can use it as a distributed cache, or expose couchbase interface as HTTP.

Here is the home page of this experiment
http://labs.couchbase.com/couchbase-nginx-module/

Back to your question, usually it should be enough to create and connect single lcb_t object, and use it for the future operations. If your application is using threads and shared memory, you should implement a queue between IO thread (which will own lcb_t and all callbacks) and the rest application.

4

Hey avsej,
First of all i would like to thank you for your comment.

anyway, I also think that this is the solution, to manage our connections via connection pool of lcb_connect instances.

we have managed to use a single lcb_create function but its still not helping much.

we need to know how to iumplement a pool that manage the lcb_connect funtion.

do you have any examples of how to implement it ? (we are using C code …)

Thank you again

Amiram.

5

The suggestions I have are in line what has been said before:
go for connection pooling, or queueing to one pre-existing connection, or both. I am going for the latter.

In reality, one often needs to talk from a synchronous environment (the app) to an asynchronous environment (libcouchbase). Some kind of queue or buffer is needed in those cases, if you go for pooling or not.

All of that is not really easy to set up in C, and I think it would be a lot easier for starters if the guys from couchbase came up with a simple but complete and realistic example. The async examples provided by couchbase are too far off from realistic use cases, and the perl interface and the nginx interface that are mentioned on several occasions are starters, but they are too impacted by the complicated environment they live in (especially the nginx module, that one is fairly opaque).
From my end, I set up a functioning queueing mechanism for a multithreaded environment in about 2 weeks time (don’t forget the needed timeouts and an exponential back-off mechanism), but it is not really ready for production yet as the finer details of the event loop are still missing, so sharing with you would be criminal :wink:

6

Depending on your application you could either use a singleton or a connection pool. One thing to bear in mind is that libcouchbase don’t use ANY locking internally, so you as a user have to ensure that you use it in a safe way in your application. The good news is that libcouchbase don’t use any global values, so you are safe as long as you don’t use the same lcb_t from multiple threads at the same time.

If your application have many threads that once in a while needs to access the Couchbase server a resource pool may be what you want. All you need to do is to create a number of instances and store them in a “pool” somewhere. Now whenever you need to perform an operation you allocate a preconfigured instance from the pool to preform your operation before you return this to the pool.

I wrote an example of a connection pool yesterday that you may use to pick some ideas from: