Correct way to use Couchbase connection in Threads

Python SDK
1

I try to connect to a Bucket in thread using LOCKMODE_WAIT and 60 seconds operation timeout. While it can connect and read/write data just fine, the connection doesn’t seems to be closed.

Connection is created using

self.cache = Bucket('couchbase://{host}/{bucket}?operation_timeout=60'.format(
                                host=setting.CONFIG_COUCHBASE,
                                bucket=bucket),
                                lockmode=LOCKMODE_WAIT)

See the following graph, the flat part is when I don’t use connection in thread. During this time, we have rather stable incoming traffic as well.

Screen Shot 2016-11-10 at 5.33.31 PM.png1568×758 91.8 KB

The scenario for this is that I’m using Bucket connection on REST API. For every request comes in, I use one Bucket connection on main thread for caching. Then another thread creating a separate connection to the same bucket for caching different data. So every request should only create 2 instances of Bucket connection in total.

As number of connections grows, so does memory of my API. What’s the best way to use Couchbase on threaded environment? Should I not do that at all?

I did read http://pythonhosted.org/couchbase/ document on Threads but I don’t totally understand the explaination.

Any idea what I should take a look at?

2

It’s better to use a framework that supports an asynchronous/non-blocking or couroutined model. Anything supporting gevent will do.

In general, the connection should be closed to the server once the actual Bucket object goes out of scope. The fact that the connection remains open means you have something keeping the Bucket alive.

The bottom line is that I wouldn’t recommend using threads where scalability and performance is concerned, but our client does support threads if you absolutely must use them.