How to close couchbase connections in multi threaded environment

Python SDK
1

I am interested in how to close a connection to Couchbase.

I am using the current version of the couchbase client in python.

Each write to the database is done by a task in celery so needs a new instance of the couchbase client to do the write.

The only way I have found to close the connection is to set it to None. There does not seem to be any warning in the documentation about doing this even though not doing so caused massive load on my server and huge disk space usage to the point that I was about to give up up on Couchbase.

The workload is one set multi, every 5 seconds with 100 documents. The data size is currently around 700 MB for 500000 documents. Before setting the client to none, the server was using 50GB to store 700MB of documents.

There is no .exit attribute on the client so it is not possible to do the standard pythonic

with Couchbase.connect(bucket=“default”) as cb:
…

Could someone enlighten me as to best practise when many connections must be made and broken? I ask this question mainly for the benefit of others as there are a lot of stories of high disk and cpu usage out there and it would be great to improve documentation in this area.

2

Currently there is no context manager available for Couchbase, and this is because internally constructing a new Couchbase instance is fairly expensive.

However depending on how you’ve configured celery there are some other alternatives:

  • If you're using individual processes, ensure the new Couchbase instance is created after the new process, not before
  • If you're using threads, the Couchbase instance is thread safe (just be sure to set the lockmode=LOCKMODE_WAIT option in the constructor).

If you absolutely must create a new Couchbase instance for each operation, there is an undocumented _close() method which will destroy any underlying network resources created with the current Couchbase object.

Can you give me some more information as to why you’re making a new object each time?

You may also wish to consider a connection pool. I believe there’s an example in the distribution under the examples directory.