Memory leak with Python-Couchbase 2.5.0

Python SDK
1

We have a service written in Python3, connecting to one or more Couchbase clusters.
During normal operations hundreds operations per second are being processed and each of them translates to one or more Couchbase operations.
The service is deployed on Ubuntu 16.04 amd64 instances, under uwsgi.

We ran the same application using:

  • libcouchbase 2.10.3-1
  • python3-couchbase 2.5.0

and with 2.3.3:

  • libcouchbase 2.8.4-1
  • python3-couchbase 2.3.3

With the old versions (2.3.3 and 2.8.4-1 for libcouchbase) the memory allocation is stable over 24h.
With the new versions (2.5.0 and 2.10.3-1 for libcouchbase) the memory grows steadily over 10-12 hours until the application needs to be restarted.
0

We used also tried python3-couchbase 2.5.5 and still, the memory leak is there.

2

Hi, is it possible for you to run Valgrind or another C profiler on the client for a few minutes? This will likely require you to build couchbase-python-client and libcouchbase in debug mode to get useful information: as follows:

  1. Build libcouchbase in debug mode:
    git clone https://github.com/couchbase/libcouchbase.git@2.10.3
    cd libcouchbase
    mkdir build
    cd build
    .../cmake/configure --enable-debug
    make && sudo make install
    cd ..
  2. Build the client
    git clone https://github.com/couchbase/couchbase-python-client.git@2.5.5
    cd couchbase-python-client
    export PYCBC_SAN_OPT_leak=TRUE or export PYCBC_SAN_OPT_address=TRUE or neither (these enable clang/gcc Leak and Address Sanitizer respectively)
    python setup.py build_ext --debug install (preferably in a clean virtualenv or similar)

Then run a profiler such as valgrind or gprof on the Python instance that uses Couchbase Python Client. Or run standalone if you used the leak/address sanitizer options, and provide the output.

Thanks,

Ellis

3

Hi Ellis,
Thanks for the advice.
Running the application using valgrind is a bit harder because we use the following setup:
uwsgi master → python3 workers

We managed to run our application under valgind (excluding uwsgi) and the most promising leaks are.
See bellow.

4

Hi Ellis,

We managed to run our application under valgind (excluding uwsgi) and the most promising leaks are:

==32187== 66 bytes in 11 blocks are possibly lost in loss record 701 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8885B: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:326)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A6954B: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 72 bytes in 12 blocks are possibly lost in loss record 710 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8885B: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:326)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 90 bytes in 10 blocks are possibly lost in loss record 715 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8883C: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:325)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A6954B: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 90 bytes in 15 blocks are possibly lost in loss record 716 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8885B: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:326)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 99 bytes in 11 blocks are possibly lost in loss record 734 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8883C: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:325)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 110 bytes in 11 blocks are possibly lost in loss record 740 of 2,641
==32187== at 0x4C2DE96: malloc (vg_replace_malloc.c:309)
==32187== by 0x50E3489: strdup (strdup.c:42)
==32187== by 0x9D88AC6: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:371)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 117 bytes in 13 blocks are possibly lost in loss record 745 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8883C: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:325)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 130 bytes in 13 blocks are possibly lost in loss record 931 of 2,641
==32187== at 0x4C2DE96: malloc (vg_replace_malloc.c:309)
==32187== by 0x50E3489: strdup (strdup.c:42)
==32187== by 0x9D88AC6: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:371)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 136 bytes in 8 blocks are possibly lost in loss record 935 of 2,641
==32187== at 0x4C2E4B6: operator new(unsigned long) (vg_replace_malloc.c:344)
==32187== by 0x9D2C124: void std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:223)
==32187== by 0xAC46C4B: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::basic_string(char const*, std::allocator const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==32187== by 0x9D887C5: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:319)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 153 bytes in 9 blocks are possibly lost in loss record 940 of 2,641
==32187== at 0x4C2E4B6: operator new(unsigned long) (vg_replace_malloc.c:344)
==32187== by 0x9D2C124: void std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:223)
==32187== by 0xAC46C4B: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::basic_string(char const*, std::allocator const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==32187== by 0x9D887C5: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:319)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 170 bytes in 10 blocks are possibly lost in loss record 956 of 2,641
==32187== at 0x4C2E4B6: operator new(unsigned long) (vg_replace_malloc.c:344)
==32187== by 0x9D2C124: void std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:223)
==32187== by 0xAC46C4B: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::basic_string(char const*, std::allocator const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==32187== by 0x9D887C5: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:319)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A6954B: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 180 bytes in 18 blocks are possibly lost in loss record 960 of 2,641
==32187== at 0x4C2DE96: malloc (vg_replace_malloc.c:309)
==32187== by 0x50E3489: strdup (strdup.c:42)
==32187== by 0x9D88AC6: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:371)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 192 bytes in 12 blocks are possibly lost in loss record 971 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x540B0A: PyEval_EvalCodeEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 260 bytes in 13 blocks are possibly lost in loss record 1,334 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 408 bytes in 17 blocks are possibly lost in loss record 1,379 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88AF0: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:375)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 480 bytes in 10 blocks are possibly lost in loss record 1,541 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8883C: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:325)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A6954B: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 528 bytes in 11 blocks are possibly lost in loss record 1,624 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8885B: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:326)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A6954B: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 528 bytes in 11 blocks are possibly lost in loss record 1,625 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8883C: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:325)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 576 bytes in 12 blocks are possibly lost in loss record 1,710 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x540B0A: PyEval_EvalCodeEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 576 bytes in 12 blocks are possibly lost in loss record 1,711 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8885B: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:326)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 576 bytes in 12 blocks are possibly lost in loss record 1,712 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8883C: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:325)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 624 bytes in 13 blocks are possibly lost in loss record 1,749 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 672 bytes in 7 blocks are possibly lost in loss record 1,795 of 2,641
==32187== at 0x4C2E4B6: operator new(unsigned long) (vg_replace_malloc.c:344)
==32187== by 0x9D87B22: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 720 bytes in 15 blocks are possibly lost in loss record 1,814 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D8885B: lcb::trace::Span::Span(lcbtrace_TRACER*, char const*, unsigned long, lcbtrace_REF_TYPE, lcb::trace::Span*) (span.cc:326)
==32187== by 0x9D87B49: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 816 bytes in 17 blocks are possibly lost in loss record 1,910 of 2,641
==32187== at 0x4C2FFAC: calloc (vg_replace_malloc.c:762)
==32187== by 0x9D88A9B: lcb::trace::Span::add_tag(char const*, int, char const*, unsigned long) (span.cc:367)
==32187== by 0x9D88A6D: lcb::trace::Span::add_tag(char const*, int, char const*) (span.cc:362)
==32187== by 0x9D87F54: lcbtrace_span_add_tag_str (span.cc:80)
==32187== by 0x9A694CB: pycbc_Context_init (ext.c:1420)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187== 864 bytes in 9 blocks are possibly lost in loss record 1,925 of 2,641
==32187== at 0x4C2E4B6: operator new(unsigned long) (vg_replace_malloc.c:344)
==32187== by 0x9D87B22: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A69467: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A77033: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x540B0A: PyEval_EvalCodeEx (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 960 bytes in 10 blocks are possibly lost in loss record 2,084 of 2,641
==32187== at 0x4C2E4B6: operator new(unsigned long) (vg_replace_malloc.c:344)
==32187== by 0x9D87B22: lcbtrace_span_start (tracer.cc:38)
==32187== by 0x9A6954B: pycbc_Context_init (ext.c:1386)
==32187== by 0x9A698BF: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A78E13: pycbc_Bucket_lookup_in (get.c:408)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x540B0A: PyEval_EvalCodeEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x4EC3F6: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5C20E6: PyObject_Call (in /home/tl/memleak/venv/bin/python3)
==32187==
==32187==
==32187== 1,037,727 (47,952 direct, 989,775 indirect) bytes in 999 blocks are definitely lost in loss record 2,640 of 2,641
==32187== at 0x4C2DE96: malloc (vg_replace_malloc.c:309)
==32187== by 0x9A6942B: pycbc_Context_init (ext.c:1378)
==32187== by 0x9A69840: pycbc_Tracer_start_span (ext.c:1518)
==32187== by 0x9A69948: pycbc_explicit_named_setup (ext.c:1276)
==32187== by 0x9A7809B: handle_single_lookup (get.c:359)
==32187== by 0x9A77059: pycbc_oputil_iter_multi (oputil.c:401)
==32187== by 0x9A78D04: sdlookup_common (get.c:388)
==32187== by 0x4EA136: PyCFunction_Call (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53D352: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x5401EE: ??? (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x53BC92: PyEval_EvalFrameEx (in /home/tl/memleak/venv/bin/python3)
==32187== by 0x540B0A: PyEval_EvalCodeEx (in /home/tl/memleak/venv/bin/python3)

5

OK, this is very useful, thanks. I’ll take a look ASAP.

6

This is coming from the tracing section of the code. If you are not using it, for the meantime you can disable tracing by passing “enable_tracing=off” in the connection string. I will work on a fix ASAP.

Thanks,

Ellis

7

Hi Elllis,

So is it a confirmed leak, or the possible leaks reported by valgrind are actually due to the “testing mode” we used ?
Thank you

8

Hello.
I observe similar memory leak with C SDK 2.10.7 + Python SDK 2.5.12 which are currently the latest.
“enable_tracing=off” workaround works for me. Just curious if anybody found a solution?

9

Hi, I didn’t manage to track down anything obvious in the Python client itself (in fact, I found a couple of places where we released tracing related memory too early, now fixed). The memory allocated appears to be allocated in libcouchbase itself, but it is possible LCB isn’t deallocating it because a tracing span isn’t being disposed of by the Python bindings. Nothing along the latter lines has been found, though. I’ll raise a ticket to give this some more attention. We were hoping to rework the tracing code in SDK3 but that’s not yet roadmapped.

10

@Serhiy.Potishuk Are you able to produce Valgrind logs, or to run with clang sanitizer by following the steps described above?

11

( JIRA issue is here: https://issues.couchbase.com/browse/PYCBC-995 ) - thanks

12

With python sdk version [3.2.4], this problem is fixed.

Just upgrade sdk to version 3.2.4 with python3 -m pip install --upgrade couchbase

1 Like
13

@berkay - That is good to hear. Thank you for the feedback!