After switch from SDK 2.6 to 3.2.0-beta.1, my Node.js backend keep showing segfault. This happens a lot on my dev env (MacOS BigSur), however, with much less occurrence in the docker containers.
The dump looks as below:
PID 93559 received SIGSEGV for address: 0x4
0 segfault-handler.node 0x00000001046bdfa0 _ZL16segfault_handleriP9__siginfoPv + 304
1 libsystem_platform.dylib 0x00007fff20543d7d _sigtramp + 29
2 ??? 0x00001549a4b0a2d9 0x0 + 23406039835353
3 couchbase_impl.node 0x0000000135148583 _ZN17lcb_QUERY_HANDLE_D2Ev + 115
4 couchbase_impl.node 0x000000013514827e _ZN17lcb_QUERY_HANDLE_D0Ev + 14
5 couchbase_impl.node 0x0000000135137ba1 _ZL14timer_callbackisPv + 49
6 node 0x00000001009e88f6 uv__run_timers + 54
7 node 0x00000001009ed167 uv_run + 247
8 node 0x00000001000e6827 _ZN4node16NodeMainInstance3RunEv + 375
9 node 0x0000000100079c56 _ZN4node5StartEiPPc + 294
10 libdyld.dylib 0x00007fff20519f3d start + 1
Does anyone have a hint on how to prevent/fix this?
Hi @sunny_arthrex do you see the same error with 3.2 and Couchbase 7.0 released last week.
And is this error related to any operation that you seek ?
I upgraded my SDK to 3.2.0, my CB server is 6.6. In my dev env I could not reproduce the crash.
However, on the production env (CB server is also 6.6), which is hosted in AWS with containers, the application became stuck in crash loop, with the dump as below:
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(+0x6fa42)[0x7fcef900da42]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(+0x86127)[0x7f63f6865127]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(+0x8f8a6)[0x7f63f686e8a6]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(_ZN3lcb6Server8try_readEP9lcbio_CTXP10rdb_IOROPE+0x3b5)[0x7f63f686c105]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(mcreq_dispatch_response+0x1d74)[0x7f63f68c7b14]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(+0x12d3af)[0x7f63f690c3af]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(_ZN3lcb8clconfig11cccp_updateEPKv10lcb_STATUSS2_mPK11lcb_host_st+0x112)[0x7f63f68eb0d2]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(ZN12CccpProvider6updateEPKcS1+0x29e)[0x7f63f68eae3e]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(_ZN3lcb8clconfig7Confmon19provider_got_configEPNS0_8ProviderEPNS0_10ConfigInfoE+0x19)[0x7f63f68f0929]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(_ZN3lcb8clconfig7Confmon11do_set_nextEPNS0_10ConfigInfoEb+0x3e)[0x7f63f68f048e]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(lcbvb_compare+0x8c)[0x7f63f68b82ec]
/tmp/43afb1c889e8030fee82db2074c78d9a6a0c7ced808b8703872861c65c8642f5/couchbase_impl.node(+0xd65a7)[0x7f63f68b55a7]
/lib/x86_64-linux-gnu/libc.so.6(+0x1576ca)[0x7f63fda7e6ca]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f63fdafa730]
Is CB server 7.0 required for running node.js SDK 3.2.0?
Hi AV25242,
This is to update you that we have resolved this issue.
The root cause of the crash is that we are running the application on Debian 10, which the code built on Debian 10. Now we build code on Debian 9 and changed the container to Debian 9, the crash loop finally stopped.
It seems that node.js SDK has a better support on Debian 9 than Debian 10.
1 Like
Thanks @sunny_arthrex this update is super useful. I have created a ticket internally to further investigate.
Hey @sunny_arthrex ,
I managed to reproduce this and have a fix in the pipeline. It should ship in our next release. Feel free to follow the ticket for more concrete details: Loading...
Cheers, Brett
1 Like
Hi Brett,
Thank you. Looking forward to your next release.
Best, Sunny
Possibly related? SIGSEGV in 3.2.0
Indeed. Looks very similar.
One caveat we noticed is that if you have a build pipeline, since the nature of CB needs a compilation, the pipeline container OS needs to be the same as the deployment container. Otherwise we will see all kind of weird segfault.
1 Like
yes for our container we run the npm ci command inside the dockerfile during build so were not running into any lib issues there. We have other processes using node and the couchbase 3.0 and 3.1 on the same container base image and just noticed a newly deployed service has been randomly going down and coming back up, the only delta is were using the 3.2.0 image for the improved typescript support. After not being able to reproduce the crash on my mac by just running npm start I decided to try running docker compose and can get it to reproduce sporadically.