Hi
Im using AWS lambda and the below configuration to try and connect to couchbase server:
Node SDK - 2.6.5 (with external C bindings for SSL)
libcouchbase.so.2.0.65
couchbases://localhost?certpath=/var/tmp/tramas/lib/couchbase-cert.pem
I get the following error:node: /builddir/build/BUILD/libcouchbase-2.10.3/src/ssl/ssl_c.c:419: Cssl_dtor: Assertion `((&cs->writes)->last == ((void *)0))' failed.
The entire stack is:
0ms [I05be4033] {1} [INFO] (instance - L:469) Version=2.10.3, Changeset=b387722192f0bc49b0a846859610493e2f8129cd
0ms [I05be4033] {1} [INFO] (instance - L:470) Effective connection string: couchbases://127.0.0.1/bucket?certpath=%2Fvar%2Ftask%2Flib%2Fcouchbase-cert.pem&enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.5%20(node%2F8.10.0%3B%20v8%2F6.2.414.50%3B%20ssl%2F1.0.2r). Bucket=bucket
0ms [I05be4033] {1} [DEBUG] (SSL - L:285) Load verify locations from "/var/task/lib/couchbase-cert.pem"
0ms [I05be4033] {1} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
0ms [I05be4033] {1} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
0ms [I05be4033] {1} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.5 (node/8.10.0; v8/6.2.414.50; ssl/1.0.2r)
5ms [I05be4033] {1} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
5ms [I05be4033] {1} [DEBUG] (instance - L:83) Adding host 127.0.0.1:18091 to initial HTTP bootstrap list
5ms [I05be4033] {1} [DEBUG] (instance - L:83) Adding host 127.0.0.1:11207 to initial CCCP bootstrap list
5ms [I05be4033] {1} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
20ms [I05be4033] {1} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
20ms [I05be4033] {1} [DEBUG] (confmon - L:87) Preparing providers (this may be called multiple times)
20ms [I05be4033] {1} [DEBUG] (confmon - L:94) Provider CCCP is ENABLED
20ms [I05be4033] {1} [DEBUG] (confmon - L:94) Provider HTTP is ENABLED
20ms [I05be4033] {1} [TRACE] (confmon - L:277) Refreshing current cluster map
20ms [I05be4033] {1} [TRACE] (confmon - L:264) Attempting to retrieve cluster map via CCCP
20ms [I05be4033] {1} [INFO] (cccp - L:151) Requesting connection to node 127.0.0.1:11207 for CCCP configuration
20ms [I05be4033] {1} [DEBUG] (lcbio_mgr - L:448) <127.0.0.1:11207> (HE=0x55cd65e3f8c0) Creating new connection because none are available in the pool
20ms [I05be4033] {1} [TRACE] (lcbio_mgr - L:359) <127.0.0.1:11207> (HE=0x55cd65e3f8c0) New pool entry: I=0x55cd65f65020
20ms [I05be4033] {1} [INFO] (connection - L:474) <127.0.0.1:11207> (SOCK=0000000000e7f303) Starting. Timeout=2000000us
29ms [I05be4033] {1} [TRACE] (connection - L:367) <127.0.0.1:11207> (SOCK=0000000000e7f303) Received completion handler. Status=0. errno=0 [Success]
29ms [I05be4033] {1} [INFO] (connection - L:147) <127.0.0.1:11207> (SOCK=0000000000e7f303) Connected established
29ms [I05be4033] {1} [WARN] (lcbio - L:112) Translating errno=0, lcb=0x2d to NETWORK_ERROR
29ms [I05be4033] {1} [INFO] (connection - L:108) <127.0.0.1:11207> (SOCK=0000000000e7f303) Couldn't set TCP_NODELAY
29ms [I05be4033] {1} [DEBUG] (connection - L:106) <127.0.0.1:11207> (SOCK=0000000000e7f303) Successfully set TCP_KEEPALIVE
30ms [I05be4033] {1} [DEBUG] (lcbio_mgr - L:318) <127.0.0.1:11207> (HE=0x55cd65e3f8c0) Received result for I=0x55cd65f65020,C=(nil); E=0x0
30ms [I05be4033] {1} [DEBUG] (lcbio_mgr - L:278) <127.0.0.1:11207> (HE=0x55cd65e3f8c0) Assigning R=0x55cd65e452e0 SOCKET=0x55cd65f5b620
30ms [I05be4033] {1} [DEBUG] (ioctx - L:101) <127.0.0.1:11207> (CTX=0x55cd65f66290,unknown) Pairing with SOCK=0000000000e7f303
30ms [I05be4033] {1} [DEBUG] (negotiation - L:440) <127.0.0.1:11207> (CTX=0x55cd65f66290,sasl,SASLREQ=0x55cd65f46b50) HELO identificator: {"a":"libcouchbase/2.10.3 (Linux-4.19.9-300.fc29.x86_64; x86_64; GNU 4.4.7) couchnode/2.6.5 (node/8.10.0; v8/6.2.414.50; ssl/1.0.2r)","i":"0000000005be4033/0000000000e7f303"}, features: 0x02 (TLS), 0x06 (XATTR), 0x0b (JSON), 0x08 (Select bucket), 0x07 (XERRO
30ms [I05be4033] {1} [TRACE] (SSL - L:235) sock=0x55cd65f5b620: ST(0x10). before/connect initialization. R(0x1)U
37ms [I05be4033] {1} [WARN] (lcbio - L:112) Translating errno=0, lcb=0x2d to NETWORK_ERROR
37ms [I05be4033] {1} [ERROR] (negotiation - L:154) <127.0.0.1:11207> (CTX=0x55cd65f66290,sasl,SASLREQ=0x55cd65f46b50) Error: 0x10, IO Error
37ms [I05be4033] {1} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
37ms [I05be4033] {1} [INFO] (confmon - L:185) Provider 'CCCP' failed
37ms [I05be4033] {1} [DEBUG] (confmon - L:226) Will try next provider in 0us
37ms [I05be4033] {1} [DEBUG] (ioctx - L:151) <127.0.0.1:11207> (CTX=0x55cd65f66290,sasl) Destroying context. Pending Writes=1, Entered=true, Socket Refcount=1
37ms [I05be4033] {1} [TRACE] (confmon - L:264) Attempting to retrieve cluster map via HTTP
node: /builddir/build/BUILD/libcouchbase-2.10.3/src/ssl/ssl_c.c:419: Cssl_dtor: Assertion `((&cs->writes)->last == ((void *)0))' failed.
When I run from an EC2 it works successfully:
0ms [I6eb06fa7] {29162} [INFO] (instance - L:469) Version=2.9.5-njs, Changeset=2bf296af07b40e1b0163f900c4f8bf388ae4e8a3
0ms [I6eb06fa7] {29162} [INFO] (instance - L:470) Effective connection string: couchbases://127.0.0.1/bucket?certpath=%2Fvar%2Ftmp%2Fmyapp%2Flib%2Fcouchbase-cert.pem&enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.5%20(node%2F8.10.0%3B%20v8%2F6.2.414.50%3B%20ssl%2F1.0.2n). Bucket=svc-transition-mgt-1
0ms [I6eb06fa7] {29162} [DEBUG] (SSL - L:285) Load verify locations from "(null)"
0ms [I6eb06fa7] {29162} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
0ms [I6eb06fa7] {29162} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
0ms [I6eb06fa7] {29162} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.5 (node/8.10.0; v8/6.2.414.50; ssl/1.0.2n)
0ms [I6eb06fa7] {29162} [INFO] (instance - L:146) DNS SRV lookup failed: The requested feature is not supported by the client, either because of settings in the configured instance, or because of options disabled at the time the library was compiled. Ignore this if not relying on DNS SRV records
0ms [I6eb06fa7] {29162} [DEBUG] (instance - L:83) Adding host 127.0.0.1:18091 to initial HTTP bootstrap list
0ms [I6eb06fa7] {29162} [DEBUG] (instance - L:83) Adding host 127.0.0.1:11207 to initial CCCP bootstrap list
0ms [I6eb06fa7] {29162} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
4ms [I6eb06fa7] {29162} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
4ms [I6eb06fa7] {29162} [DEBUG] (confmon - L:87) Preparing providers (this may be called multiple times)
4ms [I6eb06fa7] {29162} [DEBUG] (confmon - L:94) Provider CCCP is ENABLED
4ms [I6eb06fa7] {29162} [DEBUG] (confmon - L:94) Provider HTTP is ENABLED
4ms [I6eb06fa7] {29162} [TRACE] (confmon - L:277) Refreshing current cluster map
4ms [I6eb06fa7] {29162} [TRACE] (confmon - L:264) Attempting to retrieve cluster map via CCCP
4ms [I6eb06fa7] {29162} [INFO] (cccp - L:151) Requesting connection to node 127.0.0.1:11207 for CCCP configuration
4ms [I6eb06fa7] {29162} [DEBUG] (lcbio_mgr - L:448) <127.0.0.1:11207> (HE=0x3b13250) Creating new connection because none are available in the pool
4ms [I6eb06fa7] {29162} [TRACE] (lcbio_mgr - L:359) <127.0.0.1:11207> (HE=0x3b13250) New pool entry: I=0x39b23e0
4ms [I6eb06fa7] {29162} [INFO] (connection - L:474) <127.0.0.1:11207> (SOCK=00000000450f57c9) Starting. Timeout=2000000us
14ms [I6eb06fa7] {29162} [TRACE] (connection - L:367) <127.0.0.1:11207> (SOCK=00000000450f57c9) Received completion handler. Status=0. errno=0 [Success]
14ms [I6eb06fa7] {29162} [INFO] (connection - L:147) <127.0.0.1:11207> (SOCK=00000000450f57c9) Connected established
14ms [I6eb06fa7] {29162} [WARN] (lcbio - L:112) Translating errno=0, lcb=0x2d to NETWORK_ERROR
14ms [I6eb06fa7] {29162} [INFO] (connection - L:108) <127.0.0.1:11207> (SOCK=00000000450f57c9) Couldn't set TCP_NODELAY
14ms [I6eb06fa7] {29162} [DEBUG] (connection - L:106) <127.0.0.1:11207> (SOCK=00000000450f57c9) Successfully set TCP_KEEPALIVE
14ms [I6eb06fa7] {29162} [DEBUG] (lcbio_mgr - L:318) <127.0.0.1:11207> (HE=0x3b13250) Received result for I=0x39b23e0,C=(nil); E=0x0
14ms [I6eb06fa7] {29162} [DEBUG] (lcbio_mgr - L:278) <127.0.0.1:11207> (HE=0x3b13250) Assigning R=0x39b2860 SOCKET=0x3b19c60
14ms [I6eb06fa7] {29162} [DEBUG] (ioctx - L:101) <127.0.0.1:11207> (CTX=0x3b02620,unknown) Pairing with SOCK=00000000450f57c9
14ms [I6eb06fa7] {29162} [DEBUG] (negotiation - L:440) <127.0.0.1:11207> (CTX=0x3b02620,sasl,SASLREQ=0x3b21b20) HELO identificator: {"a":"libcouchbase/2.9.5-njs couchnode/2.6.5 (node/8.10.0; v8/6.2.414.50; ssl/1.0.2n)","i":"000000006eb06fa7/00000000450f57c9"}, features: 0x02 (TLS), 0x06 (XATTR), 0x0b (JSON), 0x08 (Select bucket), 0x07 (XERROR), 0x03 (TCP nodelay), 0x0a (Snappy), 0x0f (Tracing)
14ms [I6eb06fa7] {29162} [TRACE] (SSL - L:235) sock=0x3b19c60: ST(0x10). before/connect initialization. R(0x1)U
21ms [I6eb06fa7] {29162} [WARN] (lcbio - L:112) Translating errno=0, lcb=0x2d to NETWORK_ERROR
21ms [I6eb06fa7] {29162} [ERROR] (negotiation - L:154) <127.0.0.1:11207> (CTX=0x3b02620,sasl,SASLREQ=0x3b21b20) Error: 0x10, IO Error
21ms [I6eb06fa7] {29162} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
21ms [I6eb06fa7] {29162} [INFO] (confmon - L:185) Provider 'CCCP' failed
21ms [I6eb06fa7] {29162} [DEBUG] (confmon - L:226) Will try next provider in 0us
21ms [I6eb06fa7] {29162} [DEBUG] (ioctx - L:151) <127.0.0.1:11207> (CTX=0x3b02620,sasl) Destroying context. Pending Writes=1, Entered=true, Socket Refcount=1
22ms [I6eb06fa7] {29162} [TRACE] (confmon - L:264) Attempting to retrieve cluster map via HTTP
22ms [I6eb06fa7] {29162} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x3b15260
22ms [I6eb06fa7] {29162} [INFO] (connection - L:474) <127.0.0.1:18091> (SOCK=00000000383c0a01) Starting. Timeout=2000000us
29ms [I6eb06fa7] {29162} [TRACE] (connection - L:367) <127.0.0.1:18091> (SOCK=00000000383c0a01) Received completion handler. Status=0. errno=0 [Success]
30ms [I6eb06fa7] {29162} [INFO] (connection - L:147) <127.0.0.1:18091> (SOCK=00000000383c0a01) Connected established
30ms [I6eb06fa7] {29162} [WARN] (lcbio - L:112) Translating errno=0, lcb=0x2d to NETWORK_ERROR
30ms [I6eb06fa7] {29162} [INFO] (connection - L:108) <127.0.0.1:18091> (SOCK=00000000383c0a01) Couldn't set TCP_NODELAY
30ms [I6eb06fa7] {29162} [DEBUG] (connection - L:106) <127.0.0.1:18091> (SOCK=00000000383c0a01) Successfully set TCP_KEEPALIVE
30ms [I6eb06fa7] {29162} [DEBUG] (htconfig - L:306) Successfuly connected to REST API 127.0.0.1:18091
30ms [I6eb06fa7] {29162} [DEBUG] (ioctx - L:101) <127.0.0.1:18091> (CTX=0x3b29450,unknown) Pairing with SOCK=00000000383c0a01
30ms [I6eb06fa7] {29162} [TRACE] (SSL - L:235) sock=0x3b19c60: ST(0x10). before/connect initialization. R(0x1)U
74ms [I6eb06fa7] {29162} [TRACE] (SSL - L:235) sock=0x3b19c60: ST(0x20). SSL negotiation finished successfully. R(0x1)U
74ms [I6eb06fa7] {29162} [DEBUG] (SSL - L:239) sock=0x3b19c60. Using SSL version TLSv1.2. Cipher=<cipher>
89ms [I6eb06fa7] {29162} [TRACE] (htconfig - L:210) <127.0.0.1:18091> (CTX=0x3b29450,bc_http) Received 395 bytes on HTTP stream
91ms [I6eb06fa7] {29162} [TRACE] (htconfig - L:210) <127.0.0.1:18091> (CTX=0x3b29450,bc_http) Received 8565 bytes on HTTP stream
92ms [I6eb06fa7] {29162} [TRACE] (htconfig - L:210) <127.0.0.1:18091> (CTX=0x3b29450,bc_http) Received 9 bytes on HTTP stream
92ms [I6eb06fa7] {29162} [DEBUG] (htconfig - L:225) <127.0.0.1:18091> (CTX=0x3b29450,bc_http) Generation 0 -> 1
92ms [I6eb06fa7] {29162} [INFO] (confmon - L:160) Setting new configuration. Received via HTTP
92ms [I6eb06fa7] {29162} [INFO] (bootstrap - L:90) Selected network configuration: "default"
avsej
July 30, 2019, 6:43pm
3
@Evan_Karopoulos what is the OpenSSL version? Does it work on EC2 when you install the same version of node.js couchbase library? I can see that you are using different versions.
avsej
July 30, 2019, 7:16pm
4
It seems like it loads libcouchbase on Lambda. Could you make sure you don’t install libcouchbase as system library there. Couchbase SDK for node.js includes libcouchbase, so you don’t need to install it separately.
Notice that on EC2 you have Version=2.9.5-njs, while on Lambda it says Version=2.10.3
You’re absolutely right. By chance I forgot to use the external C SDK when I did an NPM install on the EC@ which meant its libcouchbase version was 2.9.5-njs. I tried doing the same on the Lambda and it works fine now.
This means even the standard Node.JS SDK has SSL capability which is great to know.
The working set up on my Lambda now is:
Node v8.10
Node Couchbase SDK version 2.6.5
Lounge version 0.26.0