Maximum connection to port 11210

Couchbase Server
1

Hi,

Currently I have issues with my query node, it can not connect to data node due to max connections in port 11210 (30k++)
I’m using couchbase 6.0.1

  1. How to increase that max conn setting?
  2. What’s the possible cause of connection leak for this port?
  3. How to recover / clean the connection?

Thanks!

2

You may want to check the memcached.log to see what clients are connecting. There can be multiple causes, but one possible one is not reusing the object representing the connection in the SDK (the Bucket or Collection, depending on the version of the SDK).

Another thing you could do is look at netstat with netstat -a | grep 11210 on UNIX-like machines. See what hosts the connections are coming from. If it’s other cluster nodes, then it’s something internal.

You can restart the server to drop the connections. You can also probably find a tcpkill like utility, but the other side may just reconnect.

I wouldn’t recommend increasing the number of connections until you know the cause.

3

Hi @ingenthr,

Thank you for your response.
I;ve tried doing netstat, and saw so many 11210 from internal node .

Yes, I’ve managed to solve it by restart the connection, but still don’t know the culprit.

From my findings, few days ago there are incoming XDCR from another cluster that flow data to my cluster.

From connection statistic, I saw total connection keep increasing for around 2 days until it reach the max connections

Is it possible if that XDCR is the culprit?

4

Hi @ingenthr,

I saw from my memcache log :
2020-07-06T05:02:00.022810+00:00 INFO 14348: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57316 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.022970+00:00 INFO 14349: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57314 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.023080+00:00 INFO 14350: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57322 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.023276+00:00 INFO 14351: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57318 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.023452+00:00 INFO 14353: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57324 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.023565+00:00 INFO 14352: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57320 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.023901+00:00 INFO 14354: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57326 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.024056+00:00 INFO 14355: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57328 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.024334+00:00 INFO 14356: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57330 - 10.129.13.186:11210 (not authenticated) ]
2020-07-06T05:02:00.024422+00:00 INFO 14357: HELO [GoMemcached] XATTR, JSON [ 10.128.16.1:57332 - 10.129.13.186:11210 (not authenticated) ]
2

2020-07-06T05:48:19.518651+00:00 INFO 13769: HELO [{“a”:“couchbase-java-client/2.7.9 (git: 2.7.9, core: 1.7.9) (Linux/3.10.0-1062.4.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_242-b08)”,“i”:“38FD822FE0DC81FB/FFFFFFFFCECB69A9”}] TCP nodelay, XATTR, XERROR, Select bucket, Snappy, Tracing [ 10.128.4.1:42675 - 10.128.9.33:11210 (not authenticated) ]
2020-07-06T05:48:20.164433+00:00 INFO 13771: HELO [{“a”:“couchbase-java-client/2.7.9 (git: 2.7.9, core: 1.7.9) (Linux/3.10.0-1062.4.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_242-b08)”,“i”:“38FD822FE0DC81FB/000000004A1C9316”}] TCP nodelay, XATTR, XERROR, Select bucket, Snappy, Tracing [ 10.128.4.1:42691 - 10.128.9.33:11210 (not authenticated) ]
2020-07-06T05:48:26.620751+00:00 INFO 13771: HELO [{“a”:“couchbase-java-client/2.7.9 (git: 2.7.9, core: 1.7.9) (Linux/3.10.0-1062.4.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_242-b08)”,“i”:“38FD822FE0DC81FB/FFFFFFFFD5134C12”}] TCP nodelay, XATTR, XERROR, Select bucket, Snappy, Tracing [ 10.128.4.1:42722 - 10.128.9.33:11210 (not authenticated) ]
2020-07-06T05:48:33.717933+00:00 INFO 13772: HELO [{“a”:“couchbase-java-client/2.7.9 (git: 2.7.9, core: 1.7.9) (Linux/3.10.0-1062.4.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_242-b08)”,“i”:“38FD822FE0DC81FB/000000007741BE6A”}] TCP nodelay, XATTR, XERROR, Select bucket, Snappy, Tracing [ 10.128.4.1:42727 - 10.128.9.33:11210 (not authenticated) ]

FYI I’ve just upgraded my java SDK as well from 2.6.1 to 2.7.13

5

That appears to be an internal connection in the cluster, as that’s what gomemcached is used for.

I’d recommend looking with netstat and/or lsof to find the process on the alternate machine using that port. I don’t know of any known issues that would cause this, but maybe @keshav_m or @jeelan.poola does. Or search issues.couchbase.com.

Of course, if you have an Enterprise subscription, you can get Couchbase Support to own problem determination.