Indexer (5.0.1) crashing

Couchbase Server
1

The indexer in one of the nodes keeps crashing with the following message. The system is running in UAT and is unlikely to be in heavy load. However these servers are newly setup and are replicating from another clusters (through XDCR). There are 4 millions documents with ~40 indexes. Each indexes is built with 1 replication

E.g.
runtime: failed to create new OS thread (have 19 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

Top taken when indexer was running (not at the moment when it crashes though)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31944 couchbas 20 0 3682m 1.3g 3436 S 255.5 1.0 6386:11 beam.smp
62997 couchbas 20 0 4948m 1.7g 10m S 142.1 1.4 2:48.34 indexer
49196 couchbas 20 0 597m 152m 6244 S 41.5 0.1 829:09.67 goxdcr
32296 couchbas 20 0 49.5g 36g 10m S 38.2 29.1 1318:45 memcached
49271 couchbas 20 0 4272m 92m 10m S 4.9 0.1 44:16.41 cbq-engine

2

Looks to be related to the environment as golang cannot spawn new OS thread. You may want to check the ulimit on the system or if the system is too overloaded.

3

As you can see, it is definitely not due to the system being overloaded. I had also added the following to limits.conf, but it still doesn’t work

couchbase hard nofile 65536
couchbase hard core unlimited
couchbase hard nproc unlimited
couchbase hard memlock unlimited

4

@wai.kwang.mak would be great if you can share cbcollectinfo.