Indexer 100% cpu at random different nodes

SQL++
1

Hi we have a cluster of 3 nodes at aws like this:
Couchbase Server Community Edition 5.1.1 build 5723.
M5.xlarge 4cpu 16gb
Ubuntu 16.04
Data Service 3000MB
Index Service 6000MB
Search Service 2000MB
Indexer Threads 2
Max Rollback Points 5

I am getting random machines with indexer service at 100%, they usually get stock there. I them go and kill -9 the indexer and some times that looks like solve the issue for a day or less time.

I haven’t found anything meaningful on the index.log or error.log or query.log .

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8002 couchba+ 20 0 10.929g 3.283g 10232 R 281.7 21.6 278:28.92 indexer
1827 couchba+ 20 0 3069752 1.590g 5364 S 4.7 10.5 210:22.26 projector
1862 couchba+ 20 0 3609660 1.527g 3060 S 23.7 10.0 518:56.59 memcached
1844 couchba+ 20 0 1808324 992.7m 9956 R 39.0 6.4 970:11.82 cbq-engine
1858 couchba+ 20 0 19.344g 881004 791828 S 1.0 5.5 83:56.27 cbft
1716 couchba+ 20 0 2141104 700552 3324 S 18.3 4.4 1249:49 beam.smp
1752 couchba+ 20 0 4132296 101764 12604 S 0.7 0.6 22:59.07 beam.smp
1348 couchba+ 20 0 1257144 23196 3320 S 0.7 0.1 20:43.20 beam.smp

Here is the query and index logs: https://mxhero.app.box.com/s/sts0ygxxwu68ae7r3yet2xpnszslc8pi

2

Found this article and started trying to figure out what was happening, after improving some queries and changed some index the issue looks solve. So I guess this was a performance issue related to index in one of the nodes.

3

@mmarmol Glad to know the article helped you. Please let us know if you face any issue with modified index definition and queries.

1 Like