High CPU and RAM usage by 'cbq-engine'

Couchbase Server
1

I am using Couchbase server with Sync Gateway.

I have a cluster with 3 nodes.
Mobile devices are adding data to couchbase server through Sync_gateway.
For few day it went smoothly. Data got accumulated in CB server.
It was around 15 million records, out of that 8 million records are not useful, so I have removed those records using N1QL delete query.

After this delete operation,
The ‘cbq-engine’ process is always showing high CPU usage and high memory usage.
The memory usage by ‘cbq-engine’ keeps on growing and it takes almost entire free memory, at this point this process is terminated by OS (shows ‘defunct’ if observed using ‘top’ command in Linux) then the ‘cbq-engine’ gets restarted.
This cycle continues and at some point that server node goes down.

Also it is observed that ‘Operation per second’ are very high, more that 12 K per second when there is not that much activity (insert recs through Sync_gateway or N1QL query etc.).

Is this problem is related to deleting records using ‘N1QL’ query ?
What can be cause of this and how to remove it ?
I have also tried by drop and create the GSI indexes. But it did not helped.

2

Hi @mdrdhaygude
I am getting the same issue. Can you please tell me if you were able to resolve this issue and how?

3

@bhuvan.aggarwal84 sorry that you are running into a problem. Can you please state what is the exact issue you are running into along with log files?
What indexes you have?
What is the query you ran for delete?
Cb_collect logs

4

I am using couchbase 5.0.1

Couchbase internal processes like beam.smp, indexer and cbq-engine are taking a lot of CPU. Can you please tell me how to tune these processes so that they take less CPU

ss1

ss2

5

@bhuvan.aggarwal84 Can you please try to run the following commands to see what’s being run?

“select * from system:active_reqeuests”
and
“select * from system:completed_requests”

6

Here are the results from the queries. Please check

select * from system:active_requests
and
select * from system:completed_requests

Active and completed requests.zip (3.1 KB)

7

I don’t see much expensive activity on the query engine.
When you have your query workbench open, it runs INFER every few mins.
You can avoid this by closing your workbench when it’s not in use.

8

For the indexer process, a couple of bugs were fixed in 5.1.1 which were causing additional CPU/memory usage on an otherwise idle system(MB-29197 and MB-29194). Upgrading to 5.1.1 should help.