Indexer high CPU

Couchbase Server
1

Hi Team,

Our ENV has 3 modes
node 1: only indexer (constantly we reaching 99% CPU)
node 2: query, search, data (reaching 60% CPU)
node 3: data, index (20% CPU)

we have High CPU with indexer on of the Node we tried various option like making replicas and replicate multiple indexes on node3, but still node1 is taking high CPU. but memory looks fine 40%.

Version: 6.5.1 build 6299

I took CPU profiler and memory PFA. please suggest which index19oct.zip (3.8 MB) is creating problem.

with regards,
Mohamed Sithik

2

@mohamedw,

Assuming that you are using enterprise edition of 6.5.1 build 6299 and running on CentOS7 (Please let us know otherwise) - I see that a lot of time is being spent in scans. CPU profile only shows which routines are actively consuming the CPU. As all indexes share the same code path, it is not possible to determine which index is consuming the CPU.

There are some stats for each index - E.g., num_requests which represent the number of scans served by each index and scan_bytes_read - number of bytes read by scans since indexer started. These statistics are available in indexer.log (or) you can query the index statistics rest endpoint - Index Statistics API | Couchbase Docs

You can use these stat to determine which is the heavily used index and then create a replica for this index for effective load balancing.

Thanks,
Varun

3

Hi Varun,

thanks please find atatched our stats fo index.CPU.zip (10.4 KB)

4

@mohamedw,

Apologies for the delay in responding. From the attached stats on node1, I see that indexes
idx_relatesToDomain_enteredDate_productinstance, adv_ALL_individualRoles_roleSpecId_individualId_type, adv_ALL_individualRoles_statusId_no_ind_type, idx_attachment_nested are most actively used - based on the stats num_requests, scan_bytes_read and total_scan_duration.

I would start with creating equivalent indexes for these indexes starting with idx_relatesToDomain_enteredDate_productinstance on node3 and drop them from node1. This should reduce the scan load on node1 and help in reduce the CPU utilisation.

Thanks,
Varun

5

Thanks Varun for information , we will try this option.

6

Hi Varun,

we have similar issue now on our env
node 1: index, query service (reaching 14% CPU )
node 2: query, search, data (reaching 100% CPU )
node 3: data, index ( 51% CPU )

we seeing issue on data node. Can you please let us know what we can check.

we took addition data.
select phaseCounts.fetch / resultCount efficiency, * from system:completed_requests where phaseCounts.indexScan is not missing and phaseCounts.fetch / resultCount > 10 order by efficiency desc;

PFA and below query with orderBy field is taking too much time,
we have around 15k data and doing order by sorting increase the CPU.

SELECT * FROM optima WHERE type = ‘processinstance’ and relatesToDomain == 0 and processTypeId == 14 and processSubTypeId not in [15,16] order by createDate desc LIMIT 6 OFFSET 0
Stats.zip (329.9 KB)

image
image1079×319 45.5 KB

with regards,
Mohamed Sithik

7

@mohamedw,

I think you should open a separate forum issue as this is not related to indexer taking high CPU.

cc: @vsr1

Thanks,
Varun

8

ok thanks. we will do it.