Indexer idling (or blocking) during heavy upsert_multi()

Couchbase Server
1

Hi,
I’m running on Couchbase CE 6.0.0, which has single bucket with around 35M documents (and growing). I notice that when I have two clients inserting documents at the same time (around 3~5k op/s). After 2~5 minutes, the "total mutation remaining " will keep increasing and “drain rate items/sec” will drop to 0 .

Here is the statistic graph:

2019-12-25 10.28.28
2019-12-25 10.28.282262×662 90.1 KB

I’ve also use htop to see if indexer process is dead. It’s still running but with only 12~15% CPU loading.

The only solution is to stop the feeding client. Then after few minutes of waiting, the indexer will start to process queued documents.

Does anyone have the same issue? How do you fix it? Thanks! :slight_smile:

2

@cb_newbies,

The system might be in heavy DGM mode. Can you check the resident percent of the index when drain rate drops to zero? If the resident percent is less ( < 20%), then you may have to increase the indexer memory quota.

If the resident percent is > 20% and still this issue is seen, then can you share the cbcollect logs. For the procedure to collect logs, you can look at Manage Logging | Couchbase Docs.

Thanks,
Varun