Slow Index Create (Any suggestion please)

Couchbase Server
1

I migrate my old server to new one using cbdatarecovery tool.
But couchbase EE (7.6.2) create my indexes so slow.
One bucket has 100M documents my server has 1TB ram and SSD drives.
It is better than old server, but Index creating more than 6 hours.
Server does not consume CPU, what is the problem?

This is my indexer.log stats
[Info] Periodic Aggregated StorageStats:
{
“num_tenant”: 1,
“num_controller”: 32,
“num_shard”: 32,
“num_shard_altid”: 0,
“num_instances”: 43,
“num_lssctx”: 86,
“num_sctxs”: 2079,
“num_active_sctxs”: 904,
“num_free_sctxs”: 1220,
“num_trim_sctxs”: 51,
“num_wctxs”: 1791,
“num_free_wctxs”: 1157,
“num_active_wctxs”: 634,
“num_trim_wctxs”: 103,
“num_readers”: 23,
“num_writers”: 825,
“cpu”: 1.28800,
“total_core”: 128,
“threads”: 179,
“avail_thread_barrier”: 8000,
“assigned_quota”: 235929600000,
“current_quota”: 235929600000,
“tenant_quota_assigned”: 0,
“tenant_quota_mandatory”: 0,
“tenant_quota_discretionary”: 0,
“tenant_quota_mutation”: 0,
“tenant_working_set”: 0,
“heap_limit”: max,
“memory_resident”: 51883655168,
“memory_jemalloc”: 48814313472,
“memory_jemalloc_active”: 48174583808,
“memory_jemalloc_allocated”: 47043261144,
“memory_jemalloc_dirty”: 2367488,
“estimated_mem_frag”: 0.14162,
“memory_stats_size_total”: 42918869390,
“memory_stats_size_global”: 42758935058,
“memory_stats_size_page”: 42416331378,
“memory_stats_size_index”: 342433730,
“memory_stats_size_bloom”: 288,
“memory_stats_size_delta”: 5322335448,
“reclaim_pending”: 53074662,
“reclaim_pending_global”: 53017751,
“buf_memused”: 107029332,
“lss_fbuf_used”: 148897792,
“lss_fbuf_quota”: 9437184000,
“under_swap”: false,
“num_freeOS”: 0,
"num_smr_ctx ": 1,
"num_smr_worker ": 256,
"num_active_smr_worker ": 0,
"num_smr_queued ": 0,
“num_total_smr”: 1804,
“num_active_smr”: 222,
“num_pending_smr”: 1612,
“num_pages”: 3921601,
“items_count”: 1050888391,
“total_records”: 1051215818,
“avg_item_size”: 55,
“total_ops”: 865122646,
“compacts”: 21259664,
“purges”: 0,
“merges_emptypages”: 0,
“lss_data_size”: 21667096257,
“lss_used_space”: 31630227129,
“lss_disk_size”: 32547307520,
“lss_open_filedesc”: 178,
“max_open_filedesc”: 200000,
“lss_fragmentation”: 31,
“lss_num_reads”: 1431770430,
“lss_read_bs”: 243583237431,
“lss_blk_read_bs”: 409942896640,
“rlss_num_reads”: 51,
“lss_rdr_reads_bs”: 4245,
“lss_blk_rdr_reads_bs”: 208896,
“lookup_num_reads”: 6004181,
“lookup_reads_bs”: 19065155400,
“lookup_blk_reads_bs”: 43526135808,
“bytes_written”: 246925647872,
“lss_flushes”: 104188,
“lss_flushdur”: 2.179039ms,
“bytes_incoming”: 45792613896,
“write_amp_avg”: 299.22,
“num_swapper_ctx”: 32,
“avg_sweep_interval”: 5m0s,
“avg_run_interval”: 66.25ms,
“avg_clock_interval”: 2m9.361314704s,
“num_swapperWorker”: 2048,
“cached_records”: 1049005472,
“resident_ratio”: 0.99790,
“num_rec_compressed”: 669165244,
“num_rec_compressible”: 898839379,
“compressed_percent”: 63,
“compressed_compressible_perc”: 74,
“compression_ratio”: 2.65128,
“num_burst_visits”: 0,
“num_periodic_visits”: 445192165,
“num_evicted”: 0,
“num_evictable”: 262389450,
“cleaner_num_reads”: 1424821450,
“cleaner_read_bs”: 220807435414,
“cleaner_blk_read_bs”: 358978494464,
“recovery_num_lssctx”: 43,
“recovery_num_sctxs”: 223,
“recovery_num_active_sctxs”: 172,
“recovery_num_free_sctxs”: 51,
“recovery_num_trim_sctxs”: 51,
“recovery_buf_memused”: 12582912,
“recovery_fbuf_used”: 58720256,
“recovery_data_size”: 1056472550,
“recovery_used_space”: 1546201458,
“recovery_disk_size”: 2031104000,
“recovery_lss_open_filedesc”: 86,
“recovery_lss_blk_read_bs”: 256949739520,
“recovery_lss_read_bs”: 133254564371,
“recovery_lss__num_reads”: 1386094664,
“recovery_bytes_written”: 139555090432,
“recovery_lss_flushes”: 533903,
“recovery_lss_flushdur”: 1.203234ms,
“recovery_lss_frag”: 31,
“recovery_cleaner_num_reads”: 1386094664,
“recovery_cleaner_read_bs”: 132854026290,
“recovery_cleaner_blk_read_bs”: 256538484736,
“mvcc_purge_ratio”: 1.00031,
“reader_purge_count”: 0,
“reader_emptypage_count”: 18,
“fbuf_tuner_status”: ,
“active_copy_ctx”: 0,
“num_shard_upload”: 0 (0),
“num_shard_restored”: 0 (0),
“upload_bytes”: 0,
“avg_upload_bw”: 0,
“download_bytes”: 0,
“avg_download_bw”: 0,
“num_upload_cleanups”: 0 (0),
“num_staging_cleanups”: 0 (0)
}

2

The time to build indexes will depend on the amount of data to be indexed and amount of indexes. Six hours to create indexes may not indicate a problem.

3

Thanks for response, but all backup done in 1 hours. This means couchbase can read all data in 1 hour. But when I try to index data, server doesn’t consume CPU, RAM and IO. This situation might be show couchbase can’t use all system sources. I thought I need to change some indexer parameters like threadcount. I need to wait 4 5 days to index all data.

5

@canaryio76 Will it be possible to share all the indexer.log files?

From the stats you shared, CPU does not seem to be utilised fully. We need logs to confirm where the bottleneck is.

The files might be large. You can create a google drive link and DM me in such case.

Thanks,
Varun

6

I shared indexer.log file with you (DM), we can share solution here after inspect.

Thanks

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.