Couchbase 6.0.0
Ejection Method full (resident ~20%)
RAM used/quota 33GB/41GB
disk used 300GB
450,000,000 items
Compression Passive
Three data nodes, two indexing nodes
I am adding items to couchbase at the rate of around 500,000 a minute. This goes fine for 8 hours or so, then I get the error below and the program that is adding items stops. No other activity is happening on the cluster I don’t start the program up again for a few hours, but when I do it runs fine for another 8 hours or so. This pattern has happened at least three times and probably 5 times on a smaller cluster. The number of items ranges from 100 million to 450 million when it happens.
This cluster is just for testing and is likely undersized, but I m not sure what the problem is since it runs file for many hours.
Each time I see these three errors (in reverse order). The time stamp is all the same.
Bucket “bucket_name” loaded on node ‘ns_1@10.3.50.210’ in 69 seconds.
Service ‘memcached’ exited with status 137. Restarting. Messages:
2019-06-25T19:17:30.993553+00:00 WARNING (bucket_name) Slow runtime for ‘Process checkpoint(s) for DCP producer eq_dcpq:replication:ns_1@10.3.50.210->ns_1@10.3.50.212:bucket_name’ on thread auxIO_worker_0: 142 ms
2019-06-25T19:17:31.500975+00:00 WARNING (bucket_name) Slow runtime for ‘Adjusting hash table sizes.’ on thread nonIO_worker_1: 106 ms
2019-06-25T19:17:32.037660+00:00 WARNING (bucket_name) Slow runtime for ‘Removing closed unreferenced checkpoints from memory’ on thread nonIO_worker_0: 54 ms
2019-06-25T19:17:32.084541+00:00 WARNING (bucket_name) Slow runtime for ‘Running a flusher loop: shard 3’ on thread writer_worker_0: 3840 ms
2019-06-25T19:17:32.467705+00:00 WARNING (bucket_name) Slow runtime for ‘Running a flusher loop: shard 0’ on thread writer_worker_2: 3059 ms
2019-06-25T19:17:33.558786+00:00 WARNING (bucket_name) Slow runtime for ‘Running a flusher loop: shard 2’ on thread writer_worker_3: 3790 ms
2019-06-25T19:17:33.55904 …show
ns_log 000
ns_1@10.3.50.210
Control connection to memcached on ‘ns_1@10.3.50.210’ disconnected: {error,closed}