Insert request failed with error kv_temporary_failure

Couchbase Server
1

I am using couchbase community edition 7.0.1 build 6102 version and trying to do a performance test on 3 node cluster.
Memory quota are allocated as follows
Data Service : 8851 MB
Index Service : 4096 MB
Search Service : 512 MB

I started execution to insert 300 million records in DB but after 123 million record insertion , I am getting below error and not able to insert any record further.

Exception in thread "main" com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x32ae907d00000001","idempotent":false,"lastChannelId":"32AE907D00000001/000000008EBB1962","lastDispatchedFrom":"10.236.172.147:59321","lastDispatchedTo":"xxxxxxxxxxxxxxxxxx:11210","reason":"TIMEOUT","requestId":112,"requestType":"InsertRequest","retried":12,"retryReasons":["KV_TEMPORARY_FAILURE","COLLECTION_MAP_REFRESH_IN_PROGRESS"],"service":{"bucket":"jcache","collection":"RecordDup","documentId":"volte#1632975655638#10000000000","opaque":"0x30","scope":"Mediation","type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":265217,"encodingMicros":9456,"totalDispatchMicros":799452,"totalMicros":2505195}}
	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
	at com.couchbase.client.java.Collection.insert(Collection.java:404)

image
image1122×274 13.6 KB

I am attaching screen shot of cluster cpu, ram and disk size.

Please suggest what is the root cause of this error. I am expecting that couchbase should evict data from memory as it keeps on getting insert request.

2

Hi, were you able to find the root cause and the fix for the issue?
Please let us know.