Hello,
I have a very strange situation when I am populating an empty bucket.
If the bucket is empty I receive a lot of Couchbase exceptions
\Couchbase\TempFailException
\Couchbase\TimeoutException
\Couchbase\NetworkException
\Couchbase\DurabilityException
as the bucket starts to fill up (at around 15 million documents), everything become normal without throwing any errors.
Normally I would say that it may be a network problem, but the network is used less than 50% and the CPU is less than 25% (across all the Couchbase servers), and on the client-side, the CPU load is higher around 90% and the network is less than 15%. But all these numbers are the same when the bucket is filled up with documents
This is something I could replicate very easily.
Below are some error logs:
[Tue Oct 06 16:44:13.883627 2020] [proxy_fcgi:error] [pid 49189] [client REDACATED-IP:30382] AH01071: Got error '
PHP message: [cb,WARN] (server L:779 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f8761d0,memcached,SRV=0x55f23f809ca0,IX=1) Failing command (pkt=0x55f23f7b1d90, opaque=1520, opcode=0x1) with error LCB_ERR_DURABLE_WRITE_IN_PROGRESS (310)
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/c3fbe9f8b01667f3/5f2","l":"REDACATED-IP:36389","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":2,"service":"kv","top":[{"last_local_address":"REDACATED-IP:36389","last_local_id":"ac720c203f1355ad/c3fbe9f8b01667f3","last_operation_id":"0x5f2","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2500126},{"last_local_address":"REDACATED-IP:36389","last_local_id":"ac720c203f1355ad/c3fbe9f8b01667f3","last_operation_id":"0x5f0","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2500099}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/c3fbe9f8b01667f3/5f5","l":"REDACATED-IP:36389","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (lcbio L:109 I:1058231725) Translating errno=0, lcb=0x401 to NETWORK_ERROR
PHP message: [cb,WARN] (server L:1053 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f8761d0,memcached,SRV=0x55f23f809ca0,IX=1) Got socket error LCB_ERR_NETWORK (1048)
PHP message: [cb,WARN] (server L:779 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f73bd40,memcached,SRV=0x55f23f809ca0,IX=1) Failing command (pkt=0x55f23f732550, opaque=1527, opcode=0x1) with error LCB_ERR_NETWORK (1048)
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":2,"service":"kv","top":[{"last_local_address":"REDACATED-IP:36389","last_local_id":"ac720c203f1355ad/c3fbe9f8b01667f3","last_operation_id":"0x5f5","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2500661},{"last_local_address":"REDACATED-IP:46326","last_local_id":"ac720c203f1355ad/75288ea37b574535","last_operation_id":"0x5f7","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2500536}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/75288ea37b574535/5fa","l":"REDACATED-IP:46326","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":1,"service":"kv","top":[{"last_local_address":"REDACATED-IP:46326","last_local_id":"ac720c203f1355ad/75288ea37b574535","last_operation_id":"0x5fa","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2501962}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/75288ea37b574535/5fc","l":"REDACATED-IP:46326","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (lcbio L:109 I:1058231725) Translating errno=0, lcb=0x401 to NETWORK_ERROR
PHP message: [cb,WARN] (server L:1053 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f73bd40,memcached,SRV=0x55f23f809ca0,IX=1) Got socket error LCB_ERR_NETWORK (1048)
PHP message: [cb,WARN] (server L:779 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f9309a0,memcached,SRV=0x55f23f809ca0,IX=1) Failing command (pkt=0x55f23f9a3670, opaque=1534, opcode=0x1) with error LCB_ERR_NETWORK (1048)
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":2,"service":"kv","top":[{"last_local_address":"REDACATED-IP:40713","last_local_id":"ac720c203f1355ad/0fba30e3b18ce29d","last_operation_id":"0x5fe","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2501994},{"last_local_address":"REDACATED-IP:46326","last_local_id":"ac720c203f1355ad/75288ea37b574535","last_operation_id":"0x5fc","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2500573}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/0fba30e3b18ce29d/601","l":"REDACATED-IP:40713","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":1,"service":"kv","top":[{"last_local_address":"REDACATED-IP:40713","last_local_id":"ac720c203f1355ad/0fba30e3b18ce29d","last_operation_id":"0x601","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2501642}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/0fba30e3b18ce29d/603","l":"REDACATED-IP:40713","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":1,"service":"kv","top":[{"last_local_address":"REDACATED-IP:40713","last_local_id":"ac720c203f1355ad/0fba30e3b18ce29d","last_operation_id":"0x603","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2501725}]}
PHP message: [cb,WARN] (lcbio L:109 I:1058231725) Translating errno=0, lcb=0x401 to NETWORK_ERROR
PHP message: [cb,WARN] (server L:1053 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f9309a0,memcached,SRV=0x55f23f809ca0,IX=1) Got socket error LCB_ERR_NETWORK (1048)
PHP message: [cb,WARN] (server L:779 I:1058231725) <SRV-HOSTNAME-REDACTED:11210> (CTX=0x55f23f9a5320,memcached,SRV=0x55f23f809ca0,IX=1) Failing command (pkt=0x55f23f73c010, opaque=1541, opcode=0x1) with error LCB_ERR_NETWORK (1048)
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":1,"service":"kv","top":[{"last_local_address":"REDACATED-IP:47576","last_local_id":"ac720c203f1355ad/900f58979e9b0cb6","last_operation_id":"0x605","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2500515}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/900f58979e9b0cb6/608","l":"REDACATED-IP:47576","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":1,"service":"kv","top":[{"last_local_address":"REDACATED-IP:47576","last_local_id":"ac720c203f1355ad/900f58979e9b0cb6","last_operation_id":"0x608","last_remote_address":"SRV-HOSTNAME-REDACTED:11210","operation_name":"upsert","server_us":0,"total_us":2501902}]}
PHP message: [cb,WARN] (server L:775 I:1058231725) Failing command with error LCB_ERR_TIMEOUT (201): {"b":"app-live","i":"ac720c203f1355ad/900f58979e9b0cb6/60a","l":"REDACATED-IP:47576","r":"SRV-HOSTNAME-REDACTED:11210","s":"kv:set","t":2500000}
PHP message: [cb,WARN] (tracer L:147 I:1058231725) Orphan responses observed: {"count":1,"service":"kv","top":[{"last_local_address":"REDACATED-IP:47576","last_local_id":"ac720c203f1355ad/900f58979e9b0cb6","last_operation_id":"0x60a","last_remote_address":"SRV-HOSTNAME-REDACTED......