It seems I faced https://issues.couchbase.com/browse/MB-19385 issue, so my question is there any workaround?
Sympthoms:
Currently I have following logs at memcached.log:
2016-08-22T09:30:37.055926Z WARNING (price) Notified the timeout on seqno persistence for vbucket 529, Check for: 38254, Persisted upto: 722, cookie 0x7fa9a8f51300 2016-08-22T09:30:37.055988Z WARNING 99: Slow SEQNO_PERSISTENCE operation on connection: 31 s ([ 127.0.0.1:59464 - 127.0.0.1:11209 (Admin) ]) 2016-08-22T09:30:37.059189Z WARNING (price) DCP (Consumer) eq_dcpq:replication:ns_1@ec2-54-82-42-217.compute-1.amazonaws.com->ns_1@ec2-52-6-195-120.compute-1.amazonaws.com:price - Got an error code 3 while trying to process mutation 2016-08-22T09:30:39.716268Z Message repeated 304 times 2016-08-22T09:30:39.716246Z WARNING (price) Notified the timeout on seqno persistence for vbucket 527, Check for: 37001, Persisted upto: 0, cookie 0x7fa9a73b8b00 2016-08-22T09:30:39.716312Z WARNING 56: Slow SEQNO_PERSISTENCE operation on connection: 31 s ([ 127.0.0.1:38021 - 127.0.0.1:11209 (Admin) ]) 2016-08-22T09:30:39.726637Z WARNING (price) DCP (Consumer) eq_dcpq:replication:ns_1@ec2-54-82-42-217.compute-1.amazonaws.com->ns_1@ec2-52-6-195-120.compute-1.amazonaws.com:price - Got an error code 3 while trying to process mutation 2016-08-22T09:30:39.745275Z WARNING (price) Notified the timeout on seqno persistence for vbucket 526, Check for: 38490, Persisted upto: 0, cookie 0x7fa9a73b8e00 2016-08-22T09:30:39.745370Z WARNING 58: Slow SEQNO_PERSISTENCE operation on connection: 31 s ([ 127.0.0.1:35097 - 127.0.0.1:11209 (Admin) ]) 2016-08-22T09:30:39.748917Z WARNING (price) DCP (Consumer) eq_dcpq:replication:ns_1@ec2-54-82-42-217.compute-1.amazonaws.com->ns_1@ec2-52-6-195-120.compute-1.amazonaws.com:price - Got an error code 3 while trying to process mutation 2016-08-22T09:30:40.315934Z Message repeated 51 times 2016-08-22T09:30:40.315912Z WARNING (price) Notified the timeout on seqno persistence for vbucket 528, Check for: 37848, Persisted upto: 0, cookie 0x7faa9f7c3c00 2016-08-22T09:30:40.315979Z WARNING 57: Slow SEQNO_PERSISTENCE operation on connection: 31 s ([ 127.0.0.1:32947 - 127.0.0.1:11209 (Admin) ]) 2016-08-22T09:30:40.322638Z WARNING (price) DCP (Consumer) eq_dcpq:replication:ns_1@ec2-54-82-42-217.compute-1.amazonaws.com->ns_1@ec2-52-6-195-120.compute-1.amazonaws.com:price - Got an error code 3 while trying to process mutation 2016-08-22T09:30:41.671261Z Message repeated 114 times 2016-08-22T09:30:41.671239Z WARNING (price) Notified the timeout on seqno persistence for vbucket 592, Check for: 37985, Persisted upto: 37389, cookie 0x7faa9f7c5100 2016-08-22T09:30:41.671319Z WARNING 55: Slow SEQNO_PERSISTENCE operation on connection: 31 s ([ 127.0.0.1:50649 - 127.0.0.1:11209 (Admin) ]) 2016-08-22T09:30:41.673111Z WARNING (price) DCP (Consumer) eq_dcpq:replication:ns_1@ec2-54-82-42-217.compute-1.amazonaws.com->ns_1@ec2-52-6-195-120.compute-1.amazonaws.com:price - Got an error code 3 while trying to process mutatio
and repalance is pending for 3 hours not moving at all:
Or should I post all of into the ticket?