Errors in logs: Failing command, Found stale packet

Couchbase Server
1

| recently found some errors in my nginx logs. Nothing changed in the topology - two servers with Couchbase 4.0.0-4051 Community Edition (they have local IPs 10.0.1.2 and 10.0.1.3).
Those errors are as follows:

  • [cb,WARN] (server L:389 I:0) <10.0.1.3:11210> (SRV=0x1a04790,IX=1) Failing command (pkt=0x18169d0, opaque=64694, opcode=0xb5) with error 0x17
  • [cb,EROR] (server L:472 I:0) <10.0.1.3:11210> (SRV=0x1a04790,IX=1) Server timed out. Some commands have failed
  • [cb,WARN] (server L:249 I:0) <10.0.1.2:11210> (SRV=0x17431d0,IX=0) Found stale packet (OP=0x0, RC=0x0, SEQ=34304)

It is hard to find anything on the web regarding those errors.

I also realised that every 2 minutes graphs in the CB console freezes, everything drops to 0 and after 4-5 seconds everything goes back online. After this period of ‘doing nothing’ each graph starts from the higher level (e.g. I have average number od ops per second aboyt 750, after this break this graph starts from 1500)

2

We managed to solve the problem by… changing the hard drive. Although we had already had SSD, according to our admins, it had journaling problem. We changed the hardware to Enterprise device and the problem dissapeared.