UnambiguousTimeoutException timeout

We are seeing this for 17 requests out of 0.5 million requests. The error is not at the start or end of the application.

SDK Version: 3.3.0

{"timestamp":"2022-06-10T19:02:24.732-07:00","message":"get","logger_name":"c.a.p.p.c.CouchbaseReadReactive","thread_name":"cb-timer-1-1","level":"ERROR","stack_trace":"com.couchbase.client.core.error.UnambiguousTimeoutException: GetRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x7caaf2f000000002","idempotent":true,"lastChannelId":"7CAAF2F000000002/00000000D5434CB3","lastDispatchedFrom":"192.168.11.115:49652","lastDispatchedTo":"lpqospdbb50755.test.com:11210","reason":"TIMEOUT","requestId":4311964,"requestType":"GetRequest","retried":0,"service":{"bucket":"<Bucket>","collection":"_default","documentId":"<Key>","opaque":"0x2a9257","scope":"_default","type":"kv","vbucket":843},"timeoutMs":2500,"timings":{"totalMicros":2506238}}\tat com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:184)\tat com.couchbase.client.core.msg.Request.cancel(Request.java:70)\tat com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)\tat com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)\tat com.couchbase.client.core.deps.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)\tat com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)\tat com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)\tat com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)\tat com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\tat java.base/java.lang.Thread.run(Thread.java:833)","trace_id":"gt8","span_id":"9ba187ce-3432-4859-aa7f-96527efe2056","key":"<Key>","bucket":"fallback","time_taken":2519}

@himanshu.mps the timeout is just the effect, never the cause. You can look at the logging output for the threshold logging tracer as well as the orphan reporter to identify what caused the slowness of the individual requests timing out and then take it from there. 17 requests out of 500.000 is not a lot, and there could be many reasons i.e. GC, network, over provisioned VMs being stalled of CPU, etc…