java.util.concurrent.TimeoutException on get and upsert operations

Java SDK
1

We are using sdk version 2.3.1 and frequently running into following issue. With retry count configured to 3 ,get and upsert were never successful, but if retry count is increased significantly (eg: 20 or 30) occurrence of following exceptions reduces to great extent.

Version: 4.1.0-5005 Enterprise Edition (build-5005)

Any help will be appreciated.

com.couchbase.client.java.error.CannotRetryException: maximum number of attempts reached after 3 retries
at com.couchbase.client.java.util.retry.RetryWithDelayHandler.call(RetryWithDelayHandler.java:101)
at com.couchbase.client.java.util.retry.RetryWithDelayHandler.call(RetryWithDelayHandler.java:42)
at rx.internal.operators.OperatorMap$MapSubscriber.onNext(OperatorMap.java:66)
at rx.internal.operators.OperatorZip$Zip.tick(OperatorZip.java:264)
at rx.internal.operators.OperatorZip$Zip$InnerSubscriber.onNext(OperatorZip.java:335)
at rx.internal.operators.OperatorMap$MapSubscriber.onNext(OperatorMap.java:74)
at rx.internal.operators.OnSubscribeRedo$3$1.onNext(OnSubscribeRedo.java:307)
at rx.internal.operators.OnSubscribeRedo$3$1.onNext(OnSubscribeRedo.java:289)
at rx.internal.operators.NotificationLite.accept(NotificationLite.java:150)
at rx.subjects.SubjectSubscriptionManager$SubjectObserver.emitNext(SubjectSubscriptionManager.java:253)
at rx.subjects.BehaviorSubject.onNext(BehaviorSubject.java:160)
at rx.internal.operators.OnSubscribeRedo$2$1.onError(OnSubscribeRedo.java:242)
at rx.internal.operators.OperatorSubscribeOn$1$1.onError(OperatorSubscribeOn.java:59)
at rx.observers.Subscribers$5.onError(Subscribers.java:224)
at rx.observers.SerializedObserver.onError(SerializedObserver.java:158)
at rx.observers.SerializedSubscriber.onError(SerializedSubscriber.java:79)
at rx.internal.operators.OperatorTimeoutBase$TimeoutSubscriber.onTimeout(OperatorTimeoutBase.java:178)
at rx.internal.operators.OperatorTimeout$1$1.call(OperatorTimeout.java:42)
at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:186)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
… 11 more

2

hi @avinjos. I have a few questions:

  • what is your workload like (in terms of number of reads and writes per second for instance)?
  • how did you configure the retry (type of delay, duration, etc…)?
  • did you try to dig on the server side see if there could be an explanation for key/value operations to time out?