After restarting database request failing and not retrying

Java SDK
1

Couchbase 4.5
AWS Server
Java 1.8 application

After restarting couchbase (for maintenance) request starting failing when trying to save new docs to the database with the exception below. I’m using the default retry strategy of BestEffortRetryStrategy. From what I’ve been able to piece together it looks like observable request (which I’m using) are handled differently than non-observables maybe this explains why the reattempts are not happening but still doesn’t explain why the request are failing to begin with. I thought maybe the failure is happening because the reconnect didn’t happen properly but I don’t think thats the case because I would assume my request would result in a TimeoutException instead.

Once I restarted my application then all things begin to play well with each other.

What I’m trying to determine is if an application restart is always needed after a couchbase restart. What is the connection pool mechanism I can use to get around a manual application restart. (Seems reconnect is out of the box and nothing is needed but I’m not sure I’m reading this right.)

com.couchbase.client.core.RequestCancelledException: Could not dispatch request, cancelling instead of retrying.
at com.couchbase.client.core.retry.RetryHelper.retryOrCancel(RetryHelper.java:51)
at com.couchbase.client.core.service.AbstractPoolingService.dispatch(AbstractPoolingService.java:60)
at com.couchbase.client.core.service.AbstractDynamicService.send(AbstractDynamicService.java:129)
at com.couchbase.client.core.node.CouchbaseNode.send(CouchbaseNode.java:170)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:202)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:74)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
2

If the request is circulating longer than the max lifetime (75) even on the best effort retry operation the request is cancelled to avoid circling around forever. Could that be something you are hitting? If you know your operation is idempotent like a get you can retry on that error (if your time budget allows for it)

3

thanks for your reply @daschl. No i dont think that is the case because even after couchbase is back up after the restart every request fails not until restarting the application will request start to working again. One thing to note is our topology has 3 nodes in our cluster could this be creating some issues

4

Which SDK version are you using?

5

Sorry can’t believe i forgot that. That app is using 2.2.1

6

@mlblount45 would it be possible for you to try with a recent version like 2.3.4?

7

Good point Yes I will give that a try

8

Hi @mlblount45, Is your issue resolved after upgrading the SDK to 2.3.4?
I am also facing the same issue with couchbase sdk 2.2.5.

9

@dasarivinod1 I’m currently using 2.3.7 SDK yes I believe this has resolved the issue for me.