While querying a view again CBL 3.0.0 using the Java Client (2.0.1-SNAPSHOT), after a few iterations of the code, the Java SDK consistently errors out and dies with the following error:
Oct 16, 2014 10:34:35 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 2225 com.couchbase.client.core.RequestEvent@49250abd
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@38b07646
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "cb-core-3-2" java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@38b07646
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@38b07646
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
... 4 more
Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.TimeoutException
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:481)
at rx.observables.BlockingObservable.single(BlockingObservable.java:348)
at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:591)
at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:564)
at com.voltaireapp.changeslistener.ChangesListener.runChangesListener(ChangesListener.java:78)
at com.voltaireapp.changeslistener.ChangesListener.main(ChangesListener.java:135)
Caused by: java.util.concurrent.TimeoutException
at rx.internal.operators.OperatorTimeoutBase$TimeoutSubscriber.onTimeout(OperatorTimeoutBase.java:169)
at rx.internal.operators.OperatorTimeout$1$1.call(OperatorTimeout.java:42)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Thanks, @daschl! I tried to search JIRA/issues before I raised this here but couldn’t find anything, sorry if this was a dupe/already known issue, not much came up, maybe I didn’t search properly.
At any rate, is there an issue # I can go follow in JIRA/Issue tracker?
FWIW - the following workaround seems to work (wrap any upsert/insert/replace) that will catch the issue, reset the client, and keep going.
Testing this out a bit more, and if it works, the workaround until the fix hits 2.0.1 will just be to build a wrapper that wraps upsert et al and does the same thing…
try {
bucket.upsert(doc);
} catch (Exception e) {
if (e.toString().contains("java.lang.IllegalStateException: Node not found for request")) {
logger.error("Ugh, couchbase is being stupid and entering Microsoft emulation mode, resetting client and trying to recover...");
cluster.disconnect();
cluster = null;
cluster = CouchbaseCluster.create(clusterString);
bucket.upsert(doc);
} else {
e.printStackTrace();
throw new RuntimeException(e);
}
}
No, this was a different exception (node addr mismatch) that was fixed in 2.0, I need to mark it appropriately. I’ll get you the right one with more info.
We’ll go a head and retry those config requests in the future to gracefully handle connect states. I’ll package a 2.0.1-pre1 very soon and get it to you upfront so you can try to see if it works.
I checked the pre-version and for me it works like a charm! Thank you. Just to be sure, i also replaced it once more with the 2.0.0 version and it almost immediately complains.
@daschl, thanks for the quick response and patch, sorry for my late reply, just had time to try this and it seems to work. Will update to 2.0.1 when it hits maven. Just as a point of interest - with what frequency (if any) is the -SNAPSHOT updated in maven? I know 2.0.1-SNAPSHOT still exhibits this issue, so I am assuming the code you posted in the zip is off some commit post-snapshot (b9021ce?)? No big deal as it seems release of 2.0.1 is imminent anyway…
Well, we are not publishing SNAPSHOTS as of now through or Jenkins, but we will do in the future. For now you have to build both the core-io and java-client from github directly through gradle, but it’s not that hard either.
Hi I updated my pom and added 2.0.1 to my project. and for some reason i am still getting the error. can anyone confirm that the issue was fixed in the new release of the Java SDK.
I included the last line of log for the last operation i ran so you get a sense of the time.
INFO 2014-11-07 00:46:57,905 [_api].connector.http.mule.default.receiver.06] org.mule.api.processor.LoggerMessageProcessor: Create collection object
Nov 07, 2014 12:50:31 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 3 com.couchbase.client.core.RequestEvent@5af345c5
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@479d308f
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Exception in thread “cb-core-3-2” java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@479d308f
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@479d308f
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
… 4 more
Nov 07, 2014 12:50:31 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 103 com.couchbase.client.core.RequestEvent@5d0a66c5
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@53cd92cc
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Exception in thread “cb-core-6-2” java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@53cd92cc
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@53cd92cc
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
… 4 more
@chaddy012 I was explicitly requesting the first lines because on bootstrap we output configuration params and all that related stuff, so if you can include these parts as well it would be great.
