Some weird exeption during ViewQuery

Java SDK
1

Need help, just upgraded from v1.4.x and updated my base code.
I have weird exception which lead to TimeoutExcption

2014-12-15 16:45:32,113 WARN [cb-io-1-1] [com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
com.couchbase.client.deps.io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: index: 32768 (expected: range(0, 32768))
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IndexOutOfBoundsException: index: 32768 (expected: range(0, 32768))
at com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1132)
at com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.getByte(AbstractByteBuf.java:338)
at com.couchbase.client.core.endpoint.view.ViewHandler.parseViewInfo(ViewHandler.java:332)
at com.couchbase.client.core.endpoint.view.ViewHandler.parseQueryResponse(ViewHandler.java:267)
at com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:189)
at com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:66)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:152)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
… 16 more

2

OK, found it
It was the viewQuery.debug(). now can someone tell me what it is and why did it caused the exception?

The docs says it helps debug view queries… LOL

3

@ohassidi this looks like a bug where a chunk is larger than expected. I need to fix this in the view parser. I created an issue to track it here: http://www.couchbase.com/issues/browse/JVMCBC-85

4

Hi @daschl
Is this bug fixed? Or is it fixed in community version higher than 3.0.0? Because I think I’m having same issue when getting a big JSON document. Thanks! I checked the ticket link but not sure which JVM is bundled in which community server?

5

May I know how to get the code of couchbase-java-client compiled without what’s in src/main/java/com/couchbase/client/dep/ (which doesn’t exit in git repo, but is required in build process)? I’m trying to build it after I built the 1.1.0 of couchbase-jvm-core, which is supposed to have the fix for this big chunk of data bug?

6

@nikkki … I think he was referring to a different bug, but it could be in the same area. Nevertheless, here is how you can build it.

You need to check out core, build and then the client and build:

  1. git clone GitHub - couchbase/couchbase-jvm-core: The JVM core for Couchbase SDKs.
  2. cd couchbase-jvm-core
  3. ./gradlew publishToMavenLocal

… then go back out of the dir and …

  1. git clone GitHub - couchbase/couchbase-java-client: The official Java client for Couchbase Server
  2. cd couchbase-java-client
  3. ./gradlew publishToMavenLocal

you will then have the master builds of both in your local maven dir (at this time 1.1.0-SNAPSHOT of the core and 2.1.0-SNAPSHOT of the client)

btw we love contributions! If you want, you can read more here: Contributing to the Couchbase Documentation | Couchbase Docs

7

Thanks!! I used gradle directly without noticing gradlew :blush:

8

just to clarify what is this dep namespace: we inline some external dependencies (namely netty) in the core jar, via a gradle plugin called shadow :wink:

9

Hi…Seeing similar exceptions in server log messages and its pretty irritating…Help/pointers/advises would be highly appreciated. The stack trace is as below-:

07:38:51,393 WARN [com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline] (cb-io-1-3) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: com.couchbase.client.deps.io.netty.handler.codec.DecoderException: rx.exceptions.OnErrorNotImplementedException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@447a718b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@54afe887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 139]
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

	Caused by: rx.exceptions.OnErrorNotImplementedException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@447a718b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@54afe887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 139]
    at rx.Observable$27.onError(Observable.java:7996)
    at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:158)
    at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:119)
    at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onError(SubjectSubscriptionManager.java:226)
    at rx.subjects.AsyncSubject.onError(AsyncSubject.java:115)

	
	Caused by: com.couchbase.client.core.CouchbaseException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@447a718b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@54afe887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 139]
    ... 22 more

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@447a718b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@54afe887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 139]

07:38:51,399 WARN [com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline] (cb-io-1-4) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.: com.couchbase.client.deps.io.netty.handler.codec.DecoderException: rx.exceptions.OnErrorNotImplementedException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6d3f29fc rejected from java.util.concurrent.ScheduledThreadPoolExecutor@70a23052[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 149]
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)

	07:39:21,413 WARN  [com.couchbase.client.core.endpoint.AbstractGenericHandler] (cb-io-1-3) [rhlappomr665.fairisaac.com/172.24.14.208:8093][QueryEndpoint]: Caught unknown exception: rx.exceptions.OnErrorNotImplementedException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@569ed0a4 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@220fdd2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 141]: com.couchbase.client.deps.io.netty.handler.codec.DecoderException: rx.exceptions.OnErrorNotImplementedException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@569ed0a4 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@220fdd2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 141]
10

Are you seeing this during shutdown or during a regular workload in a steady state? What SDK version are you using?

11

While re-deploying the application to JBOSS i am seeing this
The versions are-:
java-client -2.2.3,‘spring-data-couchbase’, version: ‘2.1.1.RELEASE’, Spring version-:4.2.5
Am using Spring to connect to Couchbase and gradle to build my application…

Thanks,
Arnab

12

@arnab.srmist a few things to try:

  • can you make sure you are shutting down everything properly and no operations are inflight when you redeploy?
  • if the issue still is there can you try upgrading to 2.2.8 of the SDK (manually overriding the 2.2.3 dep in your pomfile) and see if it helps?

This looks like an issue during shutdown/redeploy when cleaning up resources.

13

Yes…we have upgraded to 2.2.8 and we are using Spring-data-couchbase to manage the resources. So ideally everything should be closed by Spring api…Unfortunately we are still seeing the exceptions.

14

This got resolved after an SDK upgrade to java-client-2.3.4…which is the most recently released version

15

I am getting the same error in 2.3.4. Now I am checking with 2.4.3.

If this keeps happening then how we can put this in production? Please suggest else I need to switch to some other DB.

This happens once the container is up and kept idle for sometime. Suddenly this will start popping up and then all the flows will stop working. I am managing by restarting the container but this cant be done in production.

16

@abhideepchakravarty can you please share more details what your error is and in which circumstances it happens? You also might want to create another forum post instead of “hijacking” an older one :slight_smile:

17

I thought recreating a new thread for same issue is not a good idea. But if you say so, I am creating a new topic.