Error encountered while using Couchbase Java SDK

Java SDK
1

Hello,
I am getting below error and warning while using Couchbase Java SDK 3.2.4 . I am not sure of the cause for it.

java.util.concurrent.RejectedExecutionException: event executor terminated
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:923)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:825)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483)
	at com.couchbase.client.core.deps.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87)
	at com.couchbase.client.core.deps.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
	at com.couchbase.client.core.deps.io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
	at com.couchbase.client.core.deps.io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323)
	at com.couchbase.client.core.deps.io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155)
	at com.couchbase.client.core.deps.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:116)
	at com.couchbase.client.core.endpoint.BaseEndpoint.lambda$reconnect$1(BaseEndpoint.java:336)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)
	at reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4385)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4515)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4371)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4307)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4279)
	at com.couchbase.client.core.endpoint.BaseEndpoint.reconnect(BaseEndpoint.java:383)
	at com.couchbase.client.core.endpoint.BaseEndpoint.connect(BaseEndpoint.java:256)
	at com.couchbase.client.core.endpoint.BaseEndpoint.notifyChannelInactive(BaseEndpoint.java:523)
	at com.couchbase.client.core.io.netty.PipelineErrorHandler.channelInactive(PipelineErrorHandler.java:64)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at com.couchbase.client.core.io.netty.kv.KeyValueMessageHandler.channelInactive(KeyValueMessageHandler.java:302)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
	at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasksFrom(SingleThreadEventExecutor.java:425)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:374)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:760)
	at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:402)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
14:36:24.309 [cb-io-kv-5-3] ERROR c.c.c.c.d.i.n.u.c.D.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:923)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:825)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184)
	at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
	at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
	at com.couchbase.client.core.endpoint.BaseEndpoint.channelFutureIntoMono(BaseEndpoint.java:686)
	at com.couchbase.client.core.endpoint.BaseEndpoint.lambda$reconnect$1(BaseEndpoint.java:336)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77)
	at reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4385)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4515)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4371)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4307)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4279)
	at com.couchbase.client.core.endpoint.BaseEndpoint.reconnect(BaseEndpoint.java:383)
	at com.couchbase.client.core.endpoint.BaseEndpoint.connect(BaseEndpoint.java:256)
	at com.couchbase.client.core.endpoint.BaseEndpoint.notifyChannelInactive(BaseEndpoint.java:523)
	at com.couchbase.client.core.io.netty.PipelineErrorHandler.channelInactive(PipelineErrorHandler.java:64)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at com.couchbase.client.core.io.netty.kv.KeyValueMessageHandler.channelInactive(KeyValueMessageHandler.java:302)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
	at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasksFrom(SingleThreadEventExecutor.java:425)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:374)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:760)
	at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:402)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
2

It looks like your application exits without disconnecting the cluster.

3

@mreiche Yes, i have not put any code to disconnect the cluster.

4

@mreiche I am actually doing couchbase upgrade from 2.7 to 3.2 and in 2.7 we were not disconnecting the cluster.

5

@mreiche Could you please help here ?

6

I’m not sure what you’re looking for. Please open a case with customer support.

7

We’ve not seen this error before. Going from this error message and a quick internet search, my best guess is that you’re doing operations on a Cluster object that you’ve shutdown?

8

@graham.pople We are migrating from SDK2 to SDK3 and in SDK2 also we were not shutting down the cluster manually and never faced this exception.

9

Any cluster objects obtained by Cluster.connect() must be disconnected. You stated that you do not have code that does this. You must add code that does. CouchbaseClientFactories in spring data contain a cluster object, therefore any CouchbaseClientFactory created by the application must be closed/disconnected.

SDK 3 uses netty while SDK 2 did not. Therefore you would not have seen such an error using SDK 2.