Exception IllegalStateException when query view after Couchbaseclient run serval days

Java SDK
1

I have problems when I query view.
My couchbase client is manager via GenericObjectPool as follows:

class CouchClientFactory extends BasePooledObjectFactory {
private static int DEFAULT_OP_TIME_OUT = 10000;
private ArrayList nodes;
private String bucketName;
private String password;

public CouchClientFactory(ArrayList<URI> nodes, String bucketName, String password)
{
    this.nodes = nodes;
    this.bucketName = bucketName;
    this.password = password;
}

@Override
public CouchbaseClient create() {
    CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
    cfb.setOpTimeout(DEFAULT_OP_TIME_OUT);
    try {
        CouchbaseConnectionFactory cf = cfb.buildCouchbaseConnection(nodes, bucketName, password);
        CouchbaseClient client = new CouchbaseClient(cf);
        return client;
    }
    catch (Exception e)
    {
        e.printStackTrace();
    }
    return null;
}

/**
 * Use the default PooledObject implementation.
 */
@Override
public PooledObject<CouchbaseClient> wrap(CouchbaseClient client) {
    return new DefaultPooledObject<CouchbaseClient>(client);
}

/**
 * When an object is returned to the pool, clear the buffer.
 */
@Override
public void passivateObject(PooledObject<CouchbaseClient> pooledObject) {
}

// for all other methods, the no-op implementation
// in BasePooledObjectFactory will suffice

}
GenericObjectPool objectPool = new GenericObjectPool(
new CouchClientFactory(nodes, dbConfig.getBucketName(), dbConfig.getBucketPassword())
);

And when I query view failed, I will try again. But when I run my server for server days. I receive Exceptions as follow:
java.lang.RuntimeException: java.lang.RuntimeException: Timed out waiting for operation
at com.pwrd.core.couchbase.CouchAdapter.getView(CouchAdapter.java:495)
at com.pwrd.core.couchbase.CouchController.queryArrayView(CouchController.java:193)
at com.pwrd.core.couchbase.DbExecutor.queryArrayView(DbExecutor.java:72)
at com.pwrd.game.service.bo.virtualworld.VirtualWorldService.doQueryUserInfo(VirtualWorldService.java:719)
at com.pwrd.game.action.user.QueryUserMessageAction.processMessage(QueryUserMessageAction.java:16)
at com.pwrd.game.action.user.QueryUserMessageAction.processMessage(QueryUserMessageAction.java:12)
at com.pwrd.game.action.common.WorldServerPipeline.dispatchAction(WorldServerPipeline.java:62)
at com.pwrd.core.bhns.AbstractSimpleService.messageReceived(AbstractSimpleService.java:372)
at com.pwrd.game.action.common.GatewayServerPipeline.dispatchAction(GatewayServerPipeline.java:78)
at com.pwrd.net.protocol.DefaultSessionHandler.messageReceived(DefaultSessionHandler.java:53)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:716)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:75)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:75)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:427)
at com.pwrd.core.couchbase.CouchAdapter.getView(CouchAdapter.java:491)
… 19 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:93)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
… 21 more
[ERROR][15:57:33][CouchController.queryArrayView(218)]
java.lang.RuntimeException: java.lang.IllegalStateException: I/O reactor has been shut down
at com.pwrd.core.couchbase.CouchAdapter.getView(CouchAdapter.java:495)
at com.pwrd.core.couchbase.CouchController.queryArrayView(CouchController.java:193)
at com.pwrd.core.couchbase.DbExecutor.queryArrayView(DbExecutor.java:72)
at com.pwrd.game.service.bo.virtualworld.VirtualWorldService.doQueryUserInfo(VirtualWorldService.java:719)
at com.pwrd.game.action.user.QueryUserMessageAction.processMessage(QueryUserMessageAction.java:16)
at com.pwrd.game.action.user.QueryUserMessageAction.processMessage(QueryUserMessageAction.java:12)
at com.pwrd.game.action.common.WorldServerPipeline.dispatchAction(WorldServerPipeline.java:62)
at com.pwrd.core.bhns.AbstractSimpleService.messageReceived(AbstractSimpleService.java:372)
at com.pwrd.game.action.common.GatewayServerPipeline.dispatchAction(GatewayServerPipeline.java:78)
at com.pwrd.net.protocol.DefaultSessionHandler.messageReceived(DefaultSessionHandler.java:53)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:716)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:75)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: I/O reactor has been shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:226)
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:421)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:266)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:248)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:178)
at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:58)
at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:200)
at com.couchbase.client.ViewConnection.addOp(ViewConnection.java:220)
at com.couchbase.client.CouchbaseClient.addOp(CouchbaseClient.java:800)
at com.couchbase.client.CouchbaseClient.asyncGetView(CouchbaseClient.java:339)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:427)
at com.pwrd.core.couchbase.CouchAdapter.getView(CouchAdapter.java:491)
… 19 more
From the log we can find when query view first time, I receive a time out exception. And then second time, I receive a IllegalStateException. And I will always receive the IllegalStateException after that time.

2

I’m not 100% sure what’s going on based on that small piece of code (and no logs?).

Are you pooling CouchbaseClient objects? If so, for what reason?
Also, can you share some logs on the events that happened around that period?
Were it consistent timeouts or just a single one?

One thing that looks fish is:

Caused by: java.lang.IllegalStateException: I/O reactor has been shut down

This points to the fact that you shut down the CouchbaseClient and then used it again - this should not happen. Best is to create 1 CouchbaseClient per bucket and just reuse this one. In general, there is no reason to do pooling and opening/closing it all the time. It is actually more harmful than helpful.

3

Yes, I am pooling CouchbaseClient objects. Because I am using client in multithread query views and document. And I don’t whether it is thread safe.
I also think that the couchbaseclient is shutdown. But I never write any code to shutdown it. And I review the couchbaseclient code and found this Exception is raised from httpcore.

And the reason I use pool is this document: http://docs.couchbase.com/developer/dev-guide-3.0/thread-safety.html
It has advice as follow:
"
Languages such as.Net and Java have in built-in support for thread-safety, and the Couchbase Java and.Net SDKs have been certified as being thread-safe if you utilize thread-safety mechanisms provided by the two languages with the SDKs.

Note that the client object you create with a Couchbase SDK does not spawn additional threads to handle multiple requests. Therefore to provide multi-threading and client object reuse even for SDKs that are thread-safe, you will need to implement connection pools and other language-specific thread-safety measures.
"

4

Okay so then first: don’t pool them. Create one per bucket and reuse, they are 100% thread safe and meant to be used that way.

If the client was not shut down from your call there must be something else that cause the reactor to terminate. I couldn’t imagine what, can you share the full logs around that timeframe?
Thanks

5

I review the httpcore code, and I guess a reasonable cause is it run into a problem when query view in working thread in httpcore and then it shutdown the reactor. But I do not catch the log, I am looking for other log files to find some evidences.