Timeout errors since upgrading Java SDK

We recently upgraded the java SDK from 2.4.4 to 2.4.6 and since then we have been recieving timeout exceptions on certain environments. The strange thing is I have reverted the sdk to 2.4.4 and the exceptions persist. Here is the log. Can anyone help with the cause please?

We also noticed that every 3rd request fails. We opened all ports on firewall but every 3rd request fails.

cocuh.txt.zip (8.2 KB)

Full log above: but here is an excerpt:

[2017-06-20 19:13:19.006] DEBUG com.couchbase.client.core.config.ConfigurationProvider [cb-computations-1]: Received signal to proactively refresh (a maybe outdated) configuration.
[2017-06-20 19:13:19.090] DEBUG com.couchbase.client.core.config.ConfigurationProvider [cb-computations-1]: New Bucket newton-dev config proposed.
[2017-06-20 19:13:19.092] DEBUG com.couchbase.client.core.config.refresher.CarrierRefresher [cb-computations-1]: Completed refreshing config for bucket “newton-dev”
[2017-06-20 19:13:23.358] DEBUG com.couchbase.client.core.endpoint.AbstractGenericHandler [cb-io-1-2]: [newton-n1.xxxxxx.com/52.214.200.214:11210][KeyValueEndpoint]: KeepAlive fired
[2017-06-20 19:13:23.595] DEBUG com.couchbase.client.core.endpoint.AbstractGenericHandler [cb-io-1-2]: [newton-n1.xxxxxx.com/52.214.200.214:8093][QueryEndpoint]: KeepAlive fired
[2017-06-20 19:13:29.006] DEBUG com.couchbase.client.core.config.ConfigurationProvider [cb-computations-1]: Received signal to proactively refresh (a maybe outdated) configuration.
[2017-06-20 19:13:29.089] DEBUG com.couchbase.client.core.config.ConfigurationProvider [cb-computations-2]: New Bucket newton-dev config proposed.
[2017-06-20 19:13:29.091] DEBUG com.couchbase.client.core.config.refresher.CarrierRefresher [cb-computations-2]: Completed refreshing config for bucket “newton-dev”
[2017-06-20 19:13:29.236] DEBUG com.couchbase.client.core.endpoint.AbstractGenericHandler [cb-io-1-1]: [newton-n3.xxxxxx.com/34.253.107.18:11210][KeyValueEndpoint]: KeepAlive fired
[2017-06-20 19:13:38.599] ERROR org.mule.retry.notifiers.ConnectNotifier [[newton-api-dev].restAPIMainHttpConfig.worker.01]: Failed to connect/reconnect: Work Descriptor. Root Exception was: null. Type: class java.util.concurrent.TimeoutException
[2017-06-20 19:13:38.741] ERROR org.mule.exception.DefaultMessagingExceptionStrategy [[newton-api-dev].restAPIMainHttpConfig.worker.01]:

Root Exception stack trace:
java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:73)
at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:659)

Hi @ryandamiencarter,

There isn’t much known from the logs. It will be helpful to debug more if you can provide the following

  • Which server version are using
  • Can you enable the trace level logging and share the logs
  • Is it possible to reproduce the problem in isolation

Thanks.

4.6.0 clustered environment.
will add trace logs.
we have 3 nodes. we noticed if we point at node1 it fails but node2 and and node3 are fine.