Getting the AmbiguousTimeoutException from couchbase

Couchbase Server
1

We have made cluster in which 3 nodes set up and give 18GB memory [6GB per node] in our Live environment with following configuration:

  • Couchbase Server Community Edition 7.0.2 build 6703
  • OS: CentOS Stream release 8 (kernel version: 4.18.0-529.el8.x86_64)
  • Couchbase Java SDK version 3.3.4
  • Type of bucket ephemeral

We are getting below error on the time of putting the data in bucket of Couchbase.

2025-02-13 15:21:52,707 ERROR [com.test.common.util.MembaseUtil] (default task-421) Exception in makeMembaseCacheEntrySynch(key: YnbMy6iWahDYwfBYTtW6QWm1XA66a56QopK5xFAffBM=, expiryTime:600000, bucketname:myBucket):: com.couchbase.client.core.error.AmbiguousTimeoutException: UpsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xd26c099e00000001","idempotent":false,"lastChannelId":"D26C099E00000001/000000009E496673","lastDispatchedFrom":"10.10.0.4:57046","lastDispatchedTo":"11.11.11.90:11210","reason":"TIMEOUT","requestId":212599234,"requestType":"UpsertRequest","retried":19,"retryReasons":["KV_TEMPORARY_FAILURE"],"service":{"bucket":"myBucket","collection":"_default","documentId":"YnbMy6iWahDYwfBYTtW6QWm1XA66a56QopK5xFAffBM=","errorCode":{"description":"Temporary failure. Try again","name":"ETMPFAIL"},"opaque":"0xcac031c","scope":"_default","type":"kv","vbucket":74},"timeoutMs":5000,"timings":{"dispatchMicros":100,"encodingMicros":38,"totalDispatchMicros":2149,"totalMicros":5007504}}
	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
	at com.couchbase.client.java.Collection.upsert(Collection.java:431)
	at com.test.common.util.MembaseUtil.makeMembaseCacheEntrySynch(MembaseUtil.java:332)
	at com.test.newApp.security.util.AuthorizationUtil.putUSPInMembase(AuthorizationUtil.java:301)
	at com.test.newApp.security.filter.SecurityFilter.processRequest(SecurityFilter.java:143)
	at com.test.newApp.security.filter.SecurityFilter.processFilter(SecurityFilter.java:78)
	at com.test.newApp.security.filter.BaseFilter.doFilter(BaseFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
		at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:184)
		at com.couchbase.client.core.msg.Request.cancel(Request.java:70)
		at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
		at com.couchbase.client.core.deps.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
		at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		... 1 more

We have checked in web console at the time of above ERROR, but we did not get any alert or issue, and all bucket status are running. We have checked Couchbase logs but did not find any root cause regarding above error. For logs Couchbase logs

Why are we getting this type of ERROR even though Couchbase server was up?

Let us know any other details require or need to check any configuration to troubleshoot the issue.

Thanks in advance.

2

“retryReasons”:[“KV_TEMPORARY_FAILURE”]

This can happen when you make many concurrent requests. The cluster is rejecting the request because it already servicing many requests. The SDK itself tried resending the request 19 times before the (5 second) timeout expired always getting KV_TEMPORARY_FAILURE from the cluster. If the requests are in a reactive Flux, the concurrency in a flux can be limited (refer to reactor documentation). A counting semaphore can also be used to limit the number of requests in flight (acquire before making the request and release upon completion).

1 Like
3

In addition to Mike’s comments, please see Async and Reactive APIs | Couchbase Docs for some advice on using the reactive Java API to control concurrency.

(Side-note - those are very old server and SDK releases, and we’d strongly recommend upgrading.)

1 Like