Errors when executing operations

.NET SDK
1

We have a ASP.NET Core application running inside CentOS using 2.7.10 and deploying our application which connects to 2 clusters and reads and writes from both.

But randomly we see errors in our logs like:
Cannot find callback object for operation and The operation has timed out. Retried [0] times.

Even though we confirmed that the Couchbase clusters are up and running. There are other applications running in the company that are working fine but they are using full .NET framework and an older version like 2.5.3. Any help regarding the issue would be appreciated.

2

@ibrahim1 -

This can happen when a connection is closed by something between the client app and the cluster. It may or may not be a problem; the client app can always retry when it recieves this message.

You’ll want to enable logging and see how often it happens and perhaps what leads up to the connection (socket) closing.

You’ll also want to post your configuration.

-Jeff

3

@jmorris I implemented our own logging in the repository which goes to Kibana that uses the sdk and we can see the error happening every few seconds. I will implement the logging you mentioned as well.
Following is our configuration:

{
  "couchbaseConfiguration": {
    "clusters": [
      {
        "name": "Couchbase-B",
        "servers": [ "http://bk-agmcc-1b01.site.local" ],
        "userName": "sa",
        "password": "123",
        "bucketPasswordRequired": true,
        "bucketName": "test",
        "poolMaxSize": 50,
        "sendTimeout": 200,
        "connectTimeout": 1000
      },
      {
        "name": "Couchbase-A",
        "servers": [ "http://bk-agmcc-1a01.site.local" ],
        "userName": "sa",
        "password": "123",
        "bucketPasswordRequired": true,
        "bucketName": "test",
        "poolMaxSize": 50,
        "sendTimeout": 200,
        "connectTimeout": 1000
      }
    ]
  }
}

And following is how we initialize a cluster inside the app:

var cluster = new Cluster(new ClientConfiguration()
{
    Servers = ConnectionConfiguration.Servers.Select(x => new System.Uri(x)).ToList(),
    PoolConfiguration = new PoolConfiguration(null)
    {
        MaxSize = ConnectionConfiguration.PoolMaxSize,
        SendTimeout = ConnectionConfiguration.SendTimeout,
        ConnectTimeout = ConnectionConfiguration.ConnectTimeout
    },
    ConnectionPoolCreator = ConnectionPoolFactory.GetFactory<CouchbaseConnectionPool<MultiplexingConnection>>()
});
4

@ibrahim1 -

Why are you setting your sendTimeout to 200ms? That is really short and any hiccup will cause a timeout. I would leave SendTimeout and ConnectTimeout as defaults and probably lower the MaxSize to maybe 5 or 10; in most cases that is sufficient and the higher the MaxSize, the longer it will take to bootstrap the SDK.

-Jeff

1 Like
5

@jmorris Thank you for your input. Changes in the config seem to have solved some of the problems. Regarding the logging, sometimes the logging throw an exception saying:

[17:19:03]	[Step 4/10] fail: Couchbase.Authentication.SASL.PlainTextMechanism[0]
[17:19:03]	[Step 4/10] System.ObjectDisposedException: Cannot access a disposed object.
[17:19:03]	[Step 4/10] Object name: 'LoggerFactory'.
[17:19:03]	[Step 4/10]    at Microsoft.Extensions.Logging.LoggerFactory.CreateLogger(String categoryName)
[17:19:03]	[Step 4/10]    at Microsoft.Extensions.Logging.LoggerFactoryExtensions.CreateLogger(ILoggerFactory factory, Type type)
[17:19:03]	[Step 4/10]    at Couchbase.Logging.LogManager.GetLogger(Type type) in C:\Jenkins\workspace\dotnet\couchbase-net-client-scripted-build-pipeline\couchbase-net-client\Src\Couchbase\Logging\LogManager.cs:line 36
[17:19:03]	[Step 4/10]    at Couchbase.IO.Operations.OperationBase..ctor(String key, IVBucket vBucket, ITypeTranscoder transcoder, UInt32 opaque, UInt32 timeout) in C:\Jenkins\workspace\dotnet\couchbase-net-client-scripted-build-pipeline\couchbase-net-client\Src\Couchbase\IO\Operations\OperationBase.cs:line 23

I created a custom ILogger that sends logs to our logging service and added that to the ILoggerFactory from Startup.

6

@ibrahim1 -

I am not 100% sure, but perhaps your disposing the client while your still using it?

-Jeff