During the rebalancing of Couchbase(v6.5.1) cluster, .NET SDK v3.5.3 reported an error of "NotMyVBucket". Is this normal?

[erwa]

During the rebalancing of Couchbase(Community Edition 6.5.1) cluster, .NET SDK v3.5.3 reported an error of “NotMyVBucket”. Is it normal?

We have a three-node Couchbase cluster. when performing a rebalancing cluster.Our program sdk reported errors like below:

ReadWriteTest Error, Key: llt_test_424, ErrorCount:263, Percent:0.290411987500138%, Exception:
System.InvalidOperationException: No config body after NotMyVBucket, even though DedupeNotMyVBucket was not enabled.
   at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
   at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
   at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
   at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
   at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
   at ConsoleApp2.Program.ReadWriteTest[T](ICouchbaseCollection collection, String key, T value) in e:\tmp\ConsoleApp2\Program.cs:line 124
ReadWriteTest Error, Key: llt_test_424, ErrorCount:264, Percent:0.291442197297536%, Exception:
System.InvalidOperationException: No config body after NotMyVBucket, even though DedupeNotMyVBucket was not enabled.
   at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
   at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
   at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
   at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
   at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
   at ConsoleApp2.Program.ReadWriteTest[T](ICouchbaseCollection collection, String key, T value) in e:\tmp\ConsoleApp2\Program.cs:line 124
ReadWriteTest Error, Key: llt_test_424, ErrorCount:265, Percent:0.29240733997594537%, Exception:
System.InvalidOperationException: No config body after NotMyVBucket, even though DedupeNotMyVBucket was not enabled.
   at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
   at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
   at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
   at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
   at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
   at ConsoleApp2.Program.ReadWriteTest[T](ICouchbaseCollection collection, String key, T value) in e:\tmp\ConsoleApp2\Program.cs:line 124

[erwa]

Here is my test code

using Couchbase;
using Couchbase.Core.IO.Transcoders;
using Couchbase.KeyValue;
using Newtonsoft.Json;
using ProtoBuf;
using System;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading.Tasks;

namespace ConsoleApp2
{
    class Program
    {
        private static readonly Random Random = new Random();

        private static long _errorCount = 0;
        private static long _queryCount = 0;

        static void Main(string[] args)
        {
            TestCouchbase("10.227.165.4,10.227.165.5,10.227.165.6", "Administrator", "123456").ConfigureAwait(false).GetAwaiter().GetResult();
            Console.WriteLine("Totally Finish.");
            Console.ReadLine();
        }

        private static async Task TestCouchbase(string path, string username, string password)
        {
            Console.WriteLine("Path: " + path);

            try
            {
                Stopwatch sw = Stopwatch.StartNew();
                var clusterOptions = new ClusterOptions()
                    .WithConnectionString($"couchbase://{path}")
                    .WithCredentials(username, password);

                await using var cluster = await Cluster.ConnectAsync(clusterOptions);
                await using var bucket = await cluster.BucketAsync("b_bucket");
                var collection = await bucket.DefaultCollectionAsync();
                sw.Stop();
                Console.WriteLine("Connection Elapsed: " + sw.Elapsed);

                await ParallelTest(collection);
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
            finally
            {
                Console.WriteLine(path + " finish.\r\n");
            }
        }

        private static async Task SingleTest(ICouchbaseCollection collection)
        {
            try
            {
                const string key = "CouchbaseNetClient_Update_Test_Key";
                var value = $"CouchbaseNetClient_Update_Test_{DateTime.Now}";
                await ReadWriteTest(collection, key, value);
            }
            catch (Exception e)
            {
                Console.WriteLine("SingleTest Error: " + e);
            }
        }

        private static async Task SingleObjectTest(ICouchbaseCollection collection)
        {
            try
            {
                const string key = "CouchbaseNetClient_Object_Test_Key";
                var value = GetNewWriteObject();
                await ReadWriteTest(collection, key, value);
            }
            catch (Exception e)
            {
                Console.WriteLine("SingleTest Error: " + e);
            }
        }

        private static async Task ParallelTest(ICouchbaseCollection collection)
        {
            try
            {
                int num = 10000;
                ParallelOptions options = new ParallelOptions() { MaxDegreeOfParallelism = 50 };
                int[] indexes = Enumerable.Range(0, num).ToArray();

                while (true)
                {
                    await Parallel.ForEachAsync(indexes, options, async (i, _) =>
                    {
                        var locNow = i.ToString();
                        for (int j = 0; j < 200; j++)
                        {
                            var key = "llt_test_" + locNow;
                            var value = GetNewWriteObject();
                            await ReadWriteTest(collection, key, value);
                        }
                    });
                }
            }
            catch (Exception e)
            {
                Console.WriteLine("ParallelTest Error: " + e);
            }
        }

        private static async Task ReadWriteTest<T>(ICouchbaseCollection collection, string key, T value)
        {
            try
            {
                await collection.UpsertAsync(key, value);
                using var getResult = await collection.GetAsync(key);
                _queryCount++;
                var result = getResult.ContentAs<T>();
                // Console.WriteLine(result is string ? result : JsonConvert.SerializeObject(result));
            }
            catch (Exception e)
            {
                _errorCount++;
                Console.WriteLine($"ReadWriteTest Error, Key: {key}, ErrorCount:{_errorCount}, Percent:{100.0 * _errorCount / _queryCount}%, Exception:\r\n" + e);
            }
        }

        private static WriteObject GetNewWriteObject()
        {
            var indexVal = Random.Next(10000);
            return new WriteObject
            {
                Index = indexVal,
            };
        }

        private static async Task PbTest(ICouchbaseCollection collection)
        {
            try
            {
                const string key = "CouchbaseNetClient_Pb_Test_Key";
                var pbObject = GetNewPbObject();
                var pbData = PbUtil.Serialize(pbObject);
                await collection.UpsertAsync(key, pbData, new UpsertOptions().Transcoder(new RawBinaryTranscoder()));
                using var getResult = await collection.GetAsync(key, new GetOptions().Transcoder(new RawBinaryTranscoder()));
                var getData = getResult.ContentAs<byte[]>();
                var result = PbUtil.Deserialize<PbObject>(getData);
                Console.WriteLine(result is string ? result : JsonConvert.SerializeObject(result));
            }
            catch (Exception e)
            {
                Console.WriteLine("PbTest Error: " + e);
            }
        }

        private static PbObject GetNewPbObject()
        {
            var indexVal = Random.Next(10000);
            return new PbObject()
            {
                Id = indexVal,
                Name = $"Index_{indexVal}"
            };
        }

        public class WriteObject
        {
            public long? Index;
        }

        public class PbUtil
        {
            public static byte[] Serialize<T>(T obj)
            {
                if (obj == null) return null;

                using (var ms = new MemoryStream())
                {
                    Serializer.Serialize(ms, obj);
                    return ms.ToArray();
                }
            }
            public static T Deserialize<T>(byte[] bytes)
            {
                if (bytes == null) return default(T);

                using (var ms = new MemoryStream(bytes))
                {
                    return Serializer.Deserialize<T>(ms);
                }
            }
        }
        [ProtoContract]
        public class PbObject
        {
            [ProtoMember(1)]
            public long Id { get; set; }

            [ProtoMember(2)]
            public string Name { get; set; }

            [ProtoMember(3)]
            public DateTime Time { get; set; } = DateTime.Now;
        }
    }
}

[mreiche]

NotMyVBucket will occur during a rebalance between the time the vbuckets move from being active on one node to another node and the time the configuration is updated. If the configuration is updated quickly (in less time than the operation timeout), the SDK should retry the operation and it should succeed. If the configuration is not updated quickly, the operation will fail.

During the rebalancing of Couchbase(Community Edition 6.5.1) cluster, .NET SDK v3.5.3 reported an error of “NotMyVBucket”.

Server 6.x is no longer supported, and improvements in configuration updates have been made since .NET SDK 3.5.3 (especially .NET SDK 3.6.1). Migrating to server 7.6 and .NET SDK 3.6.1 will handle rebalances much more smoothly.