Authentication failed for user when bootstrapped node is removed from cluster

.NET SDK
1

Hi there,

I’m on .NET SDK client v2.6.2 with server v5.5.1 and seeing some interesting errors.

Currently testing with a 4 node cluster, nodes 1-3 are listed in the bootstrap (this is to reproduce production configuration). Using SaslPlain and MUXIO. The application is a .Net WebAPI IIS hosted application on .net framework 4.6.1.
One odd detail, there are libraries this application that reference the 2.5.4 couchbase client, the application has been updated to use 2.6.2 and we have a binding redirect in place. I’ve confirmed from SDK logging that the client on startup is recognized as 2.6.2. We are using the old authentication style from server version 4.x, so during testing our username is the bucketname and password is the old bucket password. The users are in Role “Application Access” used for the upgrade path from lower server versions.

Config example here:

<couchbaseClients>
		<couchbase forceSaslPlain="true">
			<servers>
				<add uri="http://node1:8091" />
				<add uri="http://node2:8091" />
				<add uri="http://node3:8091" />
			</servers>
			<buckets>
				<add name="bucket1" password="password1" useSsl="false">
					<connectionPool name="custom" maxSize="3" minSize="1" waitTimeout="5000" shutdownTimeout="3000">
					</connectionPool>
				</add>
				<add name="bucket2" password="password2" useSsl="false">
					<connectionPool name="custom" maxSize="3" minSize="1" waitTimeout="5000" shutdownTimeout="3000">
					</connectionPool>
				</add>
			</buckets>
		</couchbase>
	</couchbaseClients>

The Issue and Observations:

  • When failing over any node application is throwing an exception: Authentication failed for user ‘bucket1’ or Authentication failed for user ‘bucket2’.
  • This seems to mainly be happening on GET operations, although I can’t confirm if it is isolated to only GET operations or others like UPSERT are affected as well
  • These errors continually happen until the node is brought back into the cluster and rebalance is completed

Relevant SDK Logs:

2018-09-25 19:53:32,062 [373] DEBUG Couchbase.Core.Server - Sending Get1 with key document_key_1 using server 10.48.13.72:11210 2018-09-25 19:53:32,062 [373] DEBUG Couchbase.IO.SharedConnectionPool1[[Couchbase.IO.MultiplexingConnection, Couchbase.NetClient, Version=2.6.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - Trying to acquire new connection! Refs=2
2018-09-25 19:53:32,062 [373] DEBUG Couchbase.IO.ConnectionPoolBase1[[Couchbase.IO.MultiplexingConnection, Couchbase.NetClient, Version=2.6.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - 1. Checking authentication [False|False]: 10.48.13.72:11210 - b88d62cf-92fe-4bce-b6db-0416a0cb4c5e 2018-09-25 19:53:32,062 [373] DEBUG Couchbase.IO.ConnectionPoolBase1[[Couchbase.IO.MultiplexingConnection, Couchbase.NetClient, Version=2.6.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - 2. Checking authentication [False|False]: 10.48.13.72:11210 - b88d62cf-92fe-4bce-b6db-0416a0cb4c5e
2018-09-25 19:53:32,062 [373] DEBUG Couchbase.IO.ConnectionPoolBase1[[Couchbase.IO.MultiplexingConnection, Couchbase.NetClient, Version=2.6.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - 3. Checking authentication [False]: 10.48.13.72:11210 - b88d62cf-92fe-4bce-b6db-0416a0cb4c5e 2018-09-25 19:53:32,062 [373] DEBUG Couchbase.Authentication.SASL.PlainTextMechanism - Authenticating socket b88d62cf-92fe-4bce-b6db-0416a0cb4c5e 2018-09-25 19:53:32,062 [373] DEBUG Couchbase.Authentication.SASL.PlainTextMechanism - Authentication for socket b88d62cf-92fe-4bce-b6db-0416a0cb4c5e failed: 2018-09-25 19:53:32,062 [373] INFO Couchbase.IO.ConnectionPoolBase1[[Couchbase.IO.MultiplexingConnection, Couchbase.NetClient, Version=2.6.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - 4. Could not authenticate bucket1 using Couchbase.Authentication.SASL.PlainTextMechanism - 4aa5c5a0-deb8-44a3-803c-362740897c7a [10.48.13.72:11210].
2018-09-25 19:53:32,062 [373] DEBUG Couchbase.IO.ConnectionBase - Disposing b88d62cf-92fe-4bce-b6db-0416a0cb4c5e
2018-09-25 19:53:32,062 [373] INFO Couchbase.IO.ConnectionBase - Closing connection b88d62cf-92fe-4bce-b6db-0416a0cb4c5e
2018-09-25 19:53:32,062 [373] DEBUG Couchbase.Core.Buckets.CouchbaseRequestExecuter - Operation doesn’t support retries for key document_key_1
2018-09-25 19:53:32,062 [373] DEBUG Couchbase.Core.Buckets.RequestExecuterBase - Operation for key document_key_1 failed after 1 retries using vb871 from rev21362 and opaque22467. Reason: Authentication failed for user ‘bucket1’

2

@brian.hulse -

First, this sounds confusing but “Authentication failed for user ‘bucket1’” may mean the connection itself just couldn’t be made while authenticating, thus the failure. The logged message is somewhat misleading in that it may not exactly mean that the user/pass were wrong. That being said, It could definitly indicate an issue if the cluster map is stale.

Can you DM me an entire log file from start to finish? It will help me isolate if there is an issue.

I would suggest you remove these settings and just use the default.

Jeff

4

Yeah I have no idea how to DM you on this forum, sorry! help link anywhere?

5

One other thing, do you rebalance the cluster immediatly? For the client to adjust to the new topology you will have to do this.

6

Go into your profile on the top left > Messages > New Message and use my handle.

7

One time we left the cluster unbalanced for a few minutes and then kicked off the rebalanced. On another test we rebalanced immediately. We had the same results, even after rebalance the authentication failures persisted.

8

Somehow I still don’t see it, this is making me feel like an idiot haha. Can you DM me and I’ll respond? seems like that is in the Messages UI for me.