Bootstrap Error on .NET 5 and .NET 6

.NET SDK
1

I know .NET 6 isn’t officially supported yet, and this post is more to bring awareness to the issue so that it can be looked at when support for .NET 6 is added.

I downloaded the first release of .NET 6 (6.0.0) and tried it with SDK 3.2.4. It works in one situation, but not another:

  1. .Net 6 Windows Console Application With SDK 3.2.4 connecting to local 6.6 Couchbase Server ==> Able to Bootstrap + Issue Queries

  2. .Net 6 Alpine (3.14) Docker Container Application Hosted On Kubernetes connecting to cloud hosted 6.6 Couchbase Server ==> Bootstrap error below (1 occurrence each for Couchbase, Memcached, and Ephemeral):

“Cannot bootstrap bucket THEBUCKET as Couchbase. System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|187_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize)
at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte reAuthenticationData, Boolean isApm)
at Couchbase.Core.IO.Connections.ConnectionFactory.CreateAndConnectAsync(IPEndPoint endPoint, HostEndpoint hostEndpoint, CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.ConnectionPoolBase.CreateConnectionAsync(CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.<>c__DisplayClass29_0.<g__StartConnection|0>d.MoveNext()
— End of stack trace from previous location —
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.AddConnectionsAsync(Int32 count, CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.InitializeAsync(CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.InitializeAsync()
at Couchbase.Core.DI.ClusterNodeFactory.CreateAndConnectAsync(HostEndpoint endPoint, BucketType bucketType, NodeAdapter nodeAdapter, CancellationToken cancellationToken)
at Couchbase.Core.ClusterContext.CreateAndBootStrapBucketAsync(String name, HostEndpoint endpoint, BucketType type)
at Couchbase.Core.ClusterContext.GetOrCreateBucketAsync(String name)”

Unsure what differences in the environments/systems are causing the issue. Both setups are working fine with .Net Core 3.1 + SDK 3.2.4

Unsure if related/the same as this issue: .NETCore App 6 compatible

2

Thanks @obawin for brining this to our notice, @Richard_Ponton something to keep in mind.

3

@obawin

It appears you’re connecting with SSL to the Couchbase cluster, and the problem is happening somewhere in the SSL stream. .NET Core implements this using openssl on Linux under the hood, I believe. I’m wondering if there’s something about using the Alpine distribution of Linux and the version of openssl it has.

Have you tried another distro as the base for your Docker image? The default .NET 6.0 image is Debian 11 and includes openssl 1.1.1k. The Alpine image doesn’t include the openssl command-line, but appears to include libssl and libcrypto 1.1.1l.

4

Unfortunately, we are pretty firmly locked in with Alpine (chosen due to its small size, security focused approach), and we probably won’t be able to change unless there is a major reason to. It’s possible other distros have no problems, as Alpine does have some quirks given its minimalistic approach to things.

5

Hi,

Decided to run another test, and I can confirm that the same error happens on .NET 5 and SDK 3.2.4.

The variations I have tested are:

  1. .NET Core 3.1 + SDK 3.2.4 + Alpine 3.13 ==> OK

  2. .NET 5.0 + SDK 3.2.4 + Alpine 3.13 ==> Bootstrap Error as above

  3. .NET 6.0 + SDK 3.2.4 + Alpine 3.13 or Alpine 3.14 ==> Bootstrap Error as above

6

Did some digging, and I believe this is the cause of the issue in .NET 5 and .NET 6:

I assume Couchbase currently doesn’t support the most modern/restrictive ciphers yet?

Given that, the 2 options available seem to be:

  • modify the openssl config on the alpine image
    or
  • Explicitly configure some working values for CipherSuitePolicy and SslStream objects in the Couchbase SDK code

Would it be a good idea to consider implementing option #2 in a future release? The worry about option #1 is that many (most?) SDK Linux .Net5+ users will be affected, and it might be too big a hurdle/responsibility for them to have to go into the openssl configuration levels?

Perhaps a mixture of the 2 options (and a good balance of both) is to create a new option that accepts a list of allowed Cipher suites, and if explicit ones are specified, those are used in the SslStream calls. The list of available cipher suites is available via this enum:

This would give users the ability to configure/customize the used cipher suites at the SDK level, without having to alter the Alpine openssl config/defaults.

From my perspective, this would be more secure as well as we use the Alpine image for different types of containerized services. If the fix for this issue happens at the open ssl config level, images for services that don’t use Couchbase would also be affected. Even for images for services that use Couchbase, ideally the Cipher suite restriction wouldn’t be loosened on the entire image. Having the ability to change/fix at the SDK level limits the cipher change to the service that is communicating with CB.

@btburnett3 , @jmorris , what are your thoughts?

1 Like
7

I agree with your assessment, based on my research. I think we should probably expose configuration options for .NET Core 3.1 and later (the first version they are supported by HttpClient/SocketsHttpHandler and SslStream). We should also consider adjusting the defaults for greater out-of-the-box compatibility, but that has risks and needs careful consideration.

I filed this issue for tracking: [NCBC-3019] Allow SSL cipher configuration - Couchbase database

1 Like
8

Thanks @btburnett3 ! Looking forward to an upcoming release with this change, so that we can start testing SDK 3.2.4 and .NET6!

1 Like
9

hi @btburnett3 ,

I did some prototyping/testing and it looks like what’s important for our environment setup is being able to set the Enabled SslProtocols (the out-of-the box Cipher Suites in Net 6 worked fine).

Specifically, the CB SDK has TLS 1.0, 1.1, and 1.2 enabled by default. Looks like in this configuration, the lowest (available) may be used/tried. Our environment has anything lower than 1.2 disabled, which is the reason for the error. When I change the code and configure TLS 1.2 and 1.3 as enabled, the CB SDK works good on NET 6 (KV query success).

The changes I made to the CD SDK are:

  1. ClusterOptions.cs:

     /// <summary>
 /// Enabled SSL Protocols
 /// </summary>
 public SslProtocols EnabledSslProtocols { get; set; } = SslProtocols.Tls | SslProtocols.Tls11 | SslProtocols.Tls12;

#if NETCOREAPP3_1_OR_GREATER
///


/// List of supported TLS Cipher Suites. If not set, will use default .NET Cipher Suites
///
public TlsCipherSuite? SupportedTLSCipherSuites { get; set; } = null;
#endif

  1. ConnectionFactory.cs:

             //create the sslstream with appropriate authentication

#if !NETCOREAPP3_1_OR_GREATER
await sslStream.AuthenticateAsClientAsync(targetHost, certs,
_clusterOptions.EnabledSslProtocols,
_clusterOptions.EnableCertificateRevocation)
.ConfigureAwait(false);
#else
SslClientAuthenticationOptions sslOptions = new SslClientAuthenticationOptions()
{
TargetHost = targetHost,
ClientCertificates = certs,
EnabledSslProtocols = _clusterOptions.EnabledSslProtocols,
CertificateRevocationCheckMode = _clusterOptions.EnableCertificateRevocation ? X509RevocationMode.Online : X509RevocationMode.NoCheck,
CipherSuitesPolicy = _clusterOptions.SupportedTLSCipherSuites != null && _clusterOptions.SupportedTLSCipherSuites.Length > 0
? new CipherSuitesPolicy(_clusterOptions.SupportedTLSCipherSuites) : null
};
await sslStream.AuthenticateAsClientAsync(sslOptions)
.ConfigureAwait(false);
#endif

  1. CouchbaseHttpClientFactory.cs:
    private HttpMessageHandler CreateClientHandler()
    {
    #if !NETCOREAPP3_1_OR_GREATER
    …
    handler.SslProtocols = _context.ClusterOptions.EnabledSslProtocols;
    …
    #else
    //for x509 cert authentication
    if (_context.ClusterOptions.X509CertificateFactory != null)
    {
    handler.SslOptions.EnabledSslProtocols = _context.ClusterOptions.EnabledSslProtocols;
    …
    }

         ...
 	
     if (_context.ClusterOptions.SupportedTLSCipherSuites != null && _context.ClusterOptions.SupportedTLSCipherSuites.Length > 0)
     {
         handler.SslOptions.CipherSuitesPolicy = new CipherSuitesPolicy(_context.ClusterOptions.SupportedTLSCipherSuites);
     }

#endif
…

As mentioned, for the test, I only had to set options.EnabledSslProtocols = SslProtocols.Tls13 | SslProtocols.Tls12, to have it work. However, I think there is still value to have options.SupportedTLSCipherSuites as configurable too.

Is it possible to have (similar) changes as above for 3.2.5?

PS: One point of possible interest is that when I turn on SDK logging, I see this message many times:

“Error processing new clusterOptions System.ArgumentException: Comparing the same configs is not allowed. at Couchbase.Core.Configuration.Server.BucketConfigExtensions.IsNewerThan(BucketConfig newConfig, BucketConfig oldConfig) in … at Couchbase.CouchbaseBucket.ConfigUpdatedAsync(BucketConfig newConfig)”

Unsure if I added the new cluster options (EnabledSslProtocols and SupportedTLSCipherSuites) correctly/in all places (I followed how “ForceIpAsTargetHost” was implemented) and that may be causing the above?

1 Like
10

Actually, the error:

“Error processing new clusterOptions System.ArgumentException: Comparing the same configs is not allowed. at Couchbase.Core.Configuration.Server.BucketConfigExtensions.IsNewerThan(BucketConfig newConfig, BucketConfig oldConfig)”

seems like an existing issue. When logging is enabled, I see the error message in the SDK even before making the test SSL/Cipher config changes above.

It seems somewhat random when the error starts (saw it happen when service first started, but also only after first request/processing was performed). Once it starts, it logs the error messsage twice every 2-3s and seems to continue indefinitely.

11

@obawin

The ArgumentException error is a separate known issue: Loading...

As to getting SslProtocols exposed in 3.2.5, I don’t want to speak for the Couchbase SDK team, I’m just a very active community member. However, I do know that it’s the holidays here in the US right now, they are very busy on several items, and I believe the cutoff for 3.2.5 is in about a week, give or take. And I’m currently focusing my efforts on System.Text.Json support.

Therefore, I don’ think 3.2.5 is very likely. That said, community contributions to the SDK are welcome. If you wanted to polish up your experiment and put in a CR at https://review.couchbase.org/ it could get completed sooner.

1 Like
12

K, I think I got the changes/review properly pushed/created: https://review.couchbase.org/c/couchbase-net-client/+/166541

1 Like
13

@obawin

Gerrit is actually a bit funny, takes some getting used to. You should actually make your changes by amending the original commit and pushing it again. Unlike working in GitHub, this doesn’t overwrite the history of changes. Gerrit works by seeing the same Change-Id at the bottom of the commit message and links it to your original changeset. It then shows it as a new change with diffs from the previous one.

14

Thanks @btburnett3 .

Trying to get used to the Gerrit flow, but don’t think i have a really good hang of it (I usually only use Git via the Visual Studio interface with very minimal command-line usage).

Sorry, I’m a bit lost on what to do next. I thought I had pushed the changes and triggered another round of reviews on the original review page (166541), but when I logged in today, I saw that I had to click “Reply” on the code review comment responses.

I did that, then noticed there was a “Merge” conflict. Following the onscreen help/notes, I rebased, reapplied changes, then followed the patch instructions:

a) Pull down latest patchset (use the DOWNLOAD & Checkout command on the right hand side)
b) Make your changes
c) Do a git add .
d) Do a git commit --amend
e) Do a git push origin HEAD:refs/for/master

but it seems to create a new review instead of pushing the changes to this one (166541) (I’m probably doing something wrong)?

The original review 166541 still doesn’t have my pushed changes (var rename, etc.), but the new review ( 166625) does. Might be cleaner to just start fresh with the new review, so I added you and Jeff as reviewers again (sorry for the trouble!).

https://review.couchbase.org/c/couchbase-net-client/+/166625

If you guys are ok with using the new review, I’ll go and abandon the other one later?

15

I’ve reviewed the new CR and it looks good. For next time, when you do the git commit --amend make sure you keep the same Change-Id: XXXX line at the end of the commit msg. Leaving this the same and matching the original commit is what makes it update the existing review.

1 Like
16

Got it!

What steps would I need to do (if any) to close up the review/merge the code?

17

@obawin -

Looks good, thanks for the contribution! One minor change would be to make the commit message follow this format:

Motivation
----------
//Summary of reason for change

Modifications
-------------
//list of modifications

Result
------
//Behavior changes (if any) after the patch

This is consistent with convention for the repo.

Jeff

18

We will merge it on our side, but you can make as many commit amends as needed (the final needs to be reviewed). I’ll hold off on merging for a bit in case you want to make changes. Note if you add “WIP” to the title, then it won’t be merged.

19

I updated the commit message to have the standard format / info.

I don’t have anymore changes planned, so you can merge at your convenience.

Thanks Jeff and Brant for the help/review!

Btw, great work on CB SDK 3.X. I can see the amount of effort and the great improvements on it!

2 Likes
20

Thanks again @obawin!