Java SDK connection with Certificate Authentication issue

Java SDK
,
1

Hi,
I had configured certificate-based authentication for my Couchbase server (Enterprise Edition 7.0.3) by following all the steps from the link below
https://docs.couchbase.com/java-sdk/current/concept-docs/certificate-based-authentication.html

I try to use the latest Java SDK driver (3.2.6) to connect with certificate-based authentication.
But I am getting AmbiguousTimeoutException with “retryReasons”:[“GLOBAL_CONFIG_LOAD_IN_PROGRESS”]

Can someone please give me some pointer how to fix this?
Also, how to run SDK doctor to test the connection with certificate-based authentication?

The connection will be successful if I disabled TLS or enabled TLS but use user/pw authentication.

Here’s the simple code I use for testing (please note, its connecting to a single node cluster):

public class SSLcertAuth {

public static void main(String[] args) {
	
Path ksPath = Paths.get("C:\\temp\\SSL\\jssecacerts");
String ksPw ="xxxx";
Optional<String> ksType = Optional.of("JKS");		
	
	ClusterEnvironment env = ClusterEnvironment.builder()
	.securityConfig(SecurityConfig.enableTls(true).trustStore(ksPath, ksPw, ksType))
	.timeoutConfig(TimeoutConfig.builder()
            .connectTimeout(Duration.ofSeconds(60L))
            .queryTimeout(Duration.ofSeconds(30L))
            .kvTimeout(Duration.ofSeconds(30L)))
		.build();
	
	CertificateAuthenticator authenticator = CertificateAuthenticator.fromKeyStore(ksPath, ksPw, ksType);
Cluster cluster = Cluster.connect("xxxx.com:11208", ClusterOptions.clusterOptions(authenticator).environment(env));

     // Call the query() method on the cluster object and store the result.
     QueryResult result = cluster.query("select \"Hello World\" as greeting");
     System.out.println(result.rowsAsObject());
     
	cluster.disconnect();
	env.shutdown();
   }

}

Here’s the error log:

Apr 08, 2022 11:59:14 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0xcaca627800000001”,“managerPort”:“8091”,“remote”:“xxxx.com”}
Exception in thread “main” com.couchbase.client.core.error.AmbiguousTimeoutException: QueryRequest, Reason: TIMEOUT {“cancelled”:true,“completed”:true,“coreId”:“0xcaca627800000001”,“idempotent”:false,“reason”:“TIMEOUT”,“requestId”:2,“requestType”:“QueryRequest”,“retried”:68,“retryReasons”:[“GLOBAL_CONFIG_LOAD_IN_PROGRESS”],“service”:{“operationId”:“ff2b5680-64bf-415f-a324-3a7516602036”,“statement”:“select “Hello World” as greeting”,“type”:“query”},“timeoutMs”:30000,“timings”:{“totalMicros”:30012865}}
at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
at com.couchbase.client.java.Cluster.query(Cluster.java:417)
at com.couchbase.client.java.Cluster.query(Cluster.java:404)
at SSLcertAuth.main(SSLcertAuth.java:35)
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:184)
at com.couchbase.client.core.msg.Request.cancel(Request.java:70)
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
at com.couchbase.client.core.Timer$$Lambda$155/0000000010CB6900.run(Unknown Source)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
at com.couchbase.client.core.deps.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:820)

Any help would be greatly appreciated!
Thanks!

2

Hi Winnie,

It looks like you’re storing the Couchbase cluster’s trusted root CA certificate and your client identity certificate in the same keystore. Can you try storing them in separate keystores?

Thanks,
David

3

Hi David,
Yup! I had tried to put the root CA certificate and the Client certificate into two different keystore files before and I had just tired it again. But still failed with the same AmbiguousTimeoutException.

is there any waitTimeOut setting for load global config?

I saw this bug about load global config that just got fixed in the new Java SDK drivers. Not sure if its related.
https://issues.couchbase.com/browse/JVMCBC-1046

4

is there any waitTimeOut setting for load global config?

There is, you can use this:

  cluster.waitUntilReady(Duration.ofSeconds(30));

But it’s unlikely to make a difference here as this query has already been waiting for a 30 second timeout.
It’s not clear to me why the non-secure manager port 8091 is being used when you’ve specified a secure connection, but I suspect it’s related to your issue:

“managerPort”:“8091”

I wonder if it’s related to you specifying a port directly like this:

xxxx.com:11208

Is there a reason for that? Are you going through a load balancer or similar?

You might want to try using the Cluster.connect() overload that lets you specify SeedNode classes directly.

5

Hi Graham,
1.) The reason that I have to specify the KV port directly is because my Couchbase server is configured to use custom KV ports. But the ssl manage port it is still the default port(18091). Not sure why it is trying to use port 8091 when TLS is enabled.

Here’s the list of Ports that is configured on my system:
{rest_port, 9000}.
{ssl_rest_port, 18091}.
{capi_port, 9001}.
{ssl_capi_port, 18098}.
{query_port, 9002}.
{ssl_query_port, 18099}.
{fts_http_port, 9003}.
{fts_ssl_port, 18090}.
{cbas_http_port, 9004}.
{cbas_ssl_port, 18190}.
{eventing_http_port, 9005}.
{eventing_ssl_port, 18191}.
{memcached_port, 11214}.
{memcached_ssl_port, 11208}.

2.) tried using the SeedNote classes, but got the same AmbiguousTimeoutException with “retryReasons”:[“GLOBAL_CONFIG_LOAD_IN_PROGRESS”]

Here’s the code I tried:

	Set<SeedNode> seedNodes = new HashSet<>(Collections.singletonList(
			   SeedNode.create("xxxx.com", Optional.of(11208),Optional.of(18091)))  
			 );
	
	CertificateAuthenticator authenticator = CertificateAuthenticator.fromKeyStore(ksPath, ksPw, ksType);
	Cluster cluster = Cluster.connect(seedNodes, ClusterOptions.clusterOptions(authenticator).environment(env));

Here’s the full output:

Apr 11, 2022 9:57:30 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][CoreCreatedEvent] {“clientVersion”:“3.2.6”,“clientGitHash”:"${buildNumber}",“coreVersion”:“2.2.6”,“coreGitHash”:"${buildNumber}",“userAgent”:“couchbase-java/3.2.6 (Windows 10 10.0 amd64; IBM J9 VM 8.0.6.7 - pwa6480sr6fp7-20200312_01(SR6 FP7))”,“maxNumRequestsInRetry”:32768,“ioEnvironment”:{“nativeIoEnabled”:true,“eventLoopThreadCount”:6,“eventLoopGroups”:[“NioEventLoopGroup”]},“ioConfig”:{“captureTraffic”:,“mutationTokensEnabled”:true,“networkResolution”:“auto”,“dnsSrvEnabled”:true,“tcpKeepAlivesEnabled”:true,“tcpKeepAliveTimeMs”:60000,“configPollIntervalMs”:2500,“kvCircuitBreakerConfig”:“disabled”,“queryCircuitBreakerConfig”:“disabled”,“viewCircuitBreakerConfig”:“disabled”,“searchCircuitBreakerConfig”:“disabled”,“analyticsCircuitBreakerConfig”:“disabled”,“managerCircuitBreakerConfig”:“disabled”,“eventingCircuitBreakerConfig”:“disabled”,“backupCircuitBreakerConfig”:“disabled”,“numKvConnections”:1,“maxHttpConnections”:12,“idleHttpConnectionTimeoutMs”:4500,“configIdleRedialTimeoutMs”:300000,“memcachedHashingStrategy”:“StandardMemcachedHashingStrategy”},“compressionConfig”:{“enabled”:true,“minRatio”:0.83,“minSize”:32},“securityConfig”:{“tlsEnabled”:true,“nativeTlsEnabled”:true,“hostnameVerificationEnabled”:true,“hasTrustCertificates”:false,“trustManagerFactory”:“TrustManagerFactory”,“ciphers”:},“timeoutConfig”:{“kvMs”:30000,“kvDurableMs”:10000,“managementMs”:75000,“queryMs”:30000,“viewMs”:75000,“searchMs”:75000,“analyticsMs”:75000,“connectMs”:60000,“disconnectMs”:10000,“eventingMs”:75000,“backupMs”:75000},“loggerConfig”:{“customLogger”:null,“fallbackToConsole”:false,“consoleLogLevel”:{“name”:“INFO”,“resourceBundleName”:“sun.util.logging.resources.logging”,“localizedName”:“INFO”},“disableSlf4j”:false,“loggerName”:“CouchbaseLogger”,“diagnosticContextEnabled”:false},“orphanReporterConfig”:{“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024,“enabled”:true},“thresholdLoggingTracerConfig”:{“enabled”:true,“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024,“kvThresholdMs”:500,“queryThresholdMs”:1000,“searchThresholdMs”:1000,“analyticsThresholdMs”:1000,“viewThresholdMs”:1000},“loggingMeterConfig”:{“enabled”:true,“emitIntervalMs”:600000},“retryStrategy”:“BestEffortRetryStrategy”,“requestTracer”:“ThresholdLoggingTracer”,“meter”:“LoggingMeter”,“numRequestCallbacks”:0} {“coreId”:“0xc5e0828a00000001”,“seedNodes”:[{“address”:“xxxx.com”,“kvPort”:11208,“mgmtPort”:18091}]}
Apr 11, 2022 9:57:30 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0xc5e0828a00000001”,“managerPort”:“18091”,“remote”:“xxxx.com”}
Exception in thread “main” com.couchbase.client.core.error.AmbiguousTimeoutException: QueryRequest, Reason: TIMEOUT {“cancelled”:true,“completed”:true,“coreId”:“0xc5e0828a00000001”,“idempotent”:false,“reason”:“TIMEOUT”,“requestId”:2,“requestType”:“QueryRequest”,“retried”:68,“retryReasons”:[“GLOBAL_CONFIG_LOAD_IN_PROGRESS”],“service”:{“operationId”:“9c4b32b1-8004-4f79-a5c9-b5cf8129f85f”,“statement”:“select “Hello World” as greeting”,“type”:“query”},“timeoutMs”:30000,“timings”:{“totalMicros”:30019884}}
at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
at com.couchbase.client.java.Cluster.query(Cluster.java:417)
at com.couchbase.client.java.Cluster.query(Cluster.java:404)
at SSLcertAuth.main(SSLcertAuth.java:46)
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:184)
at com.couchbase.client.core.msg.Request.cancel(Request.java:70)
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
at com.couchbase.client.core.Timer$$Lambda$154/00000000118EDF20.run(Unknown Source)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
at com.couchbase.client.core.deps.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:820)

6

Specifying the SeedNodes directly has got it using your manager port at least.
Not sure what else to suggest - if your certificates are fine and your client can reach that port, config should succeed… perhaps my colleague @daschl may have an idea.

7

Hi Graham
Yes, my certificates are good. I had tested the the certificates with the “Require Client Certificate” set to Mandatory. I was able to web login to Couchbase GUI with the certificates.

Hi @daschl, do you have any advise?
Any help would be greatly appreciated!

Thanks!

8

@w_lin I don’t think the issue is with the certs - it looks like it’s timing out(not SSL failing due to certs). Do not provide the custom seed node ports - just use the regular ports and enable TLS (!) in the SecurityConfig.

If you are using custom ports inside the container, those should be configured via alternate addresses and exposed as regular ports through docker.

edit: could you share debug level logging for the failed bootstrap attempt please as well?

9

If I’m not mistaken, this message reports port 8091 even when a different port is being used.

Apr 08, 2022 11:59:14 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0xcaca627800000001”,“managerPort”:“8091”,“remote”:“[xxxx.com](http://xxxx.com/)”}
10

Is xxxx.com:11208 accessible from the client? If its in AWS and its not accessible (because of the AWS network configuration), you’ll get a timeout. You could try accessing it using curl xxxx.com:11208. If it it hangs, then the network configuration will need to allow access. If it gives an odd error like ‘empty response’ then it’s accessible (it’s not http so an error is expected). If it gives “Connection refused”, then nothing is listening on that port.

11

Hi @daschl,
To make things less confusing, I setup another Couchbase system with the default ports and also setup for certificate authentication. For the “Require client Certificate” setting, I choose Enable instead of Mandatory this time.
But the connection still failed. At lease this time it show more Exceptions. So other than the com.couchbase.client.core.error.AmbiguousTimeoutException, it also shows exception when requesting cryptographic services for ssl connection:

Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSslX509TrustManagerWrapper
FINE: Unable to access wrapped TrustManager
java.security.NoSuchProviderException: no such provider: SunJSSE

I am only allow to use IBM Java. The provider for TLS in IBM Java I think is IBMJCE .
Enforcing provider is a bad practice. Even Oracle docs say so here Java Cryptography Architecture Oracle Providers Documentation
Is there a way to work around this?

Here’s the full output:

Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.logging.InternalLoggerFactory useJdkLoggerFactory
FINE: Using java.util.logging as the default logging framework
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector
FINE: -Dcom.couchbase.client.core.deps.io.netty.leakDetection.level: simple
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector
FINE: -Dcom.couchbase.client.core.deps.io.netty.leakDetection.targetRecords: 4
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory newResourceLeakDetector
FINE: Loaded default ResourceLeakDetector: com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector@3b5e8dc0
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0 explicitNoUnsafeCause0
FINE: -Dio.netty.noUnsafe: false
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0 javaVersion0
FINE: Java version: 8
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: sun.misc.Unsafe.theUnsafe: available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: sun.misc.Unsafe.copyMemory: available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: java.nio.Buffer.address: available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: direct buffer constructor: available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: java.nio.Bits.unaligned: available, true
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent0
FINE: java.nio.DirectByteBuffer.(long, int): available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent unsafeUnavailabilityCause0
FINE: sun.misc.Unsafe: available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent maxDirectMemory0
FINE: maxDirectMemory: 536870912 bytes (maybe)
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent tmpdir0
FINE: -Dio.netty.tmpdir: C:\Users\WINNIE~1\AppData\Local\Temp (java.io.tmpdir)
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent bitMode0
FINE: -Dio.netty.bitMode: 64 (sun.arch.data.model)
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent isWindows0
FINE: Platform: Windows
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent
FINE: -Dio.netty.maxDirectMemory: 536870912 bytes
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent
FINE: -Dio.netty.uninitializedArrayAllocationThreshold: -1
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.CleanerJava6
FINE: java.nio.ByteBuffer.cleaner(): available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent
FINE: -Dio.netty.noPreferDirect: false
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.PlatformDependent$Mpsc
FINE: com.couchbase.client.core.deps.org.jctools-core.MpscChunkedArrayQueue: available
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.channel.MultithreadEventLoopGroup
FINE: -Dio.netty.eventLoopThreads: 24
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.InternalThreadLocalMap
FINE: -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.util.internal.InternalThreadLocalMap
FINE: -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop
FINE: -Dio.netty.noKeySetOptimization: false
Apr 28, 2022 1:04:13 AM com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop
FINE: -Dio.netty.selectorAutoRebuildThreshold: 512
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][DnsSrvLookupFailedEvent][14ms] DNS SRV lookup failed (name not found), trying to bootstrap from given hostname directly.
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][CoreCreatedEvent] {“clientVersion”:“3.2.6”,“clientGitHash”:“${buildNumber}”,“coreVersion”:“2.2.6”,“coreGitHash”:“${buildNumber}”,“userAgent”:“couchbase-java/3.2.6 (Windows 10 10.0 amd64; IBM J9 VM 8.0.6.7 - pwa6480sr6fp7-20200312_01(SR6 FP7))”,“maxNumRequestsInRetry”:32768,“ioEnvironment”:{“nativeIoEnabled”:true,“eventLoopThreadCount”:6,“eventLoopGroups”:[“NioEventLoopGroup”]},“ioConfig”:{“captureTraffic”:,“mutationTokensEnabled”:true,“networkResolution”:“auto”,“dnsSrvEnabled”:true,“tcpKeepAlivesEnabled”:true,“tcpKeepAliveTimeMs”:60000,“configPollIntervalMs”:2500,“kvCircuitBreakerConfig”:“disabled”,“queryCircuitBreakerConfig”:“disabled”,“viewCircuitBreakerConfig”:“disabled”,“searchCircuitBreakerConfig”:“disabled”,“analyticsCircuitBreakerConfig”:“disabled”,“managerCircuitBreakerConfig”:“disabled”,“eventingCircuitBreakerConfig”:“disabled”,“backupCircuitBreakerConfig”:“disabled”,“numKvConnections”:1,“maxHttpConnections”:12,“idleHttpConnectionTimeoutMs”:4500,“configIdleRedialTimeoutMs”:300000,“memcachedHashingStrategy”:“StandardMemcachedHashingStrategy”},“compressionConfig”:{“enabled”:true,“minRatio”:0.83,“minSize”:32},“securityConfig”:{“tlsEnabled”:true,“nativeTlsEnabled”:true,“hostnameVerificationEnabled”:true,“hasTrustCertificates”:false,“trustManagerFactory”:“TrustManagerFactory”,“ciphers”:},“timeoutConfig”:{“kvMs”:30000,“kvDurableMs”:10000,“managementMs”:75000,“queryMs”:30000,“viewMs”:75000,“searchMs”:75000,“analyticsMs”:75000,“connectMs”:60000,“disconnectMs”:10000,“eventingMs”:75000,“backupMs”:75000},“loggerConfig”:{“customLogger”:null,“fallbackToConsole”:false,“consoleLogLevel”:{“name”:“INFO”,“resourceBundleName”:“sun.util.logging.resources.logging”,“localizedName”:“INFO”},“disableSlf4j”:false,“loggerName”:“CouchbaseLogger”,“diagnosticContextEnabled”:false},“orphanReporterConfig”:{“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024,“enabled”:true},“thresholdLoggingTracerConfig”:{“enabled”:true,“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024,“kvThresholdMs”:500,“queryThresholdMs”:1000,“searchThresholdMs”:1000,“analyticsThresholdMs”:1000,“viewThresholdMs”:1000},“loggingMeterConfig”:{“enabled”:true,“emitIntervalMs”:600000},“retryStrategy”:“BestEffortRetryStrategy”,“requestTracer”:“ThresholdLoggingTracer”,“meter”:“LoggingMeter”,“numRequestCallbacks”:0} {“coreId”:“0x26fa758b00000001”,“seedNodes”:[{“address”:“sys-rh7x64-db03.isslab.usga.ibm.com”}]}
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0x26fa758b00000001”,“managerPort”:“8091”,“remote”:“xxxxxx.com”}
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.deps.io.netty.channel.DefaultChannelId
FINE: -Dio.netty.processId: 4824 (auto-detected)
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.deps.io.netty.util.NetUtil
FINE: -Djava.net.preferIPv4Stack: false
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.deps.io.netty.util.NetUtil
FINE: -Djava.net.preferIPv6Addresses: false
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.deps.io.netty.util.NetUtilInitializations determineLoopback
FINE: Loopback interface: lo (Software Loopback Interface 1, 127.0.0.1)
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.deps.io.netty.util.NetUtil$1 run
FINE: Failed to get SOMAXCONN from sysctl and file \proc\sys\net\core\somaxconn. Default: 200
Apr 28, 2022 1:04:14 AM com.couchbase.client.core.deps.io.netty.channel.DefaultChannelId
FINE: -Dio.netty.machineId: 00:05:9a:ff:fe:3c:7a:00 (auto-detected)
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.numHeapArenas: 5
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.numDirectArenas: 5
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.pageSize: 8192
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.maxOrder: 11
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.chunkSize: 16777216
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.smallCacheSize: 256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.normalCacheSize: 64
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.maxCachedBufferCapacity: 32768
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.cacheTrimInterval: 8192
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.cacheTrimIntervalMillis: 0
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.useCacheForAllThreads: true
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.PooledByteBufAllocator
FINE: -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.ByteBufUtil
FINE: -Dio.netty.allocator.type: pooled
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.ByteBufUtil
FINE: -Dio.netty.threadLocalDirectBufferSize: 0
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.ByteBufUtil
FINE: -Dio.netty.maxThreadLocalCharBufferSize: 16384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader
FINE: -Dio.netty.native.workdir: C:\Users\WINNIE~1\AppData\Local\Temp (io.netty.tmpdir)
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader
FINE: -Dio.netty.native.deleteLibAfterLoading: true
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader
FINE: -Dio.netty.native.tryPatchShadedId: true
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader
FINE: -Dio.netty.native.detectNativeLibraryDuplicates: true
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader loadLibrary
FINE: Successfully loaded the library C:\Users\WINNIE~1\AppData\Local\Temp\com_couchbase_client_core_deps_netty_tcnative_windows_x86_647304592455165599604.dll
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader loadFirstAvailable
FINE: Loaded library with name ‘netty_tcnative_windows_x86_64’
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSsl
FINE: Initialize netty-tcnative using engine: ‘default’
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSsl
FINE: netty-tcnative using native library: BoringSSL
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.AbstractByteBuf
FINE: -Dcom.couchbase.client.core.deps.io.netty.buffer.checkAccessible: true
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.buffer.AbstractByteBuf
FINE: -Dcom.couchbase.client.core.deps.io.netty.buffer.checkBounds: true
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory newResourceLeakDetector
FINE: Loaded default ResourceLeakDetector: com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector@cb032fe7
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory newResourceLeakDetector
FINE: Loaded default ResourceLeakDetector: com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector@2f2345d
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.Recycler
FINE: -Dio.netty.recycler.maxCapacityPerThread: 4096
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.Recycler
FINE: -Dio.netty.recycler.ratio: 8
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.Recycler
FINE: -Dio.netty.recycler.chunkSize: 32
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.Recycler
FINE: -Dio.netty.recycler.blocking: false
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: TLS_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.CipherSuiteConverter cacheFromOpenSsl
FINE: Cipher suite mapping: SSL_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSsl
FINE: Supported protocols (OpenSSL): [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSsl
FINE: Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256]
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSslX509TrustManagerWrapper
FINE: Unable to access wrapped TrustManager
java.security.NoSuchProviderException: no such provider: SunJSSE
at sun.security.jca.GetInstance.getService(GetInstance.java:95)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:218)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:33)
at com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSslX509TrustManagerWrapper.newSSLContext(OpenSslX509TrustManagerWrapper.java:159)
at com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSslX509TrustManagerWrapper.(OpenSslX509TrustManagerWrapper.java:64)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslContext.chooseTrustManager(ReferenceCountedOpenSslContext.java:627)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslClientContext.newSessionContext(ReferenceCountedOpenSslClientContext.java:153)
at com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSslClientContext.(OpenSslClientContext.java:196)
at com.couchbase.client.core.deps.io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:831)
at com.couchbase.client.core.deps.io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:611)
at com.couchbase.client.core.io.netty.SslHandlerFactory.get(SslHandlerFactory.java:69)
at com.couchbase.client.core.endpoint.BaseEndpoint$1.initChannel(BaseEndpoint.java:312)
at com.couchbase.client.core.deps.io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
at com.couchbase.client.core.deps.io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:820)

Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory newResourceLeakDetector
FINE: Loaded default ResourceLeakDetector: com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector@28f1666d
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory newResourceLeakDetector
FINE: Loaded default ResourceLeakDetector: com.couchbase.client.core.deps.io.netty.util.ResourceLeakDetector@5eb0e496
Apr 28, 2022 1:04:15 AM com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler setHandshakeSuccess
FINE: [id: 0xc5694466, L:/9.65.82.113:51274 - R:xxxxxx.com/9.xx.xx.15:11207] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Exception in thread “main” com.couchbase.client.core.error.AmbiguousTimeoutException: QueryRequest, Reason: TIMEOUT {“cancelled”:true,“completed”:true,“coreId”:“0x26fa758b00000001”,“idempotent”:false,“reason”:“TIMEOUT”,“requestId”:2,“requestType”:“QueryRequest”,“retried”:68,“retryReasons”:[“GLOBAL_CONFIG_LOAD_IN_PROGRESS”],“service”:{“operationId”:“4fc78643-8be7-4408-8edb-335de3226d17”,“statement”:“select "Hello World" as greeting”,“type”:“query”},“timeoutMs”:30000,“timings”:{“totalMicros”:30007840}}
at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
at com.couchbase.client.java.Cluster.query(Cluster.java:417)
at com.couchbase.client.java.Cluster.query(Cluster.java:404)
at SSLcertAuth.main(SSLcertAuth.java:65)
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:184)
at com.couchbase.client.core.msg.Request.cancel(Request.java:70)
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
at com.couchbase.client.core.Timer$$Lambda$155/00000000136C23F0.run(Unknown Source)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
at com.couchbase.client.core.deps.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:820)

Thanks!

12

Hi @mreiche,
I ran “curl xxxx.com:11208”
I got “curl: (56) Recv failure: Connection reset by peer”
what does this mean? network issue?

Thanks!

13

interesting - one thing you can try is disabling nativeTls in the SecurityConfig so it will not go through netty openssl but rather through the regular java one (enableNativeTls(boolean nativeTlsEnabled) to false) and rerun

14

Seem like something is indeed listening on that port (otherwise it would give “Connection refused”.
It sounds like @daschl is on the right path.

15

The “FINE” logging level in corresponds to DEBUG. This is not an error. According to the Netty developers, this is expected, which is why it’s logged at DEBUG level instead of a more severe level.

I’m not sure what IS causing the problem… just wanted to let you know the “no such provider” message is likely a red herring. Still, it doesn’t hurt to see if disabling native TLS like daschl suggested fixes the problem.

Thanks,
David

16

After setting the enableNativeTls to false, I got the following SSLException:

Apr 28, 2022 10:13:18 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0xaf9f2fd900000001”,“managerPort”:“8091”,“remote”:“xxxx.com”}
Apr 28, 2022 10:13:19 AM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger warn
WARNING: [com.couchbase.endpoint][EndpointConnectionFailedEvent][1446ms] Connect attempt 1 failed because of DecoderException: javax.net.ssl.SSLException: Received fatal alert: protocol_version {“circuitBreaker”:“DISABLED”,“coreId”:“0xaf9f2fd900000001”,“remote”:“xxxx.com:11207”,“type”:“KV”}
com.couchbase.client.core.deps.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: protocol_version
at java.lang.Thread.run(Thread.java:820)
Caused by: javax.net.ssl.SSLException: Received fatal alert: protocol_version
at com.ibm.jsse2.k.a(k.java:1)
at com.ibm.jsse2.as.a(as.java:423)
at com.ibm.jsse2.as.a(as.java:408)
at com.ibm.jsse2.as.j(as.java:548)
at com.ibm.jsse2.as.b(as.java:325)
at com.ibm.jsse2.as.a(as.java:224)
at com.ibm.jsse2.as.unwrap(as.java:418)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:5)
… 1 more

17

@w_lin I’m used to the IBM JVM. Which TLS version does your JVM support, and against which cluster are you connecting to?

18

TLSv1.3 & TLSv1.2 are the default TLS protocols in IBM JDK 8.0.

Couchbase Cluster (single node): Enterprise Edition 7.0.3 build 7031

Couchbase Java SDK version: 3.2.6

IBM Java version:
java version “1.8.0_321”
Java™ SE Runtime Environment (build 8.0.7.5 - pwa6480sr7fp5-20220208_01(SR7 FP5))
IBM J9 VM (build 2.9, JRE 1.8.0 Windows 10 amd64-64-Bit Compressed References 20220104_19630 (JIT enabled, AOT enabled)
OpenJ9 - 2d4c7d9
OMR - 59845b7
IBM - 3c151c1)
JCL - 20220120_01 based on Oracle jdk8u321-b07

19

@w_lin is there a way to further debug / troubleshoot the TLS setup in the IBM JDK? At this point in the code we’ve handed it off to the TLS implementation, so this is nothing we directly influence I think.

20

In the com.couchbase.client.core.deps.io.netty.handler.ssl.OpenSslX509TrustManagerWrapper.java (OpenSslX509TrustManagerWrapper.java:159) where the issue is:

private static SSLContext newSSLContext() throws NoSuchAlgorithmException, NoSuchProviderException {
    // As this depends on the implementation detail we should explicit select the correct provider.
    // See https://github.com/netty/netty/issues/10374
    return SSLContext.getInstance("TLS", "SunJSSE");
}

is it possible to change it so not to require the provider and see if it will fix it?

private static SSLContext newSSLContext() throws NoSuchAlgorithmException, NoSuchProviderException {

    return SSLContext.getInstance("TLS");
}