Could not connect cluster using Java client

Hi,

I am trying to create the secondary index using java client, below is my code snippet. It is throwing the below exception when it is trying to create the index. The same code is working in other Dev environment. can anyone please guide me what is going wrong here.

Cluster cluster = Cluster.connect( instance.getCluster().getClusterAddress(),instance.getBucket().getUserid(),
new String(Base64.getDecoder().decode(instance.getBucket().getPassword()), StandardCharsets.UTF_8) );

cluster.queryIndexes().createIndex(cluster.bucket(instance.getBucket().getBucketId()).name(),
“indexName”, Arrays.asList(fieldValuesArray));

2021-02-15 05:34:08.589 INFO 1 — [ cb-events] com.couchbase.core : [com.couchbase.core][CoreCreatedEvent] {“clientVersion”:“3.0.0”,“clientGitHash”:“a3d7a770”,“coreVersion”:“2.0.0”,“coreGitHash”:“a3d7a770”,“userAgent”:“couchbase-java/3.0.0 (Linux 4.14.209-160.335.amzn2.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_242-b08)”,“maxNumRequestsInRetry”:32768,“ioEnvironment”:{“nativeIoEnabled”:true,“eventLoopThreadCount”:2,“eventLoopGroups”:[“EpollEventLoopGroup”]},“ioConfig”:{“captureTraffic”:,“mutationTokensEnabled”:true,“networkResolution”:“auto”,“dnsSrvEnabled”:true,“tcpKeepAlivesEnabled”:true,“tcpKeepAliveTimeMs”:60000,“configPollIntervalMs”:2500,“kvCircuitBreakerConfig”:“disabled”,“queryCircuitBreakerConfig”:“disabled”,“viewCircuitBreakerConfig”:“disabled”,“searchCircuitBreakerConfig”:“disabled”,“analyticsCircuitBreakerConfig”:“disabled”,“managerCircuitBreakerConfig”:“disabled”,“numKvConnections”:1,“maxHttpConnections”:12,“idleHttpConnectionTimeoutMs”:30000,“configIdleRedialTimeoutMs”:300000},“compressionConfig”:{“enabled”:true,“minRatio”:0.83,“minSize”:32},“securityConfig”:{“tlsEnabled”:false,“nativeTlsEnabled”:true,“hasTrustCertificates”:false,“trustManagerFactory”:null},“timeoutConfig”:{“kvMs”:2500,“kvDurableMs”:10000,“managementMs”:75000,“queryMs”:75000,“viewMs”:75000,“searchMs”:75000,“analyticsMs”:75000,“connectMs”:10000,“disconnectMs”:10000},“loggerConfig”:{“customLogger”:null,“fallbackToConsole”:false,“disableSlf4j”:false,“loggerName”:“CouchbaseLogger”,“diagnosticContextEnabled”:false},“orphanReporterConfig”:{“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024},“retryStrategy”:“BestEffortRetryStrategy”,“requestTracer”:“OwnedSupplier”} {“coreId”:1}
2021-02-15 05:34:08.595 INFO 1 — [ cb-events] com.couchbase.node : [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:1,“managerPort”:“8091”,“remote”:“cluster-custom.cluster”}
2021-02-15 05:34:13.948 DEBUG 1 — [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-02-15 05:34:18.465 WARN 1 — [ cb-events] com.couchbase.endpoint : [com.couchbase.endpoint][EndpointConnectionFailedEvent][10s] Connect attempt 1 failed because of TimeoutException: Did not observe any item or terminal signal within 10000ms in ‘source(MonoDefer)’ (and no fallback has been configured) {“circuitBreaker”:“DISABLED”,“coreId”:1,“remote”:“cluster-custom.cluster:8091”,“type”:“KV”}

java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in ‘source(MonoDefer)’ (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:288) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:273) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:395) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_242]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_242]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_242]

2021-02-15 05:34:18.466 WARN 1 — [ cb-events] com.couchbase.core : [com.couchbase.core][InitGlobalConfigFailedEvent][10s] Initializing the global config failed: UNKNOWN {“coreId”:1}

com.couchbase.client.core.error.ConfigException: Caught exception while loading global config.
at com.couchbase.client.core.config.loader.GlobalLoader.lambda$load$3(GlobalLoader.java:72) ~[core-io-2.0.0.jar:na]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:134) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:134) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:134) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1829) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onError(MonoIgnoreThen.java:306) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:126) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.Operators.error(Operators.java:196) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:4252) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:97) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onError(FluxDoFinally.java:129) ~[reactor-core-3.3.12.RELEASE.jar:3.3.12.RELEASE]
at com.couchbase.client.core.Reactor$SilentMonoCompletionStage.lambda$subscribe$0(Reactor.java:175) ~[core-io-2.0.0.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[na:1.8.0_242]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[na:1.8.0_242]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_242]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_242]
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:168) ~[core-io-2.0.0.jar:na]
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157) ~[core-io-2.0.0.jar:na]
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[core-io-2.0.0.jar:na]
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[core-io-2.0.0.jar:na]
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[core-io-2.0.0.jar:na]
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[core-io-2.0.0.jar:na]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_242]
Caused by: com.couchbase.client.core.error.UnambiguousTimeoutException: CarrierGlobalConfigRequest
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:163) ~[core-io-2.0.0.jar:na]
… 6 common frames omitted

com.couchbase.client.core.deps.io.netty.channel.unix.Errors$NativeIoException: readAddress(…) failed: Connection reset by peer

2021-02-15 05:34:18.770 INFO 1 — [ cb-events] com.couchbase.node : [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:1,“managerPort”:“8091”,“remote”:“clusterid-0002.clusterid.clusterid.svc”}
2021-02-15 05:34:18.771 WARN 1 — [ cb-events] com.couchbase.io : [com.couchbase.io][GenericFailureDetectedEvent] Detected Exception in IO Layer: readAddress(…) failed: Connection reset by peer, Cause: (none) {“bucket”:“cb-bucket-2kl64fm6k”,“coreId”:1,“local”:"/10.10.111.12:42502",“remote”:“clusterid-0000.clusterid.clusterid.svc/10.10.111.123:11210”}

So there are 3 pods will get created when we create cluster, I think when it is trying to create index from java client connection reset is happening from other pods.
When I deleted two pods and tried to execute the same code with only one pod it created index successfully.
I am not sure what is going wrong here, any help would be appreciated.