Couchbase connection issue

Couchbase Server Kubernetes
1

Hi,
We have an application deployed in GCP and the problem is whenever we are deploying application though spinnaker it could not able to connect couchbase in single shot. We have to to deployed twice or thrice or may be more to get connected with couchbase. Could someone please provide the solution for it, below are the details
DNS of coucbase like : stable.ss-**.service.consul:8091 trying to connect
Error :
{“timestamp”:“2020-01-14T14:08:31.600+00:00”,“logger_name”:“com.couchbase.client.core.CouchbaseCore”,“severity”:“INFO”,“message”:“CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile=‘null’, sslTruststoreFile=‘null’, sslKeystorePassword=false, sslTruststorePassword=false, sslKeystore=null, sslTruststore=null, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=5, computationPoolSize=5, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=12, queryServiceEndpoints=12, searchServiceEndpoints=12, configPollInterval=2500, configPollFloorInterval=50, networkResolution=NetworkResolution{name=‘auto’}, ioPool=NioEventLoopGroup, kvIoPool=null, viewIoPool=null, searchIoPool=null, queryIoPool=null, analyticsIoPool=null, coreScheduler=CoreScheduler, memcachedHashingStrategy=DefaultMemcachedHashingStrategy, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.7.0 (git: 2.7.0, core: 1.7.0), retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, continuousKeepAliveEnabled=true, keepAliveErrorThreshold=4, keepAliveTimeout=10000000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=true, socketConnectTimeout=1000, callbacksOnIoPool=false, disconnectTimeout=25000, requestBufferWaitStrategy=com.couchbase.client.core.env.DefaultCoreEnvironment$4@53f0a4cb, certAuthEnabled=false, coreSendHook=null, forceSaslPlain=false, compressionMinRatio=0.83, compressionMinSize=32, compressionEnabled=true, operationTracingEnabled=true, operationTracingServerDurationEnabled=true, tracer=ThresholdLogTracer, orphanResponseReportingEnabled=true, orphanResponseReporter=DefaultOrphanResponseReporter, keyValueServiceConfig=KeyValueServiceConfig{minEndpoints=1, maxEndpoints=1, pipelined=true, idleTime=0}, queryServiceConfig=QueryServiceConfig{minEndpoints=0, maxEndpoints=12, pipelined=false, idleTime=300}, searchServiceConfig=SearchServiceConfig{minEndpoints=0, maxEndpoints=12, pipelined=false, idleTime=300}, viewServiceConfig=ViewServiceConfig{minEndpoints=0, maxEndpoints=12, pipelined=false, idleTime=300}, analyticsServiceConfig=AnalyticsServiceConfig{minEndpoints=0, maxEndpoints=12, pipelined=false, idleTime=300}, queryTimeout=1000, viewTimeout=75000, searchTimeout=75000, analyticsTimeout=75000, kvTimeout=2500, connectTimeout=5000, dnsSrvEnabled=false, propagateParentSpan=true}”,“trace”:"",“span”:"",“parent”:"",“class”:“com.couchbase.client.core.CouchbaseCore”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:31.933+00:00”,“logger_name”:“com.couchbase.client.core.endpoint.Endpoint”,“severity”:“WARN”,“message”:"[sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal:11210][KeyValueEndpoint]: Could not connect to remote socket: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:11210",“trace”:"",“span”:"",“parent”:"",“class”:“c.c.client.core.endpoint.Endpoint”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:31.946+00:00”,“logger_name”:“com.couchbase.client.core.endpoint.Endpoint”,“severity”:“WARN”,“message”:"[sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal:11210][KeyValueEndpoint]: Could not connect to remote socket: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:11210",“trace”:"",“span”:"",“parent”:"",“class”:“c.c.client.core.endpoint.Endpoint”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:31.947+00:00”,“logger_name”:“com.couchbase.client.core.endpoint.Endpoint”,“severity”:“WARN”,“message”:“Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:11210”,“trace”:"",“span”:"",“parent”:"",“class”:“c.c.client.core.endpoint.Endpoint”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:31.993+00:00”,“logger_name”:“com.couchbase.client.core.endpoint.Endpoint”,“severity”:“WARN”,“message”:"[sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:8091",“trace”:"",“span”:"",“parent”:"",“class”:“c.c.client.core.endpoint.Endpoint”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:32.006+00:00”,“logger_name”:“com.couchbase.client.core.endpoint.Endpoint”,“severity”:“WARN”,“message”:"[sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:8091",“trace”:"",“span”:"",“parent”:"",“class”:“c.c.client.core.endpoint.Endpoint”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:32.006+00:00”,“logger_name”:“com.couchbase.client.core.endpoint.Endpoint”,“severity”:“WARN”,“message”:“Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:8091”,“trace”:"",“span”:"",“parent”:"",“class”:“c.c.client.core.endpoint.Endpoint”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:32.008+00:00”,“logger_name”:“com.lowes.promoexecution.common.PromoExecutionCBConfiguration”,“severity”:“ERROR”,“message”:“failed trying to connect from couchbase Cluster “,“stack_hash”:“016dca57”,“stack_trace”:“j.n.ConnectException: Connection refused\n\t… 11 common frames omitted\nWrapped by: c.c.c.d.i.n.c.AbstractChannel$AnnotatedConnectException: Connection refused: sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal/10.148.39.230:8091\n\tat s.n.c.SocketChannelImpl.checkConnect(SocketChannelImpl.java)\n\tat s.n.c.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\tat c.c.c.d.i.n.c.s.n.NioSocketChannel.doFinishConnect(NioSocketChannel.java:225)\n\tat c.c.c.d.i.n.c.n.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)\n\t… 7 common frames omitted\nWrapped by: c.c.c.c.c.ConfigurationException: Could not open bucket.\n\tat c.c.c.c.c.DefaultConfigurationProvider$6.call(DefaultConfigurationProvider.java:306)\n\tat c.c.c.c.c.DefaultConfigurationProvider$6.call(DefaultConfigurationProvider.java:303)\n\tat r.i.o.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)\n\tat r.i.o.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)\n\tat r.i.o.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)\n\tat r.i.o.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)\n\tat r.i.o.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)\n\tat r.i.o.OperatorTake$1.onError(OperatorTake.java:66)\n\tat r.i.o.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)\n\tat r.i.o.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:658)\n\tat r.i.o.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)\n\tat r.i.o.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:855)\n\tat r.i.o.OperatorOnErrorResumeNextViaFunction$4$1.onError(OperatorOnErrorResumeNextViaFunction.java:122)\n\tat r.i.o.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)\n\tat r.i.o.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)\n\tat r.i.o.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:818)\n\tat r.i.o.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:579)\n\tat r.i.o.OperatorM…”,“trace”:””,“span”:"",“parent”:"",“class”:“c.l.p.c.PromoExecutionCBConfiguration”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:32.695+00:00”,“logger_name”:“springfox.documentation.spring.web.PropertySourcedRequestMappingHandlerMapping”,“severity”:“INFO”,“message”:“Mapped URL path [/v2/api-docs] onto method [public org.springframework.http.ResponseEntity<springfox.documentation.spring.web.json.Json> springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(java.lang.String,javax.servlet.http.HttpServletRequest)]”,“trace”:"",“span”:"",“parent”:"",“class”:“s.d.s.w.PropertySourcedRequestMappingHandlerMapping”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:32.975+00:00”,“logger_name”:“org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor”,“severity”:“INFO”,“message”:“Initializing ExecutorService ‘applicationTaskExecutor’”,“trace”:"",“span”:"",“parent”:"",“class”:“o.s.s.concurrent.ThreadPoolTaskExecutor”,“service”:“PromotionsExecutionService”}

{“timestamp”:“2020-01-14T14:08:34.174+00:00”,“logger_name”:“org.springframework.boot.web.embedded.tomcat.TomcatWebServer”,“severity”:“INFO”,“message”:“Tomcat initialized with port(s): 8081 (http)”,“trace”:"",“span”:"",“parent”:"",“class”:“o.s.b.w.embedded.tomcat.TomcatWebServer”,“service”:“PromotionsExecutionService”}

Jan 14, 2020 2:08:34 PM org.apache.coyote.AbstractProtocol init

2

Morning!

You refer to stable.ss-**.service.consul:8091 however the logs suggest it’s trying to connect to sscouchbase-pricepromo-v000-6z20.c.gcp-ushi-carbon-dbms-stage.internal, which is totally different.

First up, where is the client in relation to the database? Is it in the same Kubernetes cluster? Is there any DNAT between the two?

Second, what is the connection string you are using? (e.g. couchbase://my-cluster-srv.my-namespace.svc). If you are using an SRV record to connect (a couchbase:// scheme) what does DNS return from the client? (e.g. dig _couchbase._tcp.my-cluster-srv.my-namespace.srv.cluster.local. SRV)

Finally if you exec into a Couchbase server pod what does the cluster map look like? (e.g. kubectl exec -ti cb-example-0000 – curl localhost:8091/pools/default/nodeServices -u admin_username:password). Are all the listed endpoints resolvable and reachable from the client? (e.g. nc -vz hostname.domain.com 8091)

3

Thanks for quick response. I am not sure what are you meant to as i am new to couchbase world. Only thing i observed most of the time when it’s trying to connect index server it’s failing to establish the connection.

Below are the couchbase sdk configuration we have done to connect to couchbse

CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder().mutationTokensEnabled(true)
    .keepAliveTimeout(aliveTimeout).connectTimeout(connectionTimeout)
   .queryTimeout(queryTimeout).queryServiceConfig(null).build();
try {
  couchbaseCluster = CouchbaseCluster.create(env, nodes);
  couchbaseCluster.authenticate(userName, password);
  asyncPromoBucket = couchbaseCluster.openBucket(bucketName).async();

}  catch (Exception e) {
  log.error("failed trying to connect from couchbase Cluster ", e);
}
4

Most of my queries have nothing to do with Couchbase, it’s all networking related. Talk to your network administrator, they will understand what I’m getting at.

That code snippet is not what I want, the “connection string” looks like a URL, and is probably used to derive “nodes” in your example.