Environment:
Couchbase Server: Version: 4.1.0-5005 Enterprise Edition (build-5005)
Couchbase Client: couchbase-java-client-2.2.8.jar, couchbase-core-io-1.2.9.jar, rxjava-1.0.17.jar
3-node cluster (node155, node158, node159)
When I bring up my client on a 3-node cluster I will intermittently see connection issues to one node, or to the bootstrap node, on one or more of my applications on the same box bootstrapped at the same time. Sometimes everything works fine, this leads me to believe there may be a race condition.
2016.07.15 18:03:30:179 EDT | Info | CommonPersistency | localhost-startStop-1
[Couchbase] ioPoolSize is less than 3 (2), setting to: 3
2016.07.15 18:03:30:179 EDT | Info | CommonPersistency | localhost-startStop-1
[Couchbase] computationPoolSize is less than 3 (2), setting to: 3
2016.07.15 18:03:30:449 EDT | Info | CommonPersistency | localhost-startStop-1
[Couchbase] CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=3, computationPoolSize=3, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=2, viewServiceEndpoints=2, queryServiceEndpoints=2, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=true, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}
2016.07.15 18:03:30:451 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Diagnostics {
gc.ps marksweep.collectionCount=2,
gc.ps marksweep.collectionTime=174,
gc.ps scavenge.collectionCount=8,
gc.ps scavenge.collectionTime=130,
heap.pendingFinalize=0,
heap.used=init = 134217728(131072K) used = 38855704(37945K) committed = 238026752(232448K) max = 477626368(466432K),
mem.physical.free=302321664,
mem.physical.total=1929281536,
mem.swap.free=1658286080,
mem.swap.total=2147479552,
mem.virtual.comitted=3188330496,
offHeap.used=init = 2555904(2496K) used = 49304184(48148K) committed = 50683904(49496K) max = -1(-1K),
proc.cpu.time=5570000000,
runtime.name=5329@jberglund-PS,
runtime.spec=Oracle Corporation/Java Virtual Machine Specification: 1.8,
runtime.startTime=1468620197606,
runtime.sysProperties={awt.toolkit=sun.awt.X11.XToolkit, file.encoding.pkg=sun.io, java.specification.version=1.8, com.couchbase.client.deps.io.netty.packagePrefix=com.couchbase.client.deps., sun.cpu.isalist=, sun.jnu.encoding=ISO-8859-1, bw.nameservice.denyTimeBoundedDuplicateLookups=true, java.class.path=/usr/local/tomcat/tomcat_base/bin/bootstrap.jar:/usr/local/tomcat/tomcat_base/bin/tomcat-juli.jar, java.vm.vendor=Oracle Corporation, bw.nameservice.cachePolicy=Configured, sun.arch.data.model=64, catalina.useNaming=true, java.vendor.url=http://java.oracle.com/, sun.net.spi.nameservice.provider.1=dns,BroadSoft, user.timezone=America/New_York, os.name=Linux, java.vm.specification.version=1.8, oracle.jdbc.timezoneAsRegion=false, sun.java.launcher=SUN_STANDARD, user.country=US, java.net.preferIPv6Addresses=false, sun.boot.library.path=/usr/local/java/jdk1.8.0_72/jre/lib/amd64, sun.java.command=org.apache.catalina.startup.Bootstrap start, bw.nameservice.nCachePolicy=Configured, sun.cpu.endian=little, user.home=/var/empty/bworks, user.language=en, java.specification.vendor=Oracle Corporation, java.naming.factory.url.pkgs=org.apache.naming, java.home=/usr/local/java/jdk1.8.0_72/jre, file.separator=/, line.separator=
, java.vm.specification.vendor=Oracle Corporation, java.specification.name=Java Platform API Specification, bw.nameservice.timeBoundedPoolSize=2, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, entityExpansionLimit=1024, bw.nameservice.useAdditionalSrvRrs=true, maxNotificationTransmitterInternalQueueSize=1000, package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.tomcat., package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.naming.,org.apache.tomcat., bw.config.container.name=tomcat, sun.boot.class.path=/usr/local/java/jdk1.8.0_72/jre/lib/resources.jar:/usr/local/java/jdk1.8.0_72/jre/lib/rt.jar:/usr/local/java/jdk1.8.0_72/jre/lib/sunrsasign.jar:/usr/local/java/jdk1.8.0_72/jre/lib/jsse.jar:/usr/local/java/jdk1.8.0_72/jre/lib/jce.jar:/usr/local/java/jdk1.8.0_72/jre/lib/charsets.jar:/usr/local/java/jdk1.8.0_72/jre/lib/jfr.jar:/usr/local/java/jdk1.8.0_72/jre/classes, server.loader=, java.util.logging.config.file=/usr/local/tomcat/tomcat_base/conf/logging.properties, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, bw.nameservice.nCacheTtlSecs=600, java.runtime.version=1.8.0_72-b15, java.naming.factory.initial=org.apache.naming.java.javaURLContextFactory, user.name=bworks, java.net.preferIPv4Stack=true, path.separator=:, common.loader="${catalina.base}/lib","${catalina.base}/lib/*.jar","${catalina.home}/lib","${catalina.home}/lib/*.jar", os.version=3.10.0-327.el7.x86_64, java.endorsed.dirs=/usr/local/tomcat/tomcat_base/endorsed, java.runtime.name=Java(TM) SE Runtime Environment, com.broadsoft.webcontainer.jvm=true, file.encoding=ISO-8859-1, sun.nio.ch.bugLevel=, bw.nameservice.unreachableServerLingerSecs=60, java.vm.name=Java HotSpot(TM) 64-Bit Server VM, org.apache.catalina.STRICT_SERVLET_COMPLIANCE=true, java.vendor.url.bug=http://bugreport.sun.com/bugreport/, java.io.tmpdir=/usr/local/tomcat/tomcat_base/temp, tomcat.util.scan.StandardJarScanFilter.jarsToScan=log4j-core*.jar,log4j-taglib*.jar, catalina.home=/usr/local/tomcat/8.0.22, java.version=1.8.0_72, tomcat.util.scan.StandardJarScanFilter.jarsToSkip=bootstrap.jar,commons-daemon.jar,tomcat-juli.jar,annotations-api.jar,el-api.jar,jsp-api.jar,servlet-api.jar,websocket-api.jar,catalina.jar,catalina-ant.jar,catalina-ha.jar,catalina-storeconfig.jar,catalina-tribes.jar,jasper.jar,jasper-el.jar,ecj-*.jar,tomcat-api.jar,tomcat-util.jar,tomcat-util-scan.jar,tomcat-coyote.jar,tomcat-dbcp.jar,tomcat-jni.jar,tomcat-websocket.jar,tomcat-i18n-en.jar,tomcat-i18n-es.jar,tomcat-i18n-fr.jar,tomcat-i18n-ja.jar,tomcat-juli-adapters.jar,catalina-jmx-remote.jar,catalina-ws.jar,tomcat-jdbc.jar,tools.jar,commons-beanutils*.jar,commons-codec*.jar,commons-collections*.jar,commons-dbcp*.jar,commons-digester*.jar,commons-fileupload*.jar,commons-httpclient*.jar,commons-io*.jar,commons-lang*.jar,commons-logging*.jar,commons-math*.jar,commons-pool*.jar,jstl.jar,taglibs-standard-spec-*.jar,geronimo-spec-jaxrpc*.jar,wsdl4j*.jar,ant.jar,ant-junit*.jar,aspectj*.jar,jmx.jar,h2*.jar,hibernate*.jar,httpclient*.jar,jmx-tools.jar,jta*.jar,log4j*.jar,mail*.jar,slf4j*.jar,xercesImpl.jar,xmlParserAPIs.jar,xml-apis.jar,junit.jar,junit-*.jar,ant-launcher.jar,cobertura-*.jar,asm-*.jar,dom4j-*.jar,icu4j-*.jar,jaxen-*.jar,jdom-*.jar,jetty-*.jar,oro-*.jar,servlet-api-*.jar,tagsoup-*.jar,xmlParserAPIs-*.jar,xom-*.jar, sun.net.inetaddr.ttl=0, user.dir=/usr/local/broadworks/PS_Rel_22.0_1.916/bin, os.arch=amd64, bw.nameservice.cacheTtlSecs=86400, java.vm.specification.name=Java Virtual Machine Specification, sun.security.pkcs11.enable-solaris=false, java.awt.printerjob=sun.print.PSPrinterJob, sun.os.patch.level=unknown, sun.net.inetaddr.negative.ttl=0, catalina.base=/usr/local/tomcat/8.0.22, shared.loader=, p=tomcat, java.util.logging.manager=org.apache.juli.ClassLoaderLogManager, java.library.path=/usr/local/broadworks/bw_base/lib:/usr/local/TimesTen/tt_base/lib, java.vendor=Oracle Corporation, java.vm.info=mixed mode, java.vm.version=25.72-b15, sun.io.unicode.encoding=UnicodeLittle, java.ext.dirs=/usr/local/java/jdk1.8.0_72/jre/lib/ext:/usr/java/packages/lib/ext, tomcat.util.buf.StringCache.byte.enabled=true, java.class.version=52.0},
runtime.uptime=12881,
runtime.vm=Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM: 25.72-b15,
sys.cpu.loadAvg=5.03,
sys.cpu.num=2,
sys.os.arch=amd64,
sys.os.name=Linux,
sys.os.version=3.10.0-327.el7.x86_64,
thread.count=71,
thread.peakCount=71,
thread.startedCount=72
}
2016.07.15 18:03:30:496 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Setting seed hosts to [/10.17.4.155]
2016.07.15 18:03:30:509 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Got instructed to open bucket bwecl
2016.07.15 18:03:30:511 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Loading Config for bucket bwecl
2016.07.15 18:03:30:512 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Loading Config for bucket bwecl
2016.07.15 18:03:30:536 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Got instructed to add Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:536 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Connecting Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:537 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] [couchbase155.rtx.broadsoft.com.]: Got instructed to connect.
2016.07.15 18:03:30:538 EDT | FieldDebug | CommonPersistency | localhost-startStop-1
[Couchbase] Connect finished, registering for use.
2016.07.15 18:03:30:540 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] Successfully added Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:542 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] Got instructed to add Service BINARY, to Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:542 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] [couchbase155.rtx.broadsoft.com.]: Adding Service BINARY
2016.07.15 18:03:30:545 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] [couchbase155.rtx.broadsoft.com.]: Adding Service BINARY to registry and connecting it.
2016.07.15 18:03:30:545 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] [couchbase155.rtx.broadsoft.com.][KeyValueService]: Got instructed to connect.
2016.07.15 18:03:30:578 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] [couchbase155.rtx.broadsoft.com.][KeyValueService]: Initializing connect on Endpoint.
2016.07.15 18:03:30:578 EDT | Debug | CommonPersistency | cb-computations-3
[Couchbase] State (KeyValueEndpoint) DISCONNECTED -> CONNECTING
2016.07.15 18:03:30:578 EDT | Debug | CommonPersistency | cb-computations-3
[Couchbase] State (EndpointStateZipper) DISCONNECTED -> CONNECTING
2016.07.15 18:03:30:578 EDT | Debug | CommonPersistency | cb-computations-3
[Couchbase] State (KeyValueService) DISCONNECTED -> CONNECTING
2016.07.15 18:03:30:579 EDT | Debug | CommonPersistency | cb-computations-3
[Couchbase] State (ServiceStateZipper) DISCONNECTED -> CONNECTING
2016.07.15 18:03:30:589 EDT | Debug | CommonPersistency | cb-computations-3
[Couchbase] State (CouchbaseNode) DISCONNECTED -> CONNECTING
2016.07.15 18:03:30:616 EDT | FieldDebug | CommonPersistency | cb-computations-3
[Couchbase] [couchbase155.rtx.broadsoft.com.][KeyValueService]: Initializing connect on Endpoint.
2016.07.15 18:03:30:616 EDT | Debug | CommonPersistency | cb-computations-3
[Couchbase] State (KeyValueEndpoint) DISCONNECTED -> CONNECTING
2016.07.15 18:03:30:738 EDT | FieldDebug | CommonPersistency | cb-io-1-2
[Couchbase] Requesting supported features: [MUTATION_SEQNO, TCPNODELAY]
2016.07.15 18:03:30:740 EDT | FieldDebug | CommonPersistency | cb-io-1-1
[Couchbase] Requesting supported features: [MUTATION_SEQNO, TCPNODELAY]
2016.07.15 18:03:30:742 EDT | FieldDebug | CommonPersistency | cb-io-1-2
[Couchbase] Negotiated supported features: [MUTATION_SEQNO, TCPNODELAY]
2016.07.15 18:03:30:743 EDT | FieldDebug | CommonPersistency | cb-io-1-2
[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Connected Endpoint.
2016.07.15 18:03:30:743 EDT | Debug | CommonPersistency | cb-io-1-2
[Couchbase] State (KeyValueEndpoint) CONNECTING -> CONNECTED
2016.07.15 18:03:30:744 EDT | FieldDebug | CommonPersistency | cb-io-1-1
[Couchbase] Negotiated supported features: [MUTATION_SEQNO, TCPNODELAY]
2016.07.15 18:03:30:744 EDT | FieldDebug | CommonPersistency | cb-io-1-1
[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Connected Endpoint.
2016.07.15 18:03:30:744 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (KeyValueEndpoint) CONNECTING -> CONNECTED
2016.07.15 18:03:30:745 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (EndpointStateZipper) CONNECTING -> CONNECTED
2016.07.15 18:03:30:745 EDT | Debug | CommonPersistency | cb-io-1-2
[Couchbase] State (EndpointStateZipper) CONNECTING -> DEGRADED
2016.07.15 18:03:30:745 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (KeyValueService) CONNECTING -> CONNECTED
2016.07.15 18:03:30:746 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (ServiceStateZipper) CONNECTING -> CONNECTED
2016.07.15 18:03:30:746 EDT | Info | CommonPersistency | cb-io-1-1
[Couchbase] Connected to Node couchbase155.rtx.broadsoft.com.
2016.07.15 18:03:30:748 EDT | FieldDebug | CommonPersistency | cb-io-1-1
[Couchbase] Connected (CONNECTING) to Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:748 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (CouchbaseNode) CONNECTING -> CONNECTED
2016.07.15 18:03:30:748 EDT | FieldDebug | CommonPersistency | RxCachedThreadScheduler-1
CouchbaseEvent: NodeConnectedEvent{host=couchbase155.rtx.broadsoft.com./10.17.4.155}
2016.07.15 18:03:30:749 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (KeyValueService) CONNECTED -> DEGRADED
2016.07.15 18:03:30:749 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (ServiceStateZipper) CONNECTED -> DISCONNECTED
2016.07.15 18:03:30:749 EDT | Info | CommonPersistency | cb-io-1-1
[Couchbase] Disconnected from Node couchbase155.rtx.broadsoft.com.
2016.07.15 18:03:30:757 EDT | FieldDebug | CommonPersistency | cb-io-1-1
[Couchbase] Disconnected (CONNECTED) from Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:757 EDT | Debug | CommonPersistency | cb-io-1-1
[Couchbase] State (CouchbaseNode) CONNECTED -> DISCONNECTED
2016.07.15 18:03:30:757 EDT | FieldDebug | CommonPersistency | cb-io-1-2
[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Channel Active.
2016.07.15 18:03:30:829 EDT | FieldDebug | CommonPersistency | cb-computations-2
[Couchbase] Successfully enabled Service BINARY on Node couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:829 EDT | FieldDebug | CommonPersistency | cb-computations-2
[Couchbase] Starting to discover config through Carrier Bootstrap
2016.07.15 18:03:30:830 EDT | FieldDebug | CommonPersistency | cb-io-1-1
[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Channel Active.
2016.07.15 18:03:30:905 EDT | Info | CommonPersistency | RxCachedThreadScheduler-1
SystemEvent: Node Connected Event for host couchbase155.rtx.broadsoft.com./10.17.4.155
2016.07.15 18:03:30:905 EDT | FieldDebug | CommonPersistency | RxCachedThreadScheduler-1
CouchbaseEvent: NodeDisconnectedEvent{host=couchbase155.rtx.broadsoft.com./10.17.4.155}
2016.07.15 18:03:30:905 EDT | Info | CommonPersistency | RxCachedThreadScheduler-1
SystemEvent: Node Disconnected Event for host couchbase155.rtx.broadsoft.com./10.17.4.155