.NET Core: GKE Autonomous Operator: UnsupportedAddressFamily

foxtrotuniform6969 · November 21, 2018, 10:08pm

I am port forwarding with my cluster using cb-cluster-member-0000 in GKE with the command
$ kubectl port-forward cb-cluster-member-0000 8091:8091
and am able to access the web UI and verify the buckets I’m requesting exist.

Whenever I attempt any operation on my GKE cluster, I get:

Bootstrapping failed from http://localhost:8091/pools: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Couchbase.Utils.UnsupportedAddressFamilyException: http://cb-cluster-member-0000.cb-cluster-member.default.svc

In my appsettings.json, I have the following config:

"Couchbase": {
   "Servers": [
     "http://localhost:8091/pools" // I also tried without "/pools"
    ],
   "Username": "username",
   "Password": "password",
   "UseSsl": false

},

The full stacktrace is as follows:

info: Couchbase.Cluster[0]
      Version: 2.7.1.0
dbug: Couchbase.Configuration.Server.Monitoring.ConfigMonitor[0]
      Waiting to check configs...
info: Couchbase.Cluster[0]
      Configuration: {"UseConnectionPooling":false,"EnableDeadServiceUriPing":true,"VBucketRetrySleepTime":100,"QueryFailedThreshold":2,"QueryRequestTimeout":75000,"EnableQueryTiming":false,"SearchRequestTimeout":75000,"NodeAvailableCheckInterval":1000,"EnableTcpKeepAlives":true,"TcpKeepAliveTime":7200000,"TcpKeepAliveInterval":1000,"IOErrorThreshold":10,"IOErrorCheckInterval":500,"LoggerFactory":{},"UseSsl":false,"SslPort":11207,"ApiPort":8092,"MgmtPort":8091,"DirectPort":11210,"HttpsMgmtPort":18091,"HttpsApiPort":18092,"ObserveTimeout":500,"ObserveInterval":10,"MaxViewRetries":2,"ViewHardTimeout":30000,"Servers":["http://localhost:8091/pools"],"SerializationSettings":{"ReferenceLoopHandling":0,"MissingMemberHandling":0,"ObjectCreationHandling":0,"NullValueHandling":0,"DefaultValueHandling":0,"Converters":[],"PreserveReferencesHandling":0,"TypeNameHandling":0,"MetadataPropertyHandling":0,"TypeNameAssemblyFormat":0,"TypeNameAssemblyFormatHandling":0,"ConstructorHandling":0,"ContractResolver":{"DynamicCodeGeneration":false,"DefaultMembersSearchFlags":20,"SerializeCompilerGeneratedMembers":false,"IgnoreSerializableInterface":false,"IgnoreSerializableAttribute":true,"IgnoreIsSpecifiedMembers":false,"IgnoreShouldSerializeMembers":false,"NamingStrategy":{"ProcessDictionaryKeys":true,"ProcessExtensionDataNames":false,"OverrideSpecifiedNames":true}},"EqualityComparer":null,"ReferenceResolver":null,"ReferenceResolverProvider":null,"TraceWriter":null,"Binder":null,"SerializationBinder":null,"Error":null,"Context":{"State":0,"Context":null},"DateFormatString":"yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK","MaxDepth":null,"Formatting":0,"DateFormatHandling":0,"DateTimeZoneHandling":3,"DateParseHandling":1,"FloatFormatHandling":0,"FloatParseHandling":0,"StringEscapeHandling":0,"Culture":"(Default)","CheckAdditionalContent":false},"DeserializationSettings":{"ReferenceLoopHandling":0,"MissingMemberHandling":0,"ObjectCreationHandling":0,"NullValueHandling":0,"DefaultValueHandling":0,"Converters":[],"PreserveReferencesHandling":0,"TypeNameHandling":0,"MetadataPropertyHandling":0,"TypeNameAssemblyFormat":0,"TypeNameAssemblyFormatHandling":0,"ConstructorHandling":0,"ContractResolver":{"DynamicCodeGeneration":false,"DefaultMembersSearchFlags":20,"SerializeCompilerGeneratedMembers":false,"IgnoreSerializableInterface":false,"IgnoreSerializableAttribute":true,"IgnoreIsSpecifiedMembers":false,"IgnoreShouldSerializeMembers":false,"NamingStrategy":{"ProcessDictionaryKeys":true,"ProcessExtensionDataNames":false,"OverrideSpecifiedNames":true}},"EqualityComparer":null,"ReferenceResolver":null,"ReferenceResolverProvider":null,"TraceWriter":null,"Binder":null,"SerializationBinder":null,"Error":null,"Context":{"State":0,"Context":null},"DateFormatString":"yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK","MaxDepth":null,"Formatting":0,"DateFormatHandling":0,"DateTimeZoneHandling":3,"DateParseHandling":1,"FloatFormatHandling":0,"FloatParseHandling":0,"StringEscapeHandling":0,"Culture":"(Default)","CheckAdditionalContent":false},"BucketConfigs":{},"PoolConfiguration":{"EnableTcpKeepAlives":true,"TcpKeepAliveTime":7200000,"TcpKeepAliveInterval":1000,"MaxSize":2,"MinSize":1,"WaitTimeout":2500,"MaxAcquireIterationCount":5,"RecieveTimeout":2500,"ShutdownTimeout":10000,"OperationTimeout":2500,"UseSsl":false,"SendTimeout":15000,"ConnectTimeout":10000,"MaxCloseAttempts":5,"CloseAttemptInterval":100,"EnableOperationTiming":false,"BufferSize":16384,"Uri":null,"ForceSaslPlain":true},"HeartbeatConfigInterval":2500.0,"ConfigPollInterval":2500,"HeartbeatConfigCheckFloor":0,"ConfigPollCheckFloor":0,"ViewRequestTimeout":75000,"DefaultConnectionLimit":5,"MaxServicePointIdleTime":100,"Expect100Continue":false,"EnableConfigHeartBeat":true,"ConfigPollEnabled":true,"EnableOperationTiming":false,"BufferSize":0,"DefaultOperationLifespan":2500,"ConfigurationProviders":3,"AnalyticsRequestTimeout":75000,"ForceSaslPlain":true,"OperationTracingEnabled":true,"OperationTracingServerDurationEnabled":true,"OrphanedResponseLoggingEnabled":true,"EnableCertificateAuthentication":false,"EnableCertificateRevocation":false,"CertificateFactory":null,"HttpServerCertificateValidationCallback":null,"KvServerCertificateValidationCallback":null}
dbug: Couchbase.Core.ClusterController[0]
      Trying to bootstrap with Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.
dbug: Couchbase.Configuration.Server.Providers.ConfigProviderBase[0]
      Getting config for bucket users
dbug: Couchbase.Configuration.Server.Providers.ConfigProviderBase[0]
      Bootstrapping with 127.0.0.1:11210
dbug: Couchbase.IO.Services.PooledIOService[0]
      Creating PooledIOService 013a3f24-1ce5-4ce6-b59a-125f152d8468
dbug: Couchbase.IO.SharedConnectionPool[0]
      Trying to acquire new connection! Refs=0
dbug: Couchbase.Configuration.Server.Providers.ConfigProviderBase[0]
      Bootstrapping with 127.0.0.1:11210 failed.
warn: Couchbase.Configuration.Server.Providers.ConfigProviderBase[0]
System.Net.Sockets.SocketException (111): Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)
warn: Couchbase.Core.ClusterController[0]
System.AggregateException: Could not bootstrap with CCCP. (Connection refused) ---> System.Net.Sockets.SocketException: Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)
   --- End of inner exception stack trace ---
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)
   at Couchbase.Core.ClusterController.CreateBucketImpl(String bucketName, String password, IAuthenticator authenticator)
---> (Inner Exception #0) System.Net.Sockets.SocketException (111): Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)<---

dbug: Couchbase.Core.ClusterController[0]
      Trying to bootstrap with Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.
info: Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig[0]
      Bootstrapping from http://localhost:8091/pools
fail: Couchbase.Utils.UriExtensions[0]
      Could not resolve hostname to IP
fail: Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig[0]
      Bootstrapping failed from http://localhost:8091/pools: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Couchbase.Utils.UnsupportedAddressFamilyException: http://cb-cluster-member-0000.cb-cluster-member.default.svc
         at Couchbase.Utils.UriExtensions.GetIpAddress(Uri uri, Boolean useInterNetworkV6Addresses)
         at Couchbase.Utils.IPEndPointExtensions.GetEndPoint(String hostname, Int32 port)
         at Couchbase.Utils.IPEndPointExtensions.GetIPv4EndPoint(String server)
         at Couchbase.Configuration.Server.Serialization.VBucketServerMap.EnsureIPEndPointsAreLoaded()
         --- End of inner exception stack trace ---
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
         at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Newtonsoft.Json.Serialization.JsonContract.<>c__DisplayClass57_0.<CreateSerializationCallback>b__0(Object o, StreamingContext context)
         at Newtonsoft.Json.Serialization.JsonContract.InvokeOnDeserialized(Object o, StreamingContext context)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.OnDeserialized(JsonReader reader, JsonContract contract, Object value)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
         at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
         at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
         at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
         at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value)
         at Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig.DownloadConfigs(Uri server)
warn: Couchbase.Core.ClusterController[0]
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Couchbase.Utils.UnsupportedAddressFamilyException: http://cb-cluster-member-0000.cb-cluster-member.default.svc
   at Couchbase.Utils.UriExtensions.GetIpAddress(Uri uri, Boolean useInterNetworkV6Addresses)
   at Couchbase.Utils.IPEndPointExtensions.GetEndPoint(String hostname, Int32 port)
   at Couchbase.Utils.IPEndPointExtensions.GetIPv4EndPoint(String server)
   at Couchbase.Configuration.Server.Serialization.VBucketServerMap.EnsureIPEndPointsAreLoaded()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Newtonsoft.Json.Serialization.JsonContract.<>c__DisplayClass57_0.<CreateSerializationCallback>b__0(Object o, StreamingContext context)
   at Newtonsoft.Json.Serialization.JsonContract.InvokeOnDeserialized(Object o, StreamingContext context)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.OnDeserialized(JsonReader reader, JsonContract contract, Object value)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value)
   at Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig.DownloadConfigs(Uri server)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig.Initialize()
   at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.StartProvider(String bucketName, String username, String password)
   at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.GetConfig(String bucketName, String username, String password)
   at Couchbase.Core.ClusterController.CreateBucketImpl(String bucketName, String password, IAuthenticator authenticator)
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
      Executed action atlas_frontend.Controllers.TestController.Get (atlas-frontend) in 760.774ms
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
Couchbase.Configuration.Server.Serialization.BootstrapException: Could not bootstrap - check inner exceptions for details. (Connection refused) (Could not bootstrap with CCCP. (Connection refused)) (Exception has been thrown by the target of an invocation.) ---> System.Net.Sockets.SocketException: Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)
   --- End of inner exception stack trace ---
   at Couchbase.Core.ClusterController.CreateBucketImpl(String bucketName, String password, IAuthenticator authenticator)
   at Couchbase.Core.ClusterController.CreateBucket(String bucketName, String password, IAuthenticator authenticator)
   at Couchbase.Extensions.DependencyInjection.Internal.BucketProvider.<>c__DisplayClass5_0.<GetBucket>b__0(String name)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at atlas_frontend.Controllers.TestController..ctor(IUsersBucketProvider _users) in /home/nbaughman/Projects/atlas-frontend/Controllers/TestController.cs:line 23
   at lambda_method(Closure , IServiceProvider , Object[] )
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
---> (Inner Exception #0) System.Net.Sockets.SocketException (111): Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)<---

---> (Inner Exception #1) System.AggregateException: Could not bootstrap with CCCP. (Connection refused) ---> System.Net.Sockets.SocketException: Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)
   --- End of inner exception stack trace ---
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)
   at Couchbase.Core.ClusterController.CreateBucketImpl(String bucketName, String password, IAuthenticator authenticator)
---> (Inner Exception #0) System.Net.Sockets.SocketException (111): Connection refused
   at Couchbase.IO.DefaultConnectionFactory.<>c__0`1.<GetGeneric>b__0_0(IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
   at Couchbase.IO.SharedConnectionPool`1.CreateAndAuthConnection()
   at Couchbase.IO.SharedConnectionPool`1.Acquire()
   at Couchbase.IO.Services.PooledIOService..ctor(IConnectionPool connectionPool)
   at Couchbase.IO.IOServiceFactory.<>c__DisplayClass0_0.<GetFactory>b__0(IConnectionPool pool)
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String username, String password)<---
<---

---> (Inner Exception #2) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Couchbase.Utils.UnsupportedAddressFamilyException: http://cb-cluster-member-0000.cb-cluster-member.default.svc
   at Couchbase.Utils.UriExtensions.GetIpAddress(Uri uri, Boolean useInterNetworkV6Addresses)
   at Couchbase.Utils.IPEndPointExtensions.GetEndPoint(String hostname, Int32 port)
   at Couchbase.Utils.IPEndPointExtensions.GetIPv4EndPoint(String server)
   at Couchbase.Configuration.Server.Serialization.VBucketServerMap.EnsureIPEndPointsAreLoaded()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Newtonsoft.Json.Serialization.JsonContract.<>c__DisplayClass57_0.<CreateSerializationCallback>b__0(Object o, StreamingContext context)
   at Newtonsoft.Json.Serialization.JsonContract.InvokeOnDeserialized(Object o, StreamingContext context)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.OnDeserialized(JsonReader reader, JsonContract contract, Object value)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value)
   at Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig.DownloadConfigs(Uri server)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig.Initialize()
   at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.StartProvider(String bucketName, String username, String password)
   at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.GetConfig(String bucketName, String username, String password)
   at Couchbase.Core.ClusterController.CreateBucketImpl(String bucketName, String password, IAuthenticator authenticator)<---

After that, I repeatedly see:

dbug: Couchbase.Configuration.Server.Monitoring.ConfigMonitor[0]
      Waiting to check configs...

Which tells me it’s connected to the cluster. The following line immediately pops out to me:
Bootstrapping failed from http://localhost:8091/pools: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Couchbase.Utils.UnsupportedAddressFamilyException: http://cb-cluster-member-0000.cb-cluster-member.default.svc

It looks like it’s having trouble resolving the DNS name of the cluster I’m connected to (cluster member 0000). The other members names are 0001 and 0002. Is this because I’m port forwarding with that member, and thus addressing it as localhost:8091? If so, how do I connect to the cluster as a whole and not this specific member.

Once again, this cluster is hosted in Kubernetes inside Google COntainer Engine (GKE) using the Couchbase Autonomous Operator from the Marketplace.

EDIT:

I’ve been digging through the github repo, I think I found where the error gets thrown, but I still don;t quite know what I’m looking at here:

github.com

couchbase/couchbase-net-client/blob/master/Src/Couchbase/Configuration/Server/Providers/CarrierPublication/CarrierPublicationProvider.cs

using System.Collections.Generic;
using System.Linq;
using System.Security.Authentication;
using System.Threading;
using Couchbase.Authentication.SASL;
using Couchbase.Configuration.Client;
using Couchbase.Configuration.Server.Serialization;
using Couchbase.Core.Transcoders;
using Couchbase.IO;
using Couchbase.IO.Converters;
using Couchbase.IO.Operations;
using System;
using System.Net;
using Couchbase.Core.Buckets;
using Couchbase.Tracing;
using Couchbase.Utils;
using Newtonsoft.Json;

namespace Couchbase.Configuration.Server.Providers.CarrierPublication
{

This file has been truncated. show original

foxtrotuniform6969 · November 21, 2018, 10:39pm

The configuration returned in my stacktrace is:

Configuration: {
    "UseConnectionPooling": false,
    "EnableDeadServiceUriPing": true,
    "VBucketRetrySleepTime": 100,
    "QueryFailedThreshold": 2,
    "QueryRequestTimeout": 75000,
    "EnableQueryTiming": false,
    "SearchRequestTimeout": 75000,
    "NodeAvailableCheckInterval": 1000,
    "EnableTcpKeepAlives": true,
    "TcpKeepAliveTime": 7200000,
    "TcpKeepAliveInterval": 1000,
    "IOErrorThreshold": 10,
    "IOErrorCheckInterval": 500,
    "LoggerFactory": {},
    "UseSsl": false,
    "SslPort": 11207,
    "ApiPort": 8092,
    "MgmtPort": 8091,
    "DirectPort": 11210,
    "HttpsMgmtPort": 18091,
    "HttpsApiPort": 18092,
    "ObserveTimeout": 500,
    "ObserveInterval": 10,
    "MaxViewRetries": 2,
    "ViewHardTimeout": 30000,
    "Servers": [
        "http://localhost:8091/pools"
    ],
    "SerializationSettings": {
        "ReferenceLoopHandling": 0,
        "MissingMemberHandling": 0,
        "ObjectCreationHandling": 0,
        "NullValueHandling": 0,
        "DefaultValueHandling": 0,
        "Converters": [],
        "PreserveReferencesHandling": 0,
        "TypeNameHandling": 0,
        "MetadataPropertyHandling": 0,
        "TypeNameAssemblyFormat": 0,
        "TypeNameAssemblyFormatHandling": 0,
        "ConstructorHandling": 0,
        "ContractResolver": {
            "DynamicCodeGeneration": false,
            "DefaultMembersSearchFlags": 20,
            "SerializeCompilerGeneratedMembers": false,
            "IgnoreSerializableInterface": false,
            "IgnoreSerializableAttribute": true,
            "IgnoreIsSpecifiedMembers": false,
            "IgnoreShouldSerializeMembers": false,
            "NamingStrategy": {
                "ProcessDictionaryKeys": true,
                "ProcessExtensionDataNames": false,
                "OverrideSpecifiedNames": true
            }
        },
        "EqualityComparer": null,
        "ReferenceResolver": null,
        "ReferenceResolverProvider": null,
        "TraceWriter": null,
        "Binder": null,
        "SerializationBinder": null,
        "Error": null,
        "Context": {
            "State": 0,
            "Context": null
        },
        "DateFormatString": "yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK",
        "MaxDepth": null,
        "Formatting": 0,
        "DateFormatHandling": 0,
        "DateTimeZoneHandling": 3,
        "DateParseHandling": 1,
        "FloatFormatHandling": 0,
        "FloatParseHandling": 0,
        "StringEscapeHandling": 0,
        "Culture": "(Default)",
        "CheckAdditionalContent": false
    },
    "DeserializationSettings": {
        "ReferenceLoopHandling": 0,
        "MissingMemberHandling": 0,
        "ObjectCreationHandling": 0,
        "NullValueHandling": 0,
        "DefaultValueHandling": 0,
        "Converters": [],
        "PreserveReferencesHandling": 0,
        "TypeNameHandling": 0,
        "MetadataPropertyHandling": 0,
        "TypeNameAssemblyFormat": 0,
        "TypeNameAssemblyFormatHandling": 0,
        "ConstructorHandling": 0,
        "ContractResolver": {
            "DynamicCodeGeneration": false,
            "DefaultMembersSearchFlags": 20,
            "SerializeCompilerGeneratedMembers": false,
            "IgnoreSerializableInterface": false,
            "IgnoreSerializableAttribute": true,
            "IgnoreIsSpecifiedMembers": false,
            "IgnoreShouldSerializeMembers": false,
            "NamingStrategy": {
                "ProcessDictionaryKeys": true,
                "ProcessExtensionDataNames": false,
                "OverrideSpecifiedNames": true
            }
        },
        "EqualityComparer": null,
        "ReferenceResolver": null,
        "ReferenceResolverProvider": null,
        "TraceWriter": null,
        "Binder": null,
        "SerializationBinder": null,
        "Error": null,
        "Context": {
            "State": 0,
            "Context": null
        },
        "DateFormatString": "yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK",
        "MaxDepth": null,
        "Formatting": 0,
        "DateFormatHandling": 0,
        "DateTimeZoneHandling": 3,
        "DateParseHandling": 1,
        "FloatFormatHandling": 0,
        "FloatParseHandling": 0,
        "StringEscapeHandling": 0,
        "Culture": "(Default)",
        "CheckAdditionalContent": false
    },
    "BucketConfigs": {},
    "PoolConfiguration": {
        "EnableTcpKeepAlives": true,
        "TcpKeepAliveTime": 7200000,
        "TcpKeepAliveInterval": 1000,
        "MaxSize": 2,
        "MinSize": 1,
        "WaitTimeout": 2500,
        "MaxAcquireIterationCount": 5,
        "RecieveTimeout": 2500,
        "ShutdownTimeout": 10000,
        "OperationTimeout": 2500,
        "UseSsl": false,
        "SendTimeout": 15000,
        "ConnectTimeout": 10000,
        "MaxCloseAttempts": 5,
        "CloseAttemptInterval": 100,
        "EnableOperationTiming": false,
        "BufferSize": 16384,
        "Uri": null,
        "ForceSaslPlain": true
    },
    "HeartbeatConfigInterval": 2500.0,
    "ConfigPollInterval": 2500,
    "HeartbeatConfigCheckFloor": 0,
    "ConfigPollCheckFloor": 0,
    "ViewRequestTimeout": 75000,
    "DefaultConnectionLimit": 5,
    "MaxServicePointIdleTime": 100,
    "Expect100Continue": false,
    "EnableConfigHeartBeat": true,
    "ConfigPollEnabled": true,
    "EnableOperationTiming": false,
    "BufferSize": 0,
    "DefaultOperationLifespan": 2500,
    "ConfigurationProviders": 3,
    "AnalyticsRequestTimeout": 75000,
    "ForceSaslPlain": true,
    "OperationTracingEnabled": true,
    "OperationTracingServerDurationEnabled": true,
    "OrphanedResponseLoggingEnabled": true,
    "EnableCertificateAuthentication": false,
    "EnableCertificateRevocation": false,
    "CertificateFactory": null,
    "HttpServerCertificateValidationCallback": null,
    "KvServerCertificateValidationCallback": null
}

foxtrotuniform6969 · November 21, 2018, 11:56pm

The results of forwarding port 11210 on the same box and running sdk-doctor on an existing bucket using valid credentials are:

18:01:27.608 INFO ▶ Parsing connection string `couchbase://localhost/users`
18:01:27.609 INFO ▶ Connection string identifies the following CCCP endpoints:
18:01:27.609 INFO ▶   1. localhost:11210
18:01:27.609 INFO ▶ Connection string identifies the following HTTP endpoints:
18:01:27.609 INFO ▶   1. localhost:8091
18:01:27.609 INFO ▶ Connection string specifies bucket `users`
18:01:27.609 WARN ▶ Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
18:01:27.609 INFO ▶ Performing DNS lookup for host `localhost`
18:01:27.609 INFO ▶ Bootstrap host `localhost` refers to a server with the address `127.0.0.1`
18:01:27.609 INFO ▶ Attempting to connect to cluster via CCCP
18:01:27.609 INFO ▶ Attempting to fetch config via cccp from `localhost:11210`
18:01:27.893 WARN ▶ Bootstrap host `localhost` is not using the canonical node hostname of `cb-cluster-member-0000.cb-cluster-member.default.svc`.  This is not neccessarily an error, but has been known to result in strange and difficult-to-diagnose errors in the future when routing gets changed.
18:01:27.894 INFO ▶ Selected the following network type: default
18:01:27.894 INFO ▶ Identified the following nodes:
18:01:27.894 INFO ▶   [0] cb-cluster-member-0000.cb-cluster-member.default.svc
18:01:27.894 INFO ▶                     moxi: 11211,      indexStreamInit:  9103,   indexStreamCatchup:  9104
18:01:27.894 INFO ▶               indexHttps: 19102,              capiSSL: 18092,                 capi:  8092
18:01:27.894 INFO ▶                projector:  9999,                  fts:  8094,               ftsSSL: 18094
18:01:27.894 INFO ▶                indexHttp:  9102,            indexScan:  9101,     indexStreamMaint:  9105
18:01:27.894 INFO ▶                     n1ql:  8093,              n1qlSSL: 18093,                 mgmt:  8091
18:01:27.894 INFO ▶                  mgmtSSL: 18091,           indexAdmin:  9100,                kvSSL: 11207
18:01:27.894 INFO ▶                       kv: 11210
18:01:27.894 INFO ▶   [1] cb-cluster-member-0001.cb-cluster-member.default.svc
18:01:27.894 INFO ▶                     mgmt:  8091,            indexScan:  9101,           indexHttps: 19102
18:01:27.894 INFO ▶                  capiSSL: 18092,                 capi:  8092,                 n1ql:  8093
18:01:27.894 INFO ▶                      fts:  8094,           indexAdmin:  9100,            indexHttp:  9102
18:01:27.894 INFO ▶          indexStreamInit:  9103,   indexStreamCatchup:  9104,                kvSSL: 11207
18:01:27.894 INFO ▶                   ftsSSL: 18094,            projector:  9999,                 moxi: 11211
18:01:27.894 INFO ▶                  n1qlSSL: 18093,              mgmtSSL: 18091,     indexStreamMaint:  9105
18:01:27.894 INFO ▶                       kv: 11210
18:01:27.894 INFO ▶   [2] cb-cluster-member-0002.cb-cluster-member.default.svc
18:01:27.894 INFO ▶                     n1ql:  8093,                 mgmt:  8091,            indexHttp:  9102
18:01:27.894 INFO ▶               indexHttps: 19102,            projector:  9999,                   kv: 11210
18:01:27.894 INFO ▶                  mgmtSSL: 18091,            indexScan:  9101,   indexStreamCatchup:  9104
18:01:27.894 INFO ▶                    kvSSL: 11207,              n1qlSSL: 18093,                  fts:  8094
18:01:27.894 INFO ▶         indexStreamMaint:  9105,                 moxi: 11211,               ftsSSL: 18094
18:01:27.894 INFO ▶               indexAdmin:  9100,      indexStreamInit:  9103,              capiSSL: 18092
18:01:27.894 INFO ▶                     capi:  8092
18:01:27.894 INFO ▶ Fetching config from `cb-cluster-member-0000.cb-cluster-member.default.svc:8091`
18:01:27.894 INFO ▶ Failed to retreive cluster information (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8091/pools/default: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.895 ERRO ▶ Failed to connect to KV service at `cb-cluster-member-0000.cb-cluster-member.default.svc:11210` (error: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.895 ERRO ▶ Failed to connect to MGMT service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8091` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8091/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.895 ERRO ▶ Failed to connect to CAPI service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8092` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8092/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.896 ERRO ▶ Failed to connect to N1QL service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8093` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8093/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.896 ERRO ▶ Failed to connect to FTS service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8094` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8094/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.896 ERRO ▶ Failed to connect to KV service at `cb-cluster-member-0001.cb-cluster-member.default.svc:11210` (error: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.897 ERRO ▶ Failed to connect to MGMT service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8091` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8091/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.897 ERRO ▶ Failed to connect to CAPI service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8092` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8092/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.897 ERRO ▶ Failed to connect to N1QL service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8093` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8093/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.898 ERRO ▶ Failed to connect to FTS service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8094` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8094/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.898 ERRO ▶ Failed to connect to KV service at `cb-cluster-member-0002.cb-cluster-member.default.svc:11210` (error: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.898 ERRO ▶ Failed to connect to MGMT service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8091` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8091/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.899 ERRO ▶ Failed to connect to CAPI service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8092` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8092/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.899 ERRO ▶ Failed to connect to N1QL service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8093` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8093/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.899 ERRO ▶ Failed to connect to FTS service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8094` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8094/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
18:01:27.900 WARN ▶ Failed to perform KV connection performance analysis on `cb-cluster-member-0000.cb-cluster-member.default.svc:11210` (error: %!d(string=dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host))
18:01:27.900 WARN ▶ Failed to perform KV connection performance analysis on `cb-cluster-member-0001.cb-cluster-member.default.svc:11210` (error: %!d(string=dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host))
18:01:27.900 WARN ▶ Failed to perform KV connection performance analysis on `cb-cluster-member-0002.cb-cluster-member.default.svc:11210` (error: %!d(string=dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host))
18:01:27.900 INFO ▶ Diagnostics completed

Summary:
[WARN] Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
[WARN] Bootstrap host `localhost` is not using the canonical node hostname of `cb-cluster-member-0000.cb-cluster-member.default.svc`.  This is not neccessarily an error, but has been known to result in strange and difficult-to-diagnose errors in the future when routing gets changed.
[WARN] Failed to perform KV connection performance analysis on `cb-cluster-member-0000.cb-cluster-member.default.svc:11210` (error: %!d(string=dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host))
[WARN] Failed to perform KV connection performance analysis on `cb-cluster-member-0001.cb-cluster-member.default.svc:11210` (error: %!d(string=dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host))
[WARN] Failed to perform KV connection performance analysis on `cb-cluster-member-0002.cb-cluster-member.default.svc:11210` (error: %!d(string=dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host))
[ERRO] Failed to connect to KV service at `cb-cluster-member-0000.cb-cluster-member.default.svc:11210` (error: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to MGMT service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8091` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8091/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to CAPI service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8092` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8092/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to N1QL service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8093` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8093/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to FTS service at `cb-cluster-member-0000.cb-cluster-member.default.svc:8094` (error: Get http://cb-cluster-member-0000.cb-cluster-member.default.svc:8094/: dial tcp: lookup cb-cluster-member-0000.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to KV service at `cb-cluster-member-0001.cb-cluster-member.default.svc:11210` (error: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to MGMT service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8091` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8091/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to CAPI service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8092` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8092/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to N1QL service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8093` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8093/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to FTS service at `cb-cluster-member-0001.cb-cluster-member.default.svc:8094` (error: Get http://cb-cluster-member-0001.cb-cluster-member.default.svc:8094/: dial tcp: lookup cb-cluster-member-0001.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to KV service at `cb-cluster-member-0002.cb-cluster-member.default.svc:11210` (error: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to MGMT service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8091` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8091/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to CAPI service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8092` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8092/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to N1QL service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8093` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8093/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)
[ERRO] Failed to connect to FTS service at `cb-cluster-member-0002.cb-cluster-member.default.svc:8094` (error: Get http://cb-cluster-member-0002.cb-cluster-member.default.svc:8094/: dial tcp: lookup cb-cluster-member-0002.cb-cluster-member.default.svc on 127.0.0.53:53: no such host)

I attempted to resolve this by forwarding port 53 as well, but it failed to forward with a bind: permission denied

jmorris · November 22, 2018, 6:56pm

@foxtrotuniform6969 -

Hi, I don’t have much Kubernetes/GKE/Autonomous operator experiance, but I can possibily help isolate the issue for you.

There are two means of authenticating with the cluster which happens behind the scenes in the SDK - using TCP and memcached extended protocol or using HTTP streaming and a REST endpoint. Your stacktrace indicates that your failing both: a) TCP: “Connection refused…Cannot bootstrap with CCCP” using 127.0.0.1:11210 and b) HTTP: “Could not resolve hostname to IP…Couchbase.Utils.UnsupportedAddressFamilyException: http://cb-cluster-member-0000.cb-cluster-member.default.svc”.

So, to be clear your not connecting to the server from the SDK - the client itself is not even fetching a cluster map, so it doesn’t know what the server topology looks like. One thing that sticks out to me, is the URI that is failing - it looks nothing like what I would expect to see from the Couchbase SDK: http://cb-cluster-member-0000.cb-cluster-member.default.svc - which could be automous operator produced?

Anyways, it looks like the client cannot do the DNS lookup with that URI on the client for HTTP streaming and the client cannot via 127.0.0.1:11210. Hopefully that helps and somebody with more experiance with Kubernetes/GKE/Autonomous operator can comment as well!

-Jeff

foxtrotuniform6969 · November 22, 2018, 7:31pm

Jeff,

Thanks for the quick response, and happy Thanksgiving.

You’re absolutely right. Once I pushed the containerized application that was using Couchbase to the same cluster that CB was installed on, and used the DNS names for each node (i.e. http://cb-cluster-member-0000.cb-cluster-member.default.svc), the application functioned as expected.

I was hoping to use kubctl port forwarding to interact with the cluster in the cloud while running the consuming application on my local development box, but it seems that I went about it the wrong way.

Thanks you.

Topic		Replies	Views
Exposing Couchbase from Kubernetes via NodePort Kubernetes	0	756	December 10, 2020
I'm unable to connect Couchbase database after port forwarding and also unable to get bucket Kubernetes connections	1	871	November 28, 2022
Unable to fetch data form couchbase cluster deployed in kubernetes Kubernetes dot-net	4	1322	September 14, 2020
Unable to connect to couchbase server CE from Java SDK on Google Kubernetes Engine (GKE) Couchbase Server	3	429	May 21, 2023
Is it possible to access CB cluster on K8s with NodePort Service? Kubernetes java	3	1952	July 27, 2022

.NET Core: GKE Autonomous Operator: UnsupportedAddressFamily

Related topics