It’s a common experience to run into network connection errors when you’re developing an application – particularly if you’re connecting from your local computer SDK to a remote cluster.
Is your SDK not connected? You’re likely to run into situations where your application can connect to the server but you get errors related to the network connection or IP addresses between the nodes or your app and the Couchbase SDK.
This article discusses how you can troubleshoot and fix these errors. The example we’ll walkthrough below creates a cluster in Azure to demonstrate the initial conditions, but it should be applicable to other installations as well.
We’ll go through each step in order to replicate the error. First, we’ll install Couchbase in Azure, then run SDK Doctor to see how the error manifests itself, then update the alternate addresses, and finally run the SDK Doctor again to see the successful response.
Okay, let’s dive in.
Install Your Couchbase Cluster in Azure
Here are your steps for installing a Couchbase cluster in Azure.
- Log into the Microsoft Azure Marketplace
- Search for Couchbase
- Choose Couchbase Server and Sync Gateway
- Click Get It Now
- Choose BYOL
- Click Continue
- Click Create
- Fill out form
- Admin:
asdf88495
< Update this as appropriate - Password:
Password1!@#$
< Update this as appropriate - Complete configuration forms as appropriate
- Click Create
The process runs and looks like so once you’ve completed all the steps:
Update Firewall Rules
You likely need to update firewall rules to allow your local computer to connect to the remote Azure cluster. Here are the steps:
1. Go to Resource Group
2. Click on Virtual machine scale set
3. Click on Networking
4. Click Add inbound port rule
5. Update the configuration to use your local network information. Warning: This is very open for the purposes of this demonstration, you should follow your security guidelines when configuring the firewall.
Confirm Couchbase Is Up and Running
Your next major step is to confirm that Couchbase is up and running. Here’s how you do that:
1. Go to instances to get an IP address and connect to the Couchbase control panel.
2. Get the IP address for one of your servers.
3. In this example, the login is http://40.80.151.197:8091/ui/index.html
.
4. Login with the info used to create the admin account from the “Install Your Couchbase Cluster in Azure” section above. This was where you created the cluster in Azure and completed the form.
5. In Couchbase, add sample data to use for this demonstration. Follow the instructions in the Couchbase documentation for adding data from sample buckets.
Introducing the SDK Doctor
The Couchbase SDK doctor is a tool to diagnose application-server-side connectivity issues with your Couchbase cluster. It makes the same connections to the Couchbase Server cluster that Couchbase SDKs make during bootstrapping, and then reports on the state of the connections made – giving diagnostic information that helps you solve puzzling network issues.
Here is the full set of the instructions for using the SDK Doctor with Couchbase Server.
1. Running the SDK Doctor to Spot Errors
Following the instructions above – where you collected the IP address for your Couchbase cluster – get the IP addresses for each of the other nodes in your cluster as well. You’ll use the IP addresses in the connection string for the SDK Doctor.
The command below uses the IP addresses for the nodes in your cluster in addition to the sample data bucket name. You will need to update this information based on your own cluster IPs and the specific sample bucket you installed.
Now, run the SDK Doctor.
You’ll probably get a bunch of error messages – as you can see below – especially if you are running SDK Doctor from a local computer and not on the same network as the nodes/cluster/VMs. You’ll see the ERROR
and Failed to connect
messages. Notice the IP addresses are the internal 10 addresses, not the 40 or 13.
You may need to double-click the code sample below to see the full output (depending on your device).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 |
.\sdk-doctor-windows.exe diagnose couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample -u asdf88495 -p Password1!@#$ |====================================================================| | ___ ___ _ __ ___ ___ ___ _____ ___ ___ | | / __| \| |/ /__| \ / _ \ / __|_ _/ _ \| _ \ | | \__ \ |) | ' <___| |) | (_) | (__ | || (_) | / | | |___/___/|_|\_\ |___/ \___/ \___| |_| \___/|_|_\ | | | |====================================================================| Note: Diagnostics can only provide accurate results when your cluster is in a stable state. Active rebalancing and other cluster configuration changes can cause the output of the doctor to be inconsistent or in the worst cases, completely incorrect. 13:52:58.412 INFO â–¶ Parsing connection string `couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample` 13:52:58.413 INFO â–¶ Connection string identifies the following CCCP endpoints: 13:52:58.413 INFO â–¶ 1. 40.80.151.197:11210 13:52:58.414 INFO â–¶ 2. 40.80.146.52:11210 13:52:58.414 INFO â–¶ 3. 13.82.230.238:11210 13:52:58.414 INFO â–¶ Connection string identifies the following HTTP endpoints: 13:52:58.414 INFO â–¶ 1. 40.80.151.197:8091 13:52:58.414 INFO â–¶ 2. 40.80.146.52:8091 13:52:58.414 INFO â–¶ 3. 13.82.230.238:8091 13:52:58.415 INFO â–¶ Connection string specifies bucket `beer-sample` 13:52:58.415 INFO â–¶ Performing DNS lookup for host `40.80.151.197` 13:52:58.415 INFO â–¶ Performing DNS lookup for host `40.80.146.52` 13:52:58.415 INFO â–¶ Performing DNS lookup for host `13.82.230.238` 13:52:58.415 INFO â–¶ Attempting to connect to cluster via CCCP 13:52:58.416 INFO â–¶ Attempting to fetch config via cccp from `40.80.151.197:11210` 13:52:58.631 INFO â–¶ Attempting to fetch config via cccp from `40.80.146.52:11210` 13:52:58.846 INFO â–¶ Attempting to fetch config via cccp from `13.82.230.238:11210` 13:52:59.076 WARN â–¶ Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`. This is not necessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. 13:52:59.077 WARN â–¶ Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. 13:52:59.079 WARN â–¶ Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. 13:52:59.080 INFO â–¶ Selected the following network type: default 13:52:59.080 INFO â–¶ Identified the following nodes: 13:52:59.081 INFO â–¶ [0] 10.0.0.4 13:52:59.081 INFO â–¶ indexScan: 9101, indexStreamInit: 9103, indexStreamCatchup: 9104 13:52:59.081 INFO â–¶ kvSSL: 11207, projector: 9999, mgmt: 8091 13:52:59.082 INFO â–¶ mgmtSSL: 18091, indexAdmin: 9100, n1ql: 8093 13:52:59.082 INFO â–¶ indexHttp: 9102, indexHttps: 19102, kv: 11210 13:52:59.083 INFO â–¶ cbas: 8095, eventingSSL: 18096, ftsGRPCSSL: 19130 13:52:59.083 INFO â–¶ ftsSSL: 18094, ftsGRPC: 9130, indexStreamMaint: 9105 13:52:59.084 INFO â–¶ capi: 8092, n1qlSSL: 18093, cbasSSL: 18095 13:52:59.084 INFO â–¶ eventingDebug: 9140, fts: 8094, eventingAdminPort: 8096 13:52:59.085 INFO â–¶ capiSSL: 18092 13:52:59.085 INFO â–¶ [1] 10.0.0.5 13:52:59.086 INFO â–¶ ftsGRPCSSL: 19130, indexHttps: 19102, cbas: 8095 13:52:59.086 INFO â–¶ eventingAdminPort: 8096, eventingDebug: 9140, indexAdmin: 9100 13:52:59.087 INFO â–¶ indexStreamInit: 9103, indexStreamMaint: 9105, mgmtSSL: 18091 13:52:59.087 INFO â–¶ cbasSSL: 18095, ftsSSL: 18094, n1ql: 8093 13:52:59.088 INFO â–¶ kv: 11210, capi: 8092, capiSSL: 18092 13:52:59.088 INFO â–¶ ftsGRPC: 9130, indexScan: 9101, indexHttp: 9102 13:52:59.089 INFO â–¶ indexStreamCatchup: 9104, kvSSL: 11207, mgmt: 8091 13:52:59.089 INFO â–¶ eventingSSL: 18096, fts: 8094, projector: 9999 13:52:59.089 INFO â–¶ n1qlSSL: 18093 13:52:59.090 INFO â–¶ [2] 10.0.0.6 13:52:59.090 INFO â–¶ mgmtSSL: 18091, cbas: 8095, cbasSSL: 18095 13:52:59.091 INFO â–¶ eventingAdminPort: 8096, ftsSSL: 18094, indexAdmin: 9100 13:52:59.091 INFO â–¶ indexStreamCatchup: 9104, capi: 8092, ftsGRPC: 9130 13:52:59.092 INFO â–¶ indexHttp: 9102, indexStreamMaint: 9105, kvSSL: 11207 13:52:59.092 INFO â–¶ projector: 9999, eventingDebug: 9140, fts: 8094 13:52:59.093 INFO â–¶ ftsGRPCSSL: 19130, indexStreamInit: 9103, mgmt: 8091 13:52:59.093 INFO â–¶ eventingSSL: 18096, indexScan: 9101, indexHttps: 19102 13:52:59.093 INFO â–¶ kv: 11210, capiSSL: 18092, n1ql: 8093 13:52:59.094 INFO â–¶ n1qlSSL: 18093 13:52:59.095 INFO â–¶ Fetching config from `http://10.0.0.4:8091` 13:53:01.096 INFO â–¶ Failed to retreive cluster information (error: Get "http://10.0.0.4:8091/pools/default": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:03.097 ERRO â–¶ Failed to connect to Key Value service at `10.0.0.4:11210` (error: dial tcp 10.0.0.4:11210: i/o timeout) 13:53:05.098 ERRO â–¶ Failed to connect to Management service at `10.0.0.4:8091` (error: Get "http://10.0.0.4:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:07.099 ERRO â–¶ Failed to connect to Views service at `10.0.0.4:8092` (error: Get "http://10.0.0.4:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:09.101 ERRO â–¶ Failed to connect to Query service at `10.0.0.4:8093` (error: Get "http://10.0.0.4:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:11.109 ERRO â–¶ Failed to connect to Search service at `10.0.0.4:8094` (error: Get "http://10.0.0.4:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:13.110 ERRO â–¶ Failed to connect to Analytics service at `10.0.0.4:8095` (error: Get "http://10.0.0.4:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:15.110 ERRO â–¶ Failed to connect to Key Value service at `10.0.0.5:11210` (error: dial tcp 10.0.0.5:11210: i/o timeout) 13:53:17.119 ERRO â–¶ Failed to connect to Management service at `10.0.0.5:8091` (error: Get "http://10.0.0.5:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:19.119 ERRO â–¶ Failed to connect to Views service at `10.0.0.5:8092` (error: Get "http://10.0.0.5:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:21.120 ERRO â–¶ Failed to connect to Query service at `10.0.0.5:8093` (error: Get "http://10.0.0.5:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:23.120 ERRO â–¶ Failed to connect to Search service at `10.0.0.5:8094` (error: Get "http://10.0.0.5:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:25.121 ERRO â–¶ Failed to connect to Analytics service at `10.0.0.5:8095` (error: Get "http://10.0.0.5:8095/": dial tcp 10.0.0.5:8095: i/o timeout (Client.Timeout exceeded while awaiting headers)) 13:53:27.121 ERRO â–¶ Failed to connect to Key Value service at `10.0.0.6:11210` (error: dial tcp 10.0.0.6:11210: i/o timeout) 13:53:29.122 ERRO â–¶ Failed to connect to Management service at `10.0.0.6:8091` (error: Get "http://10.0.0.6:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:31.123 ERRO â–¶ Failed to connect to Views service at `10.0.0.6:8092` (error: Get "http://10.0.0.6:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:33.124 ERRO â–¶ Failed to connect to Query service at `10.0.0.6:8093` (error: Get "http://10.0.0.6:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:35.124 ERRO â–¶ Failed to connect to Search service at `10.0.0.6:8094` (error: Get "http://10.0.0.6:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:37.125 ERRO â–¶ Failed to connect to Analytics service at `10.0.0.6:8095` (error: Get "http://10.0.0.6:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) 13:53:39.126 WARN â–¶ Failed to perform KV connection performance analysis on `10.0.0.4:11210` (error: %!d(string=dial tcp 10.0.0.4:11210: i/o timeout)) 13:53:41.127 WARN â–¶ Failed to perform KV connection performance analysis on `10.0.0.5:11210` (error: %!d(string=dial tcp 10.0.0.5:11210: i/o timeout)) 13:53:43.128 WARN â–¶ Failed to perform KV connection performance analysis on `10.0.0.6:11210` (error: %!d(string=dial tcp 10.0.0.6:11210: i/o timeout)) 13:53:43.128 INFO â–¶ Diagnostics completed Summary: [WARN] Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. [WARN] Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. [WARN] Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. [WARN] Failed to perform KV connection performance analysis on `10.0.0.4:11210` (error: %!d(string=dial tcp 10.0.0.4:11210: i/o timeout)) [WARN] Failed to perform KV connection performance analysis on `10.0.0.5:11210` (error: %!d(string=dial tcp 10.0.0.5:11210: i/o timeout)) [WARN] Failed to perform KV connection performance analysis on `10.0.0.6:11210` (error: %!d(string=dial tcp 10.0.0.6:11210: i/o timeout)) [ERRO] Failed to connect to Key Value service at `10.0.0.4:11210` (error: dial tcp 10.0.0.4:11210: i/o timeout) [ERRO] Failed to connect to Management service at `10.0.0.4:8091` (error: Get "http://10.0.0.4:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Views service at `10.0.0.4:8092` (error: Get "http://10.0.0.4:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Query service at `10.0.0.4:8093` (error: Get "http://10.0.0.4:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Search service at `10.0.0.4:8094` (error: Get "http://10.0.0.4:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Analytics service at `10.0.0.4:8095` (error: Get "http://10.0.0.4:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Key Value service at `10.0.0.5:11210` (error: dial tcp 10.0.0.5:11210: i/o timeout) [ERRO] Failed to connect to Management service at `10.0.0.5:8091` (error: Get "http://10.0.0.5:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Views service at `10.0.0.5:8092` (error: Get "http://10.0.0.5:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Query service at `10.0.0.5:8093` (error: Get "http://10.0.0.5:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Search service at `10.0.0.5:8094` (error: Get "http://10.0.0.5:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Analytics service at `10.0.0.5:8095` (error: Get "http://10.0.0.5:8095/": dial tcp 10.0.0.5:8095: i/o timeout (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Key Value service at `10.0.0.6:11210` (error: dial tcp 10.0.0.6:11210: i/o timeout) [ERRO] Failed to connect to Management service at `10.0.0.6:8091` (error: Get "http://10.0.0.6:8091/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Views service at `10.0.0.6:8092` (error: Get "http://10.0.0.6:8092/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Query service at `10.0.0.6:8093` (error: Get "http://10.0.0.6:8093/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Search service at `10.0.0.6:8094` (error: Get "http://10.0.0.6:8094/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) [ERRO] Failed to connect to Analytics service at `10.0.0.6:8095` (error: Get "http://10.0.0.6:8095/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) Found multiple issues, see listing above. |
Using the Couchbase Alternate Address
Couchbase Server allows an alternate address to be assigned to any individual cluster-node, and an alternate port number to be assigned to any service running on that node.
The assigning of alternate addresses and port numbers – which can be accomplished by means of the CLI and the REST API – may facilitate communication with external applications that are not permitted to contact a cluster’s nodes directly. However, they may have access to a router or other networked entity that provides externally visible addresses on the nodes’ behalf.
Dive into the Couchbase documentation on using alternate addresses to learn more.
Add the Couchbase Alternate Address
There is a REST call to update the alternate address for your Couchbase cluster. (For more information, check out the Couchbase documentation on managing alternate addresses with the REST API.)
Next, you’ll want to update the alternate address for each IP address in the cluster. This maps the internal to the IP address that you specify.
1 |
curl -v -X PUT -u asdf88495:Password1!@#$ http://40.80.151.197:8091/node/controller/setupAlternateAddresses/external -d hostname=40.80.151.197 |
You should get a 200 response, as you can see below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
* Trying 13.82.230.238... * TCP_NODELAY set * Connected to 13.82.230.238 (13.82.230.238) port 8091 (#0) * Server auth using Basic with user 'asdf88495' > PUT /node/controller/setupAlternateAddresses/external HTTP/1.1 > Host: 13.82.230.238:8091 > Authorization: Basic YXNkZjg4NDk1OlBhc3N3b3JkMSFAIyQ= > User-Agent: curl/7.55.1 > Accept: */* > Content-Length: 22 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 22 out of 22 bytes < HTTP/1.1 200 OK < X-XSS-Protection: 1; mode=block < X-Permitted-Cross-Domain-Policies: none < X-Frame-Options: DENY < X-Content-Type-Options: nosniff < Server: Couchbase Server < Pragma: no-cache < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Date: Mon, 28 Jun 2021 18:59:31 GMT < Content-Length: 0 < Cache-Control: no-cache,no-store,must-revalidate |
Repeat the command for the other hosts as well, like so:
1 |
curl -v -X PUT -u asdf88495:Password1!@#$ http://40.80.146.52:8091/node/controller/setupAlternateAddresses/external -d hostname=40.80.146.52 |
1 |
curl -v -X PUT -u asdf88495:Password1!@#$ http://13.82.230.238:8091/node/controller/setupAlternateAddresses/external -d hostname=13.82.230.238 |
2. Running the SDK Doctor to Confirm Success
Now, run the SDK Doctor again to confirm success. You should get something like this below:
You may need to double-click the code sample below to see the full output (depending on your device).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 |
C:\david\bin\Couchbase> .\sdk-doctor-windows.exe diagnose couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample -u asdf88495 -p Password1!@#$ |====================================================================| | ___ ___ _ __ ___ ___ ___ _____ ___ ___ | | / __| \| |/ /__| \ / _ \ / __|_ _/ _ \| _ \ | | \__ \ |) | ' <___| |) | (_) | (__ | || (_) | / | | |___/___/|_|\_\ |___/ \___/ \___| |_| \___/|_|_\ | | | |====================================================================| Note: Diagnostics can only provide accurate results when your cluster is in a stable state. Active rebalancing and other cluster configuration changes can cause the output of the doctor to be inconsistent or in the worst cases, completely incorrect. 13:59:33.729 INFO â–¶ Parsing connection string `couchbase://40.80.151.197,40.80.146.52,13.82.230.238/beer-sample` 13:59:33.730 INFO â–¶ Connection string identifies the following CCCP endpoints: 13:59:33.730 INFO â–¶ 1. 40.80.151.197:11210 13:59:33.730 INFO â–¶ 2. 40.80.146.52:11210 13:59:33.730 INFO â–¶ 3. 13.82.230.238:11210 13:59:33.730 INFO â–¶ Connection string identifies the following HTTP endpoints: 13:59:33.731 INFO â–¶ 1. 40.80.151.197:8091 13:59:33.731 INFO â–¶ 2. 40.80.146.52:8091 13:59:33.731 INFO â–¶ 3. 13.82.230.238:8091 13:59:33.731 INFO â–¶ Connection string specifies bucket `beer-sample` 13:59:33.731 INFO â–¶ Performing DNS lookup for host `40.80.151.197` 13:59:33.731 INFO â–¶ Performing DNS lookup for host `40.80.146.52` 13:59:33.731 INFO â–¶ Performing DNS lookup for host `13.82.230.238` 13:59:33.731 INFO â–¶ Attempting to connect to cluster via CCCP 13:59:33.732 INFO â–¶ Attempting to fetch config via cccp from `40.80.151.197:11210` 13:59:33.947 INFO â–¶ Attempting to fetch config via cccp from `40.80.146.52:11210` 13:59:34.171 INFO â–¶ Attempting to fetch config via cccp from `13.82.230.238:11210` 13:59:34.391 WARN â–¶ Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. 13:59:34.392 WARN â–¶ Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. 13:59:34.393 WARN â–¶ Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. 13:59:34.394 INFO â–¶ Selected the following network type: external 13:59:34.394 INFO â–¶ Identified the following nodes: 13:59:34.394 INFO â–¶ [0] 40.80.151.197 13:59:34.395 INFO â–¶ projector: 9999, n1ql: 8093, n1qlSSL: 18093 13:59:34.395 INFO â–¶ cbasSSL: 18095, indexAdmin: 9100, kvSSL: 11207 13:59:34.396 INFO â–¶ capi: 8092, capiSSL: 18092, indexStreamCatchup: 9104 13:59:34.396 INFO â–¶ mgmtSSL: 18091, eventingAdminPort: 8096, eventingSSL: 18096 13:59:34.397 INFO â–¶ fts: 8094, ftsGRPC: 9130, eventingDebug: 9140 13:59:34.397 INFO â–¶ indexHttp: 9102, indexStreamMaint: 9105, indexStreamInit: 9103 13:59:34.398 INFO â–¶ indexHttps: 19102, kv: 11210, mgmt: 8091 13:59:34.398 INFO â–¶ cbas: 8095, ftsSSL: 18094, ftsGRPCSSL: 19130 13:59:34.399 INFO â–¶ indexScan: 9101 13:59:34.399 INFO â–¶ [1] 40.80.146.52 13:59:34.400 INFO â–¶ capiSSL: 18092, mgmtSSL: 18091, indexStreamInit: 9103 13:59:34.400 INFO â–¶ indexStreamCatchup: 9104, indexStreamMaint: 9105, indexHttps: 19102 13:59:34.400 INFO â–¶ n1ql: 8093, n1qlSSL: 18093, cbasSSL: 18095 13:59:34.401 INFO â–¶ eventingDebug: 9140, eventingSSL: 18096, fts: 8094 13:59:34.401 INFO â–¶ ftsGRPC: 9130, mgmt: 8091, cbas: 8095 13:59:34.402 INFO â–¶ eventingAdminPort: 8096, indexScan: 9101, kvSSL: 11207 13:59:34.402 INFO â–¶ capi: 8092, projector: 9999, ftsSSL: 18094 13:59:34.403 INFO â–¶ ftsGRPCSSL: 19130, indexAdmin: 9100, indexHttp: 9102 13:59:34.403 INFO â–¶ kv: 11210 13:59:34.403 INFO â–¶ [2] 13.82.230.238 13:59:34.404 INFO â–¶ kv: 11210, projector: 9999, eventingSSL: 18096 13:59:34.404 INFO â–¶ indexStreamInit: 9103, indexHttps: 19102, indexStreamMaint: 9105 13:59:34.405 INFO â–¶ kvSSL: 11207, capiSSL: 18092, eventingAdminPort: 8096 13:59:34.406 INFO â–¶ ftsGRPC: 9130, indexScan: 9101, ftsGRPCSSL: 19130 13:59:34.406 INFO â–¶ indexStreamCatchup: 9104, n1ql: 8093, n1qlSSL: 18093 13:59:34.407 INFO â–¶ mgmt: 8091, mgmtSSL: 18091, ftsSSL: 18094 13:59:34.407 INFO â–¶ fts: 8094, indexAdmin: 9100, indexHttp: 9102 13:59:34.407 INFO â–¶ capi: 8092, cbas: 8095, cbasSSL: 18095 13:59:34.408 INFO â–¶ eventingDebug: 9140 13:59:34.408 INFO â–¶ Fetching config from `http://40.80.151.197:8091` 13:59:34.511 INFO â–¶ Received cluster configuration, nodes list: [ { "addressFamily": "inet", "alternateAddresses": { "external": { "hostname": "40.80.151.197", "ports": { "capi": 8092, "capiSSL": 18092, "kv": 11210, "mgmt": 8091, "mgmtSSL": 18091 } } }, "clusterCompatibility": 393222, "clusterMembership": "active", "configuredHostname": "10.0.0.4:8091", "couchApiBase": "http://10.0.0.4:8092/", "couchApiBaseHTTPS": "https://10.0.0.4:18092/", "cpuCount": 4, "externalListeners": [ { "afamily": "inet", "nodeEncryption": false }, { "afamily": "inet6", "nodeEncryption": false } ], "hostname": "10.0.0.4:8091", "interestingStats": { "cmd_get": 0, "couch_docs_actual_disk_size": 17689291, "couch_docs_data_size": 11756324, "couch_spatial_data_size": 0, "couch_spatial_disk_size": 0, "couch_views_actual_disk_size": 22774, "couch_views_data_size": 11416, "curr_items": 2475, "curr_items_tot": 4897, "ep_bg_fetched": 0, "get_hits": 0, "mem_used": 22484624, "ops": 0, "vb_active_num_non_resident": 0, "vb_replica_curr_items": 2422 }, "mcdMemoryAllocated": 22468, "mcdMemoryReserved": 22468, "memoryFree": 27792080896, "memoryTotal": 29450268672, "nodeEncryption": false, "nodeUUID": "616c0853d0e7cbacab9cbdd6ec7a51f7", "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@10.0.0.4", "ports": { "direct": 11210, "distTCP": 21100, "distTLS": 21150, "httpsCAPI": 18092, "httpsMgmt": 18091 }, "recoveryType": "none", "services": [ "cbas", "eventing", "fts", "index", "kv", "n1ql" ], "status": "healthy", "systemStats": { "allocstall": 0, "cpu_cores_available": 4, "cpu_stolen_rate": 0, "cpu_utilization_rate": 3.282828282828283, "mem_free": 27792080896, "mem_limit": 29450268672, "mem_total": 29450268672, "swap_total": 0, "swap_used": 0 }, "thisNode": true, "uptime": "1122", "version": "6.6.2-9588-enterprise" }, { "addressFamily": "inet", "alternateAddresses": { "external": { "hostname": "40.80.146.52", "ports": { "capi": 8092, "capiSSL": 18092, "kv": 11210, "mgmt": 8091, "mgmtSSL": 18091 } } }, "clusterCompatibility": 393222, "clusterMembership": "active", "configuredHostname": "10.0.0.5:8091", "couchApiBase": "http://10.0.0.5:8092/", "couchApiBaseHTTPS": "https://10.0.0.5:18092/", "cpuCount": 4, "externalListeners": [ { "afamily": "inet", "nodeEncryption": false }, { "afamily": "inet6", "nodeEncryption": false } ], "hostname": "10.0.0.5:8091", "interestingStats": { "cmd_get": 0, "couch_docs_actual_disk_size": 17275589, "couch_docs_data_size": 11350638, "couch_spatial_data_size": 0, "couch_spatial_disk_size": 0, "couch_views_actual_disk_size": 22737, "couch_views_data_size": 11408, "curr_items": 2415, "curr_items_tot": 4862, "ep_bg_fetched": 0, "get_hits": 0, "mem_used": 22640944, "ops": 0, "vb_active_num_non_resident": 0, "vb_replica_curr_items": 2447 }, "mcdMemoryAllocated": 22468, "mcdMemoryReserved": 22468, "memoryFree": 28102778880, "memoryTotal": 29450268672, "nodeEncryption": false, "nodeUUID": "94565e32208f2d93915124d49842258c", "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@10.0.0.5", "ports": { "direct": 11210, "distTCP": 21100, "distTLS": 21150, "httpsCAPI": 18092, "httpsMgmt": 18091 }, "recoveryType": "none", "services": [ "cbas", "eventing", "fts", "index", "kv", "n1ql" ], "status": "healthy", "systemStats": { "allocstall": 0, "cpu_cores_available": 4, "cpu_stolen_rate": 0, "cpu_utilization_rate": 2.763819095477387, "mem_free": 28102778880, "mem_limit": 29450268672, "mem_total": 29450268672, "swap_total": 0, "swap_used": 0 }, "uptime": "1367", "version": "6.6.2-9588-enterprise" }, { "addressFamily": "inet", "alternateAddresses": { "external": { "hostname": "13.82.230.238", "ports": { "capi": 8092, "capiSSL": 18092, "kv": 11210, "mgmt": 8091, "mgmtSSL": 18091 } } }, "clusterCompatibility": 393222, "clusterMembership": "active", "configuredHostname": "10.0.0.6:8091", "couchApiBase": "http://10.0.0.6:8092/", "couchApiBaseHTTPS": "https://10.0.0.6:18092/", "cpuCount": 4, "externalListeners": [ { "afamily": "inet", "nodeEncryption": false }, { "afamily": "inet6", "nodeEncryption": false } ], "hostname": "10.0.0.6:8091", "interestingStats": { "cmd_get": 0, "couch_docs_actual_disk_size": 17363691, "couch_docs_data_size": 11432558, "couch_spatial_data_size": 0, "couch_spatial_disk_size": 0, "couch_views_actual_disk_size": 22756, "couch_views_data_size": 11415, "curr_items": 2413, "curr_items_tot": 4847, "ep_bg_fetched": 0, "get_hits": 0, "mem_used": 22550192, "ops": 0, "vb_active_num_non_resident": 0, "vb_replica_curr_items": 2434 }, "mcdMemoryAllocated": 22468, "mcdMemoryReserved": 22468, "memoryFree": 28038606848, "memoryTotal": 29450268672, "nodeEncryption": false, "nodeUUID": "d66d1a929cde564fa27cec0d3ef56520", "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@10.0.0.6", "ports": { "direct": 11210, "distTCP": 21100, "distTLS": 21150, "httpsCAPI": 18092, "httpsMgmt": 18091 }, "recoveryType": "none", "services": [ "cbas", "eventing", "fts", "index", "kv", "n1ql" ], "status": "healthy", "systemStats": { "allocstall": 0, "cpu_cores_available": 4, "cpu_stolen_rate": 0, "cpu_utilization_rate": 1.754385964912281, "mem_free": 28038606848, "mem_limit": 29450268672, "mem_total": 29450268672, "swap_total": 0, "swap_used": 0 }, "uptime": "1136", "version": "6.6.2-9588-enterprise" } ] 13:59:34.706 INFO â–¶ Successfully connected to Key Value service at `40.80.151.197:11210` 13:59:34.845 INFO â–¶ Successfully connected to Management service at `40.80.151.197:8091` 13:59:34.933 INFO â–¶ Successfully connected to Views service at `40.80.151.197:8092` 13:59:35.020 INFO â–¶ Successfully connected to Query service at `40.80.151.197:8093` 13:59:35.108 INFO â–¶ Successfully connected to Search service at `40.80.151.197:8094` 13:59:35.200 INFO â–¶ Successfully connected to Analytics service at `40.80.151.197:8095` 13:59:35.377 INFO â–¶ Successfully connected to Key Value service at `40.80.146.52:11210` 13:59:35.515 INFO â–¶ Successfully connected to Management service at `40.80.146.52:8091` 13:59:35.619 INFO â–¶ Successfully connected to Views service at `40.80.146.52:8092` 13:59:35.713 INFO â–¶ Successfully connected to Query service at `40.80.146.52:8093` 13:59:35.803 INFO â–¶ Successfully connected to Search service at `40.80.146.52:8094` 13:59:35.903 INFO â–¶ Successfully connected to Analytics service at `40.80.146.52:8095` 13:59:36.074 INFO â–¶ Successfully connected to Key Value service at `13.82.230.238:11210` 13:59:36.206 INFO â–¶ Successfully connected to Management service at `13.82.230.238:8091` 13:59:36.308 INFO â–¶ Successfully connected to Views service at `13.82.230.238:8092` 13:59:36.397 INFO â–¶ Successfully connected to Query service at `13.82.230.238:8093` 13:59:36.487 INFO â–¶ Successfully connected to Search service at `13.82.230.238:8094` 13:59:36.591 INFO â–¶ Successfully connected to Analytics service at `13.82.230.238:8095` 13:59:37.207 INFO â–¶ Memd Nop Pinged `40.80.151.197:11210` 10 times, 0 errors, 42ms min, 45ms max, 43ms mean 13:59:37.208 WARN â–¶ Memcached service on `40.80.151.197:11210` on average took longer than 10ms (was: 43ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. 13:59:37.209 WARN â–¶ Memcached service on `40.80.151.197:11210` maximally took longer than 20ms (was: 45ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. 13:59:37.803 INFO â–¶ Memd Nop Pinged `40.80.146.52:11210` 10 times, 0 errors, 42ms min, 42ms max, 42ms mean 13:59:37.804 WARN â–¶ Memcached service on `40.80.146.52:11210` on average took longer than 10ms (was: 42ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. 13:59:37.804 WARN â–¶ Memcached service on `40.80.146.52:11210` maximally took longer than 20ms (was: 42ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. 13:59:38.423 INFO â–¶ Memd Nop Pinged `13.82.230.238:11210` 10 times, 0 errors, 43ms min, 44ms max, 44ms mean 13:59:38.423 WARN â–¶ Memcached service on `13.82.230.238:11210` on average took longer than 10ms (was: 44ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. 13:59:38.423 WARN â–¶ Memcached service on `13.82.230.238:11210` maximally took longer than 20ms (was: 44ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. 13:59:38.424 INFO â–¶ Diagnostics completed Summary: [WARN] Bootstrap host `40.80.151.197` is not using the canonical node hostname of `10.0.0.4`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. [WARN] Bootstrap host `40.80.146.52` is not using the canonical node hostname of `10.0.0.5`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. [WARN] Bootstrap host `13.82.230.238` is not using the canonical node hostname of `10.0.0.6`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured. [WARN] Memcached service on `40.80.151.197:11210` on average took longer than 10ms (was: 43ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. [WARN] Memcached service on `40.80.151.197:11210` maximally took longer than 20ms (was: 45ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. [WARN] Memcached service on `40.80.146.52:11210` on average took longer than 10ms (was: 42ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. [WARN] Memcached service on `40.80.146.52:11210` maximally took longer than 20ms (was: 42ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. [WARN] Memcached service on `13.82.230.238:11210` on average took longer than 10ms (was: 44ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. [WARN] Memcached service on `13.82.230.238:11210` maximally took longer than 20ms (was: 44ms) to reply. This is usually due to network-related issues, and could significantly affect application performance. Found multiple issues, see listing above. |
Success!
Conclusion
Connecting to Couchbase clusters in the cloud offers a lot of advantages. But sometimes – due to various network and security requirements – the remote and distributed nature of systems can cause complications between your app SDK and the Couchbase cluster.
By using the Couchbase SDK Doctor and Alternate Addresses, you can easily troubleshoot any complications that arise and fix connection problems right away.