After install of couchbase 4.1 on OSX, it can't connect anymore

Couchbase Server
1

I used to be running Couchbase 3.0.1 on my development laptop (OSX El Capitan, v10.11.6) which was working fine. Since we want to start to use N1QL, it was time to do a reinstall (since upgrading won’t offer the new options to actually use the index and query services) I completely removed it using the instructions from here: http://developer.couchbase.com/documentation/server/current/install/install-uninstalling.html (however, I couldn’t find that Membase folder, just the Couchbase one)

Anyway, rebooted the machine, just to make sure, then started the installation for CB4, dropped the app in Applications, followed through on the web-UI and when it was finished, it ends up with:

Lost connection to server at 127.0.0.1:8091. Repeating in 1 seconds. Retry now

and that continues and that timeout just grows. I noticed that it did start memcached:

Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
stats
STAT wbufs_allocated 1
...

and a curl for http://localhost:8091/pools also gives me the expected answer:

curl http://localhost:8091/pools
{"pools":[{"name":"default","uri":"/pools/default?uuid=22f1bf1d506545e9faef02d4ae0951aa","streamingUri":"/poolsStreaming/default?uuid=22f1bf1d506545e9faef02d4ae0951aa"}],"isAdminCreds":false,"isROAdminCreds":false,"isEnterprise":false,"settings":{"maxParallelIndexers":"/settings/maxParallelIndexers?uuid=22f1bf1d506545e9faef02d4ae0951aa","viewUpdateDaemon":"/settings/viewUpdateDaemon?uuid=22f1bf1d506545e9faef02d4ae0951aa"},"uuid":"22f1bf1d506545e9faef02d4ae0951aa","implementationVersion":"4.1.0-5005-community","componentsVersion":{"lhttpc":"1.3.0","os_mon":"2.2.14","public_key":"0.21","asn1":"2.0.4","kernel":"2.16.4","ale":"4.1.0-5005-community","inets":"5.9.8","ns_server":"4.1.0-5005-community","crypto":"3.2","ssl":"5.3.3","sasl":"2.3.4","stdlib":"1.19.4"}}

so, what am I doing wrong?

after this, I uninstalled again and installed 3.1.3, which has no issues. So looks to me something is incomplete for 4.1?

2

It sounds like you followed the correct procedure to move from 3.0.1 to 4.1 on the Mac.

I have seen similar problems when the files in ~/Library/Application\ Support/Couchbase weren’t completely removed, did you check to make sure?

I find it pretty strange that the REST API on port 8091 works, while the web application does not, since they both rely on the same server data. Are there any error messages showing up in the logs? (You can find the logs here: ~/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/)

3

yeah, I checked, I also used Appcleaner to make sure any other couchbase stuff like preferences would be gone, and I rebooted after emptying the trash just to make sure that there were no open file handles that might cause interference.

where to look? I mean, there are so many logfiles there, the errors log only complains about 1 thing, a 404 on /pools/default/replications, but that would be obvious if you haven’t setup any replication at all, right?

4

the only weirdness I can find is this:

ReplicationManager 2016-08-31T10:18:36.134+10:00 [INFO] Admin port has been launched
GenericSupervisor 2016-08-31T10:18:36.134+10:00 [INFO] Adding child AdminportSupervisor to supervisor ReplicationManagerSupervisor
ReplicationManager 2016-08-31T10:18:36.134+10:00 [INFO] ReplicationManager is running
HttpServer 2016-08-31T10:18:36.134+10:00 [INFO] [xdcr:127.0.0.1:9998] new http server xdcr 127.0.0.1:9998 /
HttpServer 2016-08-31T10:18:36.134+10:00 [INFO] [xdcr:127.0.0.1:9998] ... stopped

so I checked who was using that port number, and it turns out to be beam.smp, which is another process that is started by couchbase. perhaps there is some order in which components should start that is not right?

5

I took a look at the logs in a working instance of Couchbase, and the messages you reported related to port 9998 are normal.

FYI, the ‘beam’ processes are part of ns_server, which provides the management REST API and serves up the administration web UI.

W.r.t. cleaning up the app, it should be sufficient to remove everything in ~/Library/Application Support/Couchbase. If that file is gone, then anything from the previous installation should be gone.

Can you describe in more detail the behavior you saw when you installed and launched 4.1?

  • What came up when you first went to the administration URL (http://127.0.0.1:8091)?
  • Did it show the generic “Couchbase Always scalable, always on NoSQL” message with the “Setup” button?
  • Were you able to do any configuration of the the server, e.g., specifying total memory, default buckets, etc., or did it give you the “Lost connection” message at that point?
  • what do you get from the REST API for /pools/default or /pools/default/buckets?

Thanks,

-Eben

6

so after a clean and making sure everything was empty, trashes flushed, firewall turned off, I copied a 4.1 Couchbase.app to Applications and started it.

It then goes into the menu bar at the top and opens http://127.0.0.1:8091/index.html and I got the setup screen with the setup button. It allowed me to all configuration that is possible there, so hostname, memory usage (2g), a default bucket (memcached, 256mb), the check for updates screen and setting of the administrator password, basically the entire setup, no lost connections at that point. After clicking next on that last screen, where it goes to the cluster overview (which shows), I get all kinds of red lines ‘Difficulties communicating with the cluster; displaying cached information!’ and on a yellowish background at the top ‘Lost connection to server at 127.0.0.1:8091. Repeating in xx seconds. Retry now’ and that xx counts down and then increases.

The output for /pools/default is:

{
	"storageTotals" : {
		"ram" : {
			"total" : 13191925760,
			"quotaTotal" : 2147483648,
			"quotaUsed" : 268435456,
			"used" : 12924973056,
			"usedByData" : 0,
			"quotaUsedPerNode" : 268435456,
			"quotaTotalPerNode" : 2147483648
		},
		"hdd" : {
			"total" : 249769230336,
			"quotaTotal" : 249769230336,
			"used" : 189824615055,
			"usedByData" : 0,
			"free" : 59944615281
		}
	},
	"serverGroupsUri" : "/pools/default/serverGroups?v=116193734",
	"indexMemoryQuota" : 256,
	"memoryQuota" : 2048,
	"name" : "default",
	"alerts" : [],
	"alertsSilenceURL" : "/controller/resetAlerts?token=0&uuid=e8256c8035e535a9f2a23a7eda4b780e",
	"nodes" : [{
			"systemStats" : {
				"cpu_utilization_rate" : 7.28643216080402,
				"swap_total" : 1073741824,
				"swap_used" : 129499136,
				"mem_total" : 17179869184,
				"mem_free" : 6289174528
			},
			"interestingStats" : {
				"cmd_get" : 0,
				"couch_docs_data_size" : 0,
				"curr_items" : 0,
				"curr_items_tot" : 0,
				"ep_bg_fetched" : 0,
				"get_hits" : 0,
				"mem_used" : 0,
				"ops" : 0,
				"vb_replica_curr_items" : 0
			},
			"uptime" : "313",
			"memoryTotal" : 17179869184,
			"memoryFree" : 6289174528,
			"mcdMemoryReserved" : 13107,
			"mcdMemoryAllocated" : 13107,
			"couchApiBase" : "http://myhostname:8092/",
			"otpCookie" : "eozwfuysbipqzazl",
			"clusterMembership" : "active",
			"recoveryType" : "none",
			"status" : "healthy",
			"otpNode" : "ns_1@myhostname",
			"thisNode" : true,
			"hostname" : "myhostname:8091",
			"clusterCompatibility" : 262145,
			"version" : "4.1.0-5005-community",
			"os" : "x86_64-apple-darwin13.4.0",
			"ports" : {
				"proxy" : 11211,
				"direct" : 11210
			},
			"services" : ["index", "kv", "n1ql"]
		}
	],
	"buckets" : {
		"uri" : "/pools/default/buckets?v=119323828&uuid=e8256c8035e535a9f2a23a7eda4b780e",
		"terseBucketsBase" : "/pools/default/b/",
		"terseStreamingBucketsBase" : "/pools/default/bs/"
	},
	"remoteClusters" : {
		"uri" : "/pools/default/remoteClusters?uuid=e8256c8035e535a9f2a23a7eda4b780e",
		"validateURI" : "/pools/default/remoteClusters?just_validate=1"
	},
	"controllers" : {
		"addNode" : {
			"uri" : "/controller/addNodeV2?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"rebalance" : {
			"uri" : "/controller/rebalance?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"failOver" : {
			"uri" : "/controller/failOver?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"startGracefulFailover" : {
			"uri" : "/controller/startGracefulFailover?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"reAddNode" : {
			"uri" : "/controller/reAddNode?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"reFailOver" : {
			"uri" : "/controller/reFailOver?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"ejectNode" : {
			"uri" : "/controller/ejectNode?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"setRecoveryType" : {
			"uri" : "/controller/setRecoveryType?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"setAutoCompaction" : {
			"uri" : "/controller/setAutoCompaction?uuid=e8256c8035e535a9f2a23a7eda4b780e",
			"validateURI" : "/controller/setAutoCompaction?just_validate=1"
		},
		"clusterLogsCollection" : {
			"startURI" : "/controller/startLogsCollection?uuid=e8256c8035e535a9f2a23a7eda4b780e",
			"cancelURI" : "/controller/cancelLogsCollection?uuid=e8256c8035e535a9f2a23a7eda4b780e"
		},
		"replication" : {
			"createURI" : "/controller/createReplication?uuid=e8256c8035e535a9f2a23a7eda4b780e",
			"validateURI" : "/controller/createReplication?just_validate=1"
		}
	},
	"rebalanceStatus" : "none",
	"rebalanceProgressUri" : "/pools/default/rebalanceProgress",
	"stopRebalanceUri" : "/controller/stopRebalance?uuid=e8256c8035e535a9f2a23a7eda4b780e",
	"nodeStatusesUri" : "/nodeStatuses",
	"maxBucketCount" : 10,
	"autoCompactionSettings" : {
		"parallelDBAndViewCompaction" : false,
		"databaseFragmentationThreshold" : {
			"percentage" : 30,
			"size" : "undefined"
		},
		"viewFragmentationThreshold" : {
			"percentage" : 30,
			"size" : "undefined"
		},
		"indexFragmentationThreshold" : {
			"percentage" : 30
		}
	},
	"tasks" : {
		"uri" : "/pools/default/tasks?v=44388959"
	},
	"counters" : {},
	"indexStatusURI" : "/indexStatus?v=21137658"
}

and for /pools/default/buckets:

[{
		"name" : "default",
		"bucketType" : "memcached",
		"authType" : "sasl",
		"saslPassword" : "",
		"proxyPort" : 0,
		"replicaIndex" : true,
		"uri" : "/pools/default/buckets/default?bucket_uuid=5d70f0f9efbe1c8004df5f784de04a37",
		"streamingUri" : "/pools/default/bucketsStreaming/default?bucket_uuid=5d70f0f9efbe1c8004df5f784de04a37",
		"localRandomKeyUri" : "/pools/default/buckets/default/localRandomKey",
		"controllers" : {
			"flush" : "/pools/default/buckets/default/controller/doFlush",
			"compactAll" : "/pools/default/buckets/default/controller/compactBucket",
			"compactDB" : "/pools/default/buckets/default/controller/compactDatabases",
			"purgeDeletes" : "/pools/default/buckets/default/controller/unsafePurgeBucket",
			"startRecovery" : "/pools/default/buckets/default/controller/startRecovery"
		},
		"nodes" : [{
				"systemStats" : {
					"cpu_utilization_rate" : 6.649937264742785,
					"swap_total" : 1073741824,
					"swap_used" : 129499136,
					"mem_total" : 17179869184,
					"mem_free" : 6223056896
				},
				"interestingStats" : {
					"cmd_get" : 0,
					"couch_docs_data_size" : 0,
					"curr_items" : 0,
					"curr_items_tot" : 0,
					"ep_bg_fetched" : 0,
					"get_hits" : 0,
					"mem_used" : 0,
					"ops" : 0,
					"vb_replica_curr_items" : 0
				},
				"uptime" : "323",
				"memoryTotal" : 17179869184,
				"memoryFree" : 6223056896,
				"mcdMemoryReserved" : 13107,
				"mcdMemoryAllocated" : 13107,
				"replication" : 1,
				"clusterMembership" : "active",
				"recoveryType" : "none",
				"status" : "healthy",
				"otpNode" : "ns_1@myhostname",
				"thisNode" : true,
				"hostname" : "myhostname:8091",
				"clusterCompatibility" : 262145,
				"version" : "4.1.0-5005-community",
				"os" : "x86_64-apple-darwin13.4.0",
				"ports" : {
					"proxy" : 11211,
					"direct" : 11210
				},
				"services" : ["index", "kv", "n1ql"]
			}
		],
		"stats" : {
			"uri" : "/pools/default/buckets/default/stats",
			"directoryURI" : "/pools/default/buckets/default/statsDirectory",
			"nodeStatsListURI" : "/pools/default/buckets/default/nodes"
		},
		"ddocs" : {
			"uri" : "/pools/default/buckets/default/ddocs"
		},
		"nodeLocator" : "ketama",
		"autoCompactionSettings" : false,
		"uuid" : "5d70f0f9efbe1c8004df5f784de04a37",
		"replicaNumber" : 0,
		"threadsNumber" : 3,
		"quota" : {
			"ram" : 268435456,
			"rawRAM" : 268435456
		},
		"basicStats" : {
			"quotaPercentUsed" : 0,
			"opsPerSec" : 0,
			"hitRatio" : 0,
			"itemCount" : 0,
			"memUsed" : 0
		},
		"evictionPolicy" : "valueOnly",
		"bucketCapabilitiesVer" : "",
		"bucketCapabilities" : ["cbhello", "nodesExt"]
	}
]

Obviously, I’ve replaced my hostname with ‘myhostname’ :slight_smile: