Couchbase-server official docker image starting but cant connect to it && no log errors

Couchbase Server
1

Already reported here but I am not sure if the repo is active.

I am having trouble running the Couchbase using docker.
No errors to the console and no errors to the logs.
The command I am using:

# docker run --name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase:enterprise-7.0.3

output:

Starting Couchbase Server -- Web UI available at http://<ip>:8091
and logs available in /opt/couchbase/var/lib/couchbase/logs

However, It refuses the connection whenever I connect to http://localhost:8091:

This site can’t be reached The connection was reset.
ERR_CONNECTION_RESET

Here are services statuses on the docker image:

root@07:/# service  --status-all
 [ - ]  cron
 [ ? ]  hwclock.sh
 [ - ]  procps
 [ - ]  sysstat

Here are the ports that are allocated on the docker image:

root@07:/# netstat -ltnp                  
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.11:35907        0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:21200         0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::4369                 :::*                    LISTEN      -

Here are the list of processes containing couchbase on the docker image:

root@07:/# ps aux | grep -i 'couchbase'
root          41  0.0  0.0   2380   720 ?        Ss   00:29   0:00 runsv couchbase-server
couchba+      42  0.0  9.8 6678000 1613224 ?     Sl   00:29   0:01 /opt/couchbase/lib/erlang/erts-10.7.2.7/bin/beam.smp -A 16 -sbwt none -- -root /opt/couchbase/lib/erlang -progname erl -- -home /tmp -- -smp enable -kernel logger [{handler, default, undefined}] inetrc "/opt/couchbase/etc/couchbase/hosts.cfg" dist_config_file "/opt/couchbase/var/lib/couchbase/config/dist_cfg" -hidden -name babysitter_of_ns_1@cb.local -proto_dist cb -epmd_module cb_epmd -start_epmd false -ssl_dist_optfile /opt/couchbase/etc/couchbase/ssl_dist_opts -setcookie nocookie -- -kernel global_enable_tracing false -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.cookie" -ns_babysitter nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.babysitter.node" -ns_babysitter pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" --
couchba+      61  0.0  0.0   3932   104 ?        S    00:29   0:00 /opt/couchbase/lib/erlang/erts-10.7.2.7/bin/epmd -daemon
couchba+     177  0.0  0.0   3336  1576 ?        Ssl  00:33   0:00 /opt/couchbase/bin/gosecrets
couchba+    1167  0.7  9.7 6663016 1600420 ?     Ssl  01:07   0:00 /opt/couchbase/lib/erlang/erts-10.7.2.7/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -sbwt none -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /tmp -- -smp enable -setcookie nocookie -kernel logger [{handler, default, undefined}] -user user_io -run child_erlang child_start ns_bootstrap -- -smp enable -kernel logger [{handler, default, undefined}] inetrc "/opt/couchbase/etc/couchbase/hosts.cfg" dist_config_file "/opt/couchbase/var/lib/couchbase/config/dist_cfg" -proto_dist cb -epmd_module cb_epmd -start_epmd false -ssl_dist_optfile /opt/couchbase/etc/couchbase/ssl_dist_opts -kernel global_enable_tracing false -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini --
root        1229  0.0  0.0   3312   720 pts/0    S+   01:09   0:00 grep --color=auto -i couchbase

I have tried to check the error logs but the file is empty

root@07:/# cat /opt/couchbase/var/lib/couchbase/logs/error.log

There are other logs files, Please let me know if I need to attach any of them.

The attempts I have tried and failed so far:

  • used the following images
    • couchbase:community-7.0.2
    • couchbase:enterprise-7.0.1
    • couchbase:enterprise-6.6.5
  • reinstall docker
  • upgrade && downgrade linux kernel
  • run couchbase manually from /etc/service/couchbase-server/run

SPECS:

  • OS: manjaro-21.2.5
  • Kernel: Linux 5.15.28-1
  • RAM: 16GB
  • CPU: 6 cores
  • docker: 20.10.12, build e91ed5707e
2

Hey @Kareem_Mohamed,

Thanks for your report. Could you check your network settings as well? It might be an issue with proxy/security settings stopping you from connecting to the port.

You can use the SDK doctor to help determine this:

3

Hi @Nebasuke,

Thank you for your response.
I have downloaded sdk-doctor inside the docker container as follows:

$kareem:/# docker exec -it <container_id> bash 
root@kareem:/# wget <sdk-doctor-url>
root@kareem:/# ./sdk-doctor diagnose couchbase://127.0.0.1/default

Here is the output

root@kareem-pc:/# ./sdk-doctor diagnose couchbase://127.0.0.1/default
|====================================================================|
|          ___ ___  _  __   ___   ___   ___ _____ ___  ___           |
|         / __|   \| |/ /__|   \ / _ \ / __|_   _/ _ \| _ \          |
|         \__ \ |) | ' <___| |) | (_) | (__  | || (_) |   /          |
|         |___/___/|_|\_\  |___/ \___/ \___| |_| \___/|_|_\          |
|                                                                    |
|====================================================================|

Note: Diagnostics can only provide accurate results when your cluster
 is in a stable state.  Active rebalancing and other cluster configuration
 changes can cause the output of the doctor to be inconsistent or in the
 worst cases, completely incorrect.

10:27:45.428 INFO ▶ Parsing connection string `couchbase://127.0.0.1/default`
10:27:45.428 INFO ▶ Connection string identifies the following CCCP endpoints:
10:27:45.428 INFO ▶   1. 127.0.0.1:11210
10:27:45.428 INFO ▶ Connection string identifies the following HTTP endpoints:
10:27:45.428 INFO ▶   1. 127.0.0.1:8091
10:27:45.428 INFO ▶ Connection string specifies bucket `default`
10:27:45.428 WARN ▶ Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
10:27:45.428 INFO ▶ Performing DNS lookup for host `127.0.0.1`
10:27:45.428 INFO ▶ Attempting to connect to cluster via CCCP
10:27:45.428 INFO ▶ Attempting to fetch config via cccp from `127.0.0.1:11210`
10:27:45.428 ERRO ▶ Failed to fetch configuration via cccp from `127.0.0.1:11210` (error: dial tcp 127.0.0.1:11210: connect: connection refused)
10:27:45.428 INFO ▶ Attempting to connect to cluster via HTTP (Terse)
10:27:45.428 INFO ▶ Attempting to fetch terse config via http from `127.0.0.1:8091`
10:27:45.428 ERRO ▶ Failed to fetch terse configuration via http from `127.0.0.1:8091` (error: Get "http://127.0.0.1:8091/pools/default/b/default": dial tcp 127.0.0.1:8091: connect: connection refused)
10:27:45.428 INFO ▶ Attempting to connect to cluster via HTTP (Full)
10:27:45.428 INFO ▶ Failed to connect via HTTP (Full), as it is not yet supported by the doctor
10:27:45.428 INFO ▶ Selected the following network type: 
10:27:45.428 ERRO ▶ All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible
10:27:45.428 INFO ▶ Diagnostics completed

Summary:
[WARN] Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
[ERRO] Failed to fetch configuration via cccp from `127.0.0.1:11210` (error: dial tcp 127.0.0.1:11210: connect: connection refused)
[ERRO] Failed to fetch terse configuration via http from `127.0.0.1:8091` (error: Get "http://127.0.0.1:8091/pools/default/b/default": dial tcp 127.0.0.1:8091: connect: connection refused)
[ERRO] All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible

Found multiple issues, see listing above.

Here is the content of my docker-compose.yml:

services:
  couchbase_db:
  # get the image from the couchbase/server
    image: couchbase/server:enterprise-7.0.3
    volumes:
      - ./logs:/opt/couchbase/var/lib/couchbase/logs
      - ./data:/opt/couchbase/var/lib/couchbase/data
    network_mode: "host"

Please Note: All the commands I am using in this reply or in the original post were executed inside the docker container.

4

Hi @Kareen_Mohamed,

Could you post the output of docker logs command for couchbase container and its syslog?

Thank you.

5

Hi @chedim,

These are the only logs that are displayed for the couchbase container:

/$ docker logs <container_id>
> Starting Couchbase Server -- Web UI available at http://<ip>:8091
and logs available in /opt/couchbase/var/lib/couchbase/logs

Here are my latest updates tracing this issue:

I have tested the same docker-compose file and have noticed a common factor
The systems specs I have used:

  • Manjaro-21.2.5 | Linux 5.15.28-1 (*)
  • Manjaro-21.2.5 | Linux 5.10.* (*)
  • Garuda 2204 | Linux 5.15 (*)
  • Debian 10 buster | Linux 5.10.101-amd64-desktop
  • Ubuntu 20.04.4 | Linux 5.13

NOTES:

  • The star (*), in the end, indicates the problem persisted for this device.
  • The test was conducted on different devices (Not virtual machines) using the same docker-compose file.
  • The problem seems to replicate in my case on Arch Linux based OSs only
  • Changing the kernel version seems to have no effect on this problem.
  • The Debian based systems work fine using the same configurations

For Debian
Using ps aux | grep -i 'couchbase' inside the docker container bash results in +12 processes owned by couchbase user

For Arch
Using ps aux | grep -i 'couchbase' inside the docker container bash results on only 4 processes owned by couchbase user

Seems some of the processes failed to start on archlinux for some reason.

Lastly:

I have used the following images to confirm if the problem related to couchbase. However, all these images run without problems on Archlinux based machines:

7

You might have better luck using this: GitHub - brantburnett/couchbasefakeit: Docker scripts for a base Couchbase Server image for testing/development, with support for fakeit for data generation