Server fails to start in Docker on Rocky 9

Couchbase Server
1

The Couchbase 6 and 7 images fail to start in Docker on default Rocky 9 server installations. I experienced the same behavior on multiple machines with very different resources, including a small VM and a very beefy workstation.

$ docker run --rm --name cb -p 8091-8097:8091-8097 -p 9123:9123 -p 11207:11207 -p 11210:11210 -p 11280:11280 -p 18091-18097:18091-18097 couchbase:7.0.5
Starting Couchbase Server -- Web UI available at http://<ip>:8091
and logs available in /opt/couchbase/var/lib/couchbase/logs

$ curl localhost:8091
curl: (56) Recv failure: Connection reset by peer

The server appears to start, but all connections are rejected for the UI and other services.

After a few frustrating hours of debugging, I stumbled upon this article about best practices for running it in a container, where it mentions ulimits may need adjusted in production. In this case, adjustments are necessary to run it at all. I found that the nofiles ulimit in particular needs set. The following starts the server successfully:

$ docker run --rm --name cb -p 8091-8097:8091-8097 -p 9123:9123 -p 11207:11207 -p 11210:11210 -p 11280:11280 -p 18091-18097:18091-18097 --ulimit nofile=40960:40960 couchbase:7.0.5

Strangely, the documented ulimit value for nofiles is 40960, which produces a warning upon starting, saying it must be at least 200000. Increasing it to 200000 works fine, but so does dropping it to 1024 (at least to start the server and curl the UI).

When I docker exec into the container, the default ulimit for nofiles (without specifying the docker option) is as follows:

root@0f9553d9322b:/# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 766102
max locked memory       (kbytes, -l) 8192
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1073741816
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

That value (1073741816) is quite high.

I’m not sure exactly what is going on here, but I believe this issue is pretty easy to reproduce, and it would be nice to have some sort of pre-check or error logging if there is an issue with the ulimits. If there is any specific configuration that needs done on Rocky 9 (or other RHEL 9 systems), please document that as well.

Machine Details:

  • Rocky 9.4 (kernel version 5.14.0-427.22.1.el9_4.x86_64) with 196GB RAM, 36 Xeon cores, Docker version 27.3.1, containerd version 1.7.22
  • Rocky 9.5 VM (kernel version 5.14.0-503.19.1.el9_5.x86_64) with 4GB RAM, 2 cores, Docker version 27.4.1, containerd version 1.7.24
2

There might be some useful logging in the ‘logs’ directory.

3

Unfortunately, the logs directory was completely empty for me :frowning_face:

4

Hmmm… which couchbase 7 image? Can you try a 7.2.1 image?
Edit: command shows 7.0.5.
If that doesn’t work let me know, I’ll find someone that can help.

5

Thanks. I’m running 7.0.5 locally to match my production database. 7.2.1 does seem to work though without the docker option specified.

6

I don’t find Rocky Linux in the install-platforms before 7.2. Here’s the link for 7.1

1 Like
7

Well I guess that explains it. I had looked at that page, but didn’t change the version. I also was unsure if the docker image applied to that OS table, since most images don’t have specific requirements for the container host.

1 Like