Hi,
I’m very new in Couchbase, so sorry for possible errors :).
There is a Couchbase Community Edition 5.0.1-5003-1 from Debian, with 20G RAM. The operation system runs on a Xen virtual machine - it’s a Debian 9.
Sometimes the OOM kills the cbq-engine. The course starts like this:
beam.smp invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
beam.smp cpuset=/ mems_allowed=0
CPU: 3 PID: 1048 Comm: beam.smp Not tainted 4.9.0-6-amd64 #1 Debian 4.9.82-1+deb9u3
....
Mem-Info:
active_anon:4649138 inactive_anon:357654 isolated_anon:0
active_file:112 inactive_file:145 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:4926 slab_unreclaimable:10268
mapped:1386 shmem:51446 pagetables:16139 bounce:0
free:24658 free_pcp:190 free_cma:0
...
51973 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 4194300kB
Total swap = 4194300kB
5242783 pages RAM
0 pages HighMem/MovableOnly
105740 pages reserved
0 pages hwpoisoned
...
Out of memory: Kill process 13773 (cbq-engine) score 319 or sacrifice child
Killed process 13773 (cbq-engine) total-vm:8592000kB, anon-rss:7894568kB, file-rss:0kB, shmem-rss:0kB
There is a small script, which runs in every minute, and checks the memory allocation. If there are any process, which uses much memory, starts to make the “snapshot” about the output of top command. Here is the last:
Wed Jan 22 17:50:02 CET 2020
top - 17:50:02 up 224 days, 5:22, 0 users, load average: 25.62, 7.66, 3.17
Tasks: 217 total, 2 running, 214 sleeping, 0 stopped, 1 zombie
%Cpu(s): 5.1 us, 2.5 sy, 0.0 ni, 92.1 id, 0.1 wa, 0.0 hi, 0.0 si, 0.2 st
KiB Mem: 20548172 total, 19533836 used, 1014336 free, 4052 buffers
KiB Swap: 4194300 total, 0 used, 4194300 free. 302696 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11927 couchba+ 20 0 7754484 4.848g 0 S 6.1 24.7 220:55.94 /opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
24803 wildfly 20 0 4674524 2.650g 0 S 6.1 13.5 505:29.54 java -D[Standalone] -server Duser.timezone=Europe/Budapest -Dfile.encoding=utf-8 -Xms2048m -Xmx2048M -XX:MetaspaceSize=512M XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman Djava.awt.headless=true -Dorg.jboss.boot.log.file=/opt/if3/wildfly/standalone/log/server.log Dlogging.configuration=file:/opt/if3/wildfly/standalone/configuration/logging.properties -jar /opt/if3/wildfly/jboss-m+
...
total used free shared buffers cached
Mem: 20066 19038 1028 200 3 295
-/+ buffers/cache: 18738 1327
Swap: 4095 0 4095
As you can see, the 19G of 20G total are used - but there isn’t any SWAP memory used. I know, this is an another issue, but if anybody has an idea about what can cause this heavy memory usage, please help me.
If you have any question, just let me know.
Thanks,
a.