Hi,
I’m running a Couchbase cluster, in 3.0.1 community edition, where I see the memory usage growing every day, over the quota defined per server. I tried to find out which part is eating the memory, but any Couchbase graph reports a so high usage. Here is an example of what I’m talking about. In my cluster, the per RAM server quota is set to 71Go.
On a particular node, I see this reports, from Couchbase graphs:
Meta data usage: 24 Go
User RAM usage: 45 Go
Global RAM usage: 51 Go
And from the system, a pmap -d on memcached process report 77 Go used by this processus:
> pmap -d 1412
> ...
> 00000000016de000 77381432 rw--- 0000000000000000 000:00000 [ anon ]
> ...
> mapped: 77799464K writeable/private: 77710396K shared: 0K
There is 6 Go difference between the quota and the memory really used, not reported by Couchbase, but only by the system, and it’s still growing. The system is running a Linux 3.16.0-4 under Debian. I disabled transparent huge pages, and set the swapiness to 0.
Have you any idea of the origin of the problem ?
drigby
July 3, 2015, 10:22am
2
Are you looking at Virtual (VSZ) or Resident (RSS)? Much like many applications the virtual memory footprint of the memcached process may well be higher than the quota, but the RSS should be within the Server Quota.
Examine top, or run something like the following to see the RSS and VSZ of the memcached process:
ps -o rss,vsz -p $(pgrep memcached)
For example on a test machine which is configured for 300MB quota I see (values in KB):
$ ps -o rss,vsz -p $(pgrep memcached)
RSS VSZ
178356 506520
i.e. while the virtual memory size is 494MB, the actual resident is 174MB.
Thanks for your reply. I see what you mean, unfortunately, RSS and VSZ are almost the same:
ps -o rss,vsz -p $(pgrep memcached)
RSS VSZ
77091732 77799472
drigby
July 3, 2015, 11:14am
4
Can you take a look at the memory statistics?
cbstats HOSTNAME:11210 memory
and
cbstats HOSTNAME:11210 allocator
replacing HOSTNAME with your node name (or just localhost if running cbstats locally)?
Here is the result of the two commands:
For allocation:
MALLOC: 57461808912 (54799.9 MiB) Bytes in use by application
MALLOC: + 11297824768 (10774.4 MiB) Bytes in page heap freelist
MALLOC: + 9862720336 ( 9405.8 MiB) Bytes in central cache freelist
MALLOC: + 524256 ( 0.5 MiB) Bytes in transfer cache freelist
MALLOC: + 27055552 ( 25.8 MiB) Bytes in thread cache freelists
MALLOC: + 311734432 ( 297.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 78961668256 (75303.7 MiB) Actual memory used (physical + swap)
MALLOC: + 269885440 ( 257.4 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 79231553696 (75561.1 MiB) Virtual address space used
MALLOC:
MALLOC: 4123541 Spans in use
MALLOC: 38 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Total size of freelists for per-thread caches,
transfer cache, and central cache, by size class
------------------------------------------------
class 1 [ 8 bytes ] : 52934 objs; 0.4 MiB; 0.4 cum MiB
class 2 [ 16 bytes ] : 119140 objs; 1.8 MiB; 2.2 cum MiB
class 3 [ 32 bytes ] : 855193 objs; 26.1 MiB; 28.3 cum MiB
class 4 [ 48 bytes ] : 184196 objs; 8.4 MiB; 36.8 cum MiB
class 5 [ 64 bytes ] : 11118 objs; 0.7 MiB; 37.4 cum MiB
class 6 [ 80 bytes ] : 13394 objs; 1.0 MiB; 38.5 cum MiB
class 7 [ 96 bytes ] : 2063127 objs; 188.9 MiB; 227.3 cum MiB
class 8 [ 112 bytes ] : 577069 objs; 61.6 MiB; 289.0 cum MiB
class 9 [ 128 bytes ] : 45066 objs; 5.5 MiB; 294.5 cum MiB
class 10 [ 144 bytes ] : 14008 objs; 1.9 MiB; 296.4 cum MiB
class 11 [ 160 bytes ] : 31038 objs; 4.7 MiB; 301.1 cum MiB
class 12 [ 176 bytes ] : 14382 objs; 2.4 MiB; 303.5 cum MiB
class 13 [ 192 bytes ] : 3461 objs; 0.6 MiB; 304.2 cum MiB
class 14 [ 208 bytes ] : 10907 objs; 2.2 MiB; 306.3 cum MiB
class 15 [ 224 bytes ] : 11663 objs; 2.5 MiB; 308.8 cum MiB
class 16 [ 240 bytes ] : 8463 objs; 1.9 MiB; 310.8 cum MiB
class 17 [ 256 bytes ] : 5991 objs; 1.5 MiB; 312.2 cum MiB
class 18 [ 288 bytes ] : 14746 objs; 4.1 MiB; 316.3 cum MiB
class 19 [ 320 bytes ] : 1480 objs; 0.5 MiB; 316.7 cum MiB
class 20 [ 352 bytes ] : 1152 objs; 0.4 MiB; 317.1 cum MiB
class 21 [ 384 bytes ] : 1227 objs; 0.4 MiB; 317.6 cum MiB
class 22 [ 416 bytes ] : 1087 objs; 0.4 MiB; 318.0 cum MiB
class 23 [ 448 bytes ] : 1030 objs; 0.4 MiB; 318.4 cum MiB
class 24 [ 480 bytes ] : 973 objs; 0.4 MiB; 318.9 cum MiB
class 25 [ 512 bytes ] : 1933 objs; 0.9 MiB; 319.8 cum MiB
class 26 [ 576 bytes ] : 575 objs; 0.3 MiB; 320.2 cum MiB
class 27 [ 640 bytes ] : 514 objs; 0.3 MiB; 320.5 cum MiB
class 28 [ 704 bytes ] : 569 objs; 0.4 MiB; 320.8 cum MiB
class 29 [ 768 bytes ] : 484 objs; 0.4 MiB; 321.2 cum MiB
class 30 [ 832 bytes ] : 382 objs; 0.3 MiB; 321.5 cum MiB
class 31 [ 896 bytes ] : 16082 objs; 13.7 MiB; 335.2 cum MiB
class 32 [ 960 bytes ] : 890389 objs; 815.2 MiB; 1150.4 cum MiB
class 33 [ 1024 bytes ] : 1069929 objs; 1044.9 MiB; 2195.3 cum MiB
class 34 [ 1152 bytes ] : 1633535 objs; 1794.7 MiB; 3989.9 cum MiB
class 35 [ 1280 bytes ] : 280 objs; 0.3 MiB; 3990.3 cum MiB
class 36 [ 1408 bytes ] : 254 objs; 0.3 MiB; 3990.6 cum MiB
class 37 [ 1536 bytes ] : 277 objs; 0.4 MiB; 3991.0 cum MiB
class 38 [ 1792 bytes ] : 200 objs; 0.3 MiB; 3991.4 cum MiB
class 39 [ 2048 bytes ] : 138693 objs; 270.9 MiB; 4262.2 cum MiB
class 40 [ 2304 bytes ] : 64545 objs; 141.8 MiB; 4404.1 cum MiB
class 41 [ 2560 bytes ] : 162 objs; 0.4 MiB; 4404.5 cum MiB
class 42 [ 2816 bytes ] : 18357 objs; 49.3 MiB; 4453.8 cum MiB
class 43 [ 3072 bytes ] : 98603 objs; 288.9 MiB; 4742.6 cum MiB
class 44 [ 3328 bytes ] : 60816 objs; 193.0 MiB; 4935.7 cum MiB
class 45 [ 4096 bytes ] : 122084 objs; 476.9 MiB; 5412.5 cum MiB
class 46 [ 4608 bytes ] : 77812 objs; 341.9 MiB; 5754.5 cum MiB
class 47 [ 5120 bytes ] : 84056 objs; 410.4 MiB; 6164.9 cum MiB
class 48 [ 6144 bytes ] : 139578 objs; 817.8 MiB; 6982.8 cum MiB
class 49 [ 6656 bytes ] : 68770 objs; 436.5 MiB; 7419.3 cum MiB
class 50 [ 8192 bytes ] : 70818 objs; 553.3 MiB; 7972.6 cum MiB
class 51 [ 9216 bytes ] : 51878 objs; 456.0 MiB; 8428.5 cum MiB
class 52 [ 10240 bytes ] : 12598 objs; 123.0 MiB; 8551.5 cum MiB
class 53 [ 12288 bytes ] : 503 objs; 5.9 MiB; 8557.4 cum MiB
class 54 [ 13312 bytes ] : 3 objs; 0.0 MiB; 8557.5 cum MiB
class 55 [ 16384 bytes ] : 2 objs; 0.0 MiB; 8557.5 cum MiB
class 59 [ 32768 bytes ] : 19 objs; 0.6 MiB; 8558.1 cum MiB
class 72 [ 139264 bytes ] : 10 objs; 1.3 MiB; 8559.4 cum MiB
class 75 [ 163840 bytes ] : 2 objs; 0.3 MiB; 8559.7 cum MiB
------------------------------------------------
PageHeap: 119 sizes; 10774.4 MiB free; 257.4 MiB unmapped
------------------------------------------------
1 pages * 147778 spans ~ 1154.5 MiB; 1154.5 MiB cum; unmapped: 2.5 MiB; 2.5 MiB cum
2 pages * 118252 spans ~ 1847.7 MiB; 3002.2 MiB cum; unmapped: 0.1 MiB; 2.6 MiB cum
3 pages * 32226 spans ~ 755.3 MiB; 3757.5 MiB cum; unmapped: 0.1 MiB; 2.7 MiB cum
4 pages * 30645 spans ~ 957.7 MiB; 4715.2 MiB cum; unmapped: 0.1 MiB; 2.7 MiB cum
5 pages * 5889 spans ~ 230.0 MiB; 4945.2 MiB cum; unmapped: 0.0 MiB; 2.8 MiB cum
6 pages * 4618 spans ~ 216.5 MiB; 5161.7 MiB cum; unmapped: 0.0 MiB; 2.8 MiB cum
7 pages * 632 spans ~ 34.6 MiB; 5196.2 MiB cum; unmapped: 0.1 MiB; 2.9 MiB cum
8 pages * 627 spans ~ 39.2 MiB; 5235.4 MiB cum; unmapped: 0.1 MiB; 2.9 MiB cum
9 pages * 107 spans ~ 7.5 MiB; 5242.9 MiB cum; unmapped: 0.1 MiB; 3.0 MiB cum
10 pages * 146 spans ~ 11.4 MiB; 5254.3 MiB cum; unmapped: 0.1 MiB; 3.1 MiB cum
11 pages * 34 spans ~ 2.9 MiB; 5257.3 MiB cum; unmapped: 0.0 MiB; 3.1 MiB cum
12 pages * 23 spans ~ 2.2 MiB; 5259.4 MiB cum; unmapped: 0.1 MiB; 3.2 MiB cum
13 pages * 8 spans ~ 0.8 MiB; 5260.2 MiB cum; unmapped: 0.1 MiB; 3.3 MiB cum
14 pages * 3 spans ~ 0.3 MiB; 5260.6 MiB cum; unmapped: 0.0 MiB; 3.3 MiB cum
15 pages * 1 spans ~ 0.1 MiB; 5260.7 MiB cum; unmapped: 0.0 MiB; 3.3 MiB cum
16 pages * 6 spans ~ 0.8 MiB; 5261.4 MiB cum; unmapped: 0.0 MiB; 3.3 MiB cum
18 pages * 3517 spans ~ 494.6 MiB; 5756.0 MiB cum; unmapped: 0.8 MiB; 4.1 MiB cum
19 pages * 794 spans ~ 117.9 MiB; 5873.9 MiB cum; unmapped: 0.7 MiB; 4.9 MiB cum
20 pages * 3058 spans ~ 477.8 MiB; 6351.7 MiB cum; unmapped: 0.9 MiB; 5.8 MiB cum
21 pages * 646 spans ~ 106.0 MiB; 6457.7 MiB cum; unmapped: 0.8 MiB; 6.6 MiB cum
22 pages * 2288 spans ~ 393.2 MiB; 6850.9 MiB cum; unmapped: 1.0 MiB; 7.6 MiB cum
23 pages * 527 spans ~ 94.7 MiB; 6945.6 MiB cum; unmapped: 0.9 MiB; 8.5 MiB cum
24 pages * 1597 spans ~ 299.4 MiB; 7245.0 MiB cum; unmapped: 1.1 MiB; 9.7 MiB cum
25 pages * 328 spans ~ 64.1 MiB; 7309.1 MiB cum; unmapped: 1.0 MiB; 10.6 MiB cum
26 pages * 1241 spans ~ 252.1 MiB; 7561.2 MiB cum; unmapped: 1.2 MiB; 11.9 MiB cum
27 pages * 239 spans ~ 50.4 MiB; 7611.6 MiB cum; unmapped: 1.1 MiB; 12.9 MiB cum
28 pages * 1001 spans ~ 219.0 MiB; 7830.6 MiB cum; unmapped: 1.3 MiB; 14.2 MiB cum
29 pages * 193 spans ~ 43.7 MiB; 7874.3 MiB cum; unmapped: 1.4 MiB; 15.6 MiB cum
30 pages * 783 spans ~ 183.5 MiB; 8057.8 MiB cum; unmapped: 1.4 MiB; 17.0 MiB cum
31 pages * 133 spans ~ 32.2 MiB; 8090.0 MiB cum; unmapped: 1.2 MiB; 18.2 MiB cum
32 pages * 655 spans ~ 163.8 MiB; 8253.8 MiB cum; unmapped: 1.5 MiB; 19.7 MiB cum
33 pages * 105 spans ~ 27.1 MiB; 8280.8 MiB cum; unmapped: 1.3 MiB; 21.0 MiB cum
34 pages * 526 spans ~ 139.7 MiB; 8420.6 MiB cum; unmapped: 1.6 MiB; 22.6 MiB cum
35 pages * 78 spans ~ 21.3 MiB; 8441.9 MiB cum; unmapped: 1.4 MiB; 24.0 MiB cum
36 pages * 454 spans ~ 127.7 MiB; 8569.6 MiB cum; unmapped: 1.7 MiB; 25.6 MiB cum
37 pages * 77 spans ~ 22.3 MiB; 8591.8 MiB cum; unmapped: 1.4 MiB; 27.1 MiB cum
38 pages * 396 spans ~ 117.6 MiB; 8709.4 MiB cum; unmapped: 1.8 MiB; 28.9 MiB cum
39 pages * 60 spans ~ 18.3 MiB; 8727.7 MiB cum; unmapped: 1.5 MiB; 30.4 MiB cum
40 pages * 364 spans ~ 113.8 MiB; 8841.4 MiB cum; unmapped: 1.9 MiB; 32.3 MiB cum
41 pages * 41 spans ~ 13.1 MiB; 8854.6 MiB cum; unmapped: 1.6 MiB; 33.9 MiB cum
42 pages * 294 spans ~ 96.5 MiB; 8951.0 MiB cum; unmapped: 2.0 MiB; 35.8 MiB cum
43 pages * 36 spans ~ 12.1 MiB; 8963.1 MiB cum; unmapped: 1.3 MiB; 37.2 MiB cum
44 pages * 279 spans ~ 95.9 MiB; 9059.0 MiB cum; unmapped: 1.7 MiB; 38.9 MiB cum
45 pages * 19 spans ~ 6.7 MiB; 9065.7 MiB cum; unmapped: 1.8 MiB; 40.7 MiB cum
46 pages * 211 spans ~ 75.8 MiB; 9141.5 MiB cum; unmapped: 2.2 MiB; 42.8 MiB cum
47 pages * 27 spans ~ 9.9 MiB; 9151.5 MiB cum; unmapped: 1.8 MiB; 44.7 MiB cum
48 pages * 191 spans ~ 71.6 MiB; 9223.1 MiB cum; unmapped: 1.9 MiB; 46.5 MiB cum
49 pages * 28 spans ~ 10.7 MiB; 9233.8 MiB cum; unmapped: 1.9 MiB; 48.4 MiB cum
50 pages * 188 spans ~ 73.4 MiB; 9307.2 MiB cum; unmapped: 2.3 MiB; 50.8 MiB cum
51 pages * 20 spans ~ 8.0 MiB; 9315.2 MiB cum; unmapped: 2.4 MiB; 53.2 MiB cum
52 pages * 179 spans ~ 72.7 MiB; 9387.9 MiB cum; unmapped: 2.4 MiB; 55.6 MiB cum
53 pages * 21 spans ~ 8.7 MiB; 9396.6 MiB cum; unmapped: 2.1 MiB; 57.7 MiB cum
54 pages * 157 spans ~ 66.2 MiB; 9462.9 MiB cum; unmapped: 2.5 MiB; 60.2 MiB cum
55 pages * 12 spans ~ 5.2 MiB; 9468.0 MiB cum; unmapped: 1.7 MiB; 61.9 MiB cum
56 pages * 129 spans ~ 56.4 MiB; 9524.4 MiB cum; unmapped: 2.6 MiB; 64.6 MiB cum
57 pages * 18 spans ~ 8.0 MiB; 9532.5 MiB cum; unmapped: 2.7 MiB; 67.2 MiB cum
58 pages * 125 spans ~ 56.6 MiB; 9589.1 MiB cum; unmapped: 2.3 MiB; 69.5 MiB cum
59 pages * 14 spans ~ 6.5 MiB; 9595.6 MiB cum; unmapped: 2.3 MiB; 71.8 MiB cum
60 pages * 114 spans ~ 53.4 MiB; 9649.0 MiB cum; unmapped: 2.3 MiB; 74.1 MiB cum
61 pages * 12 spans ~ 5.7 MiB; 9654.7 MiB cum; unmapped: 2.4 MiB; 76.5 MiB cum
62 pages * 97 spans ~ 47.0 MiB; 9701.7 MiB cum; unmapped: 2.4 MiB; 79.0 MiB cum
63 pages * 12 spans ~ 5.9 MiB; 9707.6 MiB cum; unmapped: 2.5 MiB; 81.4 MiB cum
64 pages * 81 spans ~ 40.5 MiB; 9748.1 MiB cum; unmapped: 2.5 MiB; 83.9 MiB cum
65 pages * 16 spans ~ 8.1 MiB; 9756.2 MiB cum; unmapped: 2.5 MiB; 86.5 MiB cum
66 pages * 80 spans ~ 41.2 MiB; 9797.5 MiB cum; unmapped: 3.1 MiB; 89.5 MiB cum
67 pages * 13 spans ~ 6.8 MiB; 9804.3 MiB cum; unmapped: 3.1 MiB; 92.7 MiB cum
68 pages * 71 spans ~ 37.7 MiB; 9842.0 MiB cum; unmapped: 3.2 MiB; 95.9 MiB cum
69 pages * 9 spans ~ 4.9 MiB; 9846.9 MiB cum; unmapped: 2.7 MiB; 98.6 MiB cum
70 pages * 63 spans ~ 34.5 MiB; 9881.3 MiB cum; unmapped: 2.7 MiB; 101.3 MiB cum
71 pages * 4 spans ~ 2.2 MiB; 9883.5 MiB cum; unmapped: 2.2 MiB; 103.5 MiB cum
72 pages * 39 spans ~ 21.9 MiB; 9905.5 MiB cum; unmapped: 2.8 MiB; 106.3 MiB cum
73 pages * 4 spans ~ 2.3 MiB; 9907.7 MiB cum; unmapped: 2.3 MiB; 108.6 MiB cum
74 pages * 53 spans ~ 30.6 MiB; 9938.4 MiB cum; unmapped: 3.5 MiB; 112.1 MiB cum
75 pages * 12 spans ~ 7.0 MiB; 9945.4 MiB cum; unmapped: 2.9 MiB; 115.0 MiB cum
76 pages * 45 spans ~ 26.7 MiB; 9972.1 MiB cum; unmapped: 3.0 MiB; 118.0 MiB cum
77 pages * 10 spans ~ 6.0 MiB; 9978.1 MiB cum; unmapped: 3.6 MiB; 121.6 MiB cum
78 pages * 37 spans ~ 22.5 MiB; 10000.7 MiB cum; unmapped: 3.0 MiB; 124.6 MiB cum
79 pages * 4 spans ~ 2.5 MiB; 10003.2 MiB cum; unmapped: 1.9 MiB; 126.5 MiB cum
80 pages * 39 spans ~ 24.4 MiB; 10027.5 MiB cum; unmapped: 2.5 MiB; 129.0 MiB cum
81 pages * 4 spans ~ 2.5 MiB; 10030.1 MiB cum; unmapped: 2.5 MiB; 131.5 MiB cum
82 pages * 35 spans ~ 22.4 MiB; 10052.5 MiB cum; unmapped: 3.8 MiB; 135.4 MiB cum
83 pages * 1 spans ~ 0.6 MiB; 10053.1 MiB cum; unmapped: 0.6 MiB; 136.0 MiB cum
84 pages * 24 spans ~ 15.8 MiB; 10068.9 MiB cum; unmapped: 3.3 MiB; 139.3 MiB cum
85 pages * 2 spans ~ 1.3 MiB; 10070.2 MiB cum; unmapped: 0.0 MiB; 139.3 MiB cum
86 pages * 37 spans ~ 24.9 MiB; 10095.1 MiB cum; unmapped: 3.4 MiB; 142.7 MiB cum
87 pages * 3 spans ~ 2.0 MiB; 10097.1 MiB cum; unmapped: 2.0 MiB; 144.7 MiB cum
88 pages * 30 spans ~ 20.6 MiB; 10117.7 MiB cum; unmapped: 3.4 MiB; 148.1 MiB cum
89 pages * 2 spans ~ 1.4 MiB; 10119.1 MiB cum; unmapped: 1.4 MiB; 149.5 MiB cum
90 pages * 19 spans ~ 13.4 MiB; 10132.5 MiB cum; unmapped: 3.5 MiB; 153.0 MiB cum
91 pages * 4 spans ~ 2.8 MiB; 10135.3 MiB cum; unmapped: 2.8 MiB; 155.9 MiB cum
92 pages * 23 spans ~ 16.5 MiB; 10151.9 MiB cum; unmapped: 3.6 MiB; 159.5 MiB cum
93 pages * 3 spans ~ 2.2 MiB; 10154.0 MiB cum; unmapped: 2.2 MiB; 161.7 MiB cum
94 pages * 17 spans ~ 12.5 MiB; 10166.5 MiB cum; unmapped: 3.7 MiB; 165.3 MiB cum
95 pages * 2 spans ~ 1.5 MiB; 10168.0 MiB cum; unmapped: 1.5 MiB; 166.8 MiB cum
96 pages * 15 spans ~ 11.2 MiB; 10179.3 MiB cum; unmapped: 3.8 MiB; 170.6 MiB cum
97 pages * 2 spans ~ 1.5 MiB; 10180.8 MiB cum; unmapped: 1.5 MiB; 172.1 MiB cum
98 pages * 15 spans ~ 11.5 MiB; 10192.3 MiB cum; unmapped: 4.6 MiB; 176.7 MiB cum
99 pages * 3 spans ~ 2.3 MiB; 10194.6 MiB cum; unmapped: 2.3 MiB; 179.0 MiB cum
100 pages * 20 spans ~ 15.6 MiB; 10210.2 MiB cum; unmapped: 4.7 MiB; 183.7 MiB cum
101 pages * 1 spans ~ 0.8 MiB; 10211.0 MiB cum; unmapped: 0.8 MiB; 184.5 MiB cum
102 pages * 18 spans ~ 14.3 MiB; 10225.3 MiB cum; unmapped: 4.0 MiB; 188.5 MiB cum
104 pages * 10 spans ~ 8.1 MiB; 10233.5 MiB cum; unmapped: 3.2 MiB; 191.7 MiB cum
106 pages * 14 spans ~ 11.6 MiB; 10245.1 MiB cum; unmapped: 4.1 MiB; 195.8 MiB cum
107 pages * 3 spans ~ 2.5 MiB; 10247.6 MiB cum; unmapped: 2.5 MiB; 198.4 MiB cum
108 pages * 11 spans ~ 9.3 MiB; 10256.9 MiB cum; unmapped: 4.2 MiB; 202.6 MiB cum
109 pages * 1 spans ~ 0.9 MiB; 10257.7 MiB cum; unmapped: 0.9 MiB; 203.4 MiB cum
110 pages * 8 spans ~ 6.9 MiB; 10264.6 MiB cum; unmapped: 4.3 MiB; 207.7 MiB cum
111 pages * 2 spans ~ 1.7 MiB; 10266.3 MiB cum; unmapped: 1.7 MiB; 209.5 MiB cum
112 pages * 5 spans ~ 4.4 MiB; 10270.7 MiB cum; unmapped: 4.4 MiB; 213.8 MiB cum
113 pages * 1 spans ~ 0.9 MiB; 10271.6 MiB cum; unmapped: 0.9 MiB; 214.7 MiB cum
114 pages * 12 spans ~ 10.7 MiB; 10282.3 MiB cum; unmapped: 4.5 MiB; 219.2 MiB cum
116 pages * 10 spans ~ 9.1 MiB; 10291.3 MiB cum; unmapped: 5.4 MiB; 224.6 MiB cum
117 pages * 1 spans ~ 0.9 MiB; 10292.2 MiB cum; unmapped: 0.9 MiB; 225.5 MiB cum
118 pages * 6 spans ~ 5.5 MiB; 10297.8 MiB cum; unmapped: 4.6 MiB; 230.1 MiB cum
120 pages * 7 spans ~ 6.6 MiB; 10304.3 MiB cum; unmapped: 5.6 MiB; 235.8 MiB cum
122 pages * 5 spans ~ 4.8 MiB; 10309.1 MiB cum; unmapped: 4.8 MiB; 240.5 MiB cum
124 pages * 6 spans ~ 5.8 MiB; 10314.9 MiB cum; unmapped: 3.9 MiB; 244.4 MiB cum
125 pages * 2 spans ~ 2.0 MiB; 10316.9 MiB cum; unmapped: 2.0 MiB; 246.3 MiB cum
126 pages * 2 spans ~ 2.0 MiB; 10318.8 MiB cum; unmapped: 2.0 MiB; 248.3 MiB cum
>255 large * 251 spans ~ 713.0 MiB; 11031.8 MiB cum; unmapped: 9.1 MiB; 257.4 MiB cum
For memory:
bytes: 57365397968
ep_blob_num: 30098210
ep_blob_overhead: 1514622591
ep_item_num: 11254
ep_kv_size: 50892806250
ep_max_size: 73400320000
ep_mem_high_wat: 62390272000
ep_mem_low_wat: 55050240000
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 2758090214
ep_storedval_num: 362772373
ep_storedval_overhead: 1514622591
ep_storedval_size: 29021789840
ep_tmp_oom_errors: 0
ep_value_size: 25498739888
mem_used: 57365397968
tcmalloc_current_thread_cache_bytes: 27647072
tcmalloc_max_thread_cache_bytes: 33554432
total_allocated_bytes: 57452351696
total_fragmentation_bytes: 9894530864
total_free_mapped_bytes: 11303051264
total_free_unmapped_bytes: 269885440
total_heap_bytes: 78919819264
With the exact value from the RSS (77091732/1024/1024), I got 73.5 Go, which is a smaller difference, but still above the limit. Maybe there is a small overhead which explain that. The growth is slow, maybe it will finally stops, but I have to wait to know that.
See this section from the TCMalloc allocator stats:
TCMalloc (the memory allocator used by Couchbase) doesn’t immediately return memory to the OS, in case it needs the memory again in the near future (and it would have been a waste of time to return it to have to then re-allocate it).
As you can see from that extract; there’s 19.7GB in TCMalloc’s freelists; should account for the difference you are seeing. That’s quite a large amount, TCMalloc doesn’t normally keep back such a quantity of memory. Has the node been up a long time?
In terms of resolving this you can always swap-rebalance out the node with a new one (and then re-insert the original node after restarting couchbase-server if you wish) - this will clear out the TCMalloc heap.
Another option is an (undocumented, unsupported) method to request TCMalloc to release it’s freelist memory which you could look into. It’s unsupported as it can have an impact on the performance of any future writes to the database as the memory allocator may not have any already free memory available). Additionally it’s not tested by Couchbase QA afiak.
If you decide to give it a try, you’ll need to run the mcctl command on the specific node(s):
/opt/couchbase/bin/mcctl -h localhost:11210 set release_free_memory
Note that this only requests TCMalloc to release memory, it doesn’t always necessarily release all free memory…
OK, thanks the detailled explaination, it explains where the missing RAM space is used. The server is not running for a long time, I restarted the whole cluster on tuesday with consecutive rebalance, to release the memory. I think TCMalloc doesn’t release the memory because of the heavy traffic received by the cluster. I create tens of millions new documents per day, and compaction is running most of the time. Currently only write operations are performed, no read.
Thanks for the tips to force TCMalloc to release memory, it can help, I will test it if the memory usage don’t stop growing.
Memory usage is stable for a while. Memory overhead usage is explained, thanks again for your help.
We seems to have similar VSZ crazyness:
ps -o pid,rss,vsz,args:200 -p $(pgrep beam.smp) PID RSS VSZ COMMAND 17891 2756708 666722476 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e10240 18042 26888 889340 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname 37400 27348 1348780 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -ho
The machine have 32Gb:
free -m total used free shared buffers cached Mem: 32170 31668 501 30 325 15336 -/+ buffers/cache: 16007 16162 Swap: 16143 0 16143
So the first process seems to use 600Gb of virtual memory…this is something really really strange…
This is a node from a 4 node cluster, other nodes have similar behaviour, one of the node I restarted this morning have:
ps -o pid,rss,vsz,args:400 -p $(pgrep beam.smp) PID RSS VSZ COMMAND 40502 25912 1282068 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -ho 40536 2148212 481494160 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e10240 40632 27004 889596 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname