I have 4.1.release installed on my Ubuntu 14 Amazon EC2 server.
Single-node cluster with turned on Elasticsearch 1.7.3 XDCR for two buckets.
Under very minor load (<5ops/sec) goxdcr process consumes 1-1.2 GB Ram 5 minutes later it started, while buckets have only 500MB each allocated RAM each.
Is it expected behavior?
Hi Rabajaba,
Goxdcr memory consumption will climb during replication but it should return to a lower level as soon as the bulk of the replication is done. Less than 5 ops per second should not impose a significant load, assuming nothing unusual is going on.
The first thing to try are the troubleshooting steps described here:
http://developer.couchbase.com/documentation/server/4.0/connectors/elasticsearch-2.1/trouble-intro.html
Take a look at your log and see if that gives you any hints about what’s going on.
You can also try reducing the XDCR Nozzles setting as described here and then re-running your test:
What seems unusual to me is that the replication doesn’t start for 5 minutes - I’m really curious about that.
It’s also possible that the issue is something fluky with your Amazon instance but if that were the case, the problem would probably disappear anyway on repeated runs.
Best,
-Will
Thanks for the response.
Troubleshooting didn’t help.
Here is my TOP for goxdcr process right after ‘service couchbase-server restart’:
18095 couchba+ 20 0 124924 4988 3972 S 0.1 0.1 0:00.01 goxdcr
18095 couchba+ 20 0 216136 8556 4664 S 0.3 0.2 0:00.02 goxdcr
18095 couchba+ 20 0 216136 8556 4664 S 0.3 0.2 0:00.03 goxdcr
18095 couchba+ 20 0 216136 8556 4664 S 0.3 0.2 0:00.04 goxdcr
18095 couchba+ 20 0 216136 8556 4664 S 0.3 0.2 0:00.05 goxdcr
18095 couchba+ 20 0 1364640 135400 6004 S 67.9 3.5 0:02.09 goxdcr
18095 couchba+ 20 0 1490736 528604 6016 S 34.6 13.7 0:03.13 goxdcr
18095 couchba+ 20 0 1505148 543328 6020 S 9.7 14.1 0:03.42 goxdcr
18095 couchba+ 20 0 1603732 576272 6036 S 11.3 14.9 0:03.76 goxdcr
18095 couchba+ 20 0 1637064 609144 6036 S 8.6 15.8 0:04.02 goxdcr
18095 couchba+ 20 0 1643600 616048 6036 S 1.7 16.0 0:04.07 goxdcr
18095 couchba+ 20 0 1644688 616048 6036 S 0.3 16.0 0:04.08 goxdcr
18095 couchba+ 20 0 1762464 668136 6048 S 7.3 17.3 0:04.30 goxdcr
18095 couchba+ 20 0 1764640 671276 6048 S 1.0 17.4 0:04.33 goxdcr
18095 couchba+ 20 0 1765732 671276 6048 S 0.3 17.4 0:04.34 goxdcr
18095 couchba+ 20 0 1822264 727528 6052 S 8.3 18.9 0:04.59 goxdcr
18095 couchba+ 20 0 1822264 727776 6052 S 0.3 18.9 0:04.60 goxdcr
18095 couchba+ 20 0 1825528 730404 6052 S 0.7 18.9 0:04.62 goxdcr
18095 couchba+ 20 0 1853076 758224 6052 S 4.7 19.7 0:04.76 goxdcr
18095 couchba+ 20 0 1877548 783020 6052 S 3.0 20.3 0:04.85 goxdcr
18095 couchba+ 20 0 1887604 792916 6052 S 2.3 20.6 0:04.92 goxdcr
18095 couchba+ 20 0 1888696 793436 6052 S 0.3 20.6 0:04.93 goxdcr
18095 couchba+ 20 0 1940616 845380 6052 S 7.3 21.9 0:05.15 goxdcr
18095 couchba+ 20 0 1943880 848264 6052 S 0.7 22.0 0:05.17 goxdcr
18095 couchba+ 20 0 1943880 848264 6052 S 0.3 22.0 0:05.18 goxdcr
18095 couchba+ 20 0 2000404 904844 6052 S 8.3 23.5 0:05.43 goxdcr
18095 couchba+ 20 0 2001492 905368 6060 S 0.3 23.5 0:05.44 goxdcr
18095 couchba+ 20 0 2004760 908248 6060 S 0.7 23.6 0:05.46 goxdcr
18095 couchba+ 20 0 2050192 953852 6060 S 6.7 24.7 0:05.66 goxdcr
18095 couchba+ 20 0 2056936 960520 6060 S 1.3 24.9 0:05.70 goxdcr
18095 couchba+ 20 0 2063636 967388 6060 S 2.0 25.1 0:05.76 goxdcr
18095 couchba+ 20 0 2116892 996.0m 6060 S 7.3 26.5 0:05.98 goxdcr
18095 couchba+ 20 0 2119068 998.8m 6060 S 1.0 26.5 0:06.01 goxdcr
18095 couchba+ 20 0 2120156 999.3m 6060 S 0.3 26.5 0:06.02 goxdcr
18095 couchba+ 20 0 2176688 1.029g 6064 S 8.3 28.0 0:06.27 goxdcr
18095 couchba+ 20 0 2179956 1.032g 6040 S 1.0 28.1 0:06.30 goxdcr
18095 couchba+ 20 0 2201800 1.054g 6040 S 3.7 28.7 0:06.41 goxdcr
18095 couchba+ 20 0 2231868 1.080g 3608 S 5.3 29.4 0:06.57 goxdcr
18095 couchba+ 20 0 2241924 1.089g 3660 S 2.3 29.6 0:06.64 goxdcr
18095 couchba+ 20 0 2241924 1.089g 3664 S 0.3 29.6 0:06.65 goxdcr
18095 couchba+ 20 0 2294932 1.139g 3696 S 7.3 31.0 0:06.87 goxdcr
18095 couchba+ 20 0 2298200 1.142g 3696 S 0.7 31.1 0:06.89 goxdcr
18095 couchba+ 20 0 2298200 1.142g 3712 S 0.3 31.1 0:06.90 goxdcr
18095 couchba+ 20 0 2354968 1.195g 3716 S 8.3 32.5 0:07.15 goxdcr
18095 couchba+ 20 0 2354936 1.196g 3868 S 31.3 32.5 0:08.09 goxdcr
18095 couchba+ 20 0 2354936 1.196g 3868 S 0.7 32.5 0:08.11 goxdcr
XDCR active work is being done on 528604 RAM.
Why process continues to consume RAM when it does nothing?
Why it needs even 500 MB RAM to replicate bucket of 43.5 MB?
Issue I’m having right now is that every 5-10 minutes system kills “goxdcr” process and it restart.
And all of it looks like there is memory leak in goxdcr process.
Situation after process is being killed:
18095 couchba+ 20 0 2354936 1.198g 0 S 0.7 32.6 0:10.48 goxdcr
18193 couchba+ 20 0 199744 8056 4536 S 0.7 0.2 0:00.02 goxdcr
18193 couchba+ 20 0 281672 8316 4692 S 0.7 0.2 0:00.04 goxdcr
18193 couchba+ 20 0 870872 56912 5932 S 28.6 1.5 0:00.90 goxdcr
18193 couchba+ 20 0 1432880 374888 6052 S 68.2 9.7 0:02.95 goxdcr
18193 couchba+ 20 0 1448124 389332 6052 S 9.3 10.1 0:03.23 goxdcr
18193 couchba+ 20 0 1457092 398816 6052 S 7.7 10.3 0:03.46 goxdcr
18193 couchba+ 20 0 1648012 457160 6060 S 13.3 11.9 0:03.86 goxdcr
18193 couchba+ 20 0 1654548 464588 6060 S 2.7 12.1 0:03.94 goxdcr
18193 couchba+ 20 0 1655636 464852 6060 S 0.3 12.1 0:03.95 goxdcr
18193 couchba+ 20 0 1773412 517188 6072 S 7.3 13.4 0:04.17 goxdcr
18193 couchba+ 20 0 1773412 517440 6072 S 0.3 13.4 0:04.18 goxdcr
18193 couchba+ 20 0 1776676 520324 6072 S 0.7 13.5 0:04.20 goxdcr
18193 couchba+ 20 0 1830776 574800 6072 S 8.0 14.9 0:04.44 goxdcr
18193 couchba+ 20 0 1832952 576816 6072 S 1.0 15.0 0:04.47 goxdcr
18193 couchba+ 20 0 1836220 579692 6076 S 0.7 15.0 0:04.49 goxdcr
18193 couchba+ 20 0 1836220 580220 6076 S 0.3 15.1 0:04.50 goxdcr
18193 couchba+ 20 0 1889584 632212 6076 S 7.3 16.4 0:04.72 goxdcr
18193 couchba+ 20 0 1899384 641864 6076 S 2.7 16.7 0:04.80 goxdcr
18193 couchba+ 20 0 1899384 642388 6076 S 0.3 16.7 0:04.81 goxdcr
18193 couchba+ 20 0 1951560 694144 6076 S 7.7 18.0 0:05.04 goxdcr
18193 couchba+ 20 0 1954828 697272 6076 S 1.0 18.1 0:05.07 goxdcr
18193 couchba+ 20 0 2009172 751556 6076 S 7.7 19.5 0:05.30 goxdcr
18193 couchba+ 20 0 2012440 754864 6084 S 1.3 19.6 0:05.34 goxdcr
18193 couchba+ 20 0 2015708 757488 6084 S 0.7 19.7 0:05.36 goxdcr
Hi Rabajaba,
Did changing the number of XDCR Nozzles have any effect?
You can try the steps listed below and either create a new ticket with Couchbase Support if you have paid support or post the logs to this forum if not. You’ll want to save them to your machine rather than uploading them to AWS in case you want to post them to the forum.
http://www.couchbase.com/wiki/display/couchbase/Working+with+the+Couchbase+Technical+Support+Team
Best,
-Will