Beam.smp memory usage

Couchbase Server
1

Hi,

We have a lone Couchbase Server 4.5.0-2601 docker container running (not connected to other node) which seems use up excessive amount of memory.

There are 2 buckets on this node.
Bucket a: Usage/Allocated 3GB/12.4GB
Bucket b: Usage/Allocated: 40MB/1GB

Cluster setting: Data: 13791MB, Index: 512MB, FullText: 512MB

As per the output from top command: beam.smp takes up ~62% of 16GB while mem cache takes ~20%.

top - 13:53:07 up 76 days,  2:30,  1 user,  load average: 0.00, 0.14, 0.15
Tasks: 168 total,   1 running, 167 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.5%us,  0.4%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:  16436176k total, 15782288k used,   653888k free,   141180k buffers
Swap:        0k total,        0k used,        0k free,  1656400k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                               
 4797 1000      20   0 11.1g 9.7g 5520 S  7.7 61.9   6651:22 beam.smp                                                                                                               
 4991 1000      20   0 3956m 3.1g  12m S  0.7 19.5 670:50.15 memcached                                                                                                              
 5935 root      20   0  577m 124m 7760 S  0.0  0.8  72:26.86 sync_gateway                                                                                                           
 4839 1000      20   0 1432m  83m  17m S  2.7  0.5 844:20.04 beam.smp                                                                                                               
 4924 1000      20   0  686m  67m  17m S  0.0  0.4 130:27.47 indexer

Also, I have noticed that memory usage has been increasing over the last month and half (since last Couchbase server docker container restart)

Screen Shot 2017-02-21 at 14.20.16.png1988×522 46.9 KB

I am quite concerned about the steady increase in memory utilisation; is this normal? If not, what measures should I take to prevent an OOM.

Thanks

1 Like
2

i have similar problem in my couchbase with version: 4.5.0 my cluster its hosted in linux AWS i have 2 nodes

each one with 32Gb RAM in first node the Beam.smp use 48% of ram and in second node use 10% of ram

i attach image of process of first node (command top)


1049×116 12.2 KB

in other post @qicb said > beam.smp is responsible for monitoring and managing all other underlying server processes such as ongoing XDCR replications, cluster operations, and views

i dont have XDCR in my cluster but i have views, in total 12 views

any idea about what is doing couchbase ??

3

Looks like you are both probably hitting https://issues.couchbase.com/browse/MB-20521 which was fixed in 4.5.1.

Would it be possible for you to try with a newer version of Couchbase Server such as 4.6?

4

There is a workaround which is to restart the cluster management process, this has the effect of emptying the stats cache, releasing the memory used. The downside of this approach is that for a very short period (while the process starts up again, usually a few minutes) there will be slight disruption:

  • The web console on that particular node will be temporarily unavailable
  • Connections on 8091 will be refused
  • The node will be seen briefly as ‘pending’ by the other nodes in the cluster.

Please note that this will not affect data access as the SDKs and other components will connect directly to each other where necessary. However, due to the node being seen as ‘pending’ during this period it is important that before conducting the workaround that you disable automatic failover, this is to prevent erroneous failovers when actually the node is still available; it will be perfectly safe to re-enable automatic failover after the cluster is marked ‘active’ again
The command to do this is as follows:

curl --data “erlang:halt().” -u username:password http://hostname:8091/diag/eval

Credit to matt.carabine for the workaround ^^

2 Likes
5

Hi guys,

I found another workaround for this issue,

check out this post Uneven Ram Usage in cluster due to Bean.smp

Andrea

6

@ianmccloy Thanks for reply
i do the workaround and works thanks
with the usage of server i believe that increase ram
i will look at possible install new version

@matt.carabine Thanks for reply
the next step is install 4.5.1 couchbase server is there way to upgrade the version Frienly ??

i think create back up
donwload the couchbase server 4.5.1
remove the 4.5.0
install new couchbase server
and do restore of data

7

@EduardoCamargo

To upgrade minor version you don’t need to remove existing installation.

Follow these simple steps:

  1. Take cb_backup of all buckets (just as a precaution)
  2. Download 4.5.1 version
  3. Stop all nodes.
  4. Start installing new version in all nodes one by one.
    It will detect existing installation and will upgrade automatically without affecting data
  5. Its all done.
8

Thanks for reply :clap: @nidhks
and if i want upgrade 4.5.1 to 4.6 its the same process?

9

@EduardoCamargo
It must be the same.
We did the same for 4.1 to 4.5.

Now due to this RAM usage issue, we are planning to upgrade to 4.5.1

10

Hello nidhks how can i stop all nodes?

do you mean sudo /etc/init.d/couchbase-server stop ?
or theres a way trhought the web portal IP_Addres:8091 ?

Kind regards!!!

11

There is no way to stop a node through admin console.

You need to do
sudo /etc/init.d/couchbase-server stop

12

Ok but im worried about the warining described on: https://developer.couchbase.com/documentation/server/3.x/admin/Misc/start-stop-couchbase.html

Warning: Never stop or restart Couchbase Server before you first remove that node from a cluster.

according to your advice:
"To upgrade minor version you don’t need to remove existing installation.

Follow these simple steps:

  1. Take cb_backup of all buckets (just as a precaution)
  2. Download 4.5.1 version
  3. Stop all nodes.
  4. Start installing new version in all nodes one by one.
    It will detect existing installation and will upgrade automatically without affecting data
  5. Its all done."

i have to remove the nodes before stopping the couchbase server? or i can skip that warning?

Kind regards!!!

13

Not sure whether the same warnings are valid for 4.x versions.

Anyway it worked for me without any issues.
You can give it a try in your test environment initially.

14

Hello nidhks, i already restarted the couchbase-server service and it stopped and started normally.

Thank you very much for your reply.

Kind regards!!!