Memory issue with couchbase server(4.5.0-2601 Community Edition)

vishal199028 · January 2, 2021, 4:37pm

We are using couchbase server with following configuration os: centos 8 version: couchbase Version: 4.5.0-2601 Community Edition (build-2601) bucket type: Memcached buckets

We can not access the our whole server due to high memory. While we are checking logs we find below error in different log files
indexer.log

       2020-12-31T14:48:20.182+00:00 [Info] memstats {"Alloc":50599240, "TotalAlloc":3056050944, "Sys":66246904, "Lookups":80870, "Mallocs":44073807,"Frees":44055350, "HeapAlloc":50599240, "HeapSys":57278464, "HeapIdle":4694016, "HeapInuse":52584448,"HeapReleased":0, "HeapObjects":18457,"GCSys":2015232, "LastGC":1609426075199119470,"PauseTotalNs":261506557, "PauseNs":[397005], "NumGC":563}
2020-12-31T14:48:20.182+00:00 [Info] PeriodicStats = {"index_not_found_errcount":0,"indexer_state":"Active","memory_quota":536870912,"memory_used":54972416,"memory_used_queue":0,"memory_used_storage":372736,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"94005 6877099903 713491757417971","uptime":"18h41m2.458393278s"}
2020-12-31T14:48:23.249+00:00 [Info] Indexer::ReadMemstats Time Taken 215.903Âµs
2020-12-31T14:48:33.250+00:00 [Info] Indexer::ReadMemstats Time Taken 75.201Âµs
2020-12-31T14:48:43.250+00:00 [Info] Indexer::ReadMemstats Time Taken 230.303Âµs
2020-12-31T14:48:53.251+00:00 [Info] Indexer::ReadMemstats Time Taken 83.401Âµs
2020-12-31T14:49:02.785+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2020-12-31T14:49:02.785+00:00 [Error] 	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/common.(*PeerPipe).doReceive.func1(0xc822142780)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:192 +0xf8
panic(0xf29280, 0xc823667dd0)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:426 +0x4e9
github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc822142780, 0x1b252d3132333435, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:255 +0x8c
github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc822142780)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:211 +0x373
created by github.com/couchbase/gometa/common.NewPeerPipe
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x142

2020-12-31T14:49:02.786+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2020-12-31T14:49:02.786+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.10.0.9:54611)
2020-12-31T14:49:03.251+00:00 [Info] Indexer::ReadMemstats Time Taken 140.901Âµs
2020-12-31T14:49:03.598+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2020-12-31T14:49:03.598+00:00 [Error] 	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/common.(*PeerPipe).doReceive.func1(0xc823591b00)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:192 +0xf8
panic(0xf29280, 0xc82360a180)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:426 +0x4e9
github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc823591b00, 0x1b252d3132333435, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:255 +0x8c
github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc823591b00)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:211 +0x373
created by github.com/couchbase/gometa/common.NewPeerPipe
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x142

2020-12-31T14:49:03.598+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2020-12-31T14:49:03.598+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.10.0.9:54622)
2020-12-31T14:49:07.855+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2020-12-31T14:49:07.856+00:00 [Error] 	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/common.(*PeerPipe).doReceive.func1(0xc822143d70)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:192 +0xf8
panic(0xf29280, 0xc823569050)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:426 +0x4e9
github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc822143d70, 0x1b252d3132333435, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:255 +0x8c
github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc822143d70)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:211 +0x373
created by github.com/couchbase/gometa/common.NewPeerPipe
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x142

2020-12-31T14:49:07.856+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2020-12-31T14:49:07.856+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.10.0.9:54694)
2020-12-31T14:49:08.379+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2020-12-31T14:49:08.379+00:00 [Error] 	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/common.(*PeerPipe).doReceive.func1(0xc82012af00)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:192 +0xf8
panic(0xf29280, 0xc823569830)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:426 +0x4e9
github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc82012af00, 0x1b252d3132333435, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:255 +0x8c
github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc82012af00)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:211 +0x373
created by github.com/couchbase/gometa/common.NewPeerPipe
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x142

2020-12-31T14:49:08.379+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate

debug.log
ns_server:error,2020-12-31T14:51:19.184Z,ns_1@10.10.0.95:index_stats_collector-index<0.4315.363>:index_rest:get_json:42]Request to (indexer) http://127.0.0.1:84/stats?async=true failed: {error,

stats.log
[stats:warn,2020-12-31T14:51:19.184Z,ns_1@10.10.0.95:index_stats_collector-index<0.4315.363>:base_stats_collector:latest_tick:69](Collector: index_stats_collector) Dropped 10 ticks

What is the reason of high memory? What is meaning of that error in indexer.log?

Topic		Replies	Views
Couchbase 4.5 Community Version MemCached Crashes on OOM Couchbase Server	1	1776	November 10, 2016
Memory used is high, but user data in RAM is low, and disk usage is low also. May be memory leak? Couchbase Server	3	2271	June 4, 2015
Memory High usage Couchbase Server	7	1151	October 3, 2017
CouchBase Server Out Of Memory (server 4.1.0 community) Couchbase Server	1	1154	February 3, 2017
Hard Out Of Memory Error. Bucket "xxx" inusual use of RAM process memcached Couchbase Server	0	862	May 29, 2018

Memory issue with couchbase server(4.5.0-2601 Community Edition)

Related topics