jda
March 29, 2021, 3:38pm
1
So I had this error some days ago - and resolved it by removing the server from the cluster (after a few attempts). Then I re-added it and thought all was good. But yesterday it started again - and over the night it just filled up the disk (log files I guess - but the server is not very communicative at the moment).
The issues started after I restored a backup of the data into our test environment. Before restoring I flushed the database and stopped the Sync.Gateway. One observation was that it took for ages to get it back up and running - I suppose that could be a symptom of the issues we see.
This is the error. What does that mean? And how do I find out what causes this so I can fix it?
Service 'memcached' exited with status 134. Restarting. Messages:
2021-03-27T19:12:02.003070+01:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f4fe15f5000+0x37ac8]
2021-03-27T19:12:02.003076+01:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f4fe41a7000+0xb786]
2021-03-27T19:12:02.003085+01:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x99824]
2021-03-27T19:12:02.003092+01:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x6d96c]
2021-03-27T19:12:02.003098+01:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core.so.2.1.8() [0x7f4fe326f000+0x17107]
2021-03-27T19:12:02.003102+01:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core.so.2.1.8(event_base_loop+0x39f) [0x7f4fe326f000+0x1767f]
2021-03-27T19:12:02.003109+01:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x6baa9]
2021-03-27T19:12:02.003113+01:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f4fe41a7000+0x8f17]
2021-03-27T19:12:02.003119+01:00 CRITICAL /lib64/libpthread.so.0() [0x7f4fe19c3000+0x7ea5]
2021-03-27T19:12:02.003147+01:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f4fe15f5000+0xfe9fd]
Using Couchbase Community Edition server 6.6.0 build 7909 on CentOS 7
jda
March 29, 2021, 3:48pm
2
It seemed that the @2i (index?) directory was filled up. So I had to delete all contents of it as there was only 20Kb left on the partition… - so I could bring up the server again.
I’ll try to remove and rebuild all indexes…
jda
March 30, 2021, 7:07pm
3
So after having manually deleted and rebuilt all indexes last night then everything has run fine - until half an hour ago. Then this was logged:
Control connection to memcached on 'ns_1@db2.dalsgaard-data.dk' disconnected. Check logs for details.
Followed b y:
Service 'memcached' exited with status 139. Restarting. Messages:
2021-03-30T20:43:34.039308+02:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x6d10f]
2021-03-30T20:43:34.039318+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f14e9ea4000+0xfce68]
2021-03-30T20:43:34.039323+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f14e9ea4000+0x158cc3]
2021-03-30T20:43:34.039328+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f14e9ea4000+0x477ef]
2021-03-30T20:43:34.039331+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f14e9ea4000+0x47d02]
2021-03-30T20:43:34.039337+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f14e9ea4000+0x18370b]
2021-03-30T20:43:34.039342+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f14e9ea4000+0x12ba64]
2021-03-30T20:43:34.039345+02:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f14f159e000+0x8f17]
2021-03-30T20:43:34.039348+02:00 CRITICAL /lib64/libpthread.so.0() [0x7f14eedba000+0x7ea5]
2021-03-30T20:43:34.039373+02:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f14ee9ec000+0xfe9fd]
And then it reloads data - and starts over again…
Any ideas??
jda
March 30, 2021, 8:05pm
4
Ok, so this happens every 5-10 minutes… So I’ll have to stop the servers and manually delete the @i2 directory and rebuild all indexes - otherwise I can see that the disk will just be filled again…
Any explanations of why this happens would be much appreciated…
jda
March 31, 2021, 9:04pm
5
After 24 hours (and a couple of minutes) without any entries in the log - then it came with this in the log:
Control connection to memcached on 'ns_1@db1.dalsgaard-data.dk' disconnected. Check logs for details.
And right after:
Service 'memcached' exited with status 134. Restarting. Messages:
2021-03-31T22:36:53.114801+02:00 CRITICAL /opt/couchbase/bin/memcached(_ZN2cb7tracing6Tracer3endEmNSt6chrono10time_pointINS2_3_V212steady_clockENS2_8durationIlSt5ratioILl1ELl1000000000EEEEEE+0xe3) [0x400000+0x103393]
2021-03-31T22:36:53.114821+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2e8f0a4000+0xe83f2]
2021-03-31T22:36:53.114826+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2e8f0a4000+0x158cb3]
2021-03-31T22:36:53.114830+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2e8f0a4000+0x477ef]
2021-03-31T22:36:53.114833+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2e8f0a4000+0x47d02]
2021-03-31T22:36:53.114838+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2e8f0a4000+0x18370b]
2021-03-31T22:36:53.114843+02:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2e8f0a4000+0x12ba64]
2021-03-31T22:36:53.114847+02:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f2e965d0000+0x8f17]
2021-03-31T22:36:53.114851+02:00 CRITICAL /lib64/libpthread.so.0() [0x7f2e93dec000+0x7ea5]
2021-03-31T22:36:53.114873+02:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f2e93a1e000+0xfe9fd]
Why does this happen? How can I investigate further to find the cause???
After having seen this issue several times it seems that the servers run Ok for a day - and then suddenly starts complaining…
