Couchbase started behaving slowing

Couchbase Server Kubernetes
1

I have couchbase cluster and 6.0.1(enterprise) with couchbase operator 1.2

I see following error

IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 3 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
9:15:50 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 2 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
9:15:19 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 2 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
9:15:14 AM Thu Sep 26, 2019
Enabled auto-failover with timeout 120 and max count 3
auto_failover 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
9:14:54 AM Thu Sep 26, 2019
Haven’t heard from a higher priority node or a master, so I’m taking over.
mb_master 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
9:14:54 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
9:14:52 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
9:14:52 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
9:14:49 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 3 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
8:31:50 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 2 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
8:31:19 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
8:31:16 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 1 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
8:31:14 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
Enabled auto-failover with timeout 120 and max count 3
auto_failover 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
Haven’t heard from a higher priority node or a master, so I’m taking over.
mb_master 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
8:30:55 AM Thu Sep 26, 2019

Can you help what can be done ?

2

nxdomain errors are caused by your DNS server. Your cloud provider should be able to assist in determining why it is happening.

3

Its in our inhouse VM Ware host

4

Please find the zip file location . I have collected the logs from the couchbase ui

5

Now the status is very bad… if i do jstack on the spark thread i see its getting timed_waiting state

“cb-computations-3” #74 daemon prio=5 os_prio=0 tid=0x000055f058fb7800 nid=0x68 waiting on condition [0x00007f70214ce000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00089a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

“cb-core-3-2” #77 daemon prio=5 os_prio=0 tid=0x000055f05a410800 nid=0x6b waiting on condition [0x00007f6fc14cb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00081d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

“cb-core-3-1” #76 daemon prio=5 os_prio=0 tid=0x000055f05a85c800 nid=0x6a waiting on condition [0x00007f6fe14cc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0008480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)