Warning: We are having troubles communicating to the indexer process. The information might be stale

Couchbase Server
1

error1.png1094×268 35.1 KB

2

In indexer.log file
2015-09-16T10:12:17.336Z+05:30 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=91$
2015-09-16T10:12:17.369Z+05:30 [Info] Indexer::NewIndexer Status INIT
2015-09-16T10:12:17.372Z+05:30 [Info] Setting maxcpus = 4
2015-09-16T10:12:17.372Z+05:30 [Info] Setting buffer block size to 16384 bytes
2015-09-16T10:12:17.372Z+05:30 [Info] Setting log level to Info
2015-09-16T10:12:17.372Z+05:30 [Info] Indexer::NewIndexer Starting with Vbuckets 1024
2015-09-16T10:12:17.436Z+05:30 [Info] [Queryport “:9101”] started …
2015-09-16T10:12:17.438Z+05:30 [Info] New settings received:
{“indexer.settings.compaction.interval”:“00:00,00:00”,“indexer.settings.persisted_snapshot.interval”:5000,“indexer.settings.log_level”:“info”,“indexer.settings.compaction.min_frag”:30,"indexer.settings.in$
2015-09-16T10:12:17.438Z+05:30 [Info] Setting maxcpus = 4
2015-09-16T10:12:17.438Z+05:30 [Info] Setting log level to Info
2015-09-16T10:12:17.961Z+05:30 [Info] ClustMgr:handleGetLocalValue Key IndexerId
2015-09-16T10:12:17.97Z+05:30 [Info] ClustMgr:handleSetLocalValue Key IndexerId Value 66:b1:82:c3:fa:5d:f:0
2015-09-16T10:12:18.045Z+05:30 [Info] Indexer Id 66:b1:82:c3:fa:5d:f:0
2015-09-16T10:12:18.055Z+05:30 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_66:b1:82:c3:fa:5d:f:0
2015-09-16T10:12:18.056Z+05:30 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_66:b1:82:c3:fa:5d:f:0. Err proto: can’t skip unknown w$
2015-09-16T10:12:18.056Z+05:30 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received proto: can’t skip unknown wire type 7 for protobuf.Error from 127.0.0.1:9999
2015-09-16T10:12:19.057Z+05:30 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received proto: can’t skip unknown wire type 7 for protobuf.Error
2015-09-16T10:12:19.057Z+05:30 [Error] Indexer::closeAllStreams Stream MAINT_STREAM Error from Projector proto: can’t skip unknown wire type 7 for protobuf.Error. Retrying.

3

Thanks for posting the log.

Is it possible that the disk storing the indexes has run out of space?

If not, what build of 4.0 are you running?

I’ve asked one of my colleagues on the query team to take a look at the log snippet you posted and it’s not an error he recognises. I’m going to ask a colleague in the index team to take a look and see if they can help us get to the bottom of it.

In the mean-time, any detail you can offer on your environment (OS, build number, etc) would be helpful.

4

Is there any chance that your cluster is running services with different versions?

5

using ubuntu 14.04 LTS version, I’m trying to install in a server machine.

6

Do you solve the issue ?
I get the same error message, after the install of a new cluster (4.0.0-4051 Community Edition (build-4051))

7

could you share the platform you are on as well as the indexer log output you have?
thanks
-cihan

8

I’m facing sim. probs. Getting the “Warning: We are having troubles communicating to the indexer process. The information might be stale.” with 4.0.0-4051 Enterprise Edition (build-4051) on CentOS6 too.

In the indexer.log i can see the following error, which makes me think the is a upper/lowercase problem in the indexer path ("/var/local/cbIndex" vs. “/var/local/cbindex”):

goroutine 12 [runnable]:
github.com/couchbase/indexing/secondary/common.ExitOnStdinClose()
        /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:313
created by main.main
        /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:61 +0x4e3
[goport] 2015/10/13 09:55:37 /opt/couchbase/bin/indexer terminated: exit status 2
2015-10-13T09:55:42.786Z+02:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/var/local/cbIndex/@2i]
panic: mkdir /var/local/cbindex: permission denied
9

Temp. changing the permissions of “var/local” solved the problem. But I still think there is a upper/lowercase problem, because now I have two subdirectories “cbIndex” and “cbindex”.

I also found another problem now in the indexer.log:

 dial tcp 10.112.12.149:9999: connection refused

According to the installation note, this port is not required to be opened:

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092, 8093, 9100 to 9105, 9998, 18091, 18092, 11214, 11215 and
from 21100 to 21299.

But in another thread I found that it needs to be open: