Unable to create indexes (windows)

Couchbase Server
1

I have a new installation of Couchbase Server 6.6 community edition and I have the problem that I can create only one index on a bucket.
The first index I create is successful but any other index I try to create get stuck in “Building 0%” phase.
I have one bucket with two documents, and 1GB data quota, 512MB index and 521MB Search quota.

I used to have Couchbase Server 6 installed earlier this year which would have this issue. I don’t remember what I did to make it work, but now absolutely nothing helps. I did a dozen reinstalls with all versions from 6 onwards, deleted everything I could find that couchbase creates, trying to fix this issue.
I’m using the default options for everything, and I’m about to give up on Couchbase. The only thing I didn’t try is reinstalling Windows, which is not even an option.

image
image1085×358 23.6 KB

After only a few minutes of running I have dozens of memcached.log.0000X.txt files with this in it:

2021-07-07T09:37:55.024143+03:00 INFO 1324: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/1 - (vb:845) ActiveStream::transitionState: Transitioning from backfilling to in-memory
2021-07-07T09:37:55.026056+03:00 INFO 1316: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/2 - (vb:846) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615, collections-manifest uid:none, sid:none
2021-07-07T09:37:55.026078+03:00 INFO 1316: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/2 - (vb:846) ActiveStream::scheduleBackfill_UNLOCKED register cursor with name "eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/2" backfill:true, seqno:1
2021-07-07T09:37:55.026085+03:00 INFO 1316: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/2 - (vb:846) ActiveStream::transitionState: Transitioning from backfilling to in-memory
2021-07-07T09:37:55.027028+03:00 INFO 1188: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/3 - (vb:847) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615, collections-manifest uid:none, sid:none
2021-07-07T09:37:55.027044+03:00 INFO 1188: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/3 - (vb:847) ActiveStream::scheduleBackfill_UNLOCKED register cursor with name "eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/3" backfill:true, seqno:1
2021-07-07T09:37:55.027049+03:00 INFO 1188: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/3 - (vb:847) ActiveStream::transitionState: Transitioning from backfilling to in-memory
2021-07-07T09:37:55.029019+03:00 INFO 1328: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/0 - (vb:848) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615, collections-manifest uid:none, sid:none
2021-07-07T09:37:55.029036+03:00 INFO 1328: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/0 - (vb:848) ActiveStream::scheduleBackfill_UNLOCKED register cursor with name "eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/0" backfill:true, seqno:1
2021-07-07T09:37:55.029041+03:00 INFO 1328: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/0 - (vb:848) ActiveStream::transitionState: Transitioning from backfilling to in-memory
2021-07-07T09:37:55.030146+03:00 INFO 1324: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/1 - (vb:849) Creating stream with start seqno 0 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615, collections-manifest uid:none, sid:none
2021-07-07T09:37:55.030169+03:00 INFO 1324: (cbltest) DCP (Producer) eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/1 - (vb:849) ActiveStream::scheduleBackfill_UNLOCKED register cursor with name "eq_dcpq:secidx:proj-cbltest-INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee-12961926129680788860/1" backfill:true, seqno:1

And here’s the indexer log that also seems to be stuck:

indexer.zip (57.2 KB)

2

Never mind. Restarting my computer fixed it. Typical.

Actually this is still very much an issue.
I tried to create another index and it got stuck again. Restarted the service and nothing. Restarted the computer and the index got created. I use windows only for development but not being able to create indexes easily, is kind of a deal breaker.

3

Hi @Rudolf_Martincsek

Sorry for the experience. Will look into the issue and get back. It will be helpful if you can collect all the logs and share it with us. You can use cbcollect and share the logs.

Thanks
Sai

4

Hi @Rudolf_Martincsek

We are not able to conclusively root cause the issue with only the indexer log. We will need projector log also for the RCA. Please share us the cbcollect logs whenever you get time. We have created indexes on windows vm in our local testing in past so we should be able to figure out whats wrong in this case given the logs. We want to check if projector process is able to communicate with indexer for the the INIT_STREAM we are not seeing STREAMBEGIN for INIT_STREAM.

Thanks
Sai

5

Hi Sai,
sorry I was not able to work on this the past week.
you can find the collected logs here

6

Hi @Rudolf_Martincsek

Thanks for the logs i am seeing errors like below in the projector logs.

2021-07-07T09:35:15.926+03:00 [Error] transport error between 127.0.0.1:62404->127.0.0.1:9103: write tcp 127.0.0.1:62404->127.0.0.1:9103: wsasend: An established connection was aborted by the software in your host machine.
2021-07-07T09:35:15.926+03:00 [Error] ENDP[<-(127.0.0.1:9103,e544)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_57fb1b5acbd397ee21165b69bd80d5ee] flushBuffers() write tcp 127.0.0.1:62404->127.0.0.1:9103: wsasend: An established connection was aborted by the software in your host machine.

Please check if 9103 port is available. In 6.6 version we create the first index using the Maintenance stream and it uses 9105 port. But from second index build onwards we use init stream (Port 9103) for the new index and maintenance stream (Port 9105) later to keep the index updated with mutations. So you were able to build one index.

Thanks
Sai

7

Please did you solve your problem???