Help! Rebalance failed and now docs are missing from view

Couchbase Server
1

To start off, this has nothing to do with the stale parameter.

I had to take down a node a few days ago and then added it back today. Before starting CB on this server, I removed everything from the data directory and then added this node back to the cluster. During the rebalance, a different node failed. I restarted this node and started the rebalance process over again.

Now certain documents in state where they are not being considered during the indexing process. I’ve even created a new test design document and created a new view. It’s like these documents are not being passed to the map function.

Any ideas would be greatly appreciated.

(using 2.0.1 community edition)

UPDATE:
I made a new view with a map function that emits the ids of all documents:

function(doc,meta)
{
emit(meta.id,null);
}

… but some documents aren’t included even though I can retrieve them with a get call and they show up in the admin page “Lookup ID” feature. I’m using the full data set and stale=false.

UPDATE 2:
I decided to start up a fourth server with a fresh install of CB and replace what I though was the “bad” node with this new one, but the indexing problem persisted. I then started trying different combinations of these four servers as 3-node clusters with hopes of finding a culprit but to no success.

My last idea was to go down to 1 node and then work my way back up. But when rebalancing from 2 to 1 node, CB-server shutdown on the destination node. I restarted/rebalanced and then both nodes failed during the rebalance process. After about 5 tries of rebalancing, all of these attempts seeming to fail for different reasons, I finally got a 3-node cluster up and running that seems to index all of the documents.

Unfortunately, I can’t post back with a reasonable solution that could help anyone else with this situation, so hopefully someone else can post, a possible reason and/or solution for what might have happened here.

2

So this issue still exists. I figured out that if I remove the node that I had originally added back to the cluster, then all of the docs are indexable again. But after purging couchbase-server on the ubuntu node that I removed, removing the couchbase directory entirely and reinstalling, the problem comes back once I add this server back into the cluster. So to sum up, with 2 nodes - everything is okay, but with 3 nodes - thousands of documents are not indexable.

Does anyone know how to fix this?

3

Also, I’m getting the following for all of the design documents in the views.* logs on the node that seems to break things:

couch_set_view:mark_partitions_unindexable([<<“bucket_name”>>,
<<"_design/dev_Mods">>,
[136]]) returned ok in 0ms

Different view log files have different partitions in the array (eg, views.1 has 136 and views.2 has 130 and 131).

4

So when you execute the count of the view on 1,2 and 3 nodes what are the results?
Any idea which document are missing?
Have you tried by recreating the view reindex everything?

5
  1. I didn’t think to look at view count to see if there was a difference between it being in a good and bad state. As stated in “Update 2”, the problem seemed to have been fixed by restarting (accidentally) the entire cluster, so I am unable to test this.
  2. The only way that I could know which documents were missing was when a user would complain that something wasn’t working (e.g. logging in).
  3. Yes, I had tried creating an entirely new designDoc and view with a map function that was unique to any other that already existed (this way an old map function with the same hash signature wouldn’t be use). But this new view was subject to the same issue as the others.
6

Hello,

I am not sure but based on our discussion in the comments, and the version you are using 2.0.1, I guess that the index is not up to date when it is called by your application (during/just after rebalance)

I will invite you to move to 2.1 or 2.2 to be able to use the new “consistency” feature we have added in it:
http://docs.couchbase.com/couchbase-manual-2.2/#disabling-consistent-query-results-on-rebalance

Regards
Tug
@tgrall