We are seeing 99+ % indexer blocked errors and large index rebuild is proceeding very slowly.
We’re new to CB, so I expect there is something trivial we are missing.
@greenj4, Please check if a proper memory quota has been allocated to the Index Service (UI->Settings).
I am not sure how to determine “proper”, we have given it 16 GB and it has 1.17 GB remaining.
We have 126M docs to index, the index on the original nodes (we added 3) used 51GB of memory
This is the first large scale CB project we have undertaken here at Dominos and I would really like it to succeed.
@greenj4, assuming you are using Standard GSI storage mode in Enterprise edition, it is recommended to allocate enough memory to keep the index 20% resident in memory. I am not clear about the index memory usage of 51GB on one of the nodes. You mentioned the memory quota is 16GB. How do you determine the index memory usage of 51GB and what is the “resident percent” of the index?
One of the things to look out for would be the size of the indexed key. We recommend keeping the total size of the index key(sum of all fields in case of a composite index, length of large arrays) to be under 1K.
Please check out the following Couchbase Connect sessions to understand more about what factors impact indexing throughput and how different stats can help pinpoint the root cause:
- https://connectonline.couchbase.com/forum/t/understanding-the-index-lifecycle-for-a-successful-deployment/91
- https://connectonline.couchbase.com/forum/t/deep-dive-into-post-production-index-manageability-and-monitoring/267
If you are already an enterprise customer, please work with support/professional service or you can share your logs here for further help.
We’re using the Community edition for the POC, evaluating Enterprise but I need to make the case.
Index key is small, way below 1K.
What we did was add 3 nodes to the existing 3 node cluster and told them to build the same indexes, so I know the exact size of the indexes we created.
I was recently informed that the new nodes are running CB 6.6 and the original nodes are on 6.5. Guessing this it likely part of the problem.
It is better to have all nodes on the same server version. Also, please note that Enterprise Edition has highly optimized storage engines which have order of magnitude better indexing throughput than the one used in Community Edition. It would be a better idea to do the evaluation using the Enterprise Edition.