Couchbase 5.1 View indexing takes very long time

Couchbase Server
1

I am using Couchbase 5.1 and I am using Views for quering documents.

My data bucket contains aroung 1.5 million documents, which also contains documents with attachments (images).

My bucket has weight aroung 800 Gb right now.

When I create new View it takes around 12 hours to be built. During this time my server is not responsive since I/O read operation is taking almos all disk throughput.

Why is it so slow? It is only 1.5 millon documents?
Does couchbase process images as well when it create Views? If so how to avoid this?

Should I move my data to separate buckets like one for images docs and other data for second bucket?

2

Hello Nikolay_Nikolaev,

Thank you for using Couchbase Server.
When you create a new view your server is not responsive, as all disk throughput is taken.
It sounds like you have a single node in your cluster and potentially not running on fast SSD/NVMe storage. For a production use-case you need to ensure that you are sized correctly with the required resources.

There’s a few things you can try,

  • Add additional disk IO
  • Add additional nodes to scale-out.
  • Define your view query to exclude documents that are out of scope for your requirements.

I’d highly suggest you also take a look at our N1QL Query and Index Services instead of using Views.
They are often more performant and allows for a range of new use-cases.

As per your question about processing images, the system doesn’t have any knowledge of what is being stored in your documents, any binary data can be stored along side JSON data as well.

You can exclude non-JSON documents from your view queries like this example:

function(doc,meta) {
if (meta.type == “json”) {
emit(doc.firstname.toLowerCase(),null);
}
}

Thanks,
Ian McCloy (Principal Product Manager, Couchbase)

3

Thanks for your reply will consider some of you advise in the future.

Final question regarding non-JSON documents. If I add this like in my View if (meta.type == “json”) does it speed up indexing process since couchbase will skip such documents?

Correct me if I am wrong