Lets say I created multiple indexes with defer_build=true. Than I want to build them all using
BUILD INDEX ON myBucket (index1,index2,`…)
If one of the indexes is already built than the whole operation will stop and fail. And the rest of the indexes will not be built. I would expect that if one index is already build, just skip it and built the rest. Than return it in the response.
@aterem, BUILD INDEX is a costly operation and once a build is in progress, the system doesn’t allow other indexes to be built for that bucket. So there is extra validation to make sure the admin is doing the right thing and help avoid wrong commands due to typo etc.
@deepkaran.salooja, thanks for the reply.
I’m not talking about calling this command during an ongoing build. I’m taking about calling it when there is no build at that moment, but, it just so happens to be, that one of the indexes is already built, so we just need to built the other indexes that aren’t.
In today’s cloud environments, there is a minimal intervention from admin/system-eng. The build of the indexes and buckets are all created on an automatic deployment scripts. So it make sense to make this call for all needed indexes, assuming that if some of them are already there, they will just be skipped.
@aterem, thanks for clarifying. This facility is going to be available in 5.0 Couchbase Server. The server will automatically filter out the indexes which are already built and build the remaining ones. You can try this out in CB 5.0 Beta.
This does not work in CE version 5 either. I guess the only solution is to execute build index query individually for all indexes.
With v6.5 there is a way to build all unbuilt indexes. Check out Point 7 in https://blog.couchbase.com/couchbase-6-5-rms-for-indexing-service/
It does not work in v5. Do you see any performance impact if we build all the indexes one by one? We have 86 indexes. We have a script file which contains 86 BUILD INDEX statements. Now, if an index is already present in ready state, the query fails and our job moves to next query. Hence, it is able to build all deferred indexes. However, i feel this is not able to exploit the performance benifit of the indexer where we build all the indexes of a bucket at once. Is my assumption correct?
Check @btburnett3 blog it will answers your question and provides a solution Rebuild All Couchbase N1QL Indexes After Restore - RandomDev
This has been provided in 6.5.0 by MB-32153
@manish_bansal Yes, you are right. Building 86 indexes one by one is not efficient and will not utilize the indexer’s provision of building multiple indexes building at the same time. I understood your concern correctly now. The issue you are facing has been addressed by Loading... and this is available in v6.5. Could you please try out your script in 6.5 version?
Thanks Pratibha. I wish i could upgrade. But what i can confirm right now is we tested scripts which contains BUILD INDEX statements for each individual index. We have total 9 buckets having multiple indexes on multiple nodes. During query execution, we do get errors like an index build is already in progress for the concerned bucket. It will be built later. And same is taken care by the server. Further, when we observe the status in system: indexes, the number of concurrent building index is always between 3 to 7. I agree this might be way less performant but its still better than breaking scripts.
Hi @manish_bansal: Yes if one is already building on a bucket, another index build on same bucket will fail but will be scheduled for building in the background. The error you might be seeing would be like “retried building in the background”. As you rightly observed, it is not ideal to build indexes one by one. When you upgrade to v6.5, please do consider upgrading your script to build all indexes in a given bucket at once. As you might already know, the benefit of building multiple at once is parallel building of indexes and better use of streaming data from the Data Service nodes with efficient network utilization.