We have couchbase CE 5.0 and have setup autocompation on for bucket , its not running automatically .
Can you please advice on this
thanks
jas
We see the compaction is triggering but its failing for index with the below error message compaction failed with reason - FDB_RESULT_NO_DB_INSTANCE and the fragmentation reported for that index is 0% but actually its higher if we restart and perform the manual compact it goes away.
Any suggestion would be appreciated.
There are couple of things that could be happening here:
- How is your auto compaction settings configured? If you have set a particular % for database fragmentation or views fragmentation, auto compaction will not be triggered until that set % of fragmentation is reached - if you have set it up based on time period, then compaction will only occur in the set time period - even if there is a need to compact outside of the set time, it will be postponed until the time period occurs
- There are two levels of compaction settings - default which applies to all buckets and bucket level settings which apply to that specific bucket - bucket level will override default settings always - how is it set up in your environment?
Will be able to guide better if I know the auto compaction settings in your environment
Thanks for the reply. We have configured circular compaction for indexes. After looking into logs we see the compaction is kicking in at the time we specified attached the image of auto-compaction setting.
2018-10-03T01:04:10.176+00:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 3874417524012202045 Data:2523993041, Disk:3725193216, ExtraSnapshotData:0, Fragmentation:32.24531199726098%, mode : circular
2018-10-03T01:04:10.176+00:00 [Info] CompactionDaemon: Compacting index instance:3874417524012202045
2018-10-03T01:04:10.194+00:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 919073. Slice Id 0, IndexInstId 3874417524012202045, IndexDefnId 9600618908746941990
2018-10-03T01:04:10.194+00:00 [INFO][FDB] Forestdb opened database file /app/couchbase/data/@2i/xxxxxx_1_3874417524012202045_0.index/data.fdb.49
Though the below fragmentation says 0% we believe the numbers are not correct and for the above we see the fragmentation is increasing every day though compaction is running.
2018-10-03T01:08:20.043+00:00 [Info] CompactionDaemon: Finished compacting index instance:3874417524012202045
2018-10-03T01:08:20.043+00:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 12189976949967969802 Data:0, Disk:32768, ExtraSnapshotData:0, Fragmentation:0%, mode : circular
2018-10-03T01:08:20.043+00:00 [Info] CompactionDaemon: Compacting index instance:12189976949967969802
2018-10-03T01:08:20.051+00:00 [Error] CompactionDaemon: Index instance:12189976949967969802 Compaction failed with reason - FDB_RESULT_NO_DB_INSTANCE
For the index instance 12189976949967969802, Data:0. As there is no data, there is nothing to compact. The error message can be improved here. FDB_RESULT_NO_DB_INSTANCE indicates there is no db instance to be compacted, which seem to be true here.
For the index instance 3874417524012202045, which has data in it, the compaction has finished successfully.
Thank you deepak
For the index 12189976949967969802 as shown below its around 14GB so nothing to compact does not make sense.
[ 12189976949967969802_0.index]# ls -ltrh
total 14G
-rw-rw---- 1 couchbase couchbase 32K Oct 2 13:54 data.fdb.17
-rw-rw---- 1 couchbase couchbase 14G Oct 5 17:12 data.fdb.18
The other index though it says the compaction is complete the fragmentation keeps on growing say for example the fragmentation grown from 32% to 48% should the compaction not fix it ?
2018-10-05T01:04:03.074+00:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 3874417524012202045 Data:2524173265, Disk:4883722240, ExtraSnapshotData:0, Fragmentation:48.3145612924948%, mode : circular
For the index
12189976949967969802as shown below its around 14GB so nothing to compact does not make sense
Can you check what is the item count for this index on the UI? As Data is being reported as 0, something is not right with this index. You may want to drop and recreate it.
if we restart and perform the manual compact it goes away
how do you perform manual compaction?
The other index though it says the compaction is complete the fragmentation keeps on growing say for example the fragmentation grown from 32% to 48% should the compaction not fix it ?
With circular block reuse, the storage engine will let the index get fragmented upto 66% and then start reusing the blocks. So once the compaction gets run, you’ll see it slowly climbing to 66% and then stay there.
Also, it is not required to run the full compaction everyday. You could do it only once a week. The block reuse won’t let it go above 66% fragmentation.
Can you check what is the item count for this index on the UI? As Data is being reported as 0, something is not right with this index. You may want to drop and recreate it ?
There is 23.9 Million items in this index.
if we restart and perform the manual compact it goes away ?
We performed couchbase-cli compact-bucket process to shrink the index which was around 150+GB. And this worked only after we did the restart of the node before restart there was no impact.
Thank you for the details on block resuse was it documented any where ? was also checking how the internal operations of compaction but did not got an good article about it.
Appreciate you are taking time in responding our issues. 
compact-bucket only works on the bucket. It doesn’t have any impact on the index compaction.
Looks like it is the restart which is leading to the index behavior you are observing. I guess you can drop/recreate the index and see if that fixes the problem.
You can read about circular block reuse here:
https://docs.couchbase.com/server/5.5/indexes/storage-modes.html#std-gsi
Want to close this thread hoping it might help others. If the compaction is not enabled at the time of creating the index then would suggest to drop and recreate them.
Now all things working fine as expected after we enabled compaction and recreating index.
Thank you folks for contributing to this issue.
1 Like