We’re seeing some relatively high I/O against a file called “scan-backfill##############” in the couchbase\server\bin directory when running queries that hit some of our larger indexes.
We’re hosted on Azure, and the Operating System drive on Azure is notoriously slow. Per Microsoft’s recommendations, you should not have any read/write files on the system drive due to it’s configuration.
This is a bit concerning as we’re seeing varying performance with N1QL queries against large indexed. The concern is that the index performance is being limited to the write-speed at which couchbase can write to the scan-backfill file.
For example, if other disk activity is low on the system drive and the scan-backfill file can write at speeds above 10MB/sec, my queries will succeed. If the system drive is busy and the scan-backfill file writes at speeds less than 10MB/sec, I run the risk on an “index scan timeout” (which we’re aware of the known issue for long-running index scans).
Obviously all our observations are anecdotal based off query performance and system stats, but is any of this expected behavior in terms of how Couchbase uses that file?
The scan-backfill files are used to spill-over results to disk if query engine cannot read/process data as fast as index server is streaming it. If you have queries returning very large result sets, please considering optimizing the queries(e.g. avoid full index scans).
Regarding its relation to scan-timeout, if all the data required by the query cannot be streamed over from index in 2mins(slow disk plays a part here), the query will result in scan timeout.
Thanks deepkaran. Is the location of the scan-backfill a configurable value? Architecturally speaking, having it write to the “bin” directory isn’t great. Ideally, this file would live on the same high-speed storage as the data or indexes.
Note that the backfill files go to OS recommended TempDir location. It is not expected to be written anywhere in Couchbase install binary directory.
As that is happening, it is worthwhile to ensure the TEMP/TMP environment variables are properly configured for Couchbase Service. On Windows, these usually point to C:\users\username\AppData\Local\Temp.
Maybe there’s a bug in 4.5 on Windows? Both the user variables for TEMP/TMP and the System variables for TEMP/TMP are configured. I would be ecstatic if this were only a matter of changing the variable location.
@jeffhoward001, sorry I didn’t see your last post. If this is still happening, please let me know. We basically use ioutil package - io/ioutil - Go Packages to determine where to write these files. I’m surprised this is creating files to our bin directory. There is also an effort to make this fully configurable (location, size etc) in next minor release.
