Scan Consistency is only working for N1QL delete … Not for N1QL insert.
...
queryOptions.scanConsistency = couchbase.QueryScanConsistency.RequestPlus
conn._cluster.query(query, queryOptions)
.catch( function onError(err) {
try {
var error = new Error(err.cause.first_error_message);
error.stack = err.cause.http_body;
onQueryCallback(error);
} catch (_err) {
console.error(_err.stack);
}
})
.then( function onResult(data, meta) {
try {
if ( typeof(data) == 'undefined' ) {
data = { rows: []}
}
onQueryCallback(false, data.rows, data.meta);
} catch (err) {
console.error(err.stack);
}
});
FYI.: I also tried to work with Collection::insert({ durabilityLevel: couchbase.DurabilityLevel.Majority }), it is like it is not working at all.
I have to add a 200ms timeout (forestdb) after an insert if I want to read it right after the operation !!
Couchbase Server Community Edition v6.6.0
NodeJS SDK v4.2.4 & v4.4.5
On MacOS
cpu: arm64
libcouchbase: v3.3.14
nodeJS: v20.19.0 LTS
nodes: x1
On Debian/amd64
cpu: arm64/amd64
libcouchbase: v3.3.12
nodeJS: v20.19.0 & v21.6.2
nodes: x3
Can you describe what you mean by “does not work”? Maybe describe what you expect to happen vs. what you observe happening. And please show what the value of query is, it’s key to understanding the intent of cluster.query(query). And also what the other queryOptions are.
See Querying with SQL++ | Couchbase Docs
By “query” parameter, understand it N1QL “statement” like “INSERT INTO …”.
While running the .query method, I was expecting a callback/promise return when indexing is done so I can retrieve my own write right away.
- Insert
- Index (REQUEST_PLUS)
- On indexing completed (at least for the current node), callback or free the promise: kind of emitting “insert completed and indexed with success”.
Normally I was going with STATEMENT_PLUS option, but it is like it is not supported anymore in SDK v4.4.5.
There is no callback for “when indexing is done”. query() completes when “INSERT INTO…” completes - it does not wait for indexing to complete.
queryOptions.scanConsistency = couchbase.QueryScanConsistency.RequestPlus
should be used on a subsequent query. It will delay execution of the query until all the mutations that were pending have been indexed. So if you inserted a document and subsequently executed a query with RequestPlus, that query would not execute until the inserted document was indexed.
cluster.query(" insert … {“a”:“stuff”} ")
// RequestPlus will delay execution of the SELECT until a:stuff has been indexed
cluster.query(“SELECT .. FROM … WHERE stuff=“a””, … RequestPlus…))
See Querying with SQL++ | Couchbase Docs
@mreiche, thank you for your answer. It is not how I understood it and how it, talking about scanConsitency.
Several links have been deleted since, but I was able to retrieve this one :
Request plus are queries where the projection includes all updates bounded from the point in time you issued the query. The indexer needs to ensure that the index within your projection is bounded to include all udpates across all partitions in the cluster at the point in time the query “request” was made. This fits our scenario above where a user profile is updated to include a recent purchase, and then immediately query that document to see a consistent list of purchased items
https://www.couchbase.com/blog/high-performance-consistency/
Please, note the update cycle every «20ms for memdb| 200ms for forestdb» precision on the picture. That’s where I got the 150/200ms timeout for insert & update when I was implementing the SDK: since I am running 6.6.x (forestdb storageMode).
You are right in your example since they did talk about “SELECT” later in the article and not about “INSERT & UPDATE”.
Still, “DELETE” operations are behaving like I was expecting to. I was thinking, maybe this could (should?) be the case for INSERT & UPDATE.
So, by what I understand, I should focus on setting { scanConsitency: 'request_plus' } For all my SELECT operations ? At least when I need to read right after an INSERT or an UPDATE ?
I am going to try this way (in your suggested order), then I will come back to you.
The blog looks correct.
For your code, it’s not the INSERT that waits for the indexer. It’s the subsequent operation that has the RequestPlus option.
The “update cycle” is how often it checks. It is not the time that the indexing is complete. The time required for indexing will vary depending on the amount of mutations.
Still, “DELETE” operations are behaving like I was expecting to.
I don’t know what your DELETE operations so I can only guess - that they are deleting by meta().id or use keys, or by a property that is not indexed. None of those use indexes, therefore are not affected by indexes.
For all my SELECT operations ? At least when I need to read right after an INSERT or an UPDATE ?
There’s a trade-off between getting the absolute latest up-to-date indexes vs returning faster.
If you are doing the mutations that might affect the query - there is also a consistentWith: MutationState in query options - if you perform the mutations with the kv api, those calls return a mutationToken, from which a MutationState can be constructed. Passing that into the query results in it waiting only for those mutations to be indexed.
@mreiche Can I hug you ? Bro, you saved me a lot of pain. By following your previous hints, after removing my timeout hack, it solved my problem.
I was focusing on the wrong operations thinking that RequestPlus should be passed for INSERT & UPDATE… Now, passing it for the SELECT operation(s) following an INSERT or an UPDATE when needed, gave me the result/behaviour that I was expecting.
For the DELETE operation, yes I am deleting using meta().id.
Yes in some cases (a lot), I am doing mutations that are affecting the query.
For the last part of yo answer, I might need a drawing. If you have a link explaining it, I would appreciate it. Meanwhile, I will look into “consistentWith”.
x1000 Thanks for your explanations and your help !
Regarding consistentWith - here is an example from the Java SDK (it will be similar in the nodejs SDK). The example shows a MutationState with one MutationToken. The MutationState can have multiple MutationTokens (i.e. if you had multiple insert(), replace(), remove() etc). Note that only kv operations return a MutationToken - query() does not.
MutationResult mr = collection.insert(id, FOO_CONTENT);
MutationState mutationState = MutationState.from(mr.mutationToken().get();
QueryOptions options = queryOptions()
.consistentWith(mutationState)
.parameters(JsonArray.from(id));
QueryResult result = cluster.query(
"select " + bucketName + ".* from " + bucketName + " where meta().id=$1",
options
);
1 Like
Got it for KV Operations vs mutationToken && consistentWith option.