Using a variable as key for N1QL query in eventing functions

[cmaster11]

Hi, I’m experimenting with eventing functions, and I need to increment a counter which is inside an object, that has a variable key. Using the standard subdoc API I’d use the counter function to increase the value.

Now, using eventing functions, I’m wondering what’s the best way to do it.

1. Exploring the docs I noticed the sentence “Other advanced Data Service operations are available as member Functions on the map object.”, but I have no clue nor further reference to understand what’s available and where.
2. I also tried some weird N1QL, like UPDATE myBucket USE KEYS["myDocId"] SET $counterKey = $counterKey + 1;, but function compilation fails on the first usage of $counterKey, as object key.

Is there something I can do about it, without fetching/setting the whole document? I cannot perform a full fetch/set operation because I noticed that, if there are many events fired in few milliseconds, the set operations overlap each other and the counter is not increased properly (I guess some workers are performing the same operation in parallel, overwriting each other documents. Also, the issue happens even if I’m using a single worker on a single node).

Also, inspecting the logs and source code I noticed that the N1QL query was translated to an expression with a function similar to execQuery at the end (maybe this one?). Would there be an alternative syntax that I could use for my case? To be able to use a variable as key for the N1QL query?

UPDATE:

So, I managed to get it working by using:

  const updated = new N1qlQuery(
    `UPDATE \`myBucket\` USE KEYS["countersDocId"] 
      SET \`${counterBaseKey}\` = 
      CASE WHEN \`${counterBaseKey}\` IS MISSING THEN 1 
      ELSE \`${counterBaseKey}\` + 1 END RETURNING \`${counterBaseKey}\``,
    {namedParams: {}}
  ).execQuery();

BUT, this sounds just slightly hackish . Is there a better and maybe also more code-stable way?

UPDATE 2:

Even though the N1QL query works, there is still some inconsistency , out of 40 runs in few milliseconds, I can see that some of the updates return the same counter value.

E.g.

2018-11-14T18:00:01.172+00:00 [INFO] "Triggered counter update" 
2018-11-14T18:00:01.174+00:00 [INFO] "Update succeeded" [{"testEvent_tag2":3}] 
2018-11-14T18:00:01.174+00:00 [INFO] "Triggered counter update"
2018-11-14T18:00:01.175+00:00 [INFO] "Update succeeded" [{"testEvent_tag2":3}]

Thanks,
Alberto

[gautham.banasandra]

Let’s assume that the document myDocId looks like this -

{
  myCounter : 1
}

You could use the following handler code to increment the field myCounter -

function OnUpdate(doc, meta) {
    UPDATE myBucket USE KEYS["myDocId"] SET myCounter = myCounter + 1;
}

The documentation is inaccurate regarding the mention of "Other advanced Data Service Operations". We have created a bug to rectify it - https://issues.couchbase.com/browse/DOC-4425 .

Please note that you should not use $myCounter as it’s not a valid N1QL syntax in this case. We have logged a bug - https://issues.couchbase.com/browse/DOC-4424 to document the right usage of named parameters in N1QL query.

The reason why you were seeing overlapping increment operations even though you had 1 worker is because, each worker is multithreaded by default - https://github.com/couchbase/eventing/blob/master/producer/depcfg_parser.go#L102

N1qlQuery is an internal artefact and should not be invoked directly. Please see -
https://docs.couchbase.com/server/6.0/eventing/eventing-language-constructs.html

[cmaster11]

Thank you for the answer!

The reason why you were seeing overlapping increment operations even though you had 1 worker is because, each worker is multithreaded by default

Ok, I verified it works if I manually set the worker as single-threaded. However, this looks quite poorly performant.

1. Is there a way, from inside the execution of a worker, to know its index? Meaning, if I say that I want 3 workers, a variable that tells me “this is worker 0, 1 or 2”? In this way, it’d be possible to make a “sharding” out of the events (e.g. hashing the document ID), while still achieving a good performance. Or even know the vBucket index of the document, in that case the sharding would be even easier.
2. Are there going to be some thread-safe operations for subdoc increments, or should I just stick to some other workarounds? (I understand the complexity of the problem here )

Thank you!

[Siri]

Hi,

It is on our short term roadmap to support other KV operations such as incrementing and decrementing Atomic Counters. If you are interested in an early drop, please let us know.

Please note that it is possible that mutations to the same document in succession are de-duplicated by KV engine. Hence, counting mutations is not possible using eventing. Event handlers are guaranteed to see the final state of a document when it is modified many times successively, but do not necessarily see every intermediate state. Ordering in Couchbase Functions - Part 1 - The Couchbase Blog

@keshav_m, does N1QL support Atomic Counters?

Thanks,
Siri

[cmaster11]

I’d definitely be interested in knowing more about atomic subdoc ops in eventing functions, especially when dealing with counters.

I’m sorry for repeating myself, I just want to be sure about the topic:

1. Is there a way, from inside the execution of a worker, to know its index ? Meaning, if I say that I want 3 workers, a variable that tells me “this is worker 0, 1 or 2”? In this way, it’d be possible to make a “sharding” out of the events (e.g. hashing the document ID), while still achieving a good performance. Or even know the vBucket index of the document, in that case the sharding would be even easier.

I was thinking, editing this area, by providing a third argument with some other meta info (e.g. current vBucket vb_no, current worker idx) could be an improvement for a case like mine (where there is the need for the fn worker to be “self-conscious” about its existence).

[asingh]

Hi,

@cmaster11

Is there a way, from inside the execution of a worker, to know its index ? Meaning, if I say that I want 3 workers, a variable that tells me “this is worker 0, 1 or 2”? In this way, it’d be possible to make a “sharding” out of the events (e.g. hashing the document ID), while still achieving a good performance. Or even know the vBucket index of the document, in that case the sharding would be even easier.

• vbucket index of document is available as meta.vb within OnUpdate() or OnDelete()
• All events mapped to a document in Data service is tied to a specific thread within a worker. In other words, all events(update/delete) specific to a meta.id would be executed in-order. So from what I understand, what you shared w.r.t. “sharding” based on doc ID is already present in the design.

type dcpMetadata struct {
	Cas     uint64 `json:"cas"`
	DocID   string `json:"id"`
	Expiry  uint32 `json:"expiration"`
	Flag    uint32 `json:"flags"`
	Vbucket uint16 `json:"vb"`
	SeqNo   uint64 `json:"seq"`
}

JFYI, mentioning some other metadata fields that you could access from Javascript - meta.cas, meta.expiration and so on.

Please let me know if I missed answering something.

Thanks,
Abhishek

[cmaster11]

Thank you for the answer!

Knowing that a worker thread is bound to a specific document id is good info, as well as that vb can be accessed.

I think I may have been wrong on the theory of a possible solution to fix the issue, and I guess I’ll have to wait for more info/updates on atomic counter operations

Even is sub-optimal, keeping a single worker, single threaded will do the job until there is a better fix.

Thanks a lot!
Alberto

[Siri]

To follow up, I checked with N1QL team and there isn’t support for KV atomic counters, though that’s on the roadmap for N1QL. I tested the suggested alternative of retrying on CAS, and ran into MB-32060 which we will have a fix very soon.

Thanks,
Siri

[jon.strabala]

Hi @cmaster11

I know it’s been a long time, but I just came across this posting, Couchbase Eventing now has special operations under Advanced Keyspace Accessors including atomic counters and safe CAS operations.

Below I show a simplistic example on how to increment an atomic counter in an Eventing Function.

function OnUpdate(doc, meta) {
    log('input meta', meta);
    log('input doc ', doc);

    // if doc.count doesn't exist it will be created
    var ctr_meta = {"id": "my_atomic_counter:1" };
    var result = couchbase.increment(src_col,ctr_meta);
    if (result.success) {
        log('success adv. increment: result',result);
    } else {
        log('failure adv. increment: id',ctr_meta.id,'result',result);
    }
}

Best

Jon Strabala
Principal Product Manager - Server‌