From N1QL to JavaScript and Back – Part 3: Iterating over Documents and DML

This earlier blog goes through the basics of executing N1QL statements inside JavaScript functions.

We now move onto…

Iterators/Processing values

So far we have purposely avoided statements that return data, and returning data from the function.

This is where we start exploring gathering data.

The N1QL() function, referenced directly, or via the transpiler magic, returns a statement handle. The handle comes in the form of a JavaScript object and has an associated iterator, for example:

I will not spend lots of time on describing iterators, so here are the basics:

• You can get the next value from an iterator using the next() method.
• The done property is true if there are no more documents to get.
• Though you can use the iterator’s next() and done members in a while loop to get all the documents, a more readable construct is the for..of loop over the statement handle itself.
• Not all values need to be retrieved – you can discard a statement at any time by calling the statement’s close() method, although, as we’ll see later, you don’t need to do this at all.

Putting it all together:

and the corresponding function:

In this example, an array named res accumulates the documents we obtain from b1 through the iterator q, and it is then used to return them to the caller.

A couple of side notes

In general terms, it probably is a better idea to process query values as they are received in the for loop, rather than just accumulating them in an array, as this has transient memory usage implications.

Conversely, returning an array value from a function allows a statement to select from the function, for example:

This is much like you would do with table functions or collection derived tables which you have encountered in the relational world.

In the N1QL world these are known as expression scans.

Of course, it goes by itself that if all you want your function to do is return some documents from a keyspace without any further logic, you are probably better off just writing an inline function.

Back to the iterator!

Once you have finished processing the values, the iterator closes itself automatically, so there’s nothing that you have to do.

If the iterator is not drained, it is good practice to close it, although this is not strictly necessary, as it will be automatically closed when the function returns, as well as in other circumstances as we’ll see later.

Inside the for … of … loop it is perfectly valid to have other N1QL statements: some things to be considered will be described in later sections.

Processing values

Let’s try and use the values returned by the select, rather than just returning them:

Nothing particularly fancy, just extracting a field from the document.

Executing this function yields:

What gives? Why does it not return f1?

We can guess the answer by executing:

And seeing, in the UI:

f1 is a field in b1, not the selected document!

Any of those does what you want:

DML statements

Consider the following function:

This function creates new documents in b1 copying them from existing documents and removes the old documents.

The fact that q1 and q2 are iterators, even when the statements do not return values shouldn’t be too much of a surprise: as we’ll see later, DML statements can return values, and even worse, the transpiler cannot always determine if values are being returned or not, hence the N1QL() function always returns an iterator.

The implication of this is that once the N1QL() call returns, the N1QL statement may very well be executing asynchronously in the background, leaving the JavaScript function free to execute other code before fetching data.

The jsevaluator removes any behavioural ambiguity by forcing statements that do not return values to execute immediately and serially, meaning that it will wait for each DML statement complete before the next statement starts.

No action on the iterator is necessary, not even closing it.

This is consistent with the behaviour we instinctively expect: we want to finish inserting a copy of the existing documents before we start deleting the old ones – or we may not be making a copy of all of the original documents!

DML statements that return values

N1QL DML statements can be made to return values, as in the function below:

Returned results are processed in much the same way as for SELECT statements.

The major difference compared to DML statements that do not return results is that when no results are returned, all the changes to disk are made as soon as the statement returns, while when results are returned the statement progresses as each individual document is collected from the iterator: you have to process all the returned values for all the changes to make it to disk.

Again, the statement is executed asynchronously – so, as you collect each individual result, you should not assume that it is the last modification made to disk: more could have already been made, with the relevant results just queued waiting to be collected.

As with DML statements that do not return results, those that do are executed serially, meaning that if you do not process all the results queued in the iterator by the time you start the next statement, the jsevaluator will complete the current statement discarding all unprocessed values.

Conclusion

We have covered processing documents through N1QL and executing data manipulation.

Next, we are going to see how to handle errors.