Bulk loading with node.js client

Node.js SDK
1

Continuing the discussion from Cannot-perform-operations-on-a-shutdown-bucket:

Indeed, increasing the timeout may resolve it for that amount of items you’re loading, but that’s not a general solution. @brett19 may have a more general suggestion for how to handle this kind of situation. I think you should be hitting TMPFAIL.

@brett19: any suggestions for how to better approach this?

2

Rather than performing 1 million operations simultaneously, you should instead performing only perform a subset of the operations at once, then as they complete, dispatch new ones.

A rough example:

var myData = ...;
function doOne() {
  if (myData.length == 0) return;
  bucket.insert(myData.pop(), function(err) {
    doOne();
  });
}
for (var i = 0; i < 10000; +++i) doOne();

Cheers, Brett

1 Like
3

For a similar bulk-load operation, we were able to use the ‘poolr’ NPM module to restrict the number of simultaneous write operations active against Couchbase. This module acts as a thread pool, and uses a Javascript array as a memory “queue” of pending function calls.

https://github.com/codingforce/poolr

1 Like