Hi Couchbase team,
i am new to eventing, and try to find a solution on one situation.
In my cb bucket we have millons of data , i have built scheduler whcih pick up records once specific criteria such a time, now in thinking how can i take an advatage of Eventing.
When I am executing my N1QL and select those records simultaneously, I also need to update one flag value , so the same records cannot be picked up in the next batch job run. While reading Eventing documents, I am seeing Eventing can not be performed on select, but do you guys have an option like execute. Can I have some info on this situation or any suggestion…?
Mehul,
Eventing responds to changes in data (or mutations). However I am not really quite following what you are trying to do. I will try my best to address your update flag question so you don’t pick something up in your next batch run.
Lets say you have 1968 documents of airport as a proxy for your 1M documents (using `travel-sample` data set) like the following:
{
"airportname": "Calais Dunkerque",
"city": "Calais",
"country": "France",
"faa": "CQF",
"geo": {
"alt": 12,
"lat": 50.962097,
"lon": 1.954764
},
"icao": "LFAC",
"id": 1254,
"type": "airport",
"tz": "Europe/Paris"
}
If you want to add a flag via N1QL we simply issue
UPDATE `travel-sample` SET needs_analysis = true WHERE type = "airport"
This will update all 1968 records. So now our sample record above looks like the following:
{
"airportname": "Calais Dunkerque",
"city": "Calais",
"country": "France",
"faa": "CQF",
"geo": {
"alt": 12,
"lat": 50.962097,
"lon": 1.954764
},
"icao": "LFAC",
"id": 1254,
"needs_analysis": true,
"type": "airport",
"tz": "Europe/Paris"
}
You can write an Eventing function that responds to any change or mutation but filters down to a condition of type == airport AND needs_analysis == true as follows:
I call the function do_analysis and make a bucket alias in the bindings section of ts_bkt to the bucket `travel-sample` in a read and write mode (this is a 6.5 or 6.5.1 feature, writing back to the source bucket).
The key thing here is that you only what process things in Eventing via the do_analysis Function when a specific field exists and is also set to true ( you can further restrict to say a specific type of airport). This is done at the beginning of the Function.
function OnUpdate(doc, meta) {
// ignore all other types
if (doc.type != "airport") return;
// ignore if we don't have the field needs_analysis or it is false
if (!doc.needs_analysis || doc.needs_analysis === false) return;
log('processing docId', meta.id);
// ---------
// Do anything you need to do I will log add the number of
// milliseconds since UNIX epoch this will be yet another new
// field - this represents a proxy for some complex work.
doc.processed_at_millis = (new Date()).getTime();
// ---------
// prevent any further work
doc.needs_analysis = false;
ts_bkt[meta.id] = doc;
}
If our function is deployed all the documents you updated will be processed needs_analysis will be set to false (to prevent a rerun) and you will see a new timestamp in milliseconds from epoch. Remember this is in milliseconds NOT seconds so if you want to use a converter like https://www.epochconverter.com/ to see the actual human readable time don’t forget to divide by 1000.
{
"airportname": "Calais Dunkerque",
"city": "Calais",
"country": "France",
"faa": "CQF",
"geo": {
"alt": 12,
"lat": 50.962097,
"lon": 1.954764
},
"icao": "LFAC",
"id": 1254,
"needs_analysis": false,
"type": "airport",
"tz": "Europe/Paris",
"processed_at_millis": 1589486439396
}
You can mutate all the documents again by say updating a field to its current value this will trigger mutations but they will be filtered out by your JavaScript since right now needs_analysis is false.
UPDATE `travel-sample` SET id = id WHERE type = "airport"
Although the above did not change the document it was a mutation that went to eventing, but the Function do_analysis saw that needs_analysis was false so it did nothing new.
You can run the N1QL statement again (a second time) to set do_analysis to true and all the timestamps in the 1968 documents will update nearly instantaneously
UPDATE `travel-sample` SET needs_analysis = true WHERE type = "airport"
If you want to remove the flag needs_analysis
UPDATE `travel-sample` UNSET needs_analysis WHERE type = "airport"
and if you want to remove the timestamp
UPDATE `travel-sample` UNSET processed_at_millis WHERE type = "airport"
Hope this helps.
If you have further questions please provide more details so I can fully understand your uses case.
Thank you so much will go through it
Went through your examples , now i got to know that eventing only works for changes in data.
- currently i am using Use Keys to update the data .
- After performing select query and picked specific records , i need to send them in to queues and then other process pic those records and perform update operation. But when queue is in slow in a response the same records picked next batch job run.
So, does CB has a feature which directly can put those records in to the queue ? i know you guys working with kafka …but beside that is there any Queue Service supported?
Hi Mehul,
Yes eventing is for responding to changes in data (or mutations) i.e. inserts, updates, deletes and expirys.
If you currently use KEYS, Eventing exposes JavaScript Maps via bindings (made at the bottom of the setting page). For example you can make any bucket say mybucket aliases to a map visible in your Eventing functions JavaScript.
Asume you alias a real bucket to the name “dest”" in a read and write mode you can not go to town using KEYS and KV (and it is wicked fast!)
function OnUpdate(doc, meta) {
// Assuming 'dest' is a bucket alias binding
var val = dest[meta.id]; // this is a bucket GET operation.
dest[meta.id] = {"status":3}; // this is a bucket SET operation.
delete dest[meta.id]; // this is a bucket DEL operation.
}
Now we are working on queues right now (hopefully a developer preview in the next version 6.6), although Kafka works just fine it is a pretty heavy weight configuration task and we want to lower the configuration threshold for 90% of the use cases. We will support a 1:1 queue with handshake and a 1:N broadcast queue.
So I m sorry as of right now there is not a simple queue construct in Eventing, however is you describe your exact use case I might be able to supply a work around . Feel free to message me directly.
Best
Jon Strabala