Customer 360 – Part 2 – Data Ingestion

So, we already took a high-level look at what’s involved in building a Customer 360 solution on Couchbase here.

That’s right, it involved a lot of thinking.

Sorry, no way around that.

Ya gotta think, can’t be one of the mindless masses…

So, assuming I haven’t managed to chase you off with the threat of thought, let’s start digging into some of the details.

In particular, the first part that needs to be thought through, getting the data into Couchbase.

I mean, let’s look back at that overall system diagram again.

Looking at the left side, you’ve got a collection of systems in a box labeled “Data Sources”.

CRM

ERP

Mainframe

(Mainframe? You’ve still got a Mainframe hanging around? Can’t we just turn the water off and let it die an ugly and unnatural death? No?!?!? Ugh…)

Fun trivia fact, Mainframes used to be water-cooled!

Yeah, when engineers realized how hot those beasts were going to get, they thought “Hey, running water through our car engines seem to keep them from over-heating. Let’s do the same thing with this baby!”

Fun trivia fact #2, back in the 90’s, the Cray Computer company, located in Minnesota, where it’s snows like 11 ½ months of the year, decided to move to a new building, only to find that they had trouble selling their old headquarters building.

It seems the place didn’t come with a heater. They had just piped the heat generated by their own computer throughout the place!

Toasty!

But I digress…

So, basically you’ve got all these systems, each with their own data model, that you’ve got to extract data from to be ingested into Couchbase.

Extraction

If this was my job, I would try and take the simple route.

It’s not because I’m lazy…

I mean, I might be, depends on your definition of lazy…

But that’s not the reason I take the easy route.

I’ve been at this a while.

And from what I’ve seen, the simpler a solution is, the easier it is to implement…

The easier it is to maintain…

The less it breaks…

And I get to sleep at night.

Something I’m quite fond of.

So maybe easy isn’t the right description…

Simple is more like it.

So, to keep things simple, I’m going to just extract any data from these source systems that I need, format it in JSON format, and insert it into Couchbase.

What could be simpler?

It could be as simple as connecting both ends to a Kafka stream, piping the data directly from the source into Couchbase.

A little configuration on each end, and voilà! I’ve got customer data in my Couchbase bucket!

Only…

It’s all disjointed and unconnected…

Not exactly achieving my goal here…

Damn! And here I thought I had managed to get by without having to think about this…

Hmm…

Transform

OK, so I’ve got all this disjointed data from various source systems in my Couchbase bucket, and I need to somehow combine them into a data model that makes sense.

There’s always some sticky detail to deal with…

But, I don’t know when any of these existing documents might be updated on the source system.

I just have to count on them coming in, willy-nilly, at random times.

How can I…

I got it! I’ll use the Couchbase Eventing system to let me know when I’ve got an update…

I’ll examine the new document to see what it is…

Look around to see if I have the other data for that customer that I need to combine it with…

And if I do, create a new document, combining the customer data.

Something like…

Woo-hoo! Pretty simple, huh?

Of course, those two functions I’m calling still have to be fleshed out…

Hope they won’t be too hairy…

Load

OK, so we’ve extracted the data from our source systems, transformed it into a combined data model, and loaded it into our target Couchbase bucket and…

Wait a minute…

Extract…Transform…Load…

ETL?!?!?

Isn’t that a dirty word around here?

We’re Couchbase!

We go around preaching Analytics without ETL!

See, any change in the data in the Couchbase Bucket just automatically shows up in the Analytics service…

…change in the Couchbase Bucket…

We…I…we…I…Uhh…

Hmm…

Guess that “No ETL” stuff only applies after you get your data into Couchbase…

Yeah, I don’t see any way to avoid it in this case.

But in this case, it could be sorta real-time.

It depends on what schedule you’re running your extracts from the source systems on.

If they’re running on a nightly basis, as a batch job, then it’s just your run-of-the-mill overnight ETL.

But if you’ve got triggers in your CRM or ERP databases that puts the updates into the Kafka queue the moment the customer’s records are updated in those systems, then it’s a fairly real-time ETL process.

Getting closer to what we preach around here…

Note: For those of you who want to dig into the nerdy details of actually putting this into action, we’ve got a very good and detailed tutorial over on our website. You can find it here.

So, Why Couchbase Again?

I mean, come on, any schema-less database will allow you to create a single, massive, uber-document containing all of your customer data. Why should you use ours?

Besides the ease of using something you likely already have in your infrastructure (Kafka) to take the data from your source system and insert it into a Couchbase bucket, then using our Eventing service to trigger some real-time functionality to ETL that data?

I mean, you can always pay big bucks to buy some magic ETL software to do some of that heavy-lifting for you…

You can then plug in a caching solution to get the speed and performance you need…

And throw in a full-text-search product to enable your users to find the information they want…

Then build a REST API to enable the data to be accessed by your mobile application…

Add another ETL system to pull the data out and toss it into some data-lake analytical system…

And then when the Rube Goldberg  machine that you’ve constructed has the slightest hic-up, run to patch things up to keep everything working…

And give up on the hobby of ever getting any sleep…

Or, you can take the simple route, and just use Couchbase for all of this.

Like I said earlier, I prefer simple solutions.