Synching Couchbase with a SQL database

Couchbase Server
1

As the title says, we are attempting to keep our couchbase environment synched with a SQL database. Not the full database mind you.

There is crucial data with entry points both from couchbase and sql that absolutely need to be synched (in real time). We already have synching from SQL to couchbase on the data we need - using triggers (issh, but a contraint we have to work with). Synching from couchbase to SQL however seems to be more complicated as it is pwoering a website and multiple changes can happen to the data we want to keep synched. At the moment we are logging all these changes with AWS SQS so that we can synch the changes down to sql without risking performance losses at peak traffic times etc…

This obviously has the huge downside of creating synching conflicts (change takes a while to come down from couchbase - enough that it is RE-changed before the trigger fires and comes back up, creating an never-ending cycle of doom)

At this point doing it ‘by hand’ is becoming more of a headache than anything else. Does couchbase have any feature that can help with synching in general between a couchbase server and another NON-couchbase server?

2

Hey @spcmnky,

Couchbase does not have a feature like this, but have you looked into Apache Spark? It is a data processing layer that can read and write from both SQL and NoSQL databases:

http://developer.couchbase.com/documentation/server/4.1/connectors/spark-1.0/spark-intro.html

Best,

3

Thanks for the detailed post. I’ll add a few more options to what @nraboy mentioned
Sounds like you already have a manual method of syncing from couchbase to RDBMS.
Another option would be to use ETL tools like Talend, Informatica or if you are using MSSQL SSIS with an ODBC/JDBC driver or REST calls to N1QL.
It would also be fairly easy to write an app that can read the data from couchbase and apply the data to RDBMS using an app in java, .net etc with both SDKs if you’d like to do custom transforms and logic.

Would these help solve the issue?
thanks
-cihan