I am attempting to develop some tools to make it easier to work with Couchbase 7.0:
couchgres-migrate is a tool for data migration from PostgreSQL and earlier versions of Couchbase to Couchbase 7.0, due to the introduction of scopes and buckets. it is limited to small-scale(5k-10k documents) at present and I am working to make it more performant.
I’d love some feedback(technical and non-technical) from the CB community and possibly any suggestions for features/improvements.
Thanks!
That looks really interesting, I’m going to give it a try, and I’ll post anything that I find as an issue in the github repo.
I’ve been working on a similar effort for SQL Server with almost the identical premise as you: GitHub - mgroves/SqlServerToCouchbase: Library to automatically best effort move and remodel data from relational databases (like SQL Server) to Couchbase
@matthew.groves,
thanks!
I have tested this only on MacOS though I think it should work on Linux systems too.
Your project looks interesting and very detailed, and I think there are a couple of features I could implement in my own.
@matthew.groves,
I added index migration support, after seeing it in your tool and it works for simple indices. However, it can be made more performant and I’m working on that.
Does index migration cause a lag in performance in your tool too? Do you have any ways of making data migration using these tools more performant?
Thanks!
The way I’m creating indexes in that tool is not deferred, so yes it takes time.
The faster (or at least more efficient) way to do it is to CREATE INDEX with defer_build: true, and then BUILD INDEX at the end. From the docs: “BUILD INDEX is capable of building multiple indexes at once, and can utilize a single scan of documents in the bucket to feed many index build operations.”
However, this is more complex to do when trying to migrate indexes over. I tried it out briefly, and I may revisit it in the future.
I’m also not sure which is faster/more efficient, or if it matters: create/build index and THEN insert data, or insert data and THEN create/build index. I suppose that would depend on how many indexes and how many fields are in them. So, in my library, I leave that as a decision for the user to make.
@aditi I mentioned your project in my latest post: SQL to NoSQL Migration: How to Convert Automatically and I would love to hear any comments or feedback you have on it
Thank you so much @matthew.groves !
I am working on the new features to make it independent of docker and add some more functionality.
The article is quite informative and looks great!