Steps to migrate self-managed Couchbase Server clusters to Couchbase Capella™ Database-as-a-Service (DBaaS)

Couchbase Capella is the easiest and fastest way to begin with Couchbase. This fully managed Database-as-a-Service eliminates your database management efforts and reduces costs while delivering flexibility across all use cases with built-in multi-model capabilities. Its memory-first architecture drives high speed data response at scale, resulting in the best price-performance of any fully managed document database.

Why migrate?

If you’re running a self-managed Couchbase Server implementation, you and your team are probably responsible for handling database maintenance tasks that take time away from developing your applications. As a managed service, Couchbase Capella removes the need to take care of installations, upgrades and general database maintenance – it’s all handled for you. And for those of you using the Community Edition of Couchbase Server, Couchbase Capella can provide a fast and easy way to level up your database with more nodes and Enterprise features. Couchbase Capella supports all Couchbase Server services and capabilities.

This blog provides steps for migrating data and indexes from your Couchbase Server clusters, running either on premises or in the cloud, to Couchbase Capella.

Prerequisites 

    • An existing Couchbase Capella account. You can create and use Couchbase Capella for free, just sign up here

    • An existing self-managed Couchbase Server environment either on premises or deployed on a cloud service provider
    • It is assumed that you already have:
      • Familiarity with administering Couchbase Server and Couchbase Capella
      • Familiarity with running commands in a command line interface (CLI)

Limitations

This guide is used to migrate data and secondary indexes from Couchbase Server to Couchbase Capella. The guide does not apply to migrating the Couchbase Eventing Service, Couchbase Full-Text Search (FTS) indexes or Couchbase Analytics.

Versions of Couchbase supported by this guide include Couchbase Server (Community Edition or Enterprise Edition) versions 6.6, 7.0.x, 7.1, and newer.

Utilities

    • Couchbase cbbackupmgr – is a Command Line Interface (CLI) tool for managing the backup and restore of Couchbase Server data. It is used for migrating data and indexes from the following versions of self-managed Couchbase Server to Capella:
      • Enterprise Edition version 6.6, version 7.x
      • Community Edition version 7.x
      • Steps for using cbbackupmgr for migration are detailed in the section of this guide titled Migrate using cbbackupmgr – option 1
    • Couchbase cbbackup and cbrestore – are CLI tools used for migrating data from the following version of self-managed Couchbase Server to Capella:
      • Community Edition version 6.6
      • Steps for using cbbackup and cbrestore for migration are detailed in the section of this guide titled Migrate using cbbackup and cbrestore – option 2
    • Couchbase XDCR – Couchbase cross data center replication (XDCR) allows data to be replicated across clusters. It is used in this guide for ongoing migration from self-managed Couchbase Server clusters to Capella clusters

Note: XDCR cannot be used with Couchbase Server Community Edition or Capella Free Tier

The Prepare for migrating section of this document provides general guidance for assessing the size of your self-managed Couchbase Server cluster and using it to identify the required configurations for your Couchbase Capella cluster. For help with a more detailed Couchbase Capella sizing exercise, please contact Couchbase.

Prepare for migrating

If you are using the Couchbase Capella free tier, you can migrate data following this guide, but you will not be able to deploy a multi-node Capella environment due to configuration restrictions in the free tier. We recommend converting the free tier to a paid account before beginning a migration in order to configure a full multi-node deployment. To convert your account, open the Billing section of the Couchbase Capella UI and then choose Add Activation ID.

Review your self-managed Couchbase Server cluster:

    • Log in to your self-managed Couchbase Server Web Console and assess your cluster’s nodes and buckets
    • Choose the Servers tab in the main navigation to show a list of cluster nodes. Record the number of nodes and then choose each node on the list to display its properties
    • Record the memory and storage for each individual node
    • Choose the Buckets tab in the main navigation and then choose each bucket in the list to display its properties
    • Record the RAM quota and conflict resolution setting for each bucket
    • You will use your self-managed Couchbase Server cluster configurations as a general guide for sizing and configuring the destination cluster on Couchbase Capella

You do not need to duplicate the exact cluster configuration on Capella. This is especially true if you are using Capella Free Tier that comes with a fixed configuration.

Note the Couchbase Service distribution on the self-managed Couchbase Server cluster:

Record the IP addresses of the self-managed Couchbase Server cluster nodes:

Skip this step if you are using Community Edition

    • Record the IP address for each node in your cluster, this is for whitelisting them on your Couchbase Capella cluster later.
    • TIP: To get the IP of each node in your self managed cluster:
      • SSH into the node
      • Issue this command: dig +short myip.opendns.com @resolver1.opendns.com

Deploy and configure a cluster on Couchbase Capella

Create and configure a cluster on Couchbase Capella:

    • Log in to the Couchbase Capella UI, choose the Clusters tab in the main navigation menu and then choose Create Cluster. (In the free tier, you should already have a cluster created for you, and you cannot further customize the sizing)
    • Use the information that you recorded from the review of your self-managed Couchbase Server cluster and choose the cluster template that meets the configuration’s requirements
    • If you don’t find an appropriate template, choose Custom Template in the Cluster Sizing editor
    • Choose and configure the nodes to match your self-managed Couchbase Server cluster environment, including number of nodes, services distribution, compute or RAM, and storage
    • Choose a support zone and support package, and then deploy the cluster. For detailed steps and instructions on creating a cluster, see Create a cluster in the Couchbase Capella documentation

Couchbase Capella uses multi-dimensional scaling guidelines, and services and nodes can only be chosen according to deployment guidelines.

Create a database credentials user:

    • A database credentials user is specific to a cluster, and consists of a username, password, and a set of bucket privileges. This user is required for accessing bucket data
    • In the Couchbase Capella UI, create a database credential user for the new cluster by following the instructions in Configure database credentials from the Couchbase Capella documentation

Record Couchbase Capella cluster connection endpoint and add IP address as “Allowed IP”.

Save Root Certificates for self-managed Couchbase Server and Couchbase Capella:

    • In the Couchbase Capella UI, choose Clusters, and then choose your destination cluster. Choose the Connect tab for the cluster and record the connection endpoint for your cluster under Wide Area Network.
    • NOTE: If your self-managed Couchbase Server is running version 7.0.x, you will need to copy the hostname URL for a DATA SERVICE NODE instead of the Wide Area Network endpoint.
    • In the Couchbase Capella UI, choose the Nodes tab for the destination cluster, all nodes in your cluster will be listed. Choose one of the nodes that is running the DATA SERVICE, and record the HOSTNAME for the node.

Add the IP address of the system where you will run the command line tools for migrating data as an Allowed IP. Under Wide Area Network, click Manage Allowed IP, then click Add Allowed IP. Enter the IP address and click Add IP. For more information about allowed IPs, see Configure allowed IP addresses in the Couchbase documentation.

Note: If adding an allowed IP fails, in most cases it is related to firewall issues. See Couchbase documentation for information on the default ports used by Couchbase Server.

Download the root certificate for your Capella cluster. Under Root Certificate, click Download. Save the root certificate as a .pem file extension in a folder on the system that will run Couchbase CLI tools.

Next, log in to your self-managed Couchbase Server Web Console. Copy the root certificate for your self-managed Couchbase Server cluster and save it as a .pem file to the same folder where you saved the root certificate file for your Couchbase Capella cluster. For more information about the root certificate, see Root certificate in the Couchbase Server documentation.

Create target buckets on Couchbase Capella:

    • Log in to the Couchbase Capella UI
    • Create one target bucket in your Couchbase Capella cluster for each source bucket by following the instructions from Create a bucket in the Couchbase Capella documentation

Important: Bucket names cannot contain an underscore.

Migrate using cbbackupmgr – option 1

This option applies to the following versions of self-managed Couchbase Server:

    • Enterprise Edition version 6.6, version 7.0.x, version 7.1
    • Community Edition version 7.0.x, version 7.1

Use cbbackupmgr CLI tool:

    • cbbackupmgr is located in the root directory of your self-managed Couchbase Server installation.
    • You use a command line terminal to work with cbbackupmgr.

Depending on the platform, cbbackupmgr is installed with Couchbase Server in the following location:

    • Linux: /opt/couchbase/bin/cbbackupmgr
    • Windows (assuming default installation): C:\Program Files\Couchbase\Server\bin\cbbackupmgr
    • Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbbackupmgr

TIP: If you do not have direct access to the install directory for Couchbase Server, you can get the cbbackupmgr command line utility by downloading Couchbase Server and installing it on your local system.

Create a backup repository on the self-managed Couchbase Server system

    • In the command line terminal, create a directory to serve as a repository for local backups from the self-managed Couchbase Server. The following command examples use Linux syntax, adjust accordingly for your command line terminal and platform
    • The following command will create the directory in your systems root folder: mkdir /backups/
    • Run the following cbbackupmgr command to create a repository in the new directory, replacing <SELF-MANAGED-BUCKET-NAME> with the actual name of the bucket you wish to migrate:

NOTE:

    • This Repository backs up only the specified bucket
    • Analytics and Eventing are explicitly disabled

Run the following command to examine the repo:

Basic information about your repository will be displayed.

Backup the self-managed Couchbase Server bucket

Run the following command to create the backup:

Note: Make the following substitutions in the command:

    • Replace <SELF-MANAGED-BUCKET-NAME> with the actual name of the bucket you wish to migrate
    • Replace <SELF-MANAGED-SERVER-ADMIN> with the administrator user name on the self-managed Couchbase Server
    • Replace <SELF-MANAGED-SERVER-ADMIN-PWD> with the administrator password on the self-managed Couchbase Server
    • Replace <FULL-PATH-TO-SELF-MANAGED-ROOT-CERT> with the full file path to the self-managed Couchbase Server root certificate (as saved in earlier step)

An example command would look like:

Output from the command will be like:

cbbackupmgr output displayed

Examine the backup using the command:

Basic information about your backup will be displayed.

Restore the backed up self-managed bucket to Capella

In the command line terminal, use the following command to restore the backed up self-managed bucket to Capella:

Note: make the following substitutions in the command:

    • Replace <SELF-MANAGED-BUCKET-NAME> with the actual name of the bucket you with to migrate
    • Replace <CAPELLA-ENDPOINT> with the Capella cluster endpoint url recorded earlier
      • If your self-managed Couchbase Server is running version 7.0.x, you will need to use the hostname URL for a DATA SERVICE NODE instead of the Wide Area Network endpoint
      • In the Couchbase Capella UI, choose the “Nodes” tab for the destination cluster, all nodes in your cluster will be listed. Choose one of the nodes that is running the DATA SERVICE, and record the HOSTNAME for the node
    • Replace <CAPELLA-DATABASE-CREDENTIALS-USER> with the database credentials user on the Capella cluster
    • Replace <CAPELLA-DATABASE-CREDENTIALS-PASSWORD> with the database credentials user password
    • Replace <FULL-PATH-TO-CAPELLA-ROOT-CERT> with the full file path to the Capella cluster root certificate (as saved in earlier step).

The command may take a few mins to run depending on the size of your cluster. Once completed you should see details about the restored backup displayed in the terminal, and a message reading: “Restore completed successfully”

Examine the migrated documents and indexes

    • In the Couchbase Capella UI, choose Tools > Documents for your target cluster
    • You should see that all of your self-managed cluster documents have been migrated to Capella
    • In the Couchbase Capella UI, choose Tools > Indexes for your target cluster
    • You should see that all of your self-managed cluster indexes have been migrated to Capella

NOTE: Any indexes with definitions set as defer_build:true will be migrated, but will be listed as CREATED. These indexes still need to be built on Capella.

Build indexes on Capella

    • In the Couchbase Capella UI, choose Tools > Query Workbench for your target cluster
    • The following query will create BUILD statements for any indexes that were created but not built during migration. Copy/paste the following query to the Query Workbench in Capella, then click Execute:

The query results will return a BUILD statement to build the created indexes. Note there will be a BUILD statement for each scope in your cluster. The results will look something like so:

Copy each BUILD statement (between the double quotes), then paste to the Query Editor and click Execute to build the indexes. Repeat for each BUILD statement,

Congratulations, you have just migrated data and indexes from your self-managed Couchbase Server cluster to Capella!

Migrate using cbbackup and cbrestore – option 2

This option applies to the following versions of self-managed Couchbase Server:

    • Community Edition version 6.6

Use the cbbackup CLI tool:

    • cbbackup is located in the root directory of your self-managed Couchbase Server installation
    • You use a command line terminal to work with cbbackup
    • Depending on the platform, cbbackup is installed with Couchbase Server in the following location:
      • Linux: /opt/couchbase/bin/cbbackup
      • Windows (assuming default installation): C:\Program Files\Couchbase\Server\bin\cbbackup
      • Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbbackup

If you do not have direct access to the install directory for Couchbase Server, you can get the cbbackup and cbrestore command line utilities by downloading Couchbase Server and installing it on your local system.

Create a backup on the self-managed Couchbase Server system

In the command line terminal, create a directory for local backups from the self-managed Couchbase Server. The following command will create the directory in your system root folder:

mkdir /backups/

Run the following cbbackup command to create the backup:

Make the following substitutions in the command:

    • Replace <SELF-MANAGED-SERVER-ADMIN> with the administrator user name on the self-managed Couchbase Server
    • Replace <SELF-MANAGED-SERVER-ADMIN-PWD> with the administrator password on the self-managed Couchbase Server
    • Replace <URL-TO-SELF-MANAGED-COUCHBASE-SERVER> with the url to your self-managed Couchbase Server
    • Replace <SELF-MANAGED-BUCKET-NAME> with the name of the bucket you wish to migrate
      • NOTE: This command backs up only the specified bucket

Run the following command to examine the backup:

    • ls /backups

For Windows use:

    • dir /backups

Basic information about your backup will be displayed

Use the cbrestore CLI tool:

  • cbrestore is located in the root directory of your self-managed Couchbase Server installation.
  • You use a command line terminal to work with cbrestore.
  • Depending on the platform, cbrestore is installed with Couchbase Server in the following location:
    • Linux: /opt/couchbase/bin/cbrestore
    • Windows (assuming default installation): C:\Program Files\Couchbase\Server\bin\cbrestore
    • Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbrestore

Restore the backup to Capella using cbrestore

Run the following cbrestore command to restore the backup on Capella:

Make the following substitutions in the command:

    • Replace <CAPELLA-NODE-HOSTNAME> with the node hostname from your Capella cluster (recorded earlier)
    • Replace <SELF-MANAGED-BUCKET-NAME> with the name of the bucket you with to migrate
    • Replace <CAPELLA-BUCKET-NAME> with the name of the target bucket on Capella
    • Replace <DATE-OF-BACKUP> with the date that you took the backup
      • NOTE: use the date format YYYY-MM-DD
    • Replace <DATE-FOLLOWING-BACKUP> with the date following the date that you took the backup
      • NOTE: use the date format YYYY-MM-DD
    • Replace <CAPELLA-DATABASE-CREDENTIALS-USER> with the database credentials user on the Capella cluster
    • Replace <CAPELLA-DATABASE-CREDENTIALS-PASSWORD> with the database credentials user password
    • Replace <FULL-PATH-TO-CAPELLA-ROOT-CERT> with the full file path to the Capella cluster root certificate (as saved in earlier step).

Here is an example of what the command should look like:

The command may take a few mins to run depending on the size of your cluster. Once completed you should see details about the restored backup displayed in the terminal, and a message reading: Done.

Examine the migrated documents and indexes:

    • In the Couchbase Capella UI, choose Tools > Documents for your target cluster
    • You should see that all of your self-managed cluster documents have been migrated to Capella
    • In the Couchbase Capella UI, choose Tools > Indexes for your target cluster
    • You should see that all of your self-managed cluster indexes have been migrated to Capella
    • NOTE: Any indexes with definitions set as defer_build:true will be migrated, but will be listed as CREATED but not READY. These indexes still need to be built on Capella

Build indexes on Capella

    • In the Couchbase Capella UI, choose Tools > Query Workbench in your target cluster
    • The following query will create BUILD statements for any indexes that were created but not built during migration.
    • Copy/paste the following query to the Query Workbench in Capella, then click Execute:

The query results will return a BUILD statement to build the created indexes. The results will look something like so:

Copy the BUILD statement (between the double quotes), then paste to the Query Editor and click Execute to build the indexes.

Congratulations, you have just migrated data and indexes from your self-managed Couchbase Server cluster to Capella!

Ongoing migration via Cross Data Center Replication (XDCR)

This option applies to the following versions of self-managed Couchbase Server:

    • Enterprise Edition version 6.6, 7.0.x, 7.1

This is an optional step. Consider using XDCR if you wish to continue migrating data for a period of time beyond the initial migration using the CLI tools.

 

Connect self-managed Couchbase Server cluster to Couchbase Capella cluster

Enterprise Edition only

    • In the Couchbase Capella UI main navigation click Clusters, then click on the target cluster for migration
    • Click the Connect tab, then under Wide Area Network click Manage Allowed IP
    • On the Allowed IP screen, click Add Allowed IP
    • On the flyout editor enter the IP address of your self-managed Couchbase Server cluster nodes (or click + Add My IP to automatically paste in your current IP address)
    • Click Add IP
    • Repeat for each cluster node IP address
    • Click < BACK to go to the previous screen

Note: If whitelisting the IP fails, in most cases it is related to firewall issues. See Couchbase documentation for information on the default ports used by Couchbase Server:

    • Under Wide Area Network, copy and save the displayed URL, this is the endpoint connection for your Capella cluster. You will use this URL later to connect your self-managed Couchbase Server cluster to Couchbase Capella
    • Under Root Certificate, click Copy (this will capture the certificate to your clipboard for pasting)
    • Log in to your self-managed Couchbase Server Web Console, and in the main navigation click XDCR. Click Add Remote

Enter the following settings:

    • Cluster Name – enter a name for the remote cluster connection
    • IP/Hostname – enter the Couchbase Capella cluster endpoint URL you copied earlier
    • Username for Remote Cluster – enter the database credentials username for your Capella target cluster
    • Password – enter the password for the database credentials user for your Capella target cluster
    • Enable Secure Connection – checked
    • Select the Full radio button
    • Paste the Capella cluster root certificate you copied earlier
    • Click Save

Set up Couchbase XDCR

Enterprise Edition only:

Couchbase cross data center replication (XDCR) allows data to be replicated across clusters that are located in different data centers. It can be used to migrate data into Couchbase Capella from self-managed Couchbase Server clusters.

NOTE: XDCR does not migrate indexes.

In your self-managed Couchbase Server Web Console click XDCR in the main navigation, then click Add Replication 

Enter the following settings:

    • Replicate From Bucket – select the source bucket for migration
    • Remote Bucket – enter the remote bucket name
    • Remote Cluster – select the remote cluster you created earlier
    • Click Save replication

The replication process should begin within a few seconds.

Verify the migration

Data migration:

    • In the Couchbase Capella UI, choose Clusters in the main navigation and then click the target cluster in your cluster list
    • Choose the Buckets tab for your target cluster. Verify that the number of Items (documents) in the target bucket match the number of items in the source bucket
    • In the target cluster, choose Documents in the Tools dropdown list. Verify that all documents were migrated

Index migration:

In the Couchbase Capella UI, choose Indexes in the Tools dropdown list for your target cluster. Verify that the indexes are migrated and built.

Query runtime:

In the Couchbase Capella UI, choose Query Workbench in the Tools dropdown list for your target cluster. 

Run a sample SQL++ query or a query used in your application to test that you receive the same results as the query in your self-managed Couchbase Server cluster.

Related resources

Preparing to migrate:

Deploy and configure resources on Couchbase Capella:

 

Author

Posted by Couchbase Product Marketing

All Posts

Leave a reply