SingleStoreDB On-Premises to SingleStoreDB Cloud Migration

Feed: SingleStore Blog.
Author: .

The reason is simple — outsourcing computing infrastructure allows organizations to focus their limited resources on differentiating their business within their given market segment, rather than maintaining computing infrastructure (i.e. digital plumbing).

This means more time can be spent improving products, acquiring/retaining customers and setting the strategic direction for the business. The following graph from Gartner compares the total revenue and revenue growth of cloud computing versus traditional/data center computing. Notice how in the coming years, cloud computing’s growth rate will continue rapid expansion, while traditional computing’s growth rate will slowly contract. 

SingleStoreDB

SingleStore was founded in 2011 as MemSQL — the world’s fastest in-memory rowstore database. Over the next decade, the product evolved past an in-memory rowstore, into a multi-model, multi-cloud, scalable, unified database for both transactions and real-time analytics. In 2020, MemSQL rebranded into SingleStore, as this name was more suitable for the evolved product and next chapter of our business. 

With SingleStoreDB Cloud, all of the backend operations are autonomous; our cloud service is self healing, self scaling and completely automated without user intervention. In addition to all of the automation, SingleStoreDB Cloud is monitored 24/7 by our SRE team who eat, sleep and breathe SingleStoreDB every day. Customers using SingleStoreDB Cloud procure compute credits, which get burned over time depending on the size of the environment deployed (billing granularity is by the second). 

When deploying SingleStoreDB On-Premises, customers are responsible for provisioning and managing SingleStoreDB software on their own x86-based virtualized infrastructure. Customers using SingleStoreDB will procure license units, which are used to install our software on their own virtualized infrastructure (VMs or containers). The following chart contrasts the responsibilities of deploying SingleStoreDB Cloud (managed service) and SingleStoreDB On-Premises (self-managed).

Migrating From SingleStoreDB On-Premises to SingleStoreDB Cloud

Many SingleStoreDB customers initially adopted our database technology via the on-premises approach we discussed in the previous section, making them responsible for all database infrastructure and to application architecture/performance. Following the larger cloud service trend, many of these customers are reconsidering their SingleStoreDB deployment methodology.

There is an increased demand from our customers to migrate their self-managed SingleStoreDB On-Premises to our cloud service, SingleStoreDB Cloud. This shift allows them to focus on their business differentiation, while we handle day-to-day database infrastructure management. In this section, we’ll discuss how to easily migrate from SingleStoreDB On-Premises to SingleStoreDB Cloud.

Prerequisites:

• SingleStoreDB On-Premises deployment, with access to sdb-admin

• Access to a cloud object storage provider (S3, Blob or GCS — S3 in our case)

• SingleStoreDB Cloud deployment

High-level steps:

• Inventory audit of current SingleStoreDB On-Premises database environment

• Gather SingleStoreDB On-Premises objects

• Configure firewall settings to allow traffic between: SingleStoreDB On-Premises, cloud object store and SingleStoreDB Cloud

• Backup all SingleStoreDB On-Premises databases to cloud object store

• Restore all databases from cloud object store to SingleStoreDB Cloud

• Create SingleStoreDB On-Premises objects in SingleStoreDB Cloud

• Validate inventory and objects in SingleStoreDB Cloud

• Perform any application level validation testing

Detailed steps:

1. Inventory audit of current SingleStoreDB On-Premises database environment. Gather counts for all databases, tables, records, functions, stored procedures, views, aggregations, pipelines, etc., from the self-managed environment.

2. Gather SingleStoreDB On-Premises objects. Use the dump functionality of sdb-admin to generate a sequence of SQL statements that can be executed to reproduce the objects defined in the cluster. Example: sdb-admin dump –output-path /path/dump_file_name.sql

3. Configure firewall settings to allow traffic between: SingleStoreDB On-Premises, cloud object store and SingleStoreDB Cloud. Ensure all firewalls, NACL’s, etc., are configured to allow data to flow from SingleStoreDB On-Premises, to the cloud object store, to SingleStoreDB Cloud.

4. Backup all SingleStoreDB On-Premises databases to cloud object store. Use SingleStoreDB’s native BACKUP DATABASE functionality to persist database backups within the cloud object store. Example:

   BACKUP DATABASE db_name TO S3 "backup_bucket/backups_path/backup_folder/"
   CONFIG '{"region":"your_cloud_region"}'
   CREDENTIALS 
   '{"aws_access_key_id":"your_access_key_id","aws_secret_access_key":"your_s
   ecret_access_key","aws_session_token":"your_session_token"}';
   

5. Restore all databases from cloud object store to SingleStoreDB Cloud. Use SingleStoreDB’s native RESTORE DATABASE functionality to restore database backups from cloud object store into SingleStoreDB Cloud. Example:

RESTORE DATABASE db_name_restored FROM S3 
"backup_bucket/backups_path/backup_folder/db_name.backup"
CONFIG '{"region":"your_cloud_region"}'
CREDENTIALS 
'{"aws_access_key_id":"your_access_key_id","aws_secret_access_key":"your_s
ecret_access_key","aws_session_token":"your_session_token"}';

6. Create SingleStoreDB On-Premises objects in SingleStoreDB Cloud. The dump command we ran in step 2 created a file called dump_file_name.sql, which now lives on your self-managed server. This file contains a list of SQL statements to recreate all objects from SingleStoreDB On-Premises. Copy and paste the contents of this dump file into SingleStoreDB Cloud SQL Editor — run the SQL statements to recreate all SingleStoreDB On-Premises objects in SingleStoreDB Cloud.

7.Validate inventory and objects in SingleStoreDB Cloud. Double check to ensure that everything has been copied over from the SingleStoreDB On-Premises environment to the SingleStoreDB Cloud environment. Validations should include things like: databases, tables, records, users, groups, aggregations, pipelines, stored procedures, etc.7.

8. Perform any application level validation testing. Now that your SingleStoreDB Cloud environment is fully hydrated and functional, start to kick the tires. Connect your application, run some benchmark queries, test concurrency — the world is your oyster!

If you’ve made it this far, you are equipped to migrate a workload from SingleStoreDB On-Premises to SingleStoreDB Cloud — well done!Please note there are some additional migration considerations, including: partition counts, database sizes, BACKUP/RESTORE vs mysqldump, objects included in the dump file, etc. This blog post is deliberately as simple as possible to maximize reach and usability. 

As always, we do not recommend doing a database migration with a production environment the first time around. QA/staging migrations are recommended as validation steps in any production migration. Please reach out to us if you are looking to migrate any workload to SingleStoreDB Cloud — we’re here to help!