Migrating to Amazon RDS with Database Migration Service
•
0 j'aime•2,879 vues
by Dhanraj Pondicherry, Sr. Manager of Solution Architecture, AWS
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Migrating to Amazon RDS with Database Migration Service
Similaire à Migrating to Amazon RDS with Database Migration Service (20)
Plus de Amazon Web Services
Plus de Amazon Web Services (20)
Dernier
Dernier (17)
Migrating to Amazon RDS with Database Migration Service
- 1. Dhanraj Pondicherry, Solution Architect Manager Database Migration Simple, Cross-Engine, and Cross-Platform Migrations with Minimal Downtime February 21, 2017
- 2. Agenda • How does the cloud help? • How do I get there? • How does it work? • What have others done? • Are there any tricks? Where is the magic? • Show me!
- 3. Migration options If you’re not switching engines and can take downtime: - SQL Server: bak file import - MySQL: read replicas - Oracle SQL Developer, Data Pump, Export/Import - PostgreSQL: pg_dump - SAP ASE: bcp
- 4. Migration used to be cost + complexity + time Commercial data migration and replication software Complex to set up and manage Application downtime Database-engine-specific application code
- 5. How can I get to the cloud? How will my on-premises data migrate to the cloud? How can I make it transparent to my users? Afterwards, how will on-premises and cloud data interact? How can I integrate my data assets within AWS? Can I get help moving off of commercial databases?
- 6. What are DMS and SCT? AWS Database Migration Service (DMS) easily and securely migrate and/or replicate your databases and data warehouses to AWS AWS Schema Conversion Tool (SCT) convert your commercial database and data warehouse schemas to open-source engines or AWS-native services, such as Amazon Aurora and Redshift We migrated over 18,000 unique databases. And counting…
- 7. When to use DMS and SCT? Modernize Migrate Replicate
- 8. When to use DMS and SCT? Modernize Modernize your Database Tier • Commercial to open-source • Commercial to Amazon Aurora Modernize your Warehouse • Commercial to Redshift Amazon Aurora Amazon Redshift
- 9. When to use DMS and SCT? Modernize Migrate Replicate Modernize your database tier – • Commercial to open-source • Commercial to Amazon Aurora Modernize your Data Warehouse – • Commercial to Redshift • Migrate business-critical applications • Migrate from Classic to VPC • Migrate data warehouse to Redshift • Upgrade to a minor version • Consolidate shards into Aurora • Create cross-regions Read Replicas • Run your analytics in the cloud • Keep your dev/test and production environment sync
- 10. New SCT Data Extractors – released 2/15 Extract Data from your data warehouse* and migrate to Amazon Redshift • Extracts through local migration agents • Data is optimized for Redshift and Saved in local files • Files are loaded to an Amazon S3 bucket (through network or Amazon Snowball) and then to Amazon Redshift Amazon Redshift AWS SCT S3 Bucket * 1st release supports Oracle v11 and up and Teradata v15 and up
- 11. Why use DMS and SCT? Secure Cost Effective Remove Barriers to Entry Allow DB Freedom Keep a Leg in the Cloud Easy to Use, but Sophisticated… Near-Zero Downtime
- 12. Who is saying What about DMS and SCT? "We migrated hundreds of our clients from our in-house data-center to Amazon RDS Oracle 12c using the AWS Data Migration Service (DMS). Due to this service, we could live-replicate the databases between our data-center and RDS before the migration. That kept the migration down-time to the very minimum. We are very happy with DMS and are planning to use it for Oracle to MySQL migration next”. ”The SCT Assessment Report was the key enabler to allow us to understand the scope of effort required to complete an Oracle to PostgreSQL migration. What was originally thought to be a largely manual task that no one was particularly excited about having to do became a very straight-forward quick and easy process." “We are in the process of migrating some databases to Amazon Aurora. The ease by which we can do this using the AWS Database Migration Service has simplified this process for us and enabled us to accelerate our migration efforts. The ability to closely monitor the process, the detailed logging feature, and the support we received from AWS have given us a great deal of confidence in a successful migration.”
- 14. SCT helps with converting tables, views, and code Sequences User-defined types Synonyms Packages Stored procedures Functions Triggers Schemas Tables Indexes Views Sort and distribution keys
- 15. Load is table by table Replication instance Source Target
- 16. Change data capture (CDC) and apply Replication instance Source Target Update t1 t2 t1 t2 Transactions Change apply after bulk load
- 17. Customer premises Application users AWS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let AWS DMS create tables, load data, and keep them in sync Switch applications over to the target at your convenience Keep your apps running during the migration AWS DMS
- 18. Multi-AZ option for high availability Customer premises or AWS AWS Internet VPN AWS DMS AWS DMS
- 19. Replication instance Source Target What else can I do? Source Source
- 20. Replication instance Source Target What else can I do? Target Target
- 21. Take it all—or not Source Target Replication instance instance
- 22. Homogenous or heterogeneous Replication instance SQL Server MySQL Replication instance Oracle Oracle Replication instance Oracle Aurora
- 23. Sources for AWS Database Migration Service Customers use the following databases as a source for data migration using AWS DMS: On-premises and Amazon EC2 instance databases: • Oracle Database 10g–12c • Microsoft SQL Server 2005–2014 • MySQL 5.5–5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.4–9.5 • SAP ASE 15.7+ RDS instance databases: • Oracle Database 11g–12c • Microsoft SQL Server 2008R2–2014. CDC operations are not supported yet. • MySQL versions 5.5–5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.4–9.5. CDC operations are not supported yet. • Amazon Aurora (MySQL-compatible data source)
- 24. Targets for AWS Database Migration Service Customers can use the following databases as a target for data replication using AWS DMS: On-premises and EC2 instance databases: • Oracle Database 10g–12c • Microsoft SQL Server 2005–2014 • MySQL 5.5–5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.3–9.5 • SAP ASE 15.7+ RDS instance databases: • Oracle Database 11g–12c • Microsoft SQL Server 2008 R2 - 2014 • MySQL 5.5–5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.3–9.5 • Amazon Aurora (MySQL-compatible data source) Amazon Redshift
- 25. AWS Database Migration service pricing T2 for developing and periodic data migration tasks C4 for large databases and minimizing time T2 pricing starts at $0.018 per hour for T2.micro C4 pricing starts at $0.154 per hour for C4.large 50 GB GP2 storage included with T2 instances 100 GB GP2 storage included with C4 instances Data transfer inbound and within AZ is free Data transfer across AZs starts at $0.01 per GB
- 26. Our customers…
- 27. Heterogeneous Migration Oracle private DC to RDS PostgreSQL migration Used the AWS Schema Conversion Tool to convert their database schema Used on-going replication (CDC) to keep databases in sync until they reached the cutover window Benefits: • Improved reliability of the cloud environment • Savings on Oracle licensing costs • SCT Assessment Report let them understand the scope of the migration
- 28. Scale Up Migration RDS MySQL to Amazon Aurora Migration Used DMS with on-going replication (CDC) to migrate the data Benefits: • Aurora handles their larger data storage requirements. Per regulations they are storing 120 TB of data for 2 years • Reduced cost and improved performance when compared to large MySQL instances
- 29. Homogeneous Migration RDS MySQL on EC Classic to VPC Database ran an end-user application so could not take downtime 70 RDS instances have been migrated Benefits: • Leveraging CDC, they could decrease outage per database to less than 5 mins • 70 RDS instances have been seamlessly migrated
- 30. Split Migration 35 million members on it’s site Many interdependent applications built over the last 15 years with unique High IOPS requirements Migration from legacy cloud service to AWS Combination of migration to MySQL on EC2 and Amazon Aurora Benefits: • DMS reduced the time required to migrate our databases by 40 percent • Realized 55 percent cost savings by moving some db’s to Amazon Aurora
- 31. Resources available to customers—DMS Getting Started Guide: Review technical documentation. Features and benefits: Highlights DMS features. Pricing: Prices for replication instances, storage, and data transfer. Support: Post your questions to our Support forum. Java SDK: Java-based API for creating and managing data migration tasks. AWS Command Line Interface: Start and stop replication tasks with simple commands.
- 32. Resources available to customers— AWS Schema Conversion Tool User Guide: Review technical docs at aws.amazon.com/documentation/SchemaConversionTool/ or choose the Download button. Download area: Get installation files for the Schema Conversion Tool. Support forums: Ask questions and review how-to guides. https://forums.aws.amazon.com/forum.jspa?forumID=208.
- 33. AWS database migration partners
- 34. Thank you!
Notes de l'éditeur
- Start with some audience interaction… How many of you have been to the AWS Database Migration website? Has anyone tried using DMS yet? Databases are special! Crown jewels of the business! They’re also the most common cause for lost sleep and IT escalations. So, its natural that people want to know,
- There are other ways of migrating too. Often the native option is the best option. They come with drawbacks though: downtime and the inability to move between engines
- - Pre DMS, costly, license-intensive replication software to move data - Were not simple to setup or use required expert resources - Long downtimes the other option - Customers who wanted to switch engines had to deal with differences in datatypes, table syntax, sp’s, functions, triggers
- Responsive to customers Migrations easier and less intrusive Migrate will low downtime to any RDS engine type Post migration, customers asked for ongoing sync Sync between on prem and also between engines This mobility addresses one of the big requests: The ability to migrate off of commercial, license-intensive database engines, onto cloud-native, non-commercial engines.
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
- The database migration process works like this. first start with the SCT.
- - Embedded SQL conversion - Cloud native code optimization
- Creates tables at target database Sets up metadata required at target Populates data from source Each process loads one entire table Can use multiple processes Can be paused When restarted will continue from where it was stopped Will reload any tables that were currently in progress
- Oracle supplemental logging MySQL row-level bin logging SQL Server bulk logged/full recovery Postgres WAL No agent Uses recovery log Native change data capture API
- Let’s explore how it is that DMS can enable near zero downtime migration. The customer starts by launching a DMS instance in their AWS account. (click) Then the customer provides database connection information to connect out to their on-premises database, and in to their AWS database. (click) Next the customer will select which tables, schemas or databases they want to migrate, (click) and DMS will load the data, and keep it in sync on an ongoing basis. (click) Finally, at the time of the customer’s choosing, they can simply change the application to point to the new AWS database, instead of their old on-premises database. (click)
- When you either migrate or replicate, you can choose the Multi-AZ option. This option provides DMS in a high availability mode–in case the primary DMS server fails, the secondary one kicks in.
- So by now it should be pretty obvious that data can move from A to B, and many have used DMS to do just that. Some customers have been pretty ingenious with DMS and have fanned in, or consolidated systems. DMS will put data where you ask it to, so if you want to merge data from a number of legacy systems into a single repository, DMS can help with this. And remember, DMS can just as easily move data between like databases and unlike databases, so you can take information from a number of different platforms and consolidate it in a single, cloud database.
- In addition to “fanning-in” or consolidating, we’ve had a customer who has taken their database and “fanned-out” different portions to different targets. For example, from their ERP system they replicate they customer list out to the CRM system and also take some of the general ledger transactions and funnel them over to a BI system. We’ve also had a customer who, for legacy reasons, used a single database for three different applications. With DMS you can split this information out to different databases. One could be PostgreSQL, one SQL Server, and one Aurora
- Some organisations have many years of historical data, and for those that are willing to make the leap, filtering can be done during a migration so that only newer information is kept. Filtering can be done on any field in the source database–not just dates.
- Just in case it wasn’t abundantly clear–DMS can move your data from pretty much anywhere to pretty much anywhere. Sort of like a courier company for the cloud
- Mention why no older engines as sources
- Where can you learn more? (CLICK) The User Guide and download option are available from the DMS section of the AWS Console (CLICK) Yeah, there’s the Download button. Click it and you’ll see something like this. (CLICK) Information on installation and use (CLICK) The Schema Conversion Tool has its own forum to ask questions and get help