AWS 마이그레이션 서비스 - 김일호 :: 2015 리인벤트 리캡 게이밍
•
8 j'aime•859 vues
11월18일 있었던 re:Invent re:Cap for Gaming 에서 김일호 솔루션즈 아키텍트가 발표한 ' AWS Database Migration Service ' 강연 발표자료입니다.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à AWS 마이그레이션 서비스 - 김일호 :: 2015 리인벤트 리캡 게이밍
Similaire à AWS 마이그레이션 서비스 - 김일호 :: 2015 리인벤트 리캡 게이밍 (20)
Plus de Amazon Web Services Korea
Plus de Amazon Web Services Korea (20)
Dernier
Dernier (20)
AWS 마이그레이션 서비스 - 김일호 :: 2015 리인벤트 리캡 게이밍
- 1. Introduction to AWS Database Migration Service 김일호, Solutions Architect
- 2. What to Expect from the Session • Learn about migrating databases with minimal downtime to Amazon RDS, Amazon Redshift and Amazon Aurora • Discuss database migrations to same and different engines • Learn about the converting schemas and stored code from Oracle and SQL Server to MySQL and Aurora • One more thing~
- 3. Embracing the cloud demands a cloud data strategy • How will my on-premises data migrate to the cloud? • How can I make it transparent to my customers? • Afterwards, how will on-premises and cloud data interact? • How can I integrate my data assets within AWS? • Can I get help moving off of commercial databases?
- 4. Historically, Migration = Cost, Time • Commercial Migration / Replication software • Complex to setup and manage • Legacy schema objects, PL/SQL or T-SQL code • Application downtime
- 5. Introducing AWS Database Migration Service
- 6. Purposes of data migration One-time data migration Between on premises and AWS Between Amazon EC2 and Amazo n RDS Ongoing Replication Replicate on premises to AWS Replicate AWS to on premises Replicate OLTP to BI Replicate for query offloading
- 7. Ways to migrate data Bulk Load AWS Database Migration Service Oracle Import/Export Oracle Data Pump Network Mode Oracle SQL*Loader Oracle Materialized Views CTAS / INSERT over dblink Ongoing Replication AWS Database Migration Service Oracle Data Pump Network Mode Oracle Materialized Views Oracle GoldenGate
- 8. High-speed database migration prior to AWS DMS EC2 Instance Linux Host On-Premises AWS Availability Zone Oracle DB RDS Oracle Tsunami Tsunami DATA_PUMP_DIR 500GB 175GB ~2.5 hours~2.5 hours Total Time ~7 hours ~3.5 hours~4 hours
- 9. Start your first migration in 10 minutes or less Keep your apps running during the migration Replicate within, to or from Amazon EC2 or RDS Move data to the same or different database engine Sign up for preview at aws.amazon.com/dms AWS Database Migration Service
- 10. 10 minutes or less to migration
- 11. Customer Premises Application Users AWS Internet VPN • Start a replication instance • Connect to source and target databases • Select tables, schemas, or databases Let AWS Database Migration Service create tables, load data, and keep them in sync Switch applications over to the target at your convenience Keep your apps running during the migration AWS Database Migration Service
- 12. After migration, use for replication and data integration • Replicate data in on-premises databases to AWS • Replicate OLTP data to Amazon Redshift • Integrate tables from third-party software into your reporting or core OLTP systems • Hybrid cloud is a stepping stone in migration to AWS
- 13. Cost-effective and no upfront costs • T2 pricing starts at $0.018 per Hour for T2.micro • C4 pricing starts at $0.154 per Hour for C4.large • 50GB GP2 storage included with T2 instances • 100GB GP2 storage included with C4 instances • • Data transfer inbound and within AZ is free • Data transfer across AZs starts at $0.01 per GB Swap Logs Cache
- 14. Migrate and replicate between database engines
- 15. Introducing AWS Schema Conversion Tool
- 16. Migrate off Oracle and SQL Server Move your tables, views, stored procedures and DM L to MySQL, MariaDB, and Amazon Aurora Know exactly where manual edits are needed Download at aws.amazon.com/dms AWS Schema Conversion Tool
- 17. Get help with converting tables, views, and code Schemas Tables Indexes Views Packages Stored Procedures Functions Triggers Sequences User Defined Types Synonyms
- 18. Move your database schema and code
- 19. Know exactly where manual edits are needed
- 20. One more thing.
- 21. RAC on Amazon EC2 would be useful • Test / dev / non-prod; allow testing to cover RAC-related regression cases • Scale out and back elastically; a good match for the cloud • Scale beyond the largest instances • High-RTO redundancy at the host/instance level; App continuity for near zero downtime • Test scaling limits; a given workload scales only to n nodes on RAC • Some applications “require” RAC • Some customers don’t want to re-engineer everything just to move to AWS • Customers want it!
- 23. Why no RAC on EC2? EBS Vol ume Shared Storage EC2 Instance X
- 24. Shared storage with iSCSI EBS Vol ume EC2 Instance EC2 Instance iSCSI Target ASM ASM ASM EC2 Instance iSCSI Target open-iscsi-2.0-873 Open iSCSI Project open-iscsi.org
- 25. Why no RAC on EC2? Multicast Network EC2 Instance EC2 Instance EC2 Instance Multicast
- 26. Multicast on EC2 EC2 Instance EC2 Instance EC2 Instance Multicast EC2 Instance N2N Edge edge0 N2N Edge edge0 N2N Edge edge0 N2N Supernode N2n VPN ntop ntop.org/n2n
- 27. RAC on EC2 prototype: aws.amazon.com/articles iSCSI Target 1 (i2.8xl) Subnet Placement Group (dedicated instances) VPC Route 53 Private Hosted Zone • DNS • SCAN • VIPs • Etc. 6400G Ephemeral SSD 4800G DATA LVM Vol tgtd iSCSI Target 2 (i2.8xl) RAC Node 1 (c3.8xl) RAC Node 2 (c3.8xl) 600G Flash Cache edge ASM (n ormal redund.) Grid Infra. DB 12c supernode 800G RECO 800G free 640G Ephemeral SSD iscsid 600G Flash Cache edge ASM (n ormal redund.) Grid Infra. DB 12c 640G Ephemeral SSD iscsid 6400G Ephemeral SSD 4800G DATA LVM Vol tgtd 800G RECO 800G free
- 28. Sign Up for AWS Database Migration Service • Sign up for AWS Database Migration Service Preview now: • aws.amazon.com/dms • Download the AWS Schema Conversion Tool: • aws.amazon.com/dms
- 29. Thank you
Notes de l'éditeur
- Introduce self and Sergei (Senior Product Manager) Migration from on-premises and traditionally hosted databases to AWS managed database services… Not only how to migrate between same engines, but also between different, like… But before you can migrate data anywhere, you need a schema; tables and objects into which to load the data. We’ll talk about how you can convert database objects, in order to support moving between engines.
- These days, we’re hearing a lot of customers tell us they want to move their on-premises applications into the cloud. But moving applications is simpler than moving the databases they depend on. Applications are usually stateless, and can be moved fairly easily using a lift and shift approach. (CLICK) But databases are stateful, and they require more care. To move databases to AWS, requires a data migration strategy. (CLICK) And when it comes to designing those strategies, customers want to be able to do it with the least possible inconvenience and visibility to their users. (CLICK) And once an application is migrated to AWS, it’s not the end of the story. Often customers have several applications, some in the cloud and some on premises or in hosted environments. Customers need to be able to synchronize their data between on-premises and cloud-based applications. (CLICK) And the same goes for applications within AWS. Those applications often share data, and customers want to be able to synchronize and replicate data between the various databases they maintain within AWS. (CLICK) And one other thing: customers moving applications to the cloud, often see it as an opportunity to break free from commercial databases, which tend to have a heavy licensing burden. We often hear customers asking us for a way to convert their commercial databases into AWS solutions, such as RDS MySQL, Postgres, Aurora and Redshift.
- Announcing preview of AWS DMS Explain what it basically is: A managed hosted data replication service engineered to support graceful migration from legacy database systems into the next generation managed databases at AWS.
- There are multiple reasons why you’d want to migrate your data. Read-only replication: reporting, read scaling Read/write replication: multi-master
- There is a multitude of approaches for migrating your data to AWS. Choose the method based on Data set size, Network Connectivity (access, latency, and bandwidth), ability to sustain downtime for source DB, Need for continuous data synchronization. If you can take or day or two of downtime, and you don’t have to do the migration process several times, you want to do a Bulk Load. If you need to minimize downtime, or when your dataset is large and you can’t shut down access to the source while you are migrating your data, you want to consider Ongoing Replication.
- In the past we recommended the following high-performance technique for moving large databases. Use DataPump and export files in parallel. Use a box that has multiple disks, to parallelize IO Compression Makes 500 GB to 175 GB Time to export 500GB is ~2.5 hours Transport compressed files to EC2 instance using UDP and the Tsunami server. Install Tsunami on both the source data host and the target EC2 host Using UDP you can achieve higher rates for transferring files that using TCP Start Upload when first files from DataPump become available. Upload in Parallel. No need to wait till all 18 files are done to start upload Time to upload 175GB is ~2.5 hours Step 3. Transfer Files to Amazon RDS DB instance Amazon RDS DB instance has an externally accessible Oracle Directory Object DATA_PUMP_DIR Use UTL_FILE. to move data files to Amazon RDS DATA_PUMP_DIR BEGIN perl_global.fh := utl_file.fopen(:dirname, :fname, 'wb', :chunk); END; BEGIN utl_file.put_raw(perl_global.fh, :data, true); END; BEGIN utl_file.fclose(perl_global.fh); END; Transfer Files as They Are Received, No need to wait till all 18 files are received in the EC2 instance. Start transfer to RDS instance as soon as the first file is received. Total time to Transfer Files to RDS ~3.5Hours Step 4. Import data into the Amazon RDS instance • Import from within Amazon RDS instance using DBMS_DATAPUMP package • Submit a job using PL/SQL script Total Time to Import Data into Amazon RDS: ~4hours But because we do everything staged and have 18 distinct files, the total duration is ~7hours. Background ------------------- Open port 46224 for Tsunami communication Tsunami UDP Protocol: A fast user-space file transfer protocol that uses TCP control and UDP data for transfer over very high speed long distance networks (≥ 1 Gbps and even 10 GE), designed to provide more throughput than possible with TCP over the same networks. Tsunami Servers: http://sourceforge.net/projects/tsunami-udp/files/latest/download?_test=goal Optimize the Data Pump Export • Reduce the data set to optimal size, avoid indexes • Use compression and parallel processing • Use multiple disks with independent I/O Optimize Data Upload • Use Tsunami for UDP-based file transfer • Use large Amazon EC2 instance with SSD or PIOPS volume • Use multiple disks with independent I/O • You could use multiple Amazon EC2 instances for parallel upload Optimize Data File Upload to RDS • Use the largest Amazon RDS DB instance possible during the import process • Avoid using Amazon RDS DB instance for any other load during this time • Provision enough storage in the Amazon RDS DB instance for the uploaded files and imported data
- * Like all AWS services, it is easy and straightforward to get started. You can get started with your first migration task in 10 min or less. You simply connect it to your source and target databases, and it copies the data over, and begins replicating changes from source to target. *That means that you can keep your apps running during the migration, then switch over at a time that is convenient for your business. * In addition to one-time database migration, you can also use DMS for ongoing data replication. Replicate within, to or from AWS EC2 or RDS databases For rinstance, After migrating your database, use the AWS Database Migration Service to replicate data into your Redshift data warehouses, cross-region to other RDS instances, or back to on-premises *Again- it is heterogeneous ~. With DMS, you can move data between engines. Supports Oracle, Microsoft SQL Server, MySQL, PostgreSQL, MariaDB, Amazon Aurora, Amazon Redshift * If you would like to sign up for the preview of DMS, go to…
- Let’s take a look at how to use the database migr. Service… From the landing page, just click “get started”. That will take you to page that describes how DMS works to migrate your data; how you connect it to a source database and target database, then define replication tasks to move the data.
- Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure. DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
- But DMSis for much more than just migration. (CLICK) DMSenables customers to adopt a hybrid approach to the cloud, maintaining some applications on premises, and others within AWS. There are dozens of compelling use cases for a hybrid cloud approach using DMS. (CLICK) for customers just getting their feet wet, AWS is a great place to keep up-to-date read-only copies of on-premises data for reporting purposes. AWS services like Aurora, Redshift and RDS are great platforms for this. (CLICK) With DMS, you can maintain copies of critical business data from third-party or ERP applications, like employee data from Peoplesoft, or financial data from Oracle E-Business Suite, in the databases used by the other applications in your enterprise. In this way, it enables application integration in the enterprise. (CLICK) Another nice thing about the hybrid cloud approach is that it lets customers become familiar with AWS technology and services gradually. DMS enables that. Moving to the cloud is much simpler if you have a way to link the data and applications that have moved to AWS with those that haven’t.
- With the AWS Database Migration Service you pay for the migration instance that moves your data from your source database to your target database.(CLICK) (Actually talk to points) Each database migration instance includes storage sufficient to support the needs of the replication engine, such as swap space, logs, and cache. (CLICK) (actually talk to points) Inbound data transfer is free. (CLICK) Additional charges only apply (CLICK) if you decide to allocate additional storage for data migration logs or when you replicate your data to a database in another region or on-premises. AWS Database Migration Service currently supports the T2 and C4 instance classes. T2 instances are low-cost standard instances designed to provide a baseline level of CPU performance with the ability to burst above the baseline. They are suitable for developing, configuring and testing your database migration process, and for periodic data migration tasks that can benefit from the CPU burst capability. C4 instances are designed to deliver the highest level of processor performance and achieve significantly higher packet per second (PPS) performance, lower network jitter, and lower network latency. You should use C4 instances if you are migrating large databases and are looking to minimize the migration time.
- Elaborate on heterogeneous use cases Database engine migration – cost savings; Move to full managed and scalable cloud-native – Ent class like Aurora Low-cost reporting, analytics and BI for systems on commercial OLTP (MySQL Postgres Aurora) Data integration – customer accounts, data like that, can be presented no only on the master platform, but also in applications that are based on non-commercial But you can’t just pick up an Oracle table and put it down in MySQL. You can’t run an Oracle PL/SQL package on Postgres. To migrate or replicate data between engines, you need a way to convert the schema, to build a set of tables and objects on the destination that is native to that engine. We’ve been working on that problem. Introduce Sergei
- The AWS Schema Conversion Tool is a development environment that you download to your desktop and use to save time when migrating from Oracle and SQL Server to next-generation cloud databases such as Amazon Aurora. You can convert database objects such as tables, indexes, views, stored procedures, and Data Manipulation Language (DML) statements like SELECT, INSERT, DELETE, UPDATE.