資料遷移的最佳實踐 - Migrating Database to the Cloud 講師: Kim Kao, Solutions Architect, AWS

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Database to the Cloud
Kim Kao, Solutions Architect

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Today
OBJECTIVES
• Understand what makes a successful DB migration project
• WHY do we migrate
• WHAT should we migrate
• HOW do we migrate? Project view
• HOW do we migrate? Engineering view
• Understand the concrete steps to initiate the migration

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How does a successful
migration project looks like?

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHECKLIST #ONE
GOALS AND OBJECTIVES

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Aurora differentiation
Provision a database in 6 minutes
Provision a MAZ database with a few mouse clicks
MAZ RPO = 0, RTO = 60-90 20-30 sec
Scale, patches with 60-90 sec near zero downtime
Performance, single node 25k 200k Write, 300k 600k Read
Add read replicas with a few mouse clicks
Read replica auto-scaling, serverless, multi master (coming)
Protect your backups and logs with 11 9’s of durability
Recover to any point in time from nightly backups + logs (5mins)
Detailed metrics, down to 1 second intervals
Secure your data with single click encryption at rest
Aurora

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Databases and Analytics
Broad and deep portfolio, purpose-built for builders
Data Lake
S3/Glacier AWS Glue
(ETL & Data Catalog)
Data Movement
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams
Non-Relational Databases
DynamoDB
ElastiCache
(Redis, Memcached)
Neptune
(Graph)
Analytics
DW | Big Data Processing | Interactive
Redshift EMR Athena
Kinesis
Analytics
Elasticsearch
Service
Real-time
Relational Databases
RDS
Aurora
Business Intelligence & Machine Learning
QuickSight SageMaker Comprehend

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Strategies for a 100 applications
“How can we start the migration? ”

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ISV solution?
Yes
Support Open
Source DB?
No
Vendor support
engine
Yes
RDS, Aurora,
compatible option
No
New build
system?
No
Biz gain >
Re-architecture
effort ?
No
Re-platform to
RDS, PIOPS,
Multi-AZ, Rehost
Yes
Re-architecture
Yes
Read/Write/Acces
s pattern
How do you query
the data
Read/Write RPS
Data volume
growth/total
Latency
requirement

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Comparing Cloud Migration Strategies
•
Time Cost Business Value # App
Retain N/A 20
Retire N/A 5
Re-host 20
Re-platform 20
Re-purchase 5
Re-architect 30
Low
High
MigrationComplexity

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pilot
Migration heat map
Easy to migrate
High business value
Low business value
Hard to migrate

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHECKLIST #TWO
CHALLENGES AND CONCERNS

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Understanding Database
Migration Projects

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s Not Just the Database!
Application code
Scripts and integration points
Backup and recovery
General management procedures
• Monitoring
• Notifications
• Auditing
• Tuning and Diagnostics
People!

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Project Planning
Project Assessment
Database Assessment
Application Assessment
Skill Sets Required
Hiring and Developing Talent
Choosing the right POCs
Project Execution

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Effort

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration
• Like to like (MSSQL to MSSQL)
• Rehost
• EC2
• Re-platform
• RDS, Elasticache
• Re-architect
• Cache layer, Fan out,
Consolidate
• Switch engine (MSSQL to PostgreSQL)
• Re-architect
• Cloud Database
• More code rewrite

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Like to like
Rehost, Re-platform

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration options
If you’re not switching engines, native tools:
• SQL Server: bak file import, log shipping, AG Async
• MySQL: mysql_dump, read replicas
• Oracle SQL Developer, Data Pump, Export/Import
• PostgreSQL: pg_dump, read replicas
• SAP ASE: bcp
• Or AWS DMS

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration – multi phase process
Phase Description Automation Effort (%)
1 Assessment 4
2 Integration with 3rd party applications 6
3 Data Migration Native tools or DMS 7
4 Functional testing of the entire system 54
5 Performance tuning 4
6 Integration and deployment 13
7 Training and knowledge 3.5
8 Documentation and version control 3.5
9 Post production support 5

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Switch engine
Re-architecture

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration process
Source
Oracle Database on premises
in EC2 or RDS
Source
Oracle Database on premises
in EC2 or RDS
AWS Schema Conversion Tool
Target
Amazon Aurora PostgreSQL
Target
Amazon Aurora PostgreSQL
Step 1
Step 2
AWS Database Migration Service

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration – multi phase process
Phase Description Automation Effort (%)
1 Assessment SCT 2
2 Database Schema Conversion SCT/DMS 14
3 Application Conversion/Remediation SCT 25
4 Scripts Conversion SCT 7
5 Integration with 3rd party applications 3
6 Data Migration DMS 4
7 Functional testing of the entire system 29
8 Performance tuning SCT 2
9 Integration and deployment 7
10 Training and knowledge 2
11 Documentation and version control 2
12 Post production support 3

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assessment

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Components of the SCT console
1. Source Schema
2. Action items
3. Target Schema
4. Schema element details
5. Edit window

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database migration assessment
Connect SCT to source
and target databases
Run assessment report
Read executive summary
Follow detailed
instructions

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCT – Assessment report

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Granularity of the messages in the report

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use SCT to convert the SQL code in your C++, C#, Java

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Supported Conversions – OLTP
*MySQL means any of Aurora, MySQL, MariaDB
Source Engine Target Engine
Microsoft SQL Server
(version 2008 and later)
Amazon Aurora (MySQL or PostgreSQL), Microsoft SQL
Server, MySQL, PostgreSQL
Oracle
(version 10.2 and later)
Amazon Aurora (MySQL or PostgreSQL), MySQL, Oracle,
PostgreSQL
PostgreSQL
(version 9.1 and later)
Amazon Aurora (MySQL), MySQL, PostgreSQL
MySQL
(version 5.6 and later)
Amazon Aurora (PostgreSQL), MySQL, PostgreSQL

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use DMS and SCT? Modernize
Modernize your database tier
• Commercial to open-source
• Commercial to Amazon Aurora
Modernize your warehouse
• Commercial to Amazon Redshift
Amazon
Aurora
Amazon
Redshift
Amazon S3
Amazon
DynamoDB

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS - Overview

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Start your first migration in 10 minutes or less
• Keep your apps running during the migration
• Replicate within, to or from Amazon EC2 or RDS
• Move data to the same or a different database engine
AWS
Database Migration
Service

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer
premises
Application users
AWS
Internet
VPN
Keep your apps running during the migration
® Start a replication instance
® Connect to source and target
databases
® Select tables, schemas, or
databases
® Let AWS Database Migration
Service (AWS DMS) create
tables, load data, and keep
them in sync
® Switch applications over to
the target at your convenience
AWS
DMS

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Scenarios

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replication
Instance
Source Target
Multiple Targets
Target
Target

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration
Server
Source Target
Multiple Sources
Source
Source

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration
Server
Source
Target
Multiple Sources and Targets
Source
Source
Target

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
You Don’t Have to Take Everything
Source
L
Target
Replication Instance

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Homogenous or Heterogeneous
Replication
Instance
SQL Server MySQL
Replication
Instance
Oracle Oracle
Replication
Instance
Oracle Aurora

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment Options
Option 1
• Let DMS Create Tables and Primary Keys
• Create needed secondary objects after initial Load has completed.
Option 2
• Create all objects
• Disable the secondary objects
• Load using DMS(Do Not Drop)
• Re-enable secondary Objects
Option 3
• Create only Tables, primary keys, views, and sequences
• Load using DMS(Do Not Drop)
• Create all secondary Objects

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replication
Instance
Source Target
Start Full Load

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source TargetReplication Instance
Update
While Loading Data Also Capture Changes

43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source Target
Replication Instance
Update
Load Complete - Apply Captured Changes

44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changes Reach Steady State
Source Target
Replication Instance
Update

45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source Target
Replication Instance
Update
Shut Down Apps & Apply Remaining Changes

46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Flip!
Source Target
Replication Instance
Update

47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load is Table by Table
Replication
Instance
Source Target
Creates tables at target database
Sets up metadata required at target
Populates data from source
Each process loads one entire table
Can use multiple processes
Can be paused
• When restarted will continue from where it was stopped
• Will reload any tables that were currently in progress

48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Server
Source Target
Update
t1 t2
t1
t2
Changes are Transactional & Come From Logs

49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS – Change Data Capture (CDC)
“No Touch” design
• Reads recovery log of source database
• Using the engine’s native change data capture API
• No agent required on the source
• Changes captured as transactions and applied to target
• Activated when load starts
• When load completes, changes are applied as soon as possible

50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS – Change Data Capture (CDC)
Some requirements
• Oracle: Supplemental logging required
• MySQL: Full image row level bin logging required
• SQL Server: Recovery model bulk logged or full
• Postgres: wal_level = logical; max_replication_slots >= 1; max_wal_Senders
>=1; wal_sender_timeout = 0
• For RDS, simply set rds.logical_replication = 1

51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary
SCT – Schema conversion tool
DMS – Database migration service

52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!