SlideShare une entreprise Scribd logo
1  sur  38
Télécharger pour lire hors ligne
Amazon RDS for MySQL: Best
Practices and Data Migration

© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
AWS Webcast - Amazon RDS for MySQL: Best Practices and Migration
Multi-AZ deployments
cname update

DNS
Primary Update

AZ1

Physical
Synchronous
Replication

AZ2

New!

Less than 2 minutes for failover
Read Replicas

Sync
Replication
Async Replication
Logical Failover – Replicas of Replicas

Sync
Replication

Async Replication
Things to consider – Read Replicas
• MySQL replication is single-threaded
• MySQL 5.1 and 5.5 – Replication may stop after crash
recovery on the master
– sync_binlog = 0 by default
– Includes Multi-AZ failover
– mysql.rds_next_master_log if potential missing info. is acceptable.

• MySQL 5.6 – Crash safe slaves
– sync_binlog = 1 by default
– Less performance impact but more replica reliability
AWS Webcast - Amazon RDS for MySQL: Best Practices and Migration
Backup and Restore
Amazon S3

Log
1 Log
2

AZ1

Log
5

AZ2
Cross Region – Snapshot Copy
Amazon
S3

Amazon
S3

AZ1

AZ3
Region 1

Region 2
Cross Region – Read Replicas

AZ1
Region 1

AZ3
Region 2
AWS Webcast - Amazon RDS for MySQL: Best Practices and Migration
Provisioned IOPS - Scale
db.m2.4xlarge - 130GB Data - Partial Random Read Workload
20,000

17,500

15,000

333GB - Regular
12,500
T
P 10,000
S
7,500

5,000

2,500

0

333GB - 1000 PIOPS

300GB - 3000 PIOPS
Provisioned IOPS - Latency

db.m2.4xlarge - 130GB Data - Partial Random Read Workload - Max Rate

9%

Percentage in Latency Bucket

8%

7.69% 7.89%

7%

6%

333GB - Regular

333GB - 1000 PIOPS

300GB - 3000 PIOPS

5%
4%

3.83%

3%

2%

1.38%

1%

0.42%
0.00%

0.01% 0.00% 0.00%

0%

3-20 ms

20-500ms

>500ms
Provisioned IOPS - Scale
db.m2.4xlarge - 130GB Data - Partial Random Read Workload
20,000

17,500

15,000

333GB - Regular

333GB - 1000 PIOPS

12,500
T
P 10,000
S
7,500

5,000

2,500

0

Target = 5000

300GB - 3000 PIOPS
Provisioned IOPS– Latency @ 5000 TPS
db.m2.4xlarge - 130GB Data - Partial Random Read Workload - 5000 TPS
9%

Percentage in Latency Bucket

8%

7.89%

7%
6%
5%

333GB - Regular

333GB - 1000 PIOPS

300GB - 3000 PIOPS

4%
3%
2%
1.17%
1%

0.35% 0.27%

0.00% 0.00%

0.04% 0.00% 0.00%

0%

3-20 ms

20-500ms

>500ms
Provisioned IOPS– 10,000 IOPS
db.m2.4xlarge - 130GB Data - Partial Random Read Workload

40,000

35,000

30,000

25,000
T
P 20,000
S
15,000

10,000

5,000

0

333GB - Regular
300GB - 3000 PIOPS

333GB - 1000 PIOPS
1000GB-10000 PIOPS
What Limit? – Amazon CloudWatch
Metrics
Bottlenecks and Scaling
Throughput
•
•
•
•

EBS Optimized (0.5 -1Gbit)
db.m1.large,db.m1.xlarge,m2.2xlarge,m2.4xlarge

db.cr1.8xlarge – 10Gbit
Regular + non optimized – shared

DB Instance Class

IOPS

•
•
•

•

db.t1.micro – db.cr1.8xlarge
1-88 ECU
0.6 – 244 GB RAM

Memory to ECU Ratio
•

1.7 to 2.8 (exclude micro)

Regular or 1,000-30,000

Storage
•

Ratio 3:1 to 10:1

5GB - 100GB - 3TB

DB Engine
•
•

Block Size 8 – 16K
Ability to utilize resources
Read/Write Benchmark

130GB Data - Partial Random 90R/10W Workload 10K IOPS

80,000
70,000

cpu
95%
r/s
6800
w/s
2600
rMB/s 104

60,000

TPS

50,000
40,000
30,000

20,000
10,000

cpu
85%
r/s
3450
w/s
600
rMB/s
52

cpu
100%
r/s
6800
w/s
950
rMB/s 104

42,000

cpu
40%
r/s
0
w/s
3200
rMB/s
0

47,000

13,200
5,850
-

db.m1.large

db.m1.xlarge

db.m2.4xlarge

db.cr1.8xlarge
Bottlenecks and Scaling – Read Replicas
Read/Write Benchmark – Using RR
130GB Data - Partial Random 90R/10W Workload – 10K IOPS

120,000

100,000

TPS

80,000
126,000

60,000

40,000

42,000

20,000
5,850

47,000

db.m2.4xlarge

db.cr1.8xlarge

13,200

-

db.m1.large

db.m1.xlarge

cr1 + 3 x 4xlarge
Things to consider – Provisioned IOPS
• Use Provisioned IOPS optimized instances
– M1.L, M3.XL, M2.2XL (500Mbps)
– M1.XL, M3.2XL, M2.4XL (1000Mbps)
– CR1.8XL (>1000 Mbps)

• Understand Channel Bandwidth
– Full duplex
– 1000 Mbps ~ 100MBps (with protocol overhead) or
– 100 MBps ~ 6250 16KB IOPS
Things to consider – Provisioned IOPS
• Max realizable IOPS
– Workload dependent
– 1:1 R/W -> Max realizable IOPS ~12.5K 16KB IOPS for M2.4XL
– 1:1 R/W -> 20K 16KB IOPS for CR1.8XL

• Provisioning more than Max can help lower latency
• IO Size
– IO Sizes <= 16KB is same
– IO Sizes > 16KB consumes more IO
– 6250 16KB IOPS = 3125 32KB IOPS
Things to consider – Provisioned IOPS
• Not able to realize IOPS provisioned?
–
–
–
–

Using Provisioned IOPS optimized instances?
Running automated backups, snapshots, scale storage?
Reviewed Queue Depth?
Database contention? Locking? Deadlocks?
Other Best Practices
• Storage Engine
– Avoid MyISAM – not transactional

• Cloudwatch alarms
– CPU, Memory, Storage, Latency, Replica Lag

• SMS/email notifications
– Failover, Replication status

• Number of Tables
– Not more than 1000 tables (standard) and 10,000 tables (PIOPS)
AWS Webcast - Amazon RDS for MySQL: Best Practices and Migration
RDS Pre-Migration Steps
•
•
•
•
•

Stop applications accessing the DB
Take a snapshot
Disable backups
Use Single-AZ instances
Configure security for cross-DB traffic
Importing from a MySQL DB Instance
Application

DB

Application

mysqldump

Staging
area

Load data

scp
Tsunami UDP
Staging server

Replication

AWS Region
Create a DB Instance for MySQL and EC2
Create DB instance for MySQL using AWS Management Console or CLI
PROMPT>rds-create-db-instance mydbinstance -s 1024 -c db.m3.2xlarge -e MySQL - u
<masterawsuser> -p <secretpassword> --backup-retention-period 3

Create Amazon EC2 (Staging server) using AWS Management Console or CLI
aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type m3.2xlarge --key-name
MyKeyPair --security-groups MySecurityGroup

Create replication user on the master
mysql> GRANT SELECT,REPLICATION USER,REPLICATION CLIENT ON *.* TO
repluser@‘<RDS Endpoint>' IDENTIFIED BY ‘<password>';
Configure the Master Database
Record the “File” and the “Position” values.
$ mysql -h localhost -u root -p
mysql> show master statusG
*************************** 1. row ***************************
File: mysql-bin.000023
Position: 107
Binlog_Do_DB: mytest
Binlog_Ignore_DB:
1 row in set (0.00 sec)
Importing from a MySQL DB Instance
Upload Files to Amazon EC2 using UDP
• Tar and compress MySQL dump file preparation to
ship to Amazon EC2 staging server.
• Update the Amazon EC2 security group to allow
UDP connection from the server where the dump file
is being created to your new MySQL client server.
• On the Amazon EC2 staging instance, untar the
tar.tgz file.
Configure the Amazon RDS database
Create the database
mysql> create database bench;

Import the database that you previously exported from the master database
Mysql> load data local infile '/reinvent/tables/customer_address.txt' into table customer_address
fields terminated by ',';
Mysql> load data local infile '/reinvent/tables/customer.txt' into table customer fields terminated by ',';

Configure the slave DB instance for MySQL, and start the slave server
mysql> call mysql.rds_set_external_master(‘<master
server>',3306,‘<replicationuser>',‘<password>','mysql-bin.000013',107,0);
mysql> call mysql.rds_start_replication;
Make Amazon RDS instance the Master
Switch over to the RDS instance
– Stop the service/application that is pointing at the Master
Database
– Once all changes have been applied to New RDS
Database. Stop replication with “call mysql.rds_stop_replication”
– Point the service/application at the New RDS Database.
– Once Migration is complete. “call mysql.
rds_reset_external_master”
RDS Post-migration Steps
•
•
•
•
•

Turn on backups
Turn on multi-az
Tighten down security
Set up alarms for key metrics
Turn on notifications for Database Events
References
• Data Import Guide for MySQL
• Best practices/operational guidelines
• Using Amazon RDS Notifications
Questions?

© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Thank You

© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Contenu connexe

En vedette

MySQL→Aurora移行セミナー
MySQL→Aurora移行セミナーMySQL→Aurora移行セミナー
MySQL→Aurora移行セミナー真吾 吉田
 
実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜
実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜
実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜真吾 吉田
 
サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)
サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)
サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)真吾 吉田
 
(DAT405) Amazon Aurora Deep Dive
(DAT405) Amazon Aurora Deep Dive(DAT405) Amazon Aurora Deep Dive
(DAT405) Amazon Aurora Deep DiveAmazon Web Services
 
jawsdays 2017 新訳-とある設計士の雲設計定石目録_3
jawsdays 2017 新訳-とある設計士の雲設計定石目録_3jawsdays 2017 新訳-とある設計士の雲設計定石目録_3
jawsdays 2017 新訳-とある設計士の雲設計定石目録_3a kyane
 
データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜
データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜
データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜Mikiya Okuno
 

En vedette (6)

MySQL→Aurora移行セミナー
MySQL→Aurora移行セミナーMySQL→Aurora移行セミナー
MySQL→Aurora移行セミナー
 
実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜
実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜
実務で活かせる AWSアーキテクチャ設計 〜AWS re:Invent 2016アップデート最新版〜
 
サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)
サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)
サーバーレスにおける開発プロセス戦略(パネルディスカッション用スライド)
 
(DAT405) Amazon Aurora Deep Dive
(DAT405) Amazon Aurora Deep Dive(DAT405) Amazon Aurora Deep Dive
(DAT405) Amazon Aurora Deep Dive
 
jawsdays 2017 新訳-とある設計士の雲設計定石目録_3
jawsdays 2017 新訳-とある設計士の雲設計定石目録_3jawsdays 2017 新訳-とある設計士の雲設計定石目録_3
jawsdays 2017 新訳-とある設計士の雲設計定石目録_3
 
データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜
データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜
データモデルについて知っておくべき7つのこと 〜NoSQLに手を出す前に〜
 

Plus de Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

Plus de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Dernier

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 

Dernier (20)

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 

AWS Webcast - Amazon RDS for MySQL: Best Practices and Migration

  • 1. Amazon RDS for MySQL: Best Practices and Data Migration © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 3. Multi-AZ deployments cname update DNS Primary Update AZ1 Physical Synchronous Replication AZ2 New! Less than 2 minutes for failover
  • 5. Logical Failover – Replicas of Replicas Sync Replication Async Replication
  • 6. Things to consider – Read Replicas • MySQL replication is single-threaded • MySQL 5.1 and 5.5 – Replication may stop after crash recovery on the master – sync_binlog = 0 by default – Includes Multi-AZ failover – mysql.rds_next_master_log if potential missing info. is acceptable. • MySQL 5.6 – Crash safe slaves – sync_binlog = 1 by default – Less performance impact but more replica reliability
  • 8. Backup and Restore Amazon S3 Log 1 Log 2 AZ1 Log 5 AZ2
  • 9. Cross Region – Snapshot Copy Amazon S3 Amazon S3 AZ1 AZ3 Region 1 Region 2
  • 10. Cross Region – Read Replicas AZ1 Region 1 AZ3 Region 2
  • 12. Provisioned IOPS - Scale db.m2.4xlarge - 130GB Data - Partial Random Read Workload 20,000 17,500 15,000 333GB - Regular 12,500 T P 10,000 S 7,500 5,000 2,500 0 333GB - 1000 PIOPS 300GB - 3000 PIOPS
  • 13. Provisioned IOPS - Latency db.m2.4xlarge - 130GB Data - Partial Random Read Workload - Max Rate 9% Percentage in Latency Bucket 8% 7.69% 7.89% 7% 6% 333GB - Regular 333GB - 1000 PIOPS 300GB - 3000 PIOPS 5% 4% 3.83% 3% 2% 1.38% 1% 0.42% 0.00% 0.01% 0.00% 0.00% 0% 3-20 ms 20-500ms >500ms
  • 14. Provisioned IOPS - Scale db.m2.4xlarge - 130GB Data - Partial Random Read Workload 20,000 17,500 15,000 333GB - Regular 333GB - 1000 PIOPS 12,500 T P 10,000 S 7,500 5,000 2,500 0 Target = 5000 300GB - 3000 PIOPS
  • 15. Provisioned IOPS– Latency @ 5000 TPS db.m2.4xlarge - 130GB Data - Partial Random Read Workload - 5000 TPS 9% Percentage in Latency Bucket 8% 7.89% 7% 6% 5% 333GB - Regular 333GB - 1000 PIOPS 300GB - 3000 PIOPS 4% 3% 2% 1.17% 1% 0.35% 0.27% 0.00% 0.00% 0.04% 0.00% 0.00% 0% 3-20 ms 20-500ms >500ms
  • 16. Provisioned IOPS– 10,000 IOPS db.m2.4xlarge - 130GB Data - Partial Random Read Workload 40,000 35,000 30,000 25,000 T P 20,000 S 15,000 10,000 5,000 0 333GB - Regular 300GB - 3000 PIOPS 333GB - 1000 PIOPS 1000GB-10000 PIOPS
  • 17. What Limit? – Amazon CloudWatch Metrics
  • 18. Bottlenecks and Scaling Throughput • • • • EBS Optimized (0.5 -1Gbit) db.m1.large,db.m1.xlarge,m2.2xlarge,m2.4xlarge db.cr1.8xlarge – 10Gbit Regular + non optimized – shared DB Instance Class IOPS • • • • db.t1.micro – db.cr1.8xlarge 1-88 ECU 0.6 – 244 GB RAM Memory to ECU Ratio • 1.7 to 2.8 (exclude micro) Regular or 1,000-30,000 Storage • Ratio 3:1 to 10:1 5GB - 100GB - 3TB DB Engine • • Block Size 8 – 16K Ability to utilize resources
  • 19. Read/Write Benchmark 130GB Data - Partial Random 90R/10W Workload 10K IOPS 80,000 70,000 cpu 95% r/s 6800 w/s 2600 rMB/s 104 60,000 TPS 50,000 40,000 30,000 20,000 10,000 cpu 85% r/s 3450 w/s 600 rMB/s 52 cpu 100% r/s 6800 w/s 950 rMB/s 104 42,000 cpu 40% r/s 0 w/s 3200 rMB/s 0 47,000 13,200 5,850 - db.m1.large db.m1.xlarge db.m2.4xlarge db.cr1.8xlarge
  • 20. Bottlenecks and Scaling – Read Replicas
  • 21. Read/Write Benchmark – Using RR 130GB Data - Partial Random 90R/10W Workload – 10K IOPS 120,000 100,000 TPS 80,000 126,000 60,000 40,000 42,000 20,000 5,850 47,000 db.m2.4xlarge db.cr1.8xlarge 13,200 - db.m1.large db.m1.xlarge cr1 + 3 x 4xlarge
  • 22. Things to consider – Provisioned IOPS • Use Provisioned IOPS optimized instances – M1.L, M3.XL, M2.2XL (500Mbps) – M1.XL, M3.2XL, M2.4XL (1000Mbps) – CR1.8XL (>1000 Mbps) • Understand Channel Bandwidth – Full duplex – 1000 Mbps ~ 100MBps (with protocol overhead) or – 100 MBps ~ 6250 16KB IOPS
  • 23. Things to consider – Provisioned IOPS • Max realizable IOPS – Workload dependent – 1:1 R/W -> Max realizable IOPS ~12.5K 16KB IOPS for M2.4XL – 1:1 R/W -> 20K 16KB IOPS for CR1.8XL • Provisioning more than Max can help lower latency • IO Size – IO Sizes <= 16KB is same – IO Sizes > 16KB consumes more IO – 6250 16KB IOPS = 3125 32KB IOPS
  • 24. Things to consider – Provisioned IOPS • Not able to realize IOPS provisioned? – – – – Using Provisioned IOPS optimized instances? Running automated backups, snapshots, scale storage? Reviewed Queue Depth? Database contention? Locking? Deadlocks?
  • 25. Other Best Practices • Storage Engine – Avoid MyISAM – not transactional • Cloudwatch alarms – CPU, Memory, Storage, Latency, Replica Lag • SMS/email notifications – Failover, Replication status • Number of Tables – Not more than 1000 tables (standard) and 10,000 tables (PIOPS)
  • 27. RDS Pre-Migration Steps • • • • • Stop applications accessing the DB Take a snapshot Disable backups Use Single-AZ instances Configure security for cross-DB traffic
  • 28. Importing from a MySQL DB Instance Application DB Application mysqldump Staging area Load data scp Tsunami UDP Staging server Replication AWS Region
  • 29. Create a DB Instance for MySQL and EC2 Create DB instance for MySQL using AWS Management Console or CLI PROMPT>rds-create-db-instance mydbinstance -s 1024 -c db.m3.2xlarge -e MySQL - u <masterawsuser> -p <secretpassword> --backup-retention-period 3 Create Amazon EC2 (Staging server) using AWS Management Console or CLI aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type m3.2xlarge --key-name MyKeyPair --security-groups MySecurityGroup Create replication user on the master mysql> GRANT SELECT,REPLICATION USER,REPLICATION CLIENT ON *.* TO repluser@‘<RDS Endpoint>' IDENTIFIED BY ‘<password>';
  • 30. Configure the Master Database Record the “File” and the “Position” values. $ mysql -h localhost -u root -p mysql> show master statusG *************************** 1. row *************************** File: mysql-bin.000023 Position: 107 Binlog_Do_DB: mytest Binlog_Ignore_DB: 1 row in set (0.00 sec)
  • 31. Importing from a MySQL DB Instance
  • 32. Upload Files to Amazon EC2 using UDP • Tar and compress MySQL dump file preparation to ship to Amazon EC2 staging server. • Update the Amazon EC2 security group to allow UDP connection from the server where the dump file is being created to your new MySQL client server. • On the Amazon EC2 staging instance, untar the tar.tgz file.
  • 33. Configure the Amazon RDS database Create the database mysql> create database bench; Import the database that you previously exported from the master database Mysql> load data local infile '/reinvent/tables/customer_address.txt' into table customer_address fields terminated by ','; Mysql> load data local infile '/reinvent/tables/customer.txt' into table customer fields terminated by ','; Configure the slave DB instance for MySQL, and start the slave server mysql> call mysql.rds_set_external_master(‘<master server>',3306,‘<replicationuser>',‘<password>','mysql-bin.000013',107,0); mysql> call mysql.rds_start_replication;
  • 34. Make Amazon RDS instance the Master Switch over to the RDS instance – Stop the service/application that is pointing at the Master Database – Once all changes have been applied to New RDS Database. Stop replication with “call mysql.rds_stop_replication” – Point the service/application at the New RDS Database. – Once Migration is complete. “call mysql. rds_reset_external_master”
  • 35. RDS Post-migration Steps • • • • • Turn on backups Turn on multi-az Tighten down security Set up alarms for key metrics Turn on notifications for Database Events
  • 36. References • Data Import Guide for MySQL • Best practices/operational guidelines • Using Amazon RDS Notifications
  • 37. Questions? © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 38. Thank You © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.