SlideShare une entreprise Scribd logo
1  sur  36
Télécharger pour lire hors ligne
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Marc Trimuschat, AWS Storage Services
April 19, 2017
Migrating Large Scale Data Sets to the Cloud
Data has gravity
…easier to move processing to the data
4k/8k
Genomics
Seismic
Financial
Logs
IoT
Cloud Data Migration
Direct
Connect
Snow* data
transport
family
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
The AWS Storage Portfolio
Object
Amazon GlacierAmazon S3
Block
Amazon EBS
(persistent)
Amazon EC2
Instance Store
(ephemeral)
File
Amazon EFS
The Snow Family:
Large batches or “edge” scenarios
AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale data
migration
AWS Snowball
Petabyte-scale data transport
• Rugged 8.5G impact case
• Rain and dust resistant
• Data encryption end-to-end
• 80 TB capacity/10G network
E-ink shipping
label
How Snowball moves data into and out of AWS
Create
a job
Connect the
Snowball
Copy data to
the Snowball
Your data
moved to
Amazon S3
In transit to you Delivered to you Delivered to AWS At AWS
Job created Job completed
AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equivalent to an
Amazon EC2 m4.4xlarge instance
• 10GBase-T, 10/25Gb SFP28, and
40Gb QSFP+ copper, and optical
networking
• Ruggedized and rack-mountable
Collect data Create job Copy data Moved to S3
Snowball Edge: Hybrid capabilities beyond data migrationMIGRATIONCOLLECTION
Create job Copy data Moved to S3
When to use AWS Snowball
Cloud
Migration
Disaster
Recovery
Data Center
Decommission
Content
Distribution
Customer: Scripps Networks Interactive
Problem Statement:
• Need storage platform to manage active archive content
• Existing content repository too large to migrate via
available network-based ingest methods
Use of AWS:
• Amazon S3 and Snowball for massively scalable ingest
• Amazon S3 for storage, Amazon Glacier for content
archive
• Snowball to securely transport existing media content
from on-premises storage and tape vault
Business Benefits:
• Petabyte-scale data transport without increased network
costs
• Massive scalability and elasticity
• Lower TCO for active archive storage
Active archive transport and archival for digital content provider
AWS Storage Gateway:
Augmenting on-premises storage with cloud
Storage Gateway hybrid storage solutions
Use standard storage protocols to access AWS storage services
AWS Storage
Gateway
Amazon EBS
snapshots
Amazon S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Files
Volumes
Tapes
On-premises
AWSCloud
File gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises
File
Gateway
Data stored and retrieved from your S3 buckets
One-to-one mapping from files-to-objects
File metadata stored in object metadata
Bucket access managed by IAM role you own and manage
Use S3 Lifecycle Policies, versioning, or CRR to manage data
Amazon
Glacier
S3
Standard
S3
Standard -
Infrequent
Access
HTTPS
NFS
v3 / v4.1
Application
Server
Enabling cloud workloads
Move data to AWS storage for Big Data, cloud bursting, or migration
“Storage Gateway has the promise to transform the way we move
data into the cloud. The NFS interface lets us easily integrate data
files from analytical instruments, and the transparent S3 storage
lets us easily connect our cloud-based applications and leverage the
powerful storage capabilities of S3.
“With Storage Gateway, we can now unleash the full power of
AWS on our instrument data.”
Application
Server
Volume gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
Block storage in S3 accessed via the volume gateway
Compression of data in-transit and at-rest
Back up on-premises volumes to EBS snapshots
Create on-premises volumes from EBS snapshots
Up to 1 PB of total volume storage per gateway
Amazon
EBS
snapshots
Storage Gateway
bucket in
Amazon S3
Customer Premises
Volume
Gateway
iSCSI HTTPS
Tiering storage into S3, Glacier, and EBS
Easily add AWS storage to your on-premises environment
“Storage Gateway is at the core of our disaster
recovery and business continuity (BCM) processes,
handling our Co-Lo'd OLTP and OLAP off site data
backups, as well as our in-office BCM. It works
transparently, in a lights out way, archiving off to a
separate AWS account with a simple grandfather-father-
son snapshot plan in place.”
Tape gateway
Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Up to 1 PB total tape storage per gateway, unlimited archive capacity
Supports leading backup applications:
Archived tapes
stored in
Amazon Glacier
MEDIA
CHANGER
TAPE
DRIVE
Customer Premises
Tape
Gateway
Virtual tapes
stored in
Amazon S3
Backup
Server
HTTPSiSCSI
Backup, archive, and disaster recovery
Cost-effective storage in AWS with local or cloud restore
“Tapes are a headache, prone with hardware
failures, offsite storage costs, and constant
maintenance needs. Storage Gateway
provided the most cost-effective and simple
alternative. We even got disaster recovery by
using a bi-coastal data center.”
Amazon EFS with AWS Direct
Connect (DX):
Working over a dedicated network
Access your EFS file system via AWS Direct Connect
Direct Connect EFS in your Amazon VPCOn-premises servers
Three scenarios for working with file data across on-
premises environments and EFS
Bursting
Migration
 Move entire data set permanently to EFS
 Access the data from applications running on EC2 instances
 Move data set temporarily to EFS
 Access the data from applications running on EC2 instances
 Move data back on premises once processing finishes
Backup and Disaster
Recovery
 Maintain copy of entire data set on EFS
 Restore the data to on premises storage or (for DR) access the
data from failed-over applications running on EC2 instances
Amazon Kinesis:
streaming data
Time Value of Money Data
Scenarios Accelerated Ingest-
Transform-Load
Continual Metrics
Generation
Responsive Data
Analysis
Data Types IT logs, applications logs, social media / clickstreams, sensor or device data, market data
Ad/Marketing
Tech
Publisher, bidder data
aggregation
Advertising metrics like
coverage, yield, conversion
Analytics on user
engagement with ads,
optimized bid / buy engines
IoT
Sensor, device telemetry
data ingestion
IT operational metrics
dashboards
Sensor operational
intelligence, alerts, and
notifications
Gaming
Online customer engagement
data aggregation
Consumer engagement
metrics for level success,
transition rates, CTR
Clickstream analytics,
leaderboard generation,
player-skill match engines
Consumer
Engagement
Online customer engagement
data aggregation
Consumer engagement
metrics like page views,
CTR
Clickstream analytics,
recommendation engines
1 2 3
Streaming Data Scenarios
Kinesis Streams
Stores data as a
continuous replayable
stream for custom
applications
Kinesis Firehose
Load streaming data into
Amazon S3, Amazon
Redshift, and Amazon
Elasticsearch Service
Kinesis Analytics
Analyze data streams
using standard SQL
queries
Amazon Kinesis
Match the Services and Scenarios
1 2 3
Kinesis StreamsKinesis Firehose Kinesis Analytics
Accelerated Ingest-
Transform-Load
Continual Metrics
Generation
Responsive Data
Analysis
Kinesis Streams
Amazon S3 Transfer Acceleration:
moving large files over long distances
Amazon S3 transfer acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Change your endpoint, not your code
Leverages 59 global edge locations
Optimized protocols
No firewall exceptions
No client software required
Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los
Angeles
Seattle Tokyo Singapore
Time[hrs]
500 GB upload from these edge locations to a bucket in Singapore
Public Internet
How fast is S3 Transfer Acceleration?
S3 Transfer Acceleration
On average, we have seen 171%
improvement over regular S3 when
uploading over long distances
Use Case: Media Upload
“S3 transfer acceleration reduces the average time it takes for
us to ingest videos from our global user base by almost half.
This gives our customers the ability to edit and share videos
sooner where speed is a critical factor. - Brian Kaiser, CTO
• Growing 30% per year
• Typical Friday during football season – over 35 hours of
video every minute is uploaded
Data in: User > S3-TA > S3 > Transcode > Redshift
Data out: S3 > CloudFront > User
S3-TA = >20% increase in upload and encoding speeds
Bursting
Migration
Tiering
Offline
Summary: Hybrid Cloud Storage Services
Backup
Online
Snowball,
Snowball Edge,
Snowmobile
Storage Gateway,
EFS over DX
Storage Gateway,
S3 Transfer Acceleration
Snowball Edge
Snowball
Storage Gateway,
3rd Parties, EFS over DX
Storage Gateway
Partner Tiering & Migration Solutions:
working with what’s already there
Storage Partner Solutions
Technology Solutions vetted by the AWS Storage Competency Program
aws.amazon.com/backup-recovery/partner-solutions/ Note: Represents a sample of storage partners
Note: Dell-EMC, IBM and Veritas have solutions and are working towards competency requirements
Backup and RecoveryPrimary Storage Archive BCDR
Solutions that leverage file, block, object,
and streamed data formats as an
extension to on-premises storage
Solutions that leverage Amazon S3 for
durable data backup
Solutions that leverage Amazon Glacier
for durable and cost-effective long-term
data backup
Solutions that utilize AWS to enable
recovery strategies focused on RTO
and RPO requirements
Thank you!

Contenu connexe

Tendances

(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics
(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics
(BDT307) Zero Infrastructure, Real-Time Data Collection, and AnalyticsAmazon Web Services
 
Building and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECSBuilding and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECSAmazon Web Services
 
ENT302 Deep Dive on AWS Management Tools and New Launches
ENT302 Deep Dive on AWS Management Tools and New LaunchesENT302 Deep Dive on AWS Management Tools and New Launches
ENT302 Deep Dive on AWS Management Tools and New LaunchesAmazon Web Services
 
AWS Data Transfer Services: Data Ingest Strategies Into the AWS Cloud
AWS Data Transfer Services: Data Ingest Strategies Into the AWS CloudAWS Data Transfer Services: Data Ingest Strategies Into the AWS Cloud
AWS Data Transfer Services: Data Ingest Strategies Into the AWS CloudAmazon Web Services
 
Data Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and ArchiveData Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and ArchiveAmazon Web Services
 
Getting started with Amazon Kinesis
Getting started with Amazon KinesisGetting started with Amazon Kinesis
Getting started with Amazon KinesisAmazon Web Services
 
Big Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWSBig Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWSAmazon Web Services
 
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceAmazon Web Services
 
WKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot Instances
WKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot InstancesWKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot Instances
WKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot InstancesAmazon Web Services
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersAmazon Web Services
 
serverless_architecture_patterns_london_loft.pdf
serverless_architecture_patterns_london_loft.pdfserverless_architecture_patterns_london_loft.pdf
serverless_architecture_patterns_london_loft.pdfAmazon Web Services
 
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...Amazon Web Services
 
Build an App on AWS for Your First 10 Million Users
Build an App on AWS for Your First 10 Million UsersBuild an App on AWS for Your First 10 Million Users
Build an App on AWS for Your First 10 Million UsersAmazon Web Services
 
AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...
AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...
AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...Amazon Web Services
 
Getting Started with Amazon DynamoDB
Getting Started with Amazon DynamoDBGetting Started with Amazon DynamoDB
Getting Started with Amazon DynamoDBAmazon Web Services
 
SRV301 Getting the most Bang for your buck with #EC2 #Winning
SRV301 Getting the most Bang for your buck with #EC2 #WinningSRV301 Getting the most Bang for your buck with #EC2 #Winning
SRV301 Getting the most Bang for your buck with #EC2 #WinningAmazon Web Services
 
BDA403 How Netflix Monitors Applications in Real-time with Amazon Kinesis
BDA403 How Netflix Monitors Applications in Real-time with Amazon KinesisBDA403 How Netflix Monitors Applications in Real-time with Amazon Kinesis
BDA403 How Netflix Monitors Applications in Real-time with Amazon KinesisAmazon Web Services
 
ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...
ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...
ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...Amazon Web Services
 

Tendances (20)

(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics
(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics
(BDT307) Zero Infrastructure, Real-Time Data Collection, and Analytics
 
Building and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECSBuilding and scaling your containerized microservices on Amazon ECS
Building and scaling your containerized microservices on Amazon ECS
 
ENT302 Deep Dive on AWS Management Tools and New Launches
ENT302 Deep Dive on AWS Management Tools and New LaunchesENT302 Deep Dive on AWS Management Tools and New Launches
ENT302 Deep Dive on AWS Management Tools and New Launches
 
AWS Data Transfer Services: Data Ingest Strategies Into the AWS Cloud
AWS Data Transfer Services: Data Ingest Strategies Into the AWS CloudAWS Data Transfer Services: Data Ingest Strategies Into the AWS Cloud
AWS Data Transfer Services: Data Ingest Strategies Into the AWS Cloud
 
Data Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and ArchiveData Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and Archive
 
Getting started with Amazon Kinesis
Getting started with Amazon KinesisGetting started with Amazon Kinesis
Getting started with Amazon Kinesis
 
Big Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWSBig Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWS
 
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
 
WKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot Instances
WKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot InstancesWKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot Instances
WKS401 Deploy a Deep Learning Framework on Amazon ECS and EC2 Spot Instances
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
 
serverless_architecture_patterns_london_loft.pdf
serverless_architecture_patterns_london_loft.pdfserverless_architecture_patterns_london_loft.pdf
serverless_architecture_patterns_london_loft.pdf
 
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...
 
Build an App on AWS for Your First 10 Million Users
Build an App on AWS for Your First 10 Million UsersBuild an App on AWS for Your First 10 Million Users
Build an App on AWS for Your First 10 Million Users
 
AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...
AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...
AWS re:Invent 2016: Big Data Architectural Patterns and Best Practices on AWS...
 
Getting Started with Amazon DynamoDB
Getting Started with Amazon DynamoDBGetting Started with Amazon DynamoDB
Getting Started with Amazon DynamoDB
 
SRV301 Getting the most Bang for your buck with #EC2 #Winning
SRV301 Getting the most Bang for your buck with #EC2 #WinningSRV301 Getting the most Bang for your buck with #EC2 #Winning
SRV301 Getting the most Bang for your buck with #EC2 #Winning
 
Self-Service Supercomputing
Self-Service SupercomputingSelf-Service Supercomputing
Self-Service Supercomputing
 
Security Best Practices
Security Best PracticesSecurity Best Practices
Security Best Practices
 
BDA403 How Netflix Monitors Applications in Real-time with Amazon Kinesis
BDA403 How Netflix Monitors Applications in Real-time with Amazon KinesisBDA403 How Netflix Monitors Applications in Real-time with Amazon Kinesis
BDA403 How Netflix Monitors Applications in Real-time with Amazon Kinesis
 
ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...
ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...
ENT201 A Tale of Two Pizzas: Accelerating Software Delivery with AWS Develope...
 

Similaire à ENT306 Migrating Large Scale Data Sets to the Cloud

Moving Data into the Cloud with AWS Transfer Services - May 2017 AWS Online ...
Moving Data into the Cloud with AWS Transfer Services  - May 2017 AWS Online ...Moving Data into the Cloud with AWS Transfer Services  - May 2017 AWS Online ...
Moving Data into the Cloud with AWS Transfer Services - May 2017 AWS Online ...Amazon Web Services
 
Storage and Data Migration - AWS Innovate Toronto
Storage and Data Migration - AWS Innovate TorontoStorage and Data Migration - AWS Innovate Toronto
Storage and Data Migration - AWS Innovate TorontoAmazon Web Services
 
AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...
AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...
AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...Amazon Web Services
 
ENT306 Migrating large Scale Data Sets to the Cloud
ENT306 Migrating large Scale Data Sets to the CloudENT306 Migrating large Scale Data Sets to the Cloud
ENT306 Migrating large Scale Data Sets to the CloudAmazon Web Services
 
AWS Storage and Data Migration: AWS Innovate Ottawa
AWS Storage and Data Migration: AWS Innovate OttawaAWS Storage and Data Migration: AWS Innovate Ottawa
AWS Storage and Data Migration: AWS Innovate OttawaAmazon Web Services
 
ENT306 Migrating Large Scale Data Sets to the Cloud
ENT306 Migrating Large Scale Data Sets to the CloudENT306 Migrating Large Scale Data Sets to the Cloud
ENT306 Migrating Large Scale Data Sets to the CloudAmazon Web Services
 
Migrating_Large_Scale_Data_Sets_to_the_Cloud
Migrating_Large_Scale_Data_Sets_to_the_CloudMigrating_Large_Scale_Data_Sets_to_the_Cloud
Migrating_Large_Scale_Data_Sets_to_the_CloudAmazon Web Services
 
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017Amazon Web Services
 
Deep Dive on AWS Cloud Data Migration Services
Deep Dive on AWS Cloud Data Migration ServicesDeep Dive on AWS Cloud Data Migration Services
Deep Dive on AWS Cloud Data Migration ServicesAmazon Web Services
 
An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...
An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...
An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...Amazon Web Services
 
Simplifying migration to aws - everett dolgner
Simplifying migration to aws -   everett dolgnerSimplifying migration to aws -   everett dolgner
Simplifying migration to aws - everett dolgnerAmazon Web Services
 
Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech Talks
Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech TalksDeep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech Talks
Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech TalksAmazon Web Services
 
AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)
AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)
AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)Amazon Web Services
 
Building Hybrid Cloud Storage Architectures with AWS @scale
Building Hybrid Cloud Storage Architectures with AWS @scaleBuilding Hybrid Cloud Storage Architectures with AWS @scale
Building Hybrid Cloud Storage Architectures with AWS @scaleAmazon Web Services
 

Similaire à ENT306 Migrating Large Scale Data Sets to the Cloud (20)

Moving Data into the Cloud with AWS Transfer Services - May 2017 AWS Online ...
Moving Data into the Cloud with AWS Transfer Services  - May 2017 AWS Online ...Moving Data into the Cloud with AWS Transfer Services  - May 2017 AWS Online ...
Moving Data into the Cloud with AWS Transfer Services - May 2017 AWS Online ...
 
Storage and Data Migration - AWS Innovate Toronto
Storage and Data Migration - AWS Innovate TorontoStorage and Data Migration - AWS Innovate Toronto
Storage and Data Migration - AWS Innovate Toronto
 
AWS Storage and Data Migration
AWS Storage and Data MigrationAWS Storage and Data Migration
AWS Storage and Data Migration
 
Migrating Large Scale Datasets
Migrating Large Scale DatasetsMigrating Large Scale Datasets
Migrating Large Scale Datasets
 
AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...
AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...
AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball ...
 
ENT306 Migrating large Scale Data Sets to the Cloud
ENT306 Migrating large Scale Data Sets to the CloudENT306 Migrating large Scale Data Sets to the Cloud
ENT306 Migrating large Scale Data Sets to the Cloud
 
AWS Storage and Data Migration: AWS Innovate Ottawa
AWS Storage and Data Migration: AWS Innovate OttawaAWS Storage and Data Migration: AWS Innovate Ottawa
AWS Storage and Data Migration: AWS Innovate Ottawa
 
ENT306 Migrating Large Scale Data Sets to the Cloud
ENT306 Migrating Large Scale Data Sets to the CloudENT306 Migrating Large Scale Data Sets to the Cloud
ENT306 Migrating Large Scale Data Sets to the Cloud
 
Migrating_Large_Scale_Data_Sets_to_the_Cloud
Migrating_Large_Scale_Data_Sets_to_the_CloudMigrating_Large_Scale_Data_Sets_to_the_Cloud
Migrating_Large_Scale_Data_Sets_to_the_Cloud
 
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
 
Deep Dive on AWS Cloud Data Migration Services
Deep Dive on AWS Cloud Data Migration ServicesDeep Dive on AWS Cloud Data Migration Services
Deep Dive on AWS Cloud Data Migration Services
 
An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...
An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...
An Overview of AWS Services for Data Storage and Migration - SRV205 - Atlanta...
 
Aws storage for media overview
Aws storage for media overview Aws storage for media overview
Aws storage for media overview
 
Data Migration Best Practices
Data Migration Best PracticesData Migration Best Practices
Data Migration Best Practices
 
Data Migration Best Practice
Data Migration Best PracticeData Migration Best Practice
Data Migration Best Practice
 
Simplifying migration to aws - everett dolgner
Simplifying migration to aws -   everett dolgnerSimplifying migration to aws -   everett dolgner
Simplifying migration to aws - everett dolgner
 
Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech Talks
Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech TalksDeep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech Talks
Deep Dive on the AWS Storage Gateway - April 2017 AWS Online Tech Talks
 
AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)
AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)
AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)
 
Cloud storage
Cloud storageCloud storage
Cloud storage
 
Building Hybrid Cloud Storage Architectures with AWS @scale
Building Hybrid Cloud Storage Architectures with AWS @scaleBuilding Hybrid Cloud Storage Architectures with AWS @scale
Building Hybrid Cloud Storage Architectures with AWS @scale
 

Plus de Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

Plus de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Dernier

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 

Dernier (20)

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 

ENT306 Migrating Large Scale Data Sets to the Cloud

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Marc Trimuschat, AWS Storage Services April 19, 2017 Migrating Large Scale Data Sets to the Cloud
  • 2. Data has gravity …easier to move processing to the data 4k/8k Genomics Seismic Financial Logs IoT
  • 3. Cloud Data Migration Direct Connect Snow* data transport family 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose The AWS Storage Portfolio Object Amazon GlacierAmazon S3 Block Amazon EBS (persistent) Amazon EC2 Instance Store (ephemeral) File Amazon EFS
  • 4. The Snow Family: Large batches or “edge” scenarios
  • 5. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
  • 6. AWS Snowball Petabyte-scale data transport • Rugged 8.5G impact case • Rain and dust resistant • Data encryption end-to-end • 80 TB capacity/10G network E-ink shipping label
  • 7. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data moved to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed
  • 8. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25Gb SFP28, and 40Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable
  • 9. Collect data Create job Copy data Moved to S3 Snowball Edge: Hybrid capabilities beyond data migrationMIGRATIONCOLLECTION Create job Copy data Moved to S3
  • 10. When to use AWS Snowball Cloud Migration Disaster Recovery Data Center Decommission Content Distribution
  • 11. Customer: Scripps Networks Interactive Problem Statement: • Need storage platform to manage active archive content • Existing content repository too large to migrate via available network-based ingest methods Use of AWS: • Amazon S3 and Snowball for massively scalable ingest • Amazon S3 for storage, Amazon Glacier for content archive • Snowball to securely transport existing media content from on-premises storage and tape vault Business Benefits: • Petabyte-scale data transport without increased network costs • Massive scalability and elasticity • Lower TCO for active archive storage Active archive transport and archival for digital content provider
  • 12. AWS Storage Gateway: Augmenting on-premises storage with cloud
  • 13. Storage Gateway hybrid storage solutions Use standard storage protocols to access AWS storage services AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Files Volumes Tapes On-premises AWSCloud
  • 14. File gateway On-premises file storage maintained as objects in Amazon S3 Customer Premises File Gateway Data stored and retrieved from your S3 buckets One-to-one mapping from files-to-objects File metadata stored in object metadata Bucket access managed by IAM role you own and manage Use S3 Lifecycle Policies, versioning, or CRR to manage data Amazon Glacier S3 Standard S3 Standard - Infrequent Access HTTPS NFS v3 / v4.1 Application Server
  • 15. Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration “Storage Gateway has the promise to transform the way we move data into the cloud. The NFS interface lets us easily integrate data files from analytical instruments, and the transparent S3 storage lets us easily connect our cloud-based applications and leverage the powerful storage capabilities of S3. “With Storage Gateway, we can now unleash the full power of AWS on our instrument data.”
  • 16. Application Server Volume gateway On-premises volume storage backed by Amazon S3 with EBS snapshots Block storage in S3 accessed via the volume gateway Compression of data in-transit and at-rest Back up on-premises volumes to EBS snapshots Create on-premises volumes from EBS snapshots Up to 1 PB of total volume storage per gateway Amazon EBS snapshots Storage Gateway bucket in Amazon S3 Customer Premises Volume Gateway iSCSI HTTPS
  • 17. Tiering storage into S3, Glacier, and EBS Easily add AWS storage to your on-premises environment “Storage Gateway is at the core of our disaster recovery and business continuity (BCM) processes, handling our Co-Lo'd OLTP and OLAP off site data backups, as well as our in-office BCM. It works transparently, in a lights out way, archiving off to a separate AWS account with a simple grandfather-father- son snapshot plan in place.”
  • 18. Tape gateway Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management Virtual tape storage in S3 and Glacier accessed via tape gateway Data compressed in-transit and at-rest Up to 1 PB total tape storage per gateway, unlimited archive capacity Supports leading backup applications: Archived tapes stored in Amazon Glacier MEDIA CHANGER TAPE DRIVE Customer Premises Tape Gateway Virtual tapes stored in Amazon S3 Backup Server HTTPSiSCSI
  • 19. Backup, archive, and disaster recovery Cost-effective storage in AWS with local or cloud restore “Tapes are a headache, prone with hardware failures, offsite storage costs, and constant maintenance needs. Storage Gateway provided the most cost-effective and simple alternative. We even got disaster recovery by using a bi-coastal data center.”
  • 20. Amazon EFS with AWS Direct Connect (DX): Working over a dedicated network
  • 21. Access your EFS file system via AWS Direct Connect Direct Connect EFS in your Amazon VPCOn-premises servers
  • 22. Three scenarios for working with file data across on- premises environments and EFS Bursting Migration  Move entire data set permanently to EFS  Access the data from applications running on EC2 instances  Move data set temporarily to EFS  Access the data from applications running on EC2 instances  Move data back on premises once processing finishes Backup and Disaster Recovery  Maintain copy of entire data set on EFS  Restore the data to on premises storage or (for DR) access the data from failed-over applications running on EC2 instances
  • 24. Time Value of Money Data
  • 25. Scenarios Accelerated Ingest- Transform-Load Continual Metrics Generation Responsive Data Analysis Data Types IT logs, applications logs, social media / clickstreams, sensor or device data, market data Ad/Marketing Tech Publisher, bidder data aggregation Advertising metrics like coverage, yield, conversion Analytics on user engagement with ads, optimized bid / buy engines IoT Sensor, device telemetry data ingestion IT operational metrics dashboards Sensor operational intelligence, alerts, and notifications Gaming Online customer engagement data aggregation Consumer engagement metrics for level success, transition rates, CTR Clickstream analytics, leaderboard generation, player-skill match engines Consumer Engagement Online customer engagement data aggregation Consumer engagement metrics like page views, CTR Clickstream analytics, recommendation engines 1 2 3 Streaming Data Scenarios
  • 26. Kinesis Streams Stores data as a continuous replayable stream for custom applications Kinesis Firehose Load streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service Kinesis Analytics Analyze data streams using standard SQL queries Amazon Kinesis
  • 27. Match the Services and Scenarios 1 2 3 Kinesis StreamsKinesis Firehose Kinesis Analytics Accelerated Ingest- Transform-Load Continual Metrics Generation Responsive Data Analysis
  • 29. Amazon S3 Transfer Acceleration: moving large files over long distances
  • 30. Amazon S3 transfer acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Change your endpoint, not your code Leverages 59 global edge locations Optimized protocols No firewall exceptions No client software required
  • 31. Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public Internet How fast is S3 Transfer Acceleration? S3 Transfer Acceleration On average, we have seen 171% improvement over regular S3 when uploading over long distances
  • 32. Use Case: Media Upload “S3 transfer acceleration reduces the average time it takes for us to ingest videos from our global user base by almost half. This gives our customers the ability to edit and share videos sooner where speed is a critical factor. - Brian Kaiser, CTO • Growing 30% per year • Typical Friday during football season – over 35 hours of video every minute is uploaded Data in: User > S3-TA > S3 > Transcode > Redshift Data out: S3 > CloudFront > User S3-TA = >20% increase in upload and encoding speeds
  • 33. Bursting Migration Tiering Offline Summary: Hybrid Cloud Storage Services Backup Online Snowball, Snowball Edge, Snowmobile Storage Gateway, EFS over DX Storage Gateway, S3 Transfer Acceleration Snowball Edge Snowball Storage Gateway, 3rd Parties, EFS over DX Storage Gateway
  • 34. Partner Tiering & Migration Solutions: working with what’s already there
  • 35. Storage Partner Solutions Technology Solutions vetted by the AWS Storage Competency Program aws.amazon.com/backup-recovery/partner-solutions/ Note: Represents a sample of storage partners Note: Dell-EMC, IBM and Veritas have solutions and are working towards competency requirements Backup and RecoveryPrimary Storage Archive BCDR Solutions that leverage file, block, object, and streamed data formats as an extension to on-premises storage Solutions that leverage Amazon S3 for durable data backup Solutions that leverage Amazon Glacier for durable and cost-effective long-term data backup Solutions that utilize AWS to enable recovery strategies focused on RTO and RPO requirements