Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Everett Dolgner
6 September 2017
AWS Data Transfer Services
Cloud Data Migration
Direct
Connect
Snow* data
transport
family
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway...
The Snow Family:
large batches or “edge” scenarios
AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale
data migration
Compute & Storage for
Hybrid/Edge workload...
AWS Snowball
Petabyte-scale data transport
• Rugged 8.5G impact case
• Rain and dust resistant
• Data encryption end-to-en...
How Snowball moves data into and out of AWS
Create
a job
Connect the
Snowball
Copy data to
the Snowball
Your data
moved to...
AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equ...
Snowball Edge key features
S3-compatible endpoint
File interface (NFS)
Clustering
Run AWS Lambda functions
Faster data tra...
Collect data Create job Copy data Moved to S3
Snowball Edge: Hybrid capabilities beyond data migrationMIGRATIONCOLLECTION
...
Snowmobile Case Study: DigitalGlobe
Use-case: Seeing a better world
DigitalGlobe takes Satellite imagery of the Earth
100P...
Snowmobile From Space
Picture taken by DigitalGlobe’s WorldView-3 satellite
http://blog.digitalglobe.com/industry/digitalg...
When to use AWS Snow*
Cloud
Migration
Disaster
Recovery
Datacenter
Decommission
Content
Distribution
S3 Transfer Acceleration:
moving large files over long distances
Amazon S3 transfer acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Change your endpoint, not your ...
Getting Started
Enable S3 Transfer Acceleration for their bucket
DNS resolution
• <bucket>.s3-accelerate.amazonaws.com res...
Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los
Angeles
Seattle Tokyo Singapore
Time[hrs]
500 G...
Use Case: Media Upload
”
“S3 transfer acceleration reduces the average time it takes for
us to ingest videos from our glob...
Storage Gateway:
augmenting existing on-premises storage with AWS
Storage Gateway hybrid storage solutions
Use standard storage protocols to access AWS storage services
AWS Storage
Gateway...
File gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises
Data stored and retrieved from ...
File Gateway Use cases
S3 Bucket
Storage
Gateway VM
NFS client
RefreshCache
Amazon
EMR
In-cloud workload
S3 Bucket
Amazon
...
Enabling cloud workloads
Move data to AWS storage for Big Data, cloud bursting, or migration
“Storage Gateway has the prom...
Application
Server
Volume gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
Block storage in S3 ac...
Tiering storage into S3 and EBS
Easily add AWS storage to your on-premises environment
“Storage Gateway is at the core of ...
Durable offsite storage
Easily store important data in a durable remote site
“As our business expanded, it became clear th...
Data center SMB server with SGW backend
SMB hosted onsite, blocks stored durably in Amazon S3
Private Datacenter
Storage
G...
High Availability Windows Server Storage
Local HA with remote DR
us-west-1
Data Center
Windows
Clients Storage
Gateway VMM...
Tape gateway
Virtual tape storage in Amazon S3 and Glacier with VTL management
Virtual tape storage in S3 and Glacier acce...
Backup, archive, and disaster recovery
Cost effective storage in AWS with local or cloud restore
“Tapes are a headache, pr...
GlacierS3 Standard S3-
Infrequent
Access
File Gateway
PetroBank
Application
ServersLTO
NAS
Active archive migration from L...
EFS with DirectConnect (DX):
Working over a dedicated network
Access your EFS file system via AWS Direct Connect
Direct Connect EFS in your Amazon VPCOn-premises Servers
Partner Tiering & Migration Solutions:
working with what’s already there
Storage Partner Solutions
Technology Solutions vetted by the AWS Storage Competency Program
aws.amazon.com/backup-recovery...
Bursting
Migration
Tiering
Offline
Summary: Hybrid Cloud Storage Services
Backup
Online
Snowball,
Snowball Edge,
Snowmobil...
Questions
Prochain SlideShare
Chargement dans…5
×

AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball Edge, and Data Transfer Services

1 667 vues

Publié le

AWS offers a suite of tools to help you surmount limitations associated to data migration from on premise to the cloud. Attend this session to learn about moving data by using networks, roads, and AWS technology partners. We will also discuss how to move data into and out of the Cloud in batches, increments, and streams.

  • Soyez le premier à commenter

AWS Data Transfer Services - AWS Storage Gateway, AWS Snowball, AWS Snowball Edge, and Data Transfer Services

  1. 1. Everett Dolgner 6 September 2017 AWS Data Transfer Services
  2. 2. Cloud Data Migration Direct Connect Snow* data transport family 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose The AWS Storage Portfolio Object Amazon GlacierAmazon S3 Block Amazon EBS (persistent) Amazon EC2 Instance Store (ephemeral) File Amazon EFS
  3. 3. The Snow Family: large batches or “edge” scenarios
  4. 4. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
  5. 5. AWS Snowball Petabyte-scale data transport • Rugged 8.5G impact case • Rain and dust resistant • Data encryption end-to-end • 80 TB capacity/10G network E-ink shipping label
  6. 6. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data moved to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed
  7. 7. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25Gb SFP28, and 40Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable
  8. 8. Snowball Edge key features S3-compatible endpoint File interface (NFS) Clustering Run AWS Lambda functions Faster data transfer Encryption
  9. 9. Collect data Create job Copy data Moved to S3 Snowball Edge: Hybrid capabilities beyond data migrationMIGRATIONCOLLECTION Create job Copy data Moved to S3
  10. 10. Snowmobile Case Study: DigitalGlobe Use-case: Seeing a better world DigitalGlobe takes Satellite imagery of the Earth 100PB image library = 6 billion square kilometers 1PB new image every year Architecture before Snowmobile: Stored data in their own data center Needed elastic compute power to retrieve and analyze images Wanted to move data to the cloud, but no feasible solution Snowmobile lets DigitalGlobe migrate 100PB of data to the cloud
  11. 11. Snowmobile From Space Picture taken by DigitalGlobe’s WorldView-3 satellite http://blog.digitalglobe.com/industry/digitalglobe-moves-to-the-cloud-with-aws-snowmobile/
  12. 12. When to use AWS Snow* Cloud Migration Disaster Recovery Datacenter Decommission Content Distribution
  13. 13. S3 Transfer Acceleration: moving large files over long distances
  14. 14. Amazon S3 transfer acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Change your endpoint, not your code Leverages 59 global edge locations Optimized protocols No firewall exceptions No client software required
  15. 15. Getting Started Enable S3 Transfer Acceleration for their bucket DNS resolution • <bucket>.s3-accelerate.amazonaws.com resolves to the nearest POP location for the client • uses the client IP and CloudFront latency based routing Customers update application / destination URI to: <bucket-name>.s3-accelerate.amazonaws.com File is uploaded to: <bucket-name> S3 Management Console, AWS SDK, AWS CLI
  16. 16. Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public Internet How fast is S3 Transfer Acceleration? S3 Transfer Acceleration On average, we have seen 171% improvement over regular S3 when uploading over long distances
  17. 17. Use Case: Media Upload ” “S3 transfer acceleration reduces the average time it takes for us to ingest videos from our global user base by almost half. This gives our customers the ability to edit and share videos sooner where speed is a critical factor. - Brian Kaiser, CTO Friday during football season – over 35 hours of video every minute is uploaded Data in: User > S3-TA > S3 > Transcode > Redshift Data out: S3 > CloudFront > User S3-TA = >20% increase in upload and encoding speeds
  18. 18. Storage Gateway: augmenting existing on-premises storage with AWS
  19. 19. Storage Gateway hybrid storage solutions Use standard storage protocols to access AWS storage services AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Files Volumes Tapes On-premises AWSCloud
  20. 20. File gateway On-premises file storage maintained as objects in Amazon S3 Customer Premises Data stored and retrieved from your S3 buckets One-to-one mapping from files-to-objects File metadata stored in object metadata Bucket access managed by IAM role you own and manage Use S3 Lifecycle Policies, versioning, or CRR to manage data GlacierS3 Standard S3 Standard - Infrequent Access HTTPS NFS v3 / v4.1 Application Server Storage Gateway Storage Gateway VM
  21. 21. File Gateway Use cases S3 Bucket Storage Gateway VM NFS client RefreshCache Amazon EMR In-cloud workload S3 Bucket Amazon Athena Amazon QuickSight Snowball + Gateway NFS client Storage Gateway VM RefreshCache Storage Gateway VM S3 BucketRead-only NFS client Cross region replication File Sharing Site A Site B NFS client
  22. 22. Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration “Storage Gateway has the promise to transform the way we move data into the cloud. The NFS interface lets us easily integrate data files from analytical instruments, and the transparent S3 storage lets us easily connect our cloud-based applications and leverage the powerful storage capabilities of S3. With Storage Gateway, we can now unleash the full power of AWS on our instrument data.”
  23. 23. Application Server Volume gateway On-premises volume storage backed by Amazon S3 with EBS snapshots Block storage in S3 accessed via the volume gateway Compression of data in-transit and at-rest Backup on-premises volumes to EBS snapshots Create on-premises volumes from EBS snapshots Up to 1PB of total volume storage per gateway Amazon EBS snapshots Customer Premises Storage Gateway Service iSCSI HTTPS Volume Gateway VM Volume Stored in Amazon S3
  24. 24. Tiering storage into S3 and EBS Easily add AWS storage to your on-premises environment “Storage Gateway is at the core of our disaster recovery and business continuity (BCM) processes, handling our Co-Lo'd OLTP and OLAP off site data backups, as well as our in-office BCM. It works transparently, in a lights out way, archiving off to a separate AWS account with a simple grandfather-father- son snapshot plan in place”.
  25. 25. Durable offsite storage Easily store important data in a durable remote site “As our business expanded, it became clear that using on-premises backup processes to store and recover these records was massively inefficient, not scalable, and very costly. Using the AWS Cached Volume Storage Gateway, we can retrieve these records very quickly.”
  26. 26. Data center SMB server with SGW backend SMB hosted onsite, blocks stored durably in Amazon S3 Private Datacenter Storage Gateway VM us-west-2 Storage Gateway Volume Windows Clients Windows Server HTTPSiSCSISMB
  27. 27. High Availability Windows Server Storage Local HA with remote DR us-west-1 Data Center Windows Clients Storage Gateway VMMicrosoft DFS HTTPSiSCSI HTTPSiSCSI SMB Storage Gateway Service us-east-1 Microsoft DFS HTTPSiSCSI Storage Gateway Service us-east-1 us-west-1
  28. 28. Tape gateway Virtual tape storage in Amazon S3 and Glacier with VTL management Virtual tape storage in S3 and Glacier accessed via tape gateway Data compressed in-transit and at-rest Up to 1 PB total tape storage per gateway, unlimited archive capacity Supports leading backup applications **3-5 hour tape retrieval from Glacier Archived Tapes stored in Amazon Glacier MEDIA CHANGER TAPE DRIVE Customer Premises Virtual Tapes stored in Amazon S3 Backup Server HTTPSiSCSI Tape Gateway Tape Gateway VM
  29. 29. Backup, archive, and disaster recovery Cost effective storage in AWS with local or cloud restore “Tapes are a headache, prone with hardware failures, offsite storage costs, and constant maintenance needs. Storage Gateway provided the most cost-effective and simple alternative. We even got disaster recovery by using a bi-coastal data center”.
  30. 30. GlacierS3 Standard S3- Infrequent Access File Gateway PetroBank Application ServersLTO NAS Active archive migration from LTO Cost effective storage in AWS with local data access AWS Direct Connect Self service loading of data Reduced time-to-data by days or weeks Storage archive costs reduced by 90% Halliburton Data Center AWS Lambda
  31. 31. EFS with DirectConnect (DX): Working over a dedicated network
  32. 32. Access your EFS file system via AWS Direct Connect Direct Connect EFS in your Amazon VPCOn-premises Servers
  33. 33. Partner Tiering & Migration Solutions: working with what’s already there
  34. 34. Storage Partner Solutions Technology Solutions vetted by the AWS Storage Competency Program aws.amazon.com/backup-recovery/partner-solutions/ Note: Represents a sample of storage partners Note: Dell-EMC, IBM and Veritas have solutions and are working towards competency requirements Backup and RecoveryPrimary Storage Archive BCDR Solutions that leverage file, block, object, and streamed data formats as an extension to on-premises storage Solutions that leverage Amazon S3 for durable data backup Solutions that leverage Amazon Glacier for durable and cost-effective long-term data backup Solutions that utilize AWS to enable recovery strategies focused on RTO and RPO requirements
  35. 35. Bursting Migration Tiering Offline Summary: Hybrid Cloud Storage Services Backup Online Snowball, Snowball Edge, Snowmobile Storage Gateway, EFS over DX Storage Gateway, S3 Transfer Acceleration Snowball Edge Snowball Storage Gateway, 3rd Parties, EFS over DX Storage Gateway
  36. 36. Questions

×