Pitt Immersion Day Module 4 - storage in AWS
•Download as PPTX, PDF•
0 likes•112 views
The document provides an overview of storage options on AWS, including block storage (EBS), shared file systems (EFS), object storage (S3), and ways to integrate on-premises storage. It discusses the characteristics, use cases, and features of each storage type. The document also covers hybrid storage solutions like Storage Gateway and Snowball that can help migrate data to and from AWS.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Pitt Immersion Day Module 4 - storage in AWS
Similar to Pitt Immersion Day Module 4 - storage in AWS (20)
More from EagleDream Technologies
More from EagleDream Technologies (10)
Recently uploaded
Recently uploaded (20)
Pitt Immersion Day Module 4 - storage in AWS
- 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.1
- 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.2 Dustin Potter, dustin.potter@eagledream.com Storage on AWS
- 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.3 Agenda • Introduction • Storage Primer • Block Storage • Shared File Systems • Object Store • On-Premises Storage Integration
- 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.4 Introduction: Why choose AWS for storage Compelling Economics Easy to Use Reduce risk Speed, Agility, Scale Pay as you go No risky capacity planning No need to provision for redundancy or overhead Self-service administration SDKs for simple integration No Commitment Durable and Secure Avoid risks of physical media handling Reduce time to market Focus on your business, not your infrastructure
- 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.5 0 Storage Primer
- 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.6 Block vs File vs Object Block Storage Raw Storage Data organized as an array of unrelated blocks Host File System places data on disk e.g.: Microsoft NTFS, Unix ZFS File Storage Unrelated data blocks managed by a file (serving) system Native file system places data on disk Object Storage Stores Virtual containers that encapsulate the data, data attributes, metadata and Object IDs API Access to data Metadata Driven, Policy-based, etc.
- 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.7 Storage - Characteristics Durability Availability Security Cost Scalability Performance Integration Measure of expected data loss Measure of expected downtime Security measures for at-rest and in- transit data Amount per storage unit, e.g. $ / GB Upward flexibility, storage size, number of users Performance metrics (bandwidth Ability to interact via API or with other services Some of the ways we look at storage
- 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.8 designed for 99.99% durability Two copies on one site designed for 99.999% durability Copies on two sites designed for 99.999999999% durability GlacierStandard IA copies in three AZ Understanding Durability
- 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.9 © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AVAILABILITY VS. DURABILITY % Availability Durability 99.999 5 minutes 15 seconds 1 in 100,000 99.9999 31 seconds 1 in 1,000,000 99.99999 3 seconds 1 in 10,000,000 99.999999999 300 uSeconds 1 in 100,000,000,000
- 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.10 AWS has a variety of storage options
- 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.11 1 Block Storage
- 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.12 What is Amazon EBS? • Block storage as a service • Create, attach volumes through an API • Service accessed over the network EC2 instance EBS volume
- 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.13 What is Amazon EBS? • Block storage as a service • Create, attach volumes through an API • Service accessed over the network EC2 instance != EBS volume
- 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.14 AWS EBS Features Durable Secure Low-latency SSD Consistent I/O Performance Stripe multiple volumes for higher I/O performance Identity and Access Policies Encryption Scalable Capacity when you need it Easily scale up and down Performance Backup Designed for 99.999 reliability Redundant storage across multiple devices within an AZ Point-in-time Snapshots Copy snapshots across AZ and Regions
- 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.15 Amazon EBS Network attached block device • Independent data lifecycle • Multiple volumes per EC2 instance • Only one EC2 instance at a time per volume • Can be detached from an instance and attached to a different one Raw block devices • Unformatted block devices • Ideal for databases, filesystems Multiple Drive Types • SSD (iops) and Magnetic (throughput)
- 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.16 Amazon EBS volume types HDDSSD Provisioned IOPS SSD io1 General Purpose SSD gp2 Throughput Optimized HDD st1 sc1 Cold HDD
- 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.17 Amazon EBS use cases HDDSSD Relational Databases MySQL, SQL Server, PostgreSQL, SAP, Oracle NoSQL Databases Cassandra, MongoDB, CouchDB Big Data , Analytics Kafka, Splunk, Hadoop, Data Warehousing File / Media CIFS/NFS, Transcoding, Encoding, Rendering
- 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.21 EBS Volume Types Comparison Magnetic General Purpose (SSD) Provisioned IOPS (SSD) Performance Lowest Cost Burstable Predictable Use Cases Infrequent Data Access Boot volumes Small to Medium DBs Dev & Test I/O Intensive Relational & NoSQL Media Magnetic (HDD) SSD SSD Max IOPS 100 on average with the ability to burst to hundreds of IOPS Baseline 3 IOPS/GB Burstable to 16,000 IOPS Consistently performed at provisioned level, up to 64,000 IOPS Price $.045/GB (st1) $.025/GB (sc1) $.10/GB/Month (gp2) $.125/GB/Month (io1) $.065/provisioned IOPS
- 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.22 Internet AWS Cloud EBS Snapshots EC2 Availability Zone EC2 Amazon S3 EBS EC2 EC2 EBS EBS EBS EBS EBS EBS Snapshot EBS Snapshot EBS Snapshot Create Snapshot Clone From Snapshot
- 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.23 EBS Volume How Do Snapshots Work? Time Snapshot 1 Snapshot 2 Snapshot 3 S3 Block 1 Block 2 Block 3 Block 4 Chunk 1 Chunk 2 Chunk 3 Chunk 4
- 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.24 What is Amazon EC2 instance store? • Local to instance • Non-persistent data store • Data not replicated (by default) • No snapshot support • SSD or HDD • >80,000 iops • >1,750 MB/S EC2 instances Physical Host Instance Store or
- 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.25 2 Shared file system
- 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.26 Elastic File System (EFS) • Fully managed file system for EC2 instances • Provides standard file system semantics • Works with standard operating system APIs • Sharable across thousands of instances • Elastically grows to petabyte scale • Delivers performance for a wide variety of workloads • Highly available and durable • NFS v4–based • Accessible from on-premise servers
- 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.27 Amazon EFS is Simple Fully managed - No hardware, network, file layer - Create a scalable file system in seconds! Seamless integration with existing tools and apps - NFS v4.1—widespread, open - Standard file system access semantics - Works with standard OS file system APIs Simple pricing = simple forecasting 1
- 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.28 Amazon EFS is Elastic • File systems grow and shrink automatically as you add and remove files • No need to provision storage capacity or performance • You pay only for the storage space you use, with no minimum fee 2
- 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.29 • File systems can grow to petabyte scale • Throughput and IOPS scale automatically as file systems grow • Consistent low latencies regardless of file system size • Support for thousands of concurrent NFS connections Amazon EFS is Scalable3
- 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.30 • Designed to sustain AZ offline conditions • Resources aggregated across multiple AZ’s • Superior to traditional NAS availability models • Appropriate for Production / Tier 0 applications Highly Durable and Highly Available
- 28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.31 Example use cases Big Data Analytics Media Workflow Processing Web Serving Content Management Home Directories
- 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.32 EFS – Mounting EFS EC2EC2 EC2 EC2EC2 EFS DNS Name availability-zone.file-system-id.efs.aws-region.amazonaws.com Mount on machine sudo mount -t nfs4 mount-target-DNS:/ ~/efs-mount-point EC2
- 30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.33 FSx for Windows Integrated with AWS Fully managed Windows file systems … … built on Windows Server
- 31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.34 Native Windows compatibility and features Native Windows compatibility DFS Namespaces and DFS Replication Integrates with Microsoft AD and supports Windows ACLs AD NTFS Windows Server Native SMB 2.0 to 3.1.1 SMB
- 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.36 3 Object Stores
- 33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.37 Amazon S3 (Simple Storage Service) • Web accessible object store (through API or HTTPS) • Highly durable (99.999999999% design) • Limitlessly scalable • Multiple Tiers to match your workload • Data Lifecycle Rules • Static Website Hosting
- 34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.38 Amazon Simple Storage Service (S3) Designed for 99.999999999% durability Parallel I/O for Max Speed Replication options across regions On-demand analytics Built-in support for SQL expressions with S3 Select Detailed data on usage patterns and access Move Data via API, HTTPS, SDK Multiple Encryption Options Automated cost reduction tools Collect AnalyzeStore
- 35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.39 Object storage classes GlacierStandard Infrequent Access 1 Zone - IA Pricing is per GB per month in the US East (N. Virginia) region Intelligent Active data Millisecond access Min 3 AZs $0.023 Archive data Minutes to Hours $0.001 - $0.004 Min 3 AZs 30 day min duration Millisecond access Min 3 AZs $0.0125 30 day min duration Millisecond access Min 1 AZ $0.01 ML to optimize Storage costs Min 3 AZs *scan cost
- 36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.40 Amazon Glacier Designed for 99.999999999% durability Replication options across regions Query-in-place analytics Expedited and bulk retrievals Deep Archive Regulatory compliance certifications Vault Lock Locking, encryption, audit and alerting tools ArchiveSecure Cost-effective
- 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.41 Object Storage Use Cases S3-IA File Sync & Share Active Archive Enterprise Backup Media Transcoding Disaster Recovery / Geo Redundancy Glacier Deep / Offline Archives Tape Vaulting Replacement WORM Compliant Data S3 Big Data Analytics Content Distribution Primary Data Temporary & Small Objects Cloud Applications Secondary Backups Easily Re-Creatable Data S3 Cross-Region Replication Target Standard IA 1Zone-IA Glacier
- 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.43 S3 Storage Management Features S3 Object Tagging manage and control access for Amazon S3 objects. S3 Analytics, Storage Class Analysis Analyze storage access patterns and transition the right data to the right storage class. S3 Inventory Simplify and speed up business workflows and big data jobs S3 CloudWatch Metrics Understand and improve the performance of your applications that use S3
- 39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.44 Amazon CloudFront Easy-to-use Content Delivery Network (CDN) Pay-as-you-go pricing Multiple origins: S3, EC2, on-premise Worldwide network of 125+ edge locations and growing • Video streaming • Geo Restriction • Custom SSL Certificates • Dynamic Content • Supports POST/PUT
- 40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.45 4 On-Premises Storage Integration
- 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.46 Storage Gateway hybrid storage solutions Enables using standard storage protocols to access AWS storage services Amazon EBS snapshots Amazon S3 Amazon Glacier Files Volumes Tapes
- 42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.47 Storage Gateway – Files, volumes, and tapes File gateway NFS (v3 and v4.1) interface On-premises file storage backed by Amazon S3 objects Tape gateway iSCSI virtual tape library interface Virtual tape storage in Amazon S3 and Glacier with VTL management Volume gateway iSCSI block interface On-premises block storage backed by S3 with EBS snapshots
- 43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.48 Storage Gateway – Common capabilities Standard storage protocols integrate with on-premises applications Local caching for low-latency access to frequently used data Efficient data transfer with buffering and bandwidth management Native data storage in AWS Stateless virtual appliance for resiliency Integrated with AWS management and security
- 44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.52 Hybrid storage use cases with Storage Gateway Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration Tiered cloud storage Easily add AWS storage to your on-premises environment Backup, archive, and disaster recovery Cost effective storage in AWS with local or cloud restore
- 45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.53 Amazon Snowball & Snowball Edge • Terabyte scale data transport • Uses secure appliances • Faster than Internet for significant data sets • Import into S3 • HIPAA Compliant
- 46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.54 What is Snowball? Terabyte scale data transport E-ink shipping label Ruggedized case “8.5G Impact” All data encrypted end-to-end 80 TB 10G network Rain & dust resistant Tamper-resistant case & electronics
- 47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.55 How fast is Snowball? • Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs, less than 1 week including shipping • Number of days to transfer 250TB via the Internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 95 190 316 632 50% 47 95 158 316 75% 32 63 105 211
- 48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.56 Amazon Snowmobile https://www.youtube.com/watch?v=8vQmTZTq7nw
- 49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.57 Any Questions?
- 50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.58
- 51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.59
Editor's Notes
- Overview of session
- Each storage option has a unique combination of performance, durability, cost, and interface
- Each storage option has a unique combination of performance, durability, cost, and interface
- Each storage option has a unique combination of performance, durability, cost, and interface. AWS SNOWBALL is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of the AWS cloud. Using Snowball addresses common challenges with large-scale data transfers including high network costs, long transfer times, and security concerns. Transferring data with Snowball is simple, fast, secure, and can be as little as one-fifth the cost of high-speed Internet. AWS SNOWMOBILE is NEW and its a secure, Exabyte-scale data transfer service used to transfer large amounts of data into and out of AWS. Each Snowmobile can transfer up to 100PB. When you order a Snowmobile it comes to your site and AWS personnel connect a removable, high-speed network switch from Snowmobile to your local network. This makes Snowmobile appear as a network attached data store. Once it is connected, secure, high-speed data transfer begins. After your data is transferred to Snowmobile, it is driven back to AWS where the data is loaded into the AWS service you select, including S3, Glacier, Redshift and others. It allows customers with large amounts of data to migrate to AWS much faster and easier.
- Amazon Web Services give you reliable, durable backup storage without the up-front capital expenditures and complex capacity-planning burden of on-premises storage. Amazon storage services remove the need for complex and time-consuming capacity planning, ongoing negotiations with multiple hardware and software vendors, specialized training, and maintenance of offsite facilities or transportation of storage media to third party offsite locations. EBS has 99.99% SLA See https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-an-increased-monthly-service-commitment-for-amazon-ec2/ & https://aws.amazon.com/compute/sla/
- Network device Data lifecycle is independent from EC2 instance lifecycle Each volume is like a hard drive on a physical server Attach multiple volumes to an EC2 instance, but only one EC2 instance per volume POSIX-compliant file systems Virtual disk ideal for: OS boot device; file systems Raw block devices Ideal for Databases (Oracle Active Storage Manager) Other raw block devices
- Describe EBS standard volumes as “best effort” and PIOPs as providing consistent performance. Mention how the most predictable performance will come by using EBS-Optimized instances to obtain dedicated storage throughput. Chart on the left describes the expected throughput and max expected 16K IOPs for various instance sizes.
- Describe how EBS snapshots work.
- Describe how snapshots are differential, but allow a full recovery from any individual snapshot based on this three snapshot scenario.
- Free with your EC2 Instance SAS and SSD options Size/type based on instance type Zero Network Overhead; local, direct attached resource. Consistent performance for sequential reads and writes Volatile Currently (09/22/2015) in Preview mode
- Describe how EFS works
- What we decided to deliver were Windows file systems – built on fully managed Windows Servers And integrated richly with other AWS services
- Built on Windows Server, you get native Windows file storage, delivering the compatibility, features, and performance needed to run Windows applications. Provides NTFS file systems that can be accessed from up to thousands of compute instances using the SMB protocol. Fully supports key NTFS file system features, including what other solutions commonly don’t support, like extended attributes, file change notifications, alternate data streams, hard links, soft links Fully supports SMB 3.1.1 (backwards compatible with all SMB 2.0+ clients) SMB performance features like SMB multichannel (a single client can establish multiple TCP connections to drive higher throughput) and SMB directory leasing (a client can own the cache for a particular directory) Amazon FSx works with Microsoft Active Directory (AD) to integrate your file system with your Windows environments, including trust relationships with on-prem ADs It supports the use of Distributed File System (DFS) Replication to enable multi-AZ deployments, and DFS Namespaces to enable namespaces that can span across many file servers
- Built on high-performance SSD storage Provides high throughout and IOPS Up to 2 GB/s of throughput and hundreds of thousands of IOPS per file system Up to tens of GB/s of throughput and hundreds of thousands of IOPS using DFS Namespaces across file systems Choose throughput independent of storage size – allows you to really cater the price/performance ratio to your application’s needs Highly consistent, sub-millisecond latencies
- 11 Nines of durability: 99.999999999%
- A chief information security officer (CISO) is the senior-level executive within an organization responsible for establishing and maintaining the enterprise vision, strategy, and program to ensure information assets and technologies are adequately protected. On-demand analytics: RedShift Spectrum, QuickSight, Athena
- Query in place: One of the most important capabilities of a data lake that is built on AWS is the ability to do in-place transformation and querying of data assets without having to provision and manage clusters. AWS Glue, as described in the previous sections, provides the data discovery and ETL capabilities, and Amazon Athena and Amazon Redshift Spectrum provide the in-place querying capabilities. Glacier Select Amazon Glacier Select allows queries to run directly on data stored in Amazon Glacier without having to retreive the entire archive. Amazon Glacier Select changes the value of archive storage by allowing you to process and find only the bytes you need out of the archive to use for analytics.
- Highlight customer architecture and how durability, avail, performance, and scalability relate to application type Amazon Glacier provides three ways to retrieve your archives to meet varying access time and cost requirements: Expedited, Standard, and Bulk retrievals. Archives requested using Expedited retrievals are typically available within 1 – 5 minutes, allowing you to quickly access your data when occasional urgent requests for a subset of archives are required. With Standard retrievals, archives typically become accessible within 3 – 5 hours. Or you can use Bulk retrievals to cost-effectively access significant portions of your data, even petabytes, for just a quarter-of-a-cent per GB.
- Object Tags are key-value pairs applied to S3 objects which can be created, updated or deleted at any time during the lifetime of the object. provide the ability to create Identity and Access Management (IAM) policies, setup S3 Lifecycle policies, and customize storage metrics. manage transitions between storage classes and expire objects in the background. automatically identifies the optimal lifecycle policy to transition less frequently accessed storage to SIA. configure a storage class analysis policy to monitor an entire bucket, a prefix, or object tag. Once an infrequent access pattern is observed, easily create a new lifecycle age policy based on the results. provides daily visualizations of your storage usage in the AWS Management Console that can be exported to an S3 bucket to analyze using the business intelligence tools of your choice, such as Amazon QuickSight. provides a CSV (Co monitoring and alarming on 13 new S3 CloudWatch Metrics receive 1-minute CloudWatch Metrics, set CloudWatch alarms, and access CloudWatch dashboards to view real-time operations and performance such as bytes downloaded and the 4xx HTTP response count of your Amazon S3 storage. For web and mobile applications that depend on cloud storage, these let you quickly identify and act on operational issues. mma Separated Values) flat-file output of your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix.
- The AWS SGW is typically deployed in your existing storage environment as a VM. You connect your existing applications, storage systems, or devices to the SGW. The SGW provides standard storage protocol interfaces so apps can connect to it without changes. The gateway in turn connects to AWS so you can store data securely and durably in Amazon S3, Glacier. The gateway optimizes data transfer from on-premises to AWS. It also provides low-latency access through a local cache so your apps can access frequently used data locally. The service is also integrated with Cloudwatch, cloudtrail, IAM, etc. so you get an extension of aws management services locally. --- “Enable cloud storage on-premises as part of your AWS platform” “Native access Industry standard protocols for file, block, and tape Secure and durable storage in Amazon S3 and Glacier Optimized data transfer from on-premises to AWS Low-latency access to frequently used data Integrated with AWS security and management services
- The tape gateway presents the Storage Gateway to your existing backup application as an industry-standard iSCSI-based virtual tape library (VTL), consisting of a virtual media changer and virtual tape drives. You can continue to use your existing backup applications and workflows while writing to a nearly limitless collection of virtual tapes. Each virtual tape is stored in Amazon S3. When you no longer require immediate or frequent access to data contained on a virtual tape, you can have your backup application archive it from the virtual tape library into Amazon Glacier, further reducing storage costs. Storage Gateway is currently compatible with most leading backup applications. The VTL interface eliminates large upfront tape automation capital expenses, multi-year maintenance contract commitments and ongoing media costs. You pay only for the capacity you use and scale as your needs grow. The need to transport storage media to offsite facilities and handle tape media manually goes away, and your archives benefit from the design and durability of the AWS cloud platform. In your VTL you can configure up to 1,500 tapes … up to 2.5 TB each (LTO-6 size) … for a total of 1PB TB per VTL
- The file gateway enables you to store and retrieve objects in Amazon S3 using industry-standard file protocols. Files are stored as objects in your S3 buckets, accessed through a Network File System (NFS) mount point. Ownership, permissions, and timestamps are durably stored in S3 in the user-metadata of the object associated with the file. Once objects are transferred to S3, they can be managed as native S3 objects, and bucket policies such as versioning, lifecycle management, and cross-region replication apply directly to objects stored in your bucket. Customers use the file interface to migrate file data into S3 for use by object-based workloads, as a cost-effective storage target for traditional backup applications, and as a tier in the cloud for on-premises file storage.
- The volume gateway presents your applications with disk volumes using the iSCSI block protocol. Data written to these volumes can be asynchronously backed up as point-in-time snapshots of your volumes, and stored in the cloud as Amazon EBS snapshots. You can set the schedule for when snapshots occur or create them via the AWS Management Console or service API. Snapshots are incremental backups that capture only changed blocks. All snapshot storage is also compressed to minimize your storage charges. When connecting with the block interface, you can run the gateway in two modes: cached and stored. In cached mode, you store your primary data in Amazon S3 and retain your frequently accessed data locally. With this mode, you can achieve substantial cost savings on primary storage, minimizing the need to scale your storage on-premises, while retaining low-latency access to your frequently accessed data. You can configure up to 32 volumes of 32 TB, for a total 1PB storage per gateway. In stored mode, you store your entire data set locally, while performing asynchronous backups of this data in Amazon S3. This mode provides durable and inexpensive offsite backups that you can recover locally or from Amazon EC2. Example applications such as databases or computational workloads … where low latency is critical and the working set of data is large, ill-defined, or constantly changing. On a stored volume gateway you can configure up to 32 volumes … up to 16 TB each … for a total of 512 TB per gateway
- We see 3 broad categories of hybrid storage where SGW helps customers. Let’s look in a little more detail at each of these.
- The priginal Snowball has 50 TB & 80TB capacity; AWS Snowball Edge, like the original Snowball, is a terabyte-scale data transfer solution, but transports more data up to 100TB of data, and retains the same embedded cryptography and security as the original Snowball. However, Snowball Edge hosts a file server and an S3-compatible endpoint that allow you to use the NFS protocol, S3 SDK or S3 CLI to transfer data directly to the device without specialized client software. Multiple units may be clustered together, forming a temporary data collection storage tier in your datacenter so you can work as data is generated without managing copies. As storage needs scale up and down, devices can be easily added or removed from the local cluster and returned to AWS.
- What is AWS Import/Export Snowball? Snowball is a new AWS Import/Export offering that provides a terabyte-scale data transfer service that uses Amazon-provided storage devices for transport. Previously customers purchased their own portable storage devices and used these devices to ship their data. With the launch of Snowball customers are now able to use highly secure, rugged Amazon-owned Network Attached Storage (NAS) devices, called Snowballs, to ship their data. Once received and set up, customers are able to copy up to 50TB data from their on prem file system to the Snowball via the Snowball client software via a 10Gbps network interface . Prior to transfer to the Snowball all data is encrypted by 256-bit GSM encryption by the client. When customers finish transferring data to the device they simply ship it back to an AWS facility where the data is ingested at high speed into Amazon S3.
- Compare and contrast Internet vs 5x Snowball. Note that with 80TB Snowball or 100TB Snowball Edge, less devices needed Snowball Edge supports up to 40GB connections (QSFP+) Also note you can also use Snowball Edge to run edge computing workloads such as performing local analysis of data on a Snowball Edge cluster and writing it to the S3-compatible endpoint
- AWS Snowmobile is a secure, Exabyte-scale data transfer service used to transfer large amounts of data into and out of AWS. Each Snowmobile can transfer up to 100PB. When you order a Snowmobile it comes to your site and AWS personnel connect a removable, high-speed network switch from Snowmobile to your local network. This makes Snowmobile appear as a network attached data store. Once it is connected, secure, high-speed data transfer begins. After your data is transferred to Snowmobile, it is driven back to AWS where the data is loaded into the AWS service you select, including S3, Glacier, Redshift and others.