Submit Search
Upload
Designing Large Scale File Storage and Serving Systems
•
Download as PPS, PDF
•
3 likes
•
654 views
AI-enhanced title
R
royans
Follow
Cal handerson's talk. Awesome http://www.iamcal.com/talks/
Read less
Read more
Technology
Report
Share
Report
Share
1 of 101
Download now
Recommended
Flickr Services
Flickr Services
royans
Flickr Php
Flickr Php
royans
Web20expo Scalable Web Arch
Web20expo Scalable Web Arch
guest18a0f1
Scalable Web Architectures - Common Patterns & Approaches
Scalable Web Architectures - Common Patterns & Approaches
Cal Henderson
Web20expo Scalable Web Arch
Web20expo Scalable Web Arch
royans
Filesystems
Filesystems
royans
Web20expo Filesystems
Web20expo Filesystems
royans
Web20expo Filesystems
Web20expo Filesystems
royans
Recommended
Flickr Services
Flickr Services
royans
Flickr Php
Flickr Php
royans
Web20expo Scalable Web Arch
Web20expo Scalable Web Arch
guest18a0f1
Scalable Web Architectures - Common Patterns & Approaches
Scalable Web Architectures - Common Patterns & Approaches
Cal Henderson
Web20expo Scalable Web Arch
Web20expo Scalable Web Arch
royans
Filesystems
Filesystems
royans
Web20expo Filesystems
Web20expo Filesystems
royans
Web20expo Filesystems
Web20expo Filesystems
royans
Optimizing the performance of WordPress
Optimizing the performance of WordPress
Josh Highland Giese
Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
adunne
Scalable and Available, Patterns for Success
Scalable and Available, Patterns for Success
Derek Collison
Scaling the Platform for Your Startup
Scaling the Platform for Your Startup
Amazon Web Services
Configuring Apache Servers for Better Web Perormance
Configuring Apache Servers for Better Web Perormance
Spark::red
Introduction to Amazon Elastic File System (EFS)
Introduction to Amazon Elastic File System (EFS)
Amazon Web Services
Deep Dive on Amazon Elastic File System (Amazon EFS)
Deep Dive on Amazon Elastic File System (Amazon EFS)
Amazon Web Services
Lamp Stack Optimization
Lamp Stack Optimization
Dave Ross
Use case for using the ElastiCache for Redis in production
Use case for using the ElastiCache for Redis in production
知教 本間
AWS re:Invent 2016: DNS Demystified: Getting Started with Amazon Route 53, fe...
AWS re:Invent 2016: DNS Demystified: Getting Started with Amazon Route 53, fe...
Amazon Web Services
Effectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEM
Norberto Leite
AWS re:Invent 2016: Deep Dive on Amazon Elastic Block Store (STG301)
AWS re:Invent 2016: Deep Dive on Amazon Elastic Block Store (STG301)
Amazon Web Services
Drupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp North
Philip Norton
Amazon Elastic File System (EFS): New Elastic File Storage Service That Makes...
Amazon Elastic File System (EFS): New Elastic File Storage Service That Makes...
Amazon Web Services
Indroduction to Web Application
Indroduction to Web Application
torny10
2015 ZendCon - Do you queue
2015 ZendCon - Do you queue
Mike Willbanks
Cloud storage with AWS
Cloud storage with AWS
Ahmad karawash
214
214
Mikhail Miguel
Steve Goes to Nashville
Steve Goes to Nashville
livintrue
How Typepad changed their architecture without taking down the service
How Typepad changed their architecture without taking down the service
royans
Hamilton International brochure
Hamilton International brochure
AmandaCorrigan_1979
Fine Art Night
Fine Art Night
sbailey1
More Related Content
What's hot
Optimizing the performance of WordPress
Optimizing the performance of WordPress
Josh Highland Giese
Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
adunne
Scalable and Available, Patterns for Success
Scalable and Available, Patterns for Success
Derek Collison
Scaling the Platform for Your Startup
Scaling the Platform for Your Startup
Amazon Web Services
Configuring Apache Servers for Better Web Perormance
Configuring Apache Servers for Better Web Perormance
Spark::red
Introduction to Amazon Elastic File System (EFS)
Introduction to Amazon Elastic File System (EFS)
Amazon Web Services
Deep Dive on Amazon Elastic File System (Amazon EFS)
Deep Dive on Amazon Elastic File System (Amazon EFS)
Amazon Web Services
Lamp Stack Optimization
Lamp Stack Optimization
Dave Ross
Use case for using the ElastiCache for Redis in production
Use case for using the ElastiCache for Redis in production
知教 本間
AWS re:Invent 2016: DNS Demystified: Getting Started with Amazon Route 53, fe...
AWS re:Invent 2016: DNS Demystified: Getting Started with Amazon Route 53, fe...
Amazon Web Services
Effectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEM
Norberto Leite
AWS re:Invent 2016: Deep Dive on Amazon Elastic Block Store (STG301)
AWS re:Invent 2016: Deep Dive on Amazon Elastic Block Store (STG301)
Amazon Web Services
Drupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp North
Philip Norton
Amazon Elastic File System (EFS): New Elastic File Storage Service That Makes...
Amazon Elastic File System (EFS): New Elastic File Storage Service That Makes...
Amazon Web Services
Indroduction to Web Application
Indroduction to Web Application
torny10
2015 ZendCon - Do you queue
2015 ZendCon - Do you queue
Mike Willbanks
Cloud storage with AWS
Cloud storage with AWS
Ahmad karawash
What's hot
(17)
Optimizing the performance of WordPress
Optimizing the performance of WordPress
Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
Scalable and Available, Patterns for Success
Scalable and Available, Patterns for Success
Scaling the Platform for Your Startup
Scaling the Platform for Your Startup
Configuring Apache Servers for Better Web Perormance
Configuring Apache Servers for Better Web Perormance
Introduction to Amazon Elastic File System (EFS)
Introduction to Amazon Elastic File System (EFS)
Deep Dive on Amazon Elastic File System (Amazon EFS)
Deep Dive on Amazon Elastic File System (Amazon EFS)
Lamp Stack Optimization
Lamp Stack Optimization
Use case for using the ElastiCache for Redis in production
Use case for using the ElastiCache for Redis in production
AWS re:Invent 2016: DNS Demystified: Getting Started with Amazon Route 53, fe...
AWS re:Invent 2016: DNS Demystified: Getting Started with Amazon Route 53, fe...
Effectively Deploying MongoDB on AEM
Effectively Deploying MongoDB on AEM
AWS re:Invent 2016: Deep Dive on Amazon Elastic Block Store (STG301)
AWS re:Invent 2016: Deep Dive on Amazon Elastic Block Store (STG301)
Drupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp North
Amazon Elastic File System (EFS): New Elastic File Storage Service That Makes...
Amazon Elastic File System (EFS): New Elastic File Storage Service That Makes...
Indroduction to Web Application
Indroduction to Web Application
2015 ZendCon - Do you queue
2015 ZendCon - Do you queue
Cloud storage with AWS
Cloud storage with AWS
Viewers also liked
214
214
Mikhail Miguel
Steve Goes to Nashville
Steve Goes to Nashville
livintrue
How Typepad changed their architecture without taking down the service
How Typepad changed their architecture without taking down the service
royans
Hamilton International brochure
Hamilton International brochure
AmandaCorrigan_1979
Fine Art Night
Fine Art Night
sbailey1
DervalResearch Food Drink London Mrs20100714
DervalResearch Food Drink London Mrs20100714
Diana Derval
Viewers also liked
(6)
214
214
Steve Goes to Nashville
Steve Goes to Nashville
How Typepad changed their architecture without taking down the service
How Typepad changed their architecture without taking down the service
Hamilton International brochure
Hamilton International brochure
Fine Art Night
Fine Art Night
DervalResearch Food Drink London Mrs20100714
DervalResearch Food Drink London Mrs20100714
Similar to Designing Large Scale File Storage and Serving Systems
Beyond the File System - Designing Large Scale File Storage and Serving
Beyond the File System - Designing Large Scale File Storage and Serving
mclee
Superior Streaming and CDN Solutions: Cloud Storage Revolutionizes Digital Media
Superior Streaming and CDN Solutions: Cloud Storage Revolutionizes Digital Media
Scality
Scalable Web Arch
Scalable Web Arch
royans
Cloud computing UNIT 2.1 presentation in
Cloud computing UNIT 2.1 presentation in
RahulBhole12
Web20expo Scalable Web Arch
Web20expo Scalable Web Arch
mclee
FOWA Scaling The Lamp Stack Workshop
FOWA Scaling The Lamp Stack Workshop
dlieberman
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Cal Henderson
AWS re:Invent 2016: Deep Dive on Amazon Elastic File System (STG202)
AWS re:Invent 2016: Deep Dive on Amazon Elastic File System (STG202)
Amazon Web Services
Dipping Your Toes: Azure Data Lake for DBAs
Dipping Your Toes: Azure Data Lake for DBAs
Bob Pusateri
Deep Dive on Elastic File System - February 2017 AWS Online Tech Talks
Deep Dive on Elastic File System - February 2017 AWS Online Tech Talks
Amazon Web Services
It's the End of Data Storage As We Know It (And I Feel Fine)
It's the End of Data Storage As We Know It (And I Feel Fine)
Stephen Foskett
Real world capacity
Real world capacity
Edward Capriolo
Web Performance & Scalability Tools
Web Performance & Scalability Tools
Folio3 Software
Distributed file systems
Distributed file systems
Sri Prasanna
Storage and Archiving Options on AWS
Storage and Archiving Options on AWS
Amazon Web Services
AWS Webinar 201 - Backup, Archive and Disaster Recovery
AWS Webinar 201 - Backup, Archive and Disaster Recovery
Amazon Web Services
Distributed file systems (from Google)
Distributed file systems (from Google)
Sri Prasanna
Knowledge share about scalable application architecture
Knowledge share about scalable application architecture
AHM Pervej Kabir
Scalable Web Architecture
Scalable Web Architecture
Aleksandr Tsertkov
Lec3 Dfs
Lec3 Dfs
mobius.cn
Similar to Designing Large Scale File Storage and Serving Systems
(20)
Beyond the File System - Designing Large Scale File Storage and Serving
Beyond the File System - Designing Large Scale File Storage and Serving
Superior Streaming and CDN Solutions: Cloud Storage Revolutionizes Digital Media
Superior Streaming and CDN Solutions: Cloud Storage Revolutionizes Digital Media
Scalable Web Arch
Scalable Web Arch
Cloud computing UNIT 2.1 presentation in
Cloud computing UNIT 2.1 presentation in
Web20expo Scalable Web Arch
Web20expo Scalable Web Arch
FOWA Scaling The Lamp Stack Workshop
FOWA Scaling The Lamp Stack Workshop
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
AWS re:Invent 2016: Deep Dive on Amazon Elastic File System (STG202)
AWS re:Invent 2016: Deep Dive on Amazon Elastic File System (STG202)
Dipping Your Toes: Azure Data Lake for DBAs
Dipping Your Toes: Azure Data Lake for DBAs
Deep Dive on Elastic File System - February 2017 AWS Online Tech Talks
Deep Dive on Elastic File System - February 2017 AWS Online Tech Talks
It's the End of Data Storage As We Know It (And I Feel Fine)
It's the End of Data Storage As We Know It (And I Feel Fine)
Real world capacity
Real world capacity
Web Performance & Scalability Tools
Web Performance & Scalability Tools
Distributed file systems
Distributed file systems
Storage and Archiving Options on AWS
Storage and Archiving Options on AWS
AWS Webinar 201 - Backup, Archive and Disaster Recovery
AWS Webinar 201 - Backup, Archive and Disaster Recovery
Distributed file systems (from Google)
Distributed file systems (from Google)
Knowledge share about scalable application architecture
Knowledge share about scalable application architecture
Scalable Web Architecture
Scalable Web Architecture
Lec3 Dfs
Lec3 Dfs
More from royans
Hadoop: Distributed data processing
Hadoop: Distributed data processing
royans
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
royans
Grid – Distributed Computing at Scale
Grid – Distributed Computing at Scale
royans
Dmk Bo2 K7 Web
Dmk Bo2 K7 Web
royans
21 Www Web Services
21 Www Web Services
royans
Web Design World Flickr
Web Design World Flickr
royans
Flickr Services
Flickr Services
royans
Web 2.0 Summit Flickr
Web 2.0 Summit Flickr
royans
Etech2005
Etech2005
royans
More from royans
(9)
Hadoop: Distributed data processing
Hadoop: Distributed data processing
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
Grid – Distributed Computing at Scale
Grid – Distributed Computing at Scale
Dmk Bo2 K7 Web
Dmk Bo2 K7 Web
21 Www Web Services
21 Www Web Services
Web Design World Flickr
Web Design World Flickr
Flickr Services
Flickr Services
Web 2.0 Summit Flickr
Web 2.0 Summit Flickr
Etech2005
Etech2005
Recently uploaded
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
wesley chun
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
Pixlogix Infotech
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
The Digital Insurer
How to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
naman860154
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Delhi Call girls
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Delhi Call girls
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
HampshireHUG
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Anna Loughnan Colquhoun
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
sudhanshuwaghmare1
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
Delhi Call girls
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Igalia
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
hans926745
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
Recently uploaded
(20)
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
How to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
Designing Large Scale File Storage and Serving Systems
1.
Beyond the File
System Designing Large Scale File Storage and Serving Cal Henderson
2.
Hello!
3.
4.
5.
6.
7.
Four buckets Storage
Serving BCP Cost
8.
Storage
9.
The storage stack
File system Block protocol RAID Hardware ext, reiserFS, NTFS SCSI, SATA, FC Mirrors, Stripes Disks and stuff File protocol NFS, CIFS, SMB
10.
11.
12.
DAS Direct attached
storage Disk shelf, connected by SCSI/SATA HP MSA30 – 14 disks in 3U
13.
14.
15.
16.
17.
18.
19.
20.
NetApp Filers Heads
and shelves, up to 500TB in 6 Cabs FC SAN with 1 or 2 NAS heads
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
Real Life Case
Studies
35.
36.
37.
GFS – Google
File System 1(a) 2(a) 1(b) Master
38.
39.
40.
41.
42.
MogileFS – OMG
Files Tracker Tracker MySQL
43.
44.
45.
46.
Flickr File System
SM SM SM
47.
48.
49.
50.
Amazon S3 Servers
Amazon
51.
Amazon S3 Servers
Amazon Users
52.
53.
The cost S3
54.
The cost S3
Regular Bandwidth
55.
56.
Serving
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
Edge networks Origin
68.
Edge networks Origin
Cache Cache Cache Cache Cache Cache Cache Cache
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
BCP
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
Cost
90.
91.
Hardware costs Cost
of hardware Usable GB Single Cost
92.
Power costs Cost
of power per year Usable GB Recurring Cost
93.
Power costs Power
installation cost Usable GB Single Cost
94.
Space costs Cost
per U Usable GB [ ] U’s needed (inc network) x Recurring Cost
95.
Network costs Cost
of network gear Usable GB Single Cost
96.
Misc costs Support
contracts + spare disks Usable GB + bus adaptors + cables [ ] Single & Recurring Costs
97.
Human costs Admin
cost per node Node count x Recurring Cost Usable GB [ ]
98.
99.
(fin)
100.
101.
Download now