SlideShare une entreprise Scribd logo
1  sur  29
Distributed Computing Seminar Lecture 3: Distributed Filesystems Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet Google, Inc.  Summer 2007 Except as otherwise noted, the content of this presentation is © Copyright University of Washington and licensed under the Creative Commons Attribution 2.5 License.
Outline ,[object Object],[object Object],[object Object]
Filesystems Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Distributed Filesystems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
NFS  ,[object Object],[object Object],[object Object]
NFS Protocol ,[object Object],[object Object],[object Object],[object Object]
Server-side Implementation ,[object Object],[object Object],[object Object],[object Object],[object Object]
NFS Locking ,[object Object],[object Object],[object Object],[object Object],[object Object]
NFS Client Caching ,[object Object],[object Object],[object Object],[object Object],[object Object]
NFS: Tradeoffs ,[object Object],[object Object],[object Object],[object Object]
The Google File System Sanjay Ghemawat, Howard Gobioff,  and Shun-Tak Leung SOSP 2003 (These slides by Alex Moshchuk, University of Washington – used with permission)
Motivation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Assumptions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS Design Decisions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS Architecture ,[object Object],[object Object],… Can anyone see a potential weakness in this design?
Single master ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Metadata (1/2) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Metadata (2/2) ,[object Object],[object Object],[object Object],[object Object]
Mutations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Atomic record append ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Relaxed consistency model (1/2) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Relaxed consistency model (2/2) ,[object Object],[object Object],[object Object],[object Object]
Master’s responsibilities (1/2) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Master’s responsibilities (2/2) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Fault Tolerance ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Performance
Deployment in Google ,[object Object],[object Object],[object Object],[object Object]
Conclusion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Next Time... ,[object Object],[object Object]

Contenu connexe

Tendances

Dfs (Distributed computing)
Dfs (Distributed computing)Dfs (Distributed computing)
Dfs (Distributed computing)Sri Prasanna
 
Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File SystemMilad Sobhkhiz
 
Presentation on nfs,afs,vfs
Presentation on nfs,afs,vfsPresentation on nfs,afs,vfs
Presentation on nfs,afs,vfsPrakriti Dubey
 
Distributed File Systems
Distributed File SystemsDistributed File Systems
Distributed File Systemsawesomesos
 
File service architecture and network file system
File service architecture and network file systemFile service architecture and network file system
File service architecture and network file systemSukhman Kaur
 
Distributed File System
Distributed File SystemDistributed File System
Distributed File SystemNtu
 
Introduction to distributed file systems
Introduction to distributed file systemsIntroduction to distributed file systems
Introduction to distributed file systemsViet-Trung TRAN
 
Distribution File System DFS Technologies
Distribution File System DFS TechnologiesDistribution File System DFS Technologies
Distribution File System DFS TechnologiesRaphael Ejike
 
Distributed file system
Distributed file systemDistributed file system
Distributed file systemNaza hamed Jan
 
file sharing semantics by Umar Danjuma Maiwada
file sharing semantics by Umar Danjuma Maiwada file sharing semantics by Umar Danjuma Maiwada
file sharing semantics by Umar Danjuma Maiwada umardanjumamaiwada
 
SUN Network File system - Design, Implementation and Experience
SUN Network File system - Design, Implementation and Experience SUN Network File system - Design, Implementation and Experience
SUN Network File system - Design, Implementation and Experience aniadkar
 
Unit 3.1 cs6601 Distributed File System
Unit 3.1 cs6601 Distributed File SystemUnit 3.1 cs6601 Distributed File System
Unit 3.1 cs6601 Distributed File SystemNandakumar P
 
Distributed File Systems
Distributed File Systems Distributed File Systems
Distributed File Systems Maurvi04
 

Tendances (20)

AFS introduction
AFS introductionAFS introduction
AFS introduction
 
Kosmos Filesystem
Kosmos FilesystemKosmos Filesystem
Kosmos Filesystem
 
Dfs (Distributed computing)
Dfs (Distributed computing)Dfs (Distributed computing)
Dfs (Distributed computing)
 
Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File System
 
Distributed file systems dfs
Distributed file systems   dfsDistributed file systems   dfs
Distributed file systems dfs
 
Presentation on nfs,afs,vfs
Presentation on nfs,afs,vfsPresentation on nfs,afs,vfs
Presentation on nfs,afs,vfs
 
AFS case study
AFS case studyAFS case study
AFS case study
 
Distributed File Systems
Distributed File SystemsDistributed File Systems
Distributed File Systems
 
File service architecture and network file system
File service architecture and network file systemFile service architecture and network file system
File service architecture and network file system
 
Distributed File System
Distributed File SystemDistributed File System
Distributed File System
 
Introduction to distributed file systems
Introduction to distributed file systemsIntroduction to distributed file systems
Introduction to distributed file systems
 
Distribution File System DFS Technologies
Distribution File System DFS TechnologiesDistribution File System DFS Technologies
Distribution File System DFS Technologies
 
12. dfs
12. dfs12. dfs
12. dfs
 
Distributed file system
Distributed file systemDistributed file system
Distributed file system
 
file sharing semantics by Umar Danjuma Maiwada
file sharing semantics by Umar Danjuma Maiwada file sharing semantics by Umar Danjuma Maiwada
file sharing semantics by Umar Danjuma Maiwada
 
SUN Network File system - Design, Implementation and Experience
SUN Network File system - Design, Implementation and Experience SUN Network File system - Design, Implementation and Experience
SUN Network File system - Design, Implementation and Experience
 
Unit 3.1 cs6601 Distributed File System
Unit 3.1 cs6601 Distributed File SystemUnit 3.1 cs6601 Distributed File System
Unit 3.1 cs6601 Distributed File System
 
Gfs final
Gfs finalGfs final
Gfs final
 
Coda file system
Coda file systemCoda file system
Coda file system
 
Distributed File Systems
Distributed File Systems Distributed File Systems
Distributed File Systems
 

En vedette

Parallel Programming Primer 1
Parallel Programming Primer 1Parallel Programming Primer 1
Parallel Programming Primer 1mobius.cn
 
Lec4 Clustering
Lec4 ClusteringLec4 Clustering
Lec4 Clusteringmobius.cn
 
Lecture 1 (distributed systems)
Lecture 1 (distributed systems)Lecture 1 (distributed systems)
Lecture 1 (distributed systems)Fazli Amin
 
Lec5 Pagerank
Lec5 PagerankLec5 Pagerank
Lec5 Pagerankmobius.cn
 

En vedette (6)

Parallel Programming Primer 1
Parallel Programming Primer 1Parallel Programming Primer 1
Parallel Programming Primer 1
 
Lec1 Intro
Lec1 IntroLec1 Intro
Lec1 Intro
 
Lec2 Mapred
Lec2 MapredLec2 Mapred
Lec2 Mapred
 
Lec4 Clustering
Lec4 ClusteringLec4 Clustering
Lec4 Clustering
 
Lecture 1 (distributed systems)
Lecture 1 (distributed systems)Lecture 1 (distributed systems)
Lecture 1 (distributed systems)
 
Lec5 Pagerank
Lec5 PagerankLec5 Pagerank
Lec5 Pagerank
 

Similaire à Lec3 Dfs

Distributed file systems (from Google)
Distributed file systems (from Google)Distributed file systems (from Google)
Distributed file systems (from Google)Sri Prasanna
 
CS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMSCS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMSKathirvel Ayyaswamy
 
GOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMGOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMJYoTHiSH o.s
 
Advance google file system
Advance google file systemAdvance google file system
Advance google file systemLalit Rastogi
 
Hadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data AnalyticsHadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data AnalyticsDrPDShebaKeziaMalarc
 
Chapter 17 - Distributed File Systems
Chapter 17 - Distributed File SystemsChapter 17 - Distributed File Systems
Chapter 17 - Distributed File SystemsWayne Jones Jnr
 
Ch16 OS
Ch16 OSCh16 OS
Ch16 OSC.U
 
storage-systems.pptx
storage-systems.pptxstorage-systems.pptx
storage-systems.pptxShimoFcis
 
GFS - Google File System
GFS - Google File SystemGFS - Google File System
GFS - Google File Systemtutchiio
 
PARALLEL FILE SYSTEM FOR LINUX CLUSTERS
PARALLEL FILE SYSTEM FOR LINUX CLUSTERSPARALLEL FILE SYSTEM FOR LINUX CLUSTERS
PARALLEL FILE SYSTEM FOR LINUX CLUSTERSRaheemUnnisa1
 
Rhel cluster gfs_improveperformance
Rhel cluster gfs_improveperformanceRhel cluster gfs_improveperformance
Rhel cluster gfs_improveperformancesprdd
 
Big data interview questions and answers
Big data interview questions and answersBig data interview questions and answers
Big data interview questions and answersKalyan Hadoop
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFSzihad164
 

Similaire à Lec3 Dfs (20)

Distributed file systems (from Google)
Distributed file systems (from Google)Distributed file systems (from Google)
Distributed file systems (from Google)
 
Google
GoogleGoogle
Google
 
CS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMSCS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMS
 
Gfs介绍
Gfs介绍Gfs介绍
Gfs介绍
 
GOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMGOOGLE FILE SYSTEM
GOOGLE FILE SYSTEM
 
Distributed File System.ppt
Distributed File System.pptDistributed File System.ppt
Distributed File System.ppt
 
Advance google file system
Advance google file systemAdvance google file system
Advance google file system
 
Hadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data AnalyticsHadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data Analytics
 
Hadoop
HadoopHadoop
Hadoop
 
Chapter 17 - Distributed File Systems
Chapter 17 - Distributed File SystemsChapter 17 - Distributed File Systems
Chapter 17 - Distributed File Systems
 
OSCh16
OSCh16OSCh16
OSCh16
 
Ch16 OS
Ch16 OSCh16 OS
Ch16 OS
 
OS_Ch16
OS_Ch16OS_Ch16
OS_Ch16
 
storage-systems.pptx
storage-systems.pptxstorage-systems.pptx
storage-systems.pptx
 
GFS - Google File System
GFS - Google File SystemGFS - Google File System
GFS - Google File System
 
XFS.ppt
XFS.pptXFS.ppt
XFS.ppt
 
PARALLEL FILE SYSTEM FOR LINUX CLUSTERS
PARALLEL FILE SYSTEM FOR LINUX CLUSTERSPARALLEL FILE SYSTEM FOR LINUX CLUSTERS
PARALLEL FILE SYSTEM FOR LINUX CLUSTERS
 
Rhel cluster gfs_improveperformance
Rhel cluster gfs_improveperformanceRhel cluster gfs_improveperformance
Rhel cluster gfs_improveperformance
 
Big data interview questions and answers
Big data interview questions and answersBig data interview questions and answers
Big data interview questions and answers
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFS
 

Lec3 Dfs

  • 1. Distributed Computing Seminar Lecture 3: Distributed Filesystems Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet Google, Inc. Summer 2007 Except as otherwise noted, the content of this presentation is © Copyright University of Washington and licensed under the Creative Commons Attribution 2.5 License.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11. The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 (These slides by Alex Moshchuk, University of Washington – used with permission)
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 27.
  • 28.
  • 29.