Goes into details about the prior knowledge required to design a file system. Concludes with a demo that explains how these concepts are used by Second Extended File System (ext2fs) A demo is also provided. Presented at Florence, Italy on 6th December 2017

1. How to design a
FileSystem
A Beginner’s Tale
V N Nikhil Anurag

2. Who are you?

3. Who are you?
● Curious about File Systems
● Interested in low level systems
● Building an OS from scratch is hard!
● Not sure how to approach such a task
● C/C++ is not your forte

4. Who are you?
What we really want
● Feasible
● Practical
● Yet involves complexity of low level system design

5. Why not design a
File System (FS)?

6. Who am I?

7. Who am I?
● Software Engineer (REST & API)
● Tools of trade: Go, Python & Clojure
● Github: https://github.com/last-ent
● Twitter: @last_ent
● Site: www.last-ent.com
● Learns by Doing

8. Who am I?

9. Beginner’s Tale

10. File System & Operating System

11. File System & Operating System
Hard Disk

12. File System & Operating System
OS
● Where does the FS sit?
● How does the OS know
about it?
● How is OS stored?
Hard Disk

13. File System & Operating System
OS
File System
Hard Disk

14. File System & Operating System
FS
Operations
FS
Formatted
Data
File System Process
OS
(in-memory)

15. Creating
Or
Initialization
Mounting
Or
Opening
Unmounting
Or
Closing
Reading Writing Deleting
File System ✔ ✔ ✔
Files ✔ ✔ ✔ ✔ ✔ ✔
Directories ✔ ✔ ✔ ✔ ✔ ✔
File System & Operating System
Operations
Type
File System Operations

16. How is a
File System
mounted?

17. How can an
Operating System
load
Hard Disk
as a
File System?

18. Investigating the Hard Disk

19. Investigating the Hard Disk
/dev/sda - Hard Disk
/dev/sdaX - Partitions on the Hard Disk

20. Unix Philosophy:
Everything is a File

21. Investigating the Hard Disk: less -f /dev/sda1

22. Investigating the Hard Disk: less -f /dev/sda

23. Investigating the Hard Disk
Conclusion:
1. Linux/Unix-like OSes view all hard disks as files.
2. A Hard Disk can be opened with standard “Open
File” commands!
3. A Hard Disk contains a stream of bytes.

24. How to
parse
the bytes
on the
hard disk?

25. File System Constructs

26. File System Constructs
Block
The smallest unit of bytes available to the OS & FS.
It is a multiple of disk’s block size.
Block Size = n * 512 Bytes
Block Group
Boot Block
Contiguous set of blocks that form a logical group.
A Block Group consists of FS metadata & file data.
Used by the computer to bootstrap kernel loading process.
(Irrelevant for our purposes)

27. File System Constructs
Boot Block Block Group 0 Block Group N
Formatted Hard Disk or Partition

28. File System Constructs: Block Group
Superblock Consists of all the metadata required by
the OS to load the File System
Inode
Consists of all the metadata required by File System
to load a File or a Directory.
Metadata Bitmap
Provides a minimap for the state of Blocks or other things.

29. File System Constructs: Block Group
Boot Block Block Group 0 Block Group N
Superblock
Inode
map
Inodes Bitmap
Data Bitmap
Data Blocks
Inodes

30. How does the
Operating System
identify
a
File System?

31. How does the OS identify a FS?
● A Hard Disk or Partition is just a stream of bytes.
● The OS can’t store the partition’s FS type anywhere
else.

32. How does the OS identify a FS?
File System Magic Signature
● Unique for each FS
● Stored at a unique place for each FS

33. File System Constructs: Block Group
Boot Block Block Group 0 Block Group N
Superblock
Inode
map
Inodes Bitmap
Data Bitmap
Data Blocks
Inodes
$FS

34. We mounted the
File System.
What’s next?

35. Look at
all the
Files & Directories
in the
File System

36. Let’s start with Root Directory
“/”

37. Root Directory
● Where is the Root Directory?
● How does the OS know about it.
● How is it represented?

38. Root Directory
Inode
Consists of all the metadata required by File System
to load a File or a Directory.
Inode
map
Look Up table to quickly reach Inodes
based on their Inode#
Root Directory is at a predefined position
on the File System.

39. Root Directory
● Root Directory is at a predefined position on the
File System.
● It can be looked up in the Inode map.
● An Inode can be interpreted differently based on
whether it is a Directory or a File.

40. Root Directory
Inode map
Location of Root Directory
Root
Directory
Inode

41. Directory Inode

42. Directory Inode
Directory
Inode
Expected
● Directory Name
● Directory Contents
Actual
● Directory Name
● Directory Entries
○ Inode
○ Name
○ Type

43. Root Directory’s
name is
“/”
So we are safe!

44. Block0
Root Directory: Data Blocks
/
Inode: _
Block1
BlockN

45. Block0
Root Directory: Dentries in Block0
/
Inode: _
dir1
Inode:
#inode1
dirN
Inode:
#inodeN
file1
Inode:
#inodeF1
fileN
Inode:
#inodeFN

46. File Inode

47. File Inode
File Inode
Expected
● File Name
● File Size
● File Contents
Actual
● File Name
● File Size
● File’s Data Blocks

48. Creating a File
&
Directory Look Up

49. Creating a File & Directory Look Up
Create
Directory: /dir1/subdir1
FileName: file.txt
/
dir0
dir1 subdir1
File Inode
Name: file.txt
Create Inode
for file

50. Creating a File
● Find Directory Inode
● Create File Inode
○ Write all relevant metadata
Don’t add File Content yet!

51. Case Study:
ext2 File System

52. File System Constructs
Block
The smallest unit of bytes available to the OS & FS.
It is a multiple of disk’s block size.
Block Size = n * 512 Bytes
Block Group
Boot Block
Contiguous set of blocks that form a logical group.
A Block Group consists of FS metadata & file data.
Used by the computer to bootstrap kernel loading process.
(Irrelevant for our purposes)

53. File System Constructs: Block Group
Superblock Consists of all the metadata required by
the OS to load the File System
Inode
Consists of all the metadata required by File System
to load a File or a Directory.
Metadata Bitmap
Provides a minimap for the state of Blocks or other things.

54. ext2 File System: Layout
Boot Block Block Group 0 Block Group N
Superblock
Inode
map
Inodes Bitmap
Data Bitmap
Data Blocks
Inodes
$FS
Group Descriptor

55. ext2 File System: Layout
● Magic Signature: 0xEF53
● Root Folder Inode# 1
● Number of Inodes: Size in Bytes ➗ Block Size

56. Demo

57. Demo Instructions
1. dd if=/dev/zero of=demo.ex2 bs=1024 count=131072
2. mke2fs ./demo.ex2
3. sudo mount ./demo.ext2 /demo-mnt
4. go run github.com/last-ent/ext2-explorer/main.go
5. Enter file path in browser:
/home/entux/Documents/Code/demo/demo.ext2
6. View Ext2 Data
7. sudo umount /demo-mnt

58. How are File
System operations
connected to OS?

59. Operating System & FS Operations
OS
(in-memory)
ext4
Open
Close
Dentries
Inode
... btrfs
Virtual File System - Used in Kernel Space
FUSE - Used in User Space

60. Summary
● File System is just a program.
● It is well structured and there is no magic.
● Easy to implement a practical low level System.

61. References
Understanding the Linux Kernel, Third Edition 3rd Edition
Practical File System Design
Demo Project - https://github.com/last-ent/ext2-explorer