This document discusses file systems and distributed file systems. It describes how file systems work, including hardware, partitions, logical volume management (LVM), and basic and distributed file systems. It focuses on GlusterFS and NFS distributed file systems. GlusterFS allows various volume types including distributed, replicated, distributed-replicated and stripe. NFS provides network access but no redundancy. The document also discusses storage solutions for AI training workloads, including Pure Storage FlashBlade and AIRI systems optimized for high-performance needs of AI.

1. Introductionto
filesystem

2. Outline
What is File System
What’s GlusterFS and NFS
How aurora works

3. FileSystem
User-Application
File System
Kernel Block I/O
Device(HDD/
SDD)
Device(HDD
/SDD)
Kernel Driver
Hardware
Kernel (OS)
User-Space
Partition Partition Partition Partition

4. Hardware
HDD/SDD
We won’t use the whole disk
You need to plan how to use the disk
○ How many partition
○ Size of each partition
/dev/sda (100G)
○ /dev/sda1 –(50G)
○ /dev/sda2 –(25G)
○ /dev/sda3 – (25G)

6. FileSystem
Basic FS
○ EXT4/ZFS/NFS/NTFS/BTRFS
Logical FS
○ LVM
Distributed FS
○ GlusterFS/Ceph/OpenEBS/BeeGFS/Lizard
FS
Only focus on GlusterFS and NFS
today

7. Write/READPath
User-Application
File System
Kernel Block I/O
Device(HDD/
SDD)
Device(HDD
/SDD)
Kernel Driver
Partition Partition Partition Partition
Write
Read

8. LVM
It’s impossible to predict the usage of
each partition.
You need to re-partition the whole
disk sometimes to fit the usage of
your user.
We can use the LVM
○ Logical Volume management

9. LVM

10. RAID
RAID (software RAID)
○ MDADM
○ Block Level
○ Also integrate with LVM to support more
flexible management
File System implement
○ BTRFS
○ ZFS

11. RAID

13. NFS
Network File System
○ No Any RAID Support
Read/Write via Network
NFS
Client
NFS Server
NFS
Client
NFS
Client
NFS
Client
Switch

14. Theproblemoffilesystem
I/OPS (Input / Output per second)
○ Slow
Raid Mechanism isn’t for larger
number of disk
○ 100+ (only broken 2/3)

15. WhyDFS
Scale up V.S Scale out

16. Distributedfilesystem
Allows access to files from multiple
hosts sharing via a computer network.
Multiple users on multiple machines
to share files and storage resources.
Components
○ Server
○ Client
○ Metadata Servers
Different features
○ Security/Redundancy

20. GlusterFS
Distributed File System
Volume
○ Distributed
○ Replicated
○ Distributed-Replicated
○ Stripe
○ Disperse
Metrics
○ Large/Small files
○ Random/Sequence location

21. Distributed
File Based
Like RAID-0
Volume = V1+V2+…Vn
IOPS
○ Read: v * n
○ Write: v * n

22. Replicated
File Based
Like RAID-1
Assume Replication Factor =2
Volume = sum(V)/2
IOPS
○ Read: v*n
○ Write: v*n/2

23. Distribuuted-replicated
File-Based
The server will mirror by itself (in the
same Replication)
Volume: sum(V)/2
IOPS:
○ Read: n *v
○ Write: n*v/2

24. Stripe
Object-Based
Split file into many bricks
Volume: sum(v)
IOPS:
○ Read: n*v
○ Write: n*v

25. DiSPERSE
Erasure Coding
Volume: based on config

26. ErasureCoding
Erasure Coding
○ N = k + m (data + redundancy)
○ Take 6=4 + 2 as example
10 MB File
2.5 MB 2.5 MB
Server 1 Server 2
2.5 MB
Server 3
2.5 MB
Server 4
2.5 MB
Server 5
2.5 MB
Server 6
Data Data Data DataRedundancyRedundancy

27. Aurora
How Kubernetes works with volume
○ local
○ GlusterFS
○ NFS

28. LocalMount
Read/Write from/to local disk
○ Memory
○ Disk
Can’t share data cross nodes
Share data in the same node
○ Access control
○ Read-Write many
○ Read-Write once
○ Read-Only

29. Server Server
MountLocalVolume
Server
Platform (K8S)
Pod Pod Pod Pod Pod Pod
Mount
local
volume
Local Local Local
No ShareShare

30. GlusterFS
Read/Write from/to GlusterFS
○ Memory
○ Disk/Network Access
Share data cross nodes (same name)
○ Access Control
■ Read-Write Once
■ Read-Write Many
■ Read-Only
Ask the volume size from GlusterFS

31. Server Server
MountGlusterFS
Server
Platform (K8S)
Pod Pod Pod Pod Pod Pod
Mount
Remote
Volume
Local Local Local
Gluster-FS
Local LocalLocal
5G 10G 25G
Server
Local Loc
al

32. NFS
Read/Write from/to NFS
○ Memory
○ Network Access
Share data cross nodes (same name)
○ Access Control (no…since the NFS doesn’t
support those feature)
Ask the data size
○ It don’t support this feature..lol

33. Server Server
MountNFS
Server
Platform (K8S)
Pod Pod Pod Pod Pod Pod
NFS
NFS
Server
Performance bottleneck

34. HowaboutstorageforAItraining

35. Storage systems available today are
optimized for a design point that’s
different to what AI truly requires
They are optimized for structured
workloads – predictable, sequential
access, not random patterns.

36. Pure-Storage(FlashBlade)
A single storage server for NVIDIA
DGX-1
Based on NFS
Storage
○ Flash-Array
○ PB
○ 17 GB/s bandwidth
○ 1.5M IOPS
Network
○ 8 * 40Gb/s
Price <$1 per GB

37. Pure-Storage(AIRI)
Four DGX-1 systems
Based on NFS
Storage
○ Flash-Array
○ PB
○ 17 GB/s bandwidth
○ 1.5M IOPS
Network
○ 100G
○ GPUDirect RDMA
Price
○ > 1M

39. Pure-Storage
You can use GPFS/Lustre and other
HPC storage.
Btu data-scientists don’t want to deal
with data-center infrastructure