Presentation held at Locaweb Tech Talks

1. A distributed storage system

2. whoami
● Italo Santos
● @ Locaweb since 2007
● Sysadmin @ Storage Team

3. Introduction
● Single Storage System
● Scalable
● Reliable
● Self-healing
● Fault Tolerant
● NO single point of failure

4. Architecture

5. RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-
managing, intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
RBD
A reliable and fully-
distributed block
device, with a
Linux kernel client
and a QEMU/KVM
driver
CEPH FS
A POSIX-compliant
distributed file
system, with a
Linux kernel client
and support for
FUSE
RADOSGW
A bucket-based
REST gateway,
compatible with S3
and Swift
APP APP HOST/VM CLIENT

6. Ceph Storage Cluster

7. OSDs
M
Monitors MDS

8. OSDs

9. OSDs
● One per disk
● Store data
● Replication
● Recovery
● Backfilling
● Rebalancing
● OSDs heartbeat

10. DISK
FS
DISK DISK
OSD
DISK DISK
OSD OSD OSD OSD
FS FS FSFS btrfs
xfs
ext4
MMM

11. M
Ceph Monitors

12. Monitors
● Cluster map
● Monitors map
● OSDs map
● Placement Group map
● CRUSH map

13. Metadata Server (MDS)

14. MDS
● Used only by CephFS
● POSIX-compliant shared filesystem
● Manage metadata
○ Directory hierarchy
○ File metadata
● Stores metadata on RADOS

15. CRUSH

16. CRUSH
● Pseudo-random placement algorithm
○ Fast calculation
○ Deterministic
● Statistically uniform distribution
● Limited data migration on change
● Rule-based configuration

17. 10 10 01 01 10 10 01 11 01 10
10 10 01 01 10 10 01 11 01 10
CRUSH(pg, cluster state, rule set)
hash(object name) % num pg

20. CLIENT
??

21. Placement Groups (PGs)

22. Placement Groups
● Logical collection of objects
● Maps PGs to OSDs dynamically
● Computationally less expensive
○ Reduce number of process
○ Less of per-object metadata
● Dynamically rebalance

23. Placement Groups

24. Placement Groups
● Increase PGs reduces per-osd load
● ~100 PGs per OSD
(i.e., OSD per object = Number of replicas)
● Defined on pool creation
● PGs with multiple pools
○ Balance PGs per pool with PGs per OSD

25. Pools

26. Pools
● Replicated
○ Object replicated N times (i.e., default size = 3)
○ Object + 2 protection replicas
● Erasure Coded
○ Stores objects as K+M chunks (i.e., size = K+M)
○ Divided into K data chunks and M coding chunks

29. Ceph Clients

30. RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-
managing, intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
RBD
A reliable and fully-
distributed block
device, with a
Linux kernel client
and a QEMU/KVM
driver
CEPH FS
A POSIX-compliant
distributed file
system, with a
Linux kernel client
and support for
FUSE
RADOSGW
A bucket-based
REST gateway,
compatible with S3
and Swift
APP APP HOST/VM CLIENT

31. RadosGW
Ceph Object Gateway

32. RadosGW
● Object Storage Interface
● Apache + FastCGI
● S3-compatible
● Swift-compatible
● Common namespace
● Store data on Ceph cluster

33. RBD
Rados Block Device

34. RBD
● Block device interface
● Data striped on ceph cluster
● Thin-provisioned
● Snapshot support
● Linux Kernel-based (librbd)
● Cloud native support

35. CephFS
Ceph File System

36. CephFS
● POSIX-compliant filesystem
● Shared filesystem
● Directory hierarchy
● File metadata (owner, timestamps, mode, etc.)
● Ceph MDS required
● NOT production ready!

38. Thanks
Italo Santos @ Storage Team