true

1. Storing VMs with Cinder and
Ceph RBD

2. Growing With Hardware Appliances
C D First PB C D Second PB
C D • Proprietary C D • Proprietary storage
C D storage hardware C D hardware
C D • Well-known C D • Same storage
C D storage vendor C D vendor
C D C D
C D C D
$14 b’zillion Another
C D C D
C D C D
$14 b’zillion
C D C D
C D C D
C D C D
47

3. C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D
52

4. X
C D
C D
C C D
C D
D
C D
C D
C++ C D
C D
C D
C D
C D
53

5. C D
C D
C D
C D
C D
HUMAN !! C D
[DEVELOPER] C D
C D
C D
C D
C D
C D
54

6. Hard Drives Are Tiny Record Players and They Fail Often
jon_a_ross, Flickr / CC BY 2.0 71

7. D D
D D
D D =
D D
x 1 MILLION
55 times / day
72

8. 73

9. philosophy design
OPEN SOURCE SCALABLE
COMMUNITY-FOCUSED NO SINGLE POINT OF FAILURE
SOFTWARE BASED
SELF-MANAGING

10. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
79

11. OSD OSD OSD OSD OSD
btrfs
FS FS FS FS FS
xfs
ext4
DISK DISK DISK DISK DISK
M M M
81

12. HUMAN
M
M M
82

13. Monitors:
M
• Maintain cluster map
• Provide consensus for
distributed decision-making
• Must have an odd number
• These do not serve stored
objects to clients
OSDs:
• One per disk (recommended)
• At least three in a cluster
• Serve stored objects to
clients
• Intelligently peer to perform
replication tasks
• Supports object classes
83

14. C D
C D
C D
C D
C D
??
APP C D
C D
C D
C D
C D
C D
C D

15. C D
C D
C D
C D
C D
APP C D
C D
C D
C D
C D
C D
C D

16. C D
C D A-G
C D
C D
C D H-N
F
APP * C D
C D
C D O-T
C D
C D
C D U-Z
C D

17. 10 10 01 01 10 10 01 11 01 10
hash(object name) % num pg
10 10 01 01 10 10 01 11 01 10
CRUSH(pg, cluster state, rule set)
107

18. 10 10 01 01 10 10 01 11 01 10
10 10 01 01 10 10 01 11 01 10
108

19. CRUSH
• Pseudo-random placement
algorithm
• Ensures even distribution
• Repeatable, deterministic
• Rule-based configuration
• Replica count
• Infrastructure topology
• Weighting
109

20. CLIENT
??
110

21. 112

22. CLIENT
??
113

23. 111

24. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
84

25. APP
LIBRADOS
native
M
M M
85

26. LIBRADOS
L
• Provides direct access to
RADOS for applications
• C, C++, Python, PHP, Java
• No HTTP overhead

27. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
87

28. APP APP
REST
RADOSGW RADOSGW
LIBRADOS LIBRADOS
native
M
M M
88

29. RADOS Gateway:
• REST-based interface to
RADOS
• Supports buckets,
accounting
• Compatible with S3 and
Swift applications
89

30. APP APP HOST/VM CLIENT
RADOSGW RBD CEPH FS
LIBRADOS
A bucket-based REST A reliable and fully- A POSIX-compliant
A library allowing gateway, compatible distributed block distributed file
apps to directly with S3 and Swift device, with a Linux system, with a Linux
access RADOS, kernel client and a kernel client and
with support for QEMU/KVM driver support for FUSE
C, C++, Java,
Python, Ruby,
and PHP
RADOS
A reliable, autonomous, distributed object store comprised of self-healing, self-managing,
intelligent storage nodes
90

31. VM
VIRTUALIZATION CONTAINER
LIBRBD
LIBRADOS
M
M M
91

32. CONTAINER VM CONTAINER
LIBRBD LIBRBD
LIBRADOS LIBRADOS
M
M M
92

33. HOST
KRBD (KERNEL MODULE)
LIBRADOS
M
M M
93

34. RADOS Block Device:
• Storage of virtual disks in RADOS
• Allows decoupling of VMs and
containers
• Live migration!
• Images are striped across the
cluster
• Thin-provisioning
• Snapshots and cloning

35. VM
VIRTUALIZATION CONTAINER
LIBRBD
LIBRADOS
M
M M
115

36. HOW DO YOU
SPIN UP
THOUSANDS OF VMs
INSTANTLY
AND
EFFICIENTLY?
116

37. instant copy
144 0 0 0 0 = 144
117

38. write
CLIENT
write
write
write
144 4 = 148
118

39. read
read
CLIENT
read
144 4 = 148
119

40. old-style VM image creation
local disk Nova Glance
(VM images) compute (templates)
read X
● ephemeral
● expensive to create
X
X'
29

41. Why use block storage?
• Persistent
•
More familiar to users
• Not tied to a single host
•
Decouples compute and storage
•
Enables Live migration
•
Extra capabilities of storage system
•
Efficient snapshots
•
Different types of storage available
• Cloning for fast restore or scaling

42. Cinder volume creation
Cinder Cinder volume Glance
API volume driver (templates)
create image from X
locate X
location of X
read X
X
flexibility in where VM
images are stored
X'
reference to X'
31

43. Efficient volume creation
Cinder Cinder volume Glance
API volume driver (templates)
create image from X
locate X
location of X
clone X to X'
X
fast CoW clone
X'
X' complete
reference to X'
32

44. Questions?
Josh Durgin
josh.durgin@inktank.com
jdurgin on freenode
inktank.com | ceph.com