Oder: Wie speicher man terabyteweise Dokumente. Mein Talk auf der data2day in Karlsruhe, 28.11.2014.

1. Ceph Object Store
oder: Wie speichert man terabyteweise Dokumente
Daniel Schneller
daniel.schneller@centerdevice.de
@dschneller

2. Wer sind wir?

3. @dschneller
@drivebytesting

4. Was machen wir?

6. Wo kamen wir her?

7. Warum wollten wir
da weg?

8. Wohin wollten wir?

9. Und der Weg?

10. Ceph Grundlagen

11. “Unified, distributed storage system designed for
excellent performance, reliability and scalability”

12. Stark skalierbar
Commodity Hardware
Kein Single Point of Failure

13. Ceph Komponenten

14. OSD Daemons
Object Storage Device Daemons

15. CRUSH Algorithmus
Intelligente Objektverteilung ohne zentrale Metadaten

16. RADOS
Reliable Autonomous Distributed Object Store

17. Objekte

18. Data Pools
Sammelbecken für Objekte mit gleichen Anforderungen

20. Placement Groups

23. Monitors
Erste Anlaufstelle für Clients

24. Hardware Setup

25. Storage Virtualization
Bare Metal Hardware
Compute Virtualization
Network Virtualization
Virtual Infrastructure
Application

26. Storage Virtualization
Bare Metal Hardware
Compute Virtualization
Network Virtualization
Virtual Infrastructure
Application

28. Baseline Benchmarks
Erwartungen definieren

29. Storage
Disk I/O pro Node

32. Netzwerk

33. IEEE 802.3ad != IEEE 802.3ad
> cat /etc/network/interfaces
...
auto bond2
iface bond2 inet manual
bond-slaves p2p3 p2p4 # interfaces to bond
bond-mode 802.3ad # activate LACP
bond-miimon 100 # monitor link health
bond-xmit_hash_policy layer3+4 # use Layer 3+4 for link selection
pre-up ip link set dev bond2 mtu 9000 # set Jumbo Frames
auto vlan-ceph-clust
iface vlan-ceph-clust inet static
pre-up ip link add link bond2 name vlan-ceph-clust type vlan id 105
pre-up ip link set dev vlan-ceph-clust mtu 9000 # Jumbo Frames
post-down ip link delete vlan-ceph-clust
address ...
netmask ...
network ...
broadcast ...
...

34. IEEE 802.3ad != IEEE 802.3ad
[node01] > iperf -s -B node01.ceph-cluster
[node02] > iperf -c node01.ceph-cluster -P 2
[node03] > iperf -c node01.ceph-cluster -P 2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address node01.ceph-cluster
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49412
[ 5] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49413
[ 6] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59947
[ 7] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59946
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 342 MBytes 286 Mbits/sec
[ 5] 0.0-10.0 sec 271 MBytes 227 Mbits/sec
[SUM] 0.0-10.0 sec 613 MBytes 513 Mbits/sec
[ 6] 0.0-10.0 sec 293 MBytes 246 Mbits/sec
[ 7] 0.0-10.0 sec 338 MBytes 283 Mbits/sec
[SUM] 0.0-10.0 sec 631 MBytes 529 Mbits/sec

35. IEEE 802.3ad != IEEE 802.3ad
[node01] > iperf -s -B node01.ceph-cluster
[node02] > iperf -c node01.ceph-cluster -P 2
[node03] > iperf -c node01.ceph-cluster -P 2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address node01.ceph-cluster
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49412
[ 5] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49413
[ 6] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59947
[ 7] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59946
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 342 MBytes 286 Mbits/sec
[ 5] 0.0-10.0 sec 271 MBytes 227 Mbits/sec
[SUM] 0.0-10.0 sec 613 MBytes 513 Mbits/sec
[ 6] 0.0-10.0 sec 293 MBytes 246 Mbits/sec
[ 7] 0.0-10.0 sec 338 MBytes 283 Mbits/sec
[SUM] 0.0-10.0 sec 631 MBytes 529 Mbits/sec ???

36. Messen!
…und die Ergebnisse verstehen

37. CenterDevice

38. Gesamtarchitektur
Node 1
OSD 1
…
…
Node 2
…
…
…
Node 3
…
…
…
Node 4
…
OSD 48
…
Bare Metal
Ceph

39. Gesamtarchitektur
Node 1
OSD 1
…
…
Rados GW
Node 2
…
…
…
Rados GW
Node 3
…
…
…
Rados GW
Node 4
…
OSD 48
…
Rados GW
Bare Metal
Ceph

40. Gesamtarchitektur
Node 1
OSD 1
…
…
Rados GW
Node 2
…
…
…
Rados GW
Node 3
…
…
…
Rados GW
Node 4
…
OSD 48
…
Rados GW
VM 1
HAProxy
VM 1
HAProxy
VM 1
HAProxy
VM …
HAProxy
VMs
Bare Metal
Ceph

41. Gesamtarchitektur
Node 1
OSD 1
…
…
Rados GW
Node 2
…
…
…
Rados GW
Node 3
…
…
…
Rados GW
Node 4
…
OSD 48
…
Rados GW
VMs
Bare Metal
VM 1
Ceph
HAProxy
CenterDevice
Swift
VM 1
HAProxy
CenterDevice
Swift
VM 1
HAProxy
CenterDevice
Swift
VM …
HAProxy
CenterDevice
Swift

42. Vorteile

43. Nachteile

44. Caveats

45. CephFS
Not recommended for production data.

46. Scrubbing
Integrität hat ihren Preis. Aber man kann handeln!

47. Zukunft

48. Rados Gateway
Ceph Caching Tier
SSD based Journaling
10GBit/s Networking

49. Zum Schluss

50. Folien bei Slideshare
http://www.slideshare.net/dschneller

51. Handout bei
CenterDevice
https://public.centerdevice.de/399612bf-ce31-489f-
bd58-04e8d030be52

52. @drivebytesting
@dschneller

53. Ende
Daniel Schneller
daniel.schneller@centerdevice.de
@dschneller