Operational aspects of distributed Linux file systems like Ceph and/or GlusterFS

1. Cloud storage reloaded:
Some aspects on operating distributed Linux file systems
like Ceph and GlusterFS
Udo Seidel

2. Agenda
● Introduction/motivation
● Distributed storage
● Ceph and GlusterFS
● Operational considerations
● Summary
LinuxCon Europe 05-07/11 2012 2

3. Introduction
LinuxCon Europe 05-07/11 2012 3

4. Me ;-)
● Teacher of mathematics & physics
● PhD in experimental physics
● Started with Linux in 1996
● Linux/UNIX trainer
● Solution engineer in HPC and CAx environment
● Head of a international team with people in
Erding, Miami and Sydney @Amadeus
LinuxCon Europe 05-07/11 2012 4

5. Shared File Systems
● Normal 'business' for Linux
● Several implementation approaches
● Network based, e.g. NFS, CIFS
● 'clustered'
– Shared disk, e.g. OCFS2, GFS2
– Parallel/distributed, e.g. Ceph, GlusterFS
LinuxCon Europe 05-07/11 2012 5

6. Distributed File Systems
● 'Recent' attention on distributed storage
● Cloud hype
● Big Data
● Which solution to choose?
● Just storage or file system too?
● Knock-on effects
● Technical
● Non-technical
LinuxCon Europe 05-07/11 2012 6

7. Operations
● Important part of the life cycle
● Geek vs. Enterprise
● Technical challenges and more
LinuxCon Europe 05-07/11 2012 7

8. Distributed Storage
LinuxCon Europe 05-07/11 2012 8

9. History
● Reviewing storage task responsibilities
● Block allocation
● Space management
● Extension of SCSI standard
● Object based storage
● Meta-Data handling separated from data
management
LinuxCon Europe 05-07/11 2012 9

10. Object based storage
● Storage objects quite general
● Partition, file, ...
● Unique identifier
● OSD (Object based Storage Device)
● Hardware -> original trigger
● Software -> common implementation
● Interfaces to existing technology
● Part of the distributed storage solution
● Additional development of existing approaches
LinuxCon Europe 05-07/11 2012 10

11. Distributed storage:
Paradigm changes
● Block -> Object
● Central -> Distributed
● Few -> Many
● Big -> Small
LinuxCon Europe 05-07/11 2012 11

12. Distributed storage – Now what?!?
● Several implementations
● Different functions
● Support models
● Storage vendors initiatives
● Relation to Linux distributions
LinuxCon Europe 05-07/11 2012 12

13. Ceph and GlusterFS
LinuxCon Europe 05-07/11 2012 13

14. Ceph and GlusterFS
● Sufficient community presence
● Picked up by Enterprise Linux vendors
● Good amount of commonalities but still quite
different
LinuxCon Europe 05-07/11 2012 14

15. Ceph and GlusterFS – High level
● Server part
● Block storage -> storage server (OSD)
● (POSIX) file system -> Meta data
● HA -> replication and distribution
● Client part
● (POSIX) file system
● Storage layer
LinuxCon Europe 05-07/11 2012 15

16. Ceph
LinuxCon Europe 05-07/11 2012 16

17. GlusterFS
LinuxCon Europe 05-07/11 2012 17

18. Ceph and GlusterFS – some details
CEPH GLUSTERFS
Server implementation userspace userspace
Client implementation userspace/FUSE and userspace/FUSE
native kernel
File system access CEPH protocol suite GlusterFS and NFS
protocol
Block device access yes RFP ongoing
QEMU integration yes Under development
Replication traffic OSD network Client-server network
Interconnect IP based IP based and/or
Infiniband
Encryption no not really
Back-end file system XFS ..... BTRFS ... ext4 XFS ........ext4
Storage object CRUSH object file (except for striping)
GEO replication not really yes
LinuxCon Europe 05-07/11 2012 18

19. Ceph and GlusterFS – more details
CEPH GLUSTERFS
Online growth/shrink yes yes
POSIX yes alike
Concurrent workload technically yes technically yes
NAS replacement technically yes technically yes
SAN replacment technically yes no
Hadoop integration yes yes
Openstack integration yes yes
Community version yes yes
available
Production readiness Block layer File system layer
Service provider Vendor and other Vendor and other
... ... ...
LinuxCon Europe 05-07/11 2012 19

20. Operational considerations
LinuxCon Europe 05-07/11 2012 20

21. Status quo “pre distributed storage”
● Server and Storage
● Clear responsibilities
● Known area: SCSI, NFS, FC
● Strict border lines
– Technically
– organizationally
● Storage hardware and topology
● Less focus on local drives
● Dedicated/separated storage network
LinuxCon Europe 05-07/11 2012 21

22. Changes with distributed storage
● New paradigm
● Big overlap between server and storage
● Technical implication
● Organizational structure
● Strong network dependency
● Topology
● physics
● Support and supported setups
LinuxCon Europe 05-07/11 2012 22

23. Challenge one: server vs. storage
● Need for
● Server admin with strong storage skills
● Storage admin with strong O/S skills
● Review of
● Current organizational setup
● Strategy for traditional storage
● Operation procedures
● HA architecture
LinuxCon Europe 05-07/11 2012 23

24. Challenge one: possible answers
● 'DevOps model' for O/S and storage admins
● Best of both worlds
● Better common understanding
● Potential of process simplification
● BUT organizational changes needed!
LinuxCon Europe 05-07/11 2012 24

25. Challenge one: possible answers
● 'Appliance model' for O/S and storage admins
● Less internal changes
● Potential of reusing existing processes
● BUT less control and visibility
LinuxCon Europe 05-07/11 2012 25

26. Challenge two: infrastructure
● Dedicated vs. Shared
● Responsibilities
● server vs. storage vs. network
● Transport medium
● Local setup
● Spinning disks vs. SSD
● RAID?
● HA configuration of single server
LinuxCon Europe 05-07/11 2012 26

27. Challenge two: possible answers
● Different flavour of performance vs. cost topic
● Dedicated vs. Shared
● Transport medium
● Spinning disks vs. SSD
● HA on server level
LinuxCon Europe 05-07/11 2012 27

28. Challenge two: possible answers
● Team up with network organization
● Single server full blown SPOF on purpose
● Think big
LinuxCon Europe 05-07/11 2012 28

29. Challenge three: support
● Service provider
● Application support
LinuxCon Europe 05-07/11 2012 29

30. Challenge three: possible answers
● Combine with Linux vendor support
● Single point of contact
● Leverage vendor QA
● Hide the new technology
● XaaS
● Appliance approach
● File system level
● Reminder: Think big
LinuxCon Europe 05-07/11 2012 30

31. Summary ... kind of ...
LinuxCon Europe 05-07/11 2012 31

32. The message
● New world with several challenges
● Don't underestimate them
● Don't overestimate them
● Technical revolution
● requires non-technical changes
● Don't underestimate the potential
● Think big! ;-)
● Get started!
LinuxCon Europe 05-07/11 2012 32

33. References
● http://www.ceph.com
● http://www.gluster.org
● “your business/company”
LinuxCon Europe 05-07/11 2012 33

34. Thank you!
LinuxCon Europe 05-07/11 2012 34