RedHat built a distributed object storage solution named Ceph which first debuted ten years ago. Now we are seeing rapid developments in the industry and we want to take advantage of them. In this talk, we will briefly introduce Ceph, revisit the problems we are seeing when profiling its I/O performance with flash device, and explain why we want to embrace the future by switching to Seastar. We’ll share our experiences with the audience of how and when we are porting our software to this framework.

1. Rebuild the Ceph Distributed
Storage with Seastar
Kefu Chai
Software Engineer, Red Hat

2. Kefu Chai
He is a software engineer who has been focusing on storage
systems for over a decade.
Currently, he is working on Ceph, an open source, distributed
storage solution.

3. What is Ceph anyway?

4. About Ceph
▪ A platform
• Object, block and filesystem
▪ Scalable
• from 10 to 10,000 nodes (OSDs)
▪ No single point failure
▪ Free and Open Source (LGPL)

5. Ceph Components

6. CRUSH

7. CRUSH (cont.)

8. good not not good enough

9. The latency analysis for write path
courtesy of Myongwon Oh, et al., ALCA: Locality-Aware Lock Contention
Avoidance for NVMe-Based Scale-out Storage System

10. Blank Slide for full size diagram
Storage Backend
Network Messenger
Thread Pool serving requests

11. A closer look at the thread pool

12. Random R/W performance
The performance of random I/O is throttled by CPU

14. Latencies in I/O path
▪ TCP/IP stack in kernel space
▪ Coarse grained locks
▪ Internal fragmentation
▪ Memory copy for direct-I/O

15. Seastar to the Rescue

16. Non-seastar
▪ Thread pool
▪ Lock
▪ Share state
▪ TCP/IP in kernel
▪ Callbacks
▪ Multiple-threaded
▪ Fiber
▪ Lockless
▪ Message passing
▪ TCP/IP in userspace (0-copy)
▪ future/promise
▪ single-threaded
Seastar

18. A message from the alien

19. A message from the alien
▪ std::future versus seastar::future
// in rocksdb
Status SequentialFile::read(size_t n,
char* buf)
{
return alien::submit_to(io_shard,
[n,buf,this]
{
return do_dma_read(n, buf);
}).wait();
}
▪ Dedicated cores for the alien territory

20. Backfill with priority
▪ Critical path first
• Basic I/O paths
▪ Supporting features later
• recovery
• Metric reporting
• …

21. Thank You
Any Questions ?
Please stay in touch
kchai@redhat.com
@tchaik0v