Hadoop was born much earlier than the Cloud Native era. But the question is still the same: what can it offer in the time of Kubernetes, containerization and hybrid clouds? Apache Hadoop Ozone is a new subproject of Hadoop. It has a generic low-level binary layer, the Hadoop Distributed Data Storage (HDDS) and a S3 compatible Object Store implementation on top of it. But the HDDS data storage layer is not just for the object store. It could be used for multiple purposes: to enhance the scalability the HDFS or provide block level access to the managed storage space. With this approach the same Hadoop Ozone cluster could provide hadoop file system based storage, object store space and block level storage. Storage is still a hot topic with Kubernetes and in Cloud Native environments. Container Storage Interface specification is a vendor neutral standard to provide storage plugin for multiple container orchestration system. Quadra provides block level access on top of the Hadoop Distributed Data Storage layer and it’s first class citizen of the containerized word. It implements the Container Storage Interface and can work as a Kubernetes dynamic volume provisioner. In this talk we will demonstrate how the Hadoop Ozone storage could be used from containers. We will explain the basic storage type of Kubernetes clusters and show how Hadoop Ozone and Quadra could help to solve the storage problem in an industry standard way.

1. © Cloudera, Inc. All rights reserved.
HADOOP STORAGE IN THE CLOUD NATIVE ERA
Nandakumar Vadivelu
nanda@apache.org

2. © Cloudera, Inc. All rights reserved. 2© Cloudera, Inc. All rights reserved.
Hadoop Storage
Container
Orchestrator
Application
(Container)
Application
(Container)
Application
(Container)
Application
(Container)
Application
(Container)

3. © Cloudera, Inc. All rights reserved. 3
HDDS
Ozone
Quadra
CSI

4. © Cloudera, Inc. All rights reserved. 4© Cloudera, Inc. All rights reserved.
HADOOP DISTRIBUTED DATA STORE

5. © Cloudera, Inc. All rights reserved. 5
Namespace
+
Blockspace
HDFS - NameNode

6. © Cloudera, Inc. All rights reserved. 6
Namespace
File -> B1, B2, B3
Block Management Layer
B1 -> Dn1, Dn2, Dn3
Namenode
Layering

7. © Cloudera, Inc. All rights reserved. 7
Namespace
HDDS
Block Storage

8. © Cloudera, Inc. All rights reserved. 8© Cloudera, Inc. All rights reserved.
BlocksMetadata
STORAGE CONTAINER

9. © Cloudera, Inc. All rights reserved. 9© Cloudera, Inc. All rights reserved.
ARCHITECTURE
Storage
Container
Manager
DatanodeDatanode Datanode

10. © Cloudera, Inc. All rights reserved. 10© Cloudera, Inc. All rights reserved.
Container Protocol
● Create Container
● Get Container
● List Container
● Close Container
● Delete Container
Container Operations
● Read Block
● Write Block
● Delete Block

11. © Cloudera, Inc. All rights reserved. 11© Cloudera, Inc. All rights reserved.
OZONE

12. © Cloudera, Inc. All rights reserved. 12
Ozone Manager
Key 1 -> List <block Id>
Key 2 -> List <block Id>
Key 3 -> List <block Id>
Key 4 -> List <block Id>
Key 5 -> List <block Id>
…
Key n -> List <block Id>
Block Id -> [Container Id + Local Id]

13. © Cloudera, Inc. All rights reserved. 13© Cloudera, Inc. All rights reserved.
Storage
Container Manager
DatanodeDatanode Datanode
HDDS
Ozone Manager
ARCHITECTURE

14. © Cloudera, Inc. All rights reserved. 14© Cloudera, Inc. All rights reserved.
INTERFACES
Object Store API
(RPC)
OzoneFileSystem (HCFS)
Connector
S3 Connector

15. © Cloudera, Inc. All rights reserved. 15© Cloudera, Inc. All rights reserved.
NAMENODE’
HDFS-10419

16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved.
Storage
Container Manager
DatanodeDatanode Datanode
HDDS
NameNode’
ARCHITECTURE

17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved.
QUADRA
HDFS-11118

18. © Cloudera, Inc. All rights reserved. 18© Cloudera, Inc. All rights reserved.
QUADRA
• LUN like Raw-Block Storage
• Backed by HDDS
• Mountable disk FS volume
• Volume: A raw-block device that can be used to create mountable disk
• Can create filesystems like ext4 or XFS on the volumes
• POSIX semantics

19. © Cloudera, Inc. All rights reserved. 19© Cloudera, Inc. All rights reserved.
Storage
Container Manager
DatanodeDatanode Datanode
HDDS
Quadra Manager
iSCSI Server
ARCHITECTURE

20. © Cloudera, Inc. All rights reserved. 20© Cloudera, Inc. All rights reserved.
USAGE
• Create a Volume
• quadra -c foo datavolume 4TB
• Mount the volume
• iscsiadm -m node -o new -T foo:datavolume -p localhost:3260
• Format the Volume
• mkfs.ext4 -b 4096 /dev/sdb
• Mount the filesystem
• mkdir datavol; mount /dev/sdb datavol

21. © Cloudera, Inc. All rights reserved. 21© Cloudera, Inc. All rights reserved.
Storage
Container
Manager
HDDS
Datanode DatanodeDatanode
Quadra
Volume
Manager
Quadra
Plugin
JSCSI
Kernel
User
SCSI Initiator
Volume API
Data Path
HOST

22. © Cloudera, Inc. All rights reserved. 22© Cloudera, Inc. All rights reserved.
Storage
Container Manager
DatanodeDatanode Datanode
HDDS
Quadra Manager Ozone Manager NameNode’
Block Store Object Store File Store
HADOOP STORAGE ECOSYSTEM

23. © Cloudera, Inc. All rights reserved. 23© Cloudera, Inc. All rights reserved.
CONTAINER STORAGE INTERFACE

24. © Cloudera, Inc. All rights reserved. 24© Cloudera, Inc. All rights reserved.
Pluggable Storage
Interface
Pluggable Storage
Interface
Pluggable Storage
Interface
Storage Provider has to write a plugin for each container orchestrator
WHY?

25. © Cloudera, Inc. All rights reserved. 25© Cloudera, Inc. All rights reserved.
CONTAINER STORAGE INTERFACE
• Specification
• Interoperable
• Vendor neutral
• Control plane only

26. © Cloudera, Inc. All rights reserved. 26© Cloudera, Inc. All rights reserved.
Pluggable Storage Interface
Storage Provider
CSI

27. © Cloudera, Inc. All rights reserved. 27© Cloudera, Inc. All rights reserved.
PLUGINS
Control
Plugin
Node
Plugin
Container
Orchestrator
Storage Provider

28. © Cloudera, Inc. All rights reserved. 28© Cloudera, Inc. All rights reserved.
• It can run anywhere
• Handles storage volume
creation and deletion
Control
Plugin
Node
Plugin
• Runs on all the nodes
• Handles storage volume
mounting and unmounting

29. © Cloudera, Inc. All rights reserved. 29

30. © Cloudera, Inc. All rights reserved. 30© Cloudera, Inc. All rights reserved.
CSI DRIVER FOR HADOOP STORAGE
HDDS-1382

31. © Cloudera, Inc. All rights reserved. 31
DatanodeDatanode Datanode
HDDS
Quadra Manager
iSCSI Server
Storage
Container Manager
Hadoop
Storage
Container
Orchestrator
Application Application Application Application Application
CSI Driver

32. © Cloudera, Inc. All rights reserved. 32© Cloudera, Inc. All rights reserved.
Control
Plugin
Node
Plugin
Hadoop CSI Driver
Storage
Container
Manager
HDDS
DatanodeDatanode
Quadra
Volume
Manager
Quadra
Plugin
JSCSI
Volume API
Data Path
Datanode

33. © Cloudera, Inc. All rights reserved. 33© Cloudera, Inc. All rights reserved.
DEMO

35. © Cloudera, Inc. All rights reserved. 35© Cloudera, Inc. All rights reserved.
CURRENT STATUS
● Apache Hadoop Ozone 0.4.0-alpha – Released on May 7
● Implementing Namenode on top of HDDS (HDFS-10419) – Design Discussion
● Quadra (HDFS-11118) – Design Discussion (POC)
● CSI Server for Ozone (HDDS-1382) – In development

36. © Cloudera, Inc. All rights reserved. 36© Cloudera, Inc. All rights reserved.
Q & A

37. © Cloudera, Inc. All rights reserved.
THANK YOU