Alluxio Online Community Office Hours Jan 28, 2020 Speakers: Bin Fan, Alluxio Calvin Jia, Alluxio Alluxio 2.0 release was the biggest update since the birth of the project “Tachyon” from UC Berkley’s AmpLab. Gathering feedback from our Open Source Community and enterprise users, Alluxio 2.0 expands the system in three major directions including improving the operability of the system, having more advanced data management, as well as re-architecting the system to be able to scale to 1 billion + file. The system is now cloud native on AWS, Google Cloud, and allow users to enable native deployment with K8s. The new advanced data management enables data migration and replication from diff storage systems. In this office hour, we introduce what’s new in the Alluxio 2 release, and dive deeper in each major direction the system has expanded on. In this Office Hour, we will go over: - Introduction and motivation of focus areas of Alluxio 2 - Overview of cloud native deployment methods - New data management features - System scalability improvements

1. What’s new in Alluxio 2
Bin Fan & Calvin Jia | Founding Engineers | Alluxio

2. Seamless
Operations
Alluxio 2 Directions
Advanced
Data Management
2
Hyper-scale
Architecture

3. Seamless Operations

4. Cloud Native on AWS: AMI, CFT, EMR
Presto Hive
Cluster Metadata &
Data cache
Presto Hive
Metadata &
Data cache
Compute-driven
Continuous sync
Compute-driven
Continuous sync
4
§ Alluxio AMI in the Marketplace
§ Alluxio Cloud Formation Template for cluster deployment
§ AWS EMR with Alluxio with bootstrap script
Enable one-click to deploy Alluxio on AWS

5. Cloud Native on Google Cloud: Dataproc
Presto Hive
Metadata &
Data cache
Presto Hive
Metadata &
Data cache
Compute-driven
Continuous sync
Compute-driven
Continuous sync
5
§ Google Dataproc with Alluxio (init action integration available)
Google
Dataproc
Cluster
Enable one-click to deploy Alluxio on Google Cloud

6. Native Deployment with Kubernetes
Alluxio Worker
Kubernetes
Cluster
Host
Machine
6
Alluxio Master Alluxio Worker
Host
Machine
Journal Volume
Application
ApplicationApplicationApplicationApplication

7. Self-Managed Quorum
7
Available in 2.0.0
Distributed
Storage
(ie. HDFS)
Alluxio Standby
Master
Distributed
Quorum
(Zookeeper)
Alluxio Master
Alluxio Standby
Master
Alluxio Standby
Master
Alluxio Master
RAFT
No major external dependencies

8. Hyper-scale architecture

9. § Challenge:
• 1 file metadata takes 1KB of on-heap storage
• 1 billion files would take 1 TB of heap space, GC becomes a big problem
§ Solution:
• Add new tier with embedded RocksDB to store inode tree
• Keep an in-memory cache of frequently used inodes
9
Scaling to 1 Billion+ Files
Scale to one billion files and beyond, with performance comparable
to previous on-heap implementation

10. Scaling to 1 Billion+ Files
10
Available in 2.0.0
Alluxio Master
Local Disk
RocksDB (Embedded)
● Inode Table
● Edge Table
● Block Table
● Block to Worker Table
● Worker to Block Table
On Heap
● Inode Cache
● Mount Table
● Locks
Inode ID Metadata (Binary)
12392 010101101101
12393 110110110100
… …
Edge (ID, name) Inode ID
12392,foo 12393
… …

11. Efficient cluster communication with gRPC
11
Available in 2.0.0
Thrift (Metadata)
Netty (IO)
Alluxio Master
Alluxio Worker
Alluxio Worker
Alluxio Client
Alluxio Master
Alluxio Worker
Alluxio Worker
Alluxio Client
gRPC (Metadata + IO)

12. Advanced Data Management

13. § New Alluxio Catalog Service
• Provides the Abstraction of Structured Data
• Attaching a Hive MetaStore like Mounting a File system
• Understand and Serve Schema of Files or Objects
§ New Alluxio Data Transformation Service
• Tranform csv à parquet
• Compact many files à fewer files
Deeper Integration with Presto
13
Presto Alluxio Connector Based off the Hive Connector
Now available as Developer Preview

14. Policy Driven Data Management
14
Available in 2.0.0
Alluxio
Master
Alluxio Policy Engine
Example Policy
Move files older than 90
days from HDFS to S3
Application
Apps access the same path regardless
of where the actual data is stored
Decouple logical file system namespace with physical storage systems

15. Replicated Asynchronous Writes
15
RAM / SSD / HDD
Network Speed Write of Data
Application
Alluxio
Client
Alluxio
Worker
RAM / SSD / HDD
Alluxio
Worker
Under Store
Available in 2.0.0
Fast and reliable writes to Alluxio, with data persisted in background

16. Questions?