The talk I shared at PyCon APAC 2022. https://tw.pycon.org/2022/en-us/conference/talk/245

1. Distributing your pandas
ETL job using Ray and Modin
李泓旻(Andrew Li)

2. 2
About me
- Data Engineer at DDT, Cathay Financial Holdings
- Former one-stop engineer for data science(Manufacturing)
- Former Chemical Engineer
- Polymer material, Genetic engineering, Bacterial fermentation
- First prize, Genius For Home competition, MediaTek, 2018
- D4SG (Data for Social Good) #4, winter 2018
- : orcahmlee

3. WHY
3

4. 4
Downloads: 1.7B
Downloads per month: 92M
Ecosystem: 30+ packages

5. 5

6. 6

7. 7

8. Two cases I want to handle in the real world
8
1. Many small datasets which share the same business logic
2. An out-of-core dataset without rewriting the ETL script

9. HOW
to handle
9
Many small datasets which share the same business logic

10. CASE I
10

11. CASE I
11

12. CASE I
12

13. CASE I
13

14. CASE I
14

15. You are already know
how to use Ray!
15

16. Ray 16

17. Ray 17

18. 18
Programming model
- Tasks
- A task executes on a stateless worker
- A future representing the result of the task is returned immediately
- Futures can be passed to other remote functions
- Idempotence
- Actors
- An actor executes on a stateful worker
- Each actor exposes methods that can be executed
- An actor’s method execution is similar to a task
- A handle to an actor can be passed to other actors or tasks
Ray: A Distributed Framework for Emerging AI Applications
[COSCUP 2011] Programming for the Future, Introduction to the Actor Model and Akka Framework

19. 19
Function & Class

20. 20
Task & Actor

21. ray.init()
21
Ray Core API

22. Something I want to share
22
- Components
- Global Control Store(GCS)
- Bottom-Up Distributed Scheduler
- In-Memory Distributed Object Store
- Features
- Handling Dependencies

23. Architecture
23
Ray: A Distributed Framework for Emerging AI Applications

24. Global Control Store(GCS)
24
Ray: A Distributed Framework for Emerging AI Applications

25. - Maintains the entire control state of the system
- Key-value store with pub-sub functionality
- Redis as storage(< v1.11.0)
- v1.11.0+: No longer starts Redis as default
- Enables every components in the system to be stateless
- The primary reasons: fault tolerance and low latency
Global Control Store(GCS)
25
Ray: A Distributed Framework for Emerging AI Applications

26. Fault tolerance
- Heartbeat table
- Job table
- Task table
- Actor table
- Decouple the durable lineage storage from other system
components
Global Control Store(GCS)
26
Ray: A Distributed Framework for Emerging AI Applications

27. Global Control Store(GCS)
27
Ray: A Distributed Framework for Emerging AI Applications
Low latency
- Centralized scheduler couple task scheduling and task
dispatch(Dask, Spark, CIEL)
- Involving the scheduler in each object transfer is
prohibitively expensive
- Ray store the object metadata in GCS rather than in the
scheduler, fully decoupling task dispatch from task scheduling

28. Bottom-Up Distributed Scheduler
28
Ray: A Distributed Framework for Emerging AI Applications

29. Bottom-Up Distributed Scheduler
29
Ray: A Distributed Framework for Emerging AI Applications

30. In-Memory Distributed Object Store
30
Ray: A Distributed Framework for Emerging AI Applications

31. - Plasma: A High-Performance Shared-Memory Object Store
- Plasma was initially developed as part of Ray that is being developed as part of
Apache Arrow(https://arrow.apache.org/docs/python/plasma.html)
- To minimize task latency, Ray has an in-memory
distributed storage system to store the inputs and
outputs of every task, or stateless computation.
- On each node, Ray has the object store via shared memory.
This allows zero-copy data sharing between tasks running
on the same node.
In-Memory Distributed Object Store
31
Ray: A Distributed Framework for Emerging AI Applications

32. In-Memory Distributed Object Store
- Spilling objects to external storage once the capacity of
the object store is used up(v1.3+)
- Two types of external storage supported by default:
- Local storage, S3
- Ray recovers any needed objects through lineage
re-execution. The lineage stored in the GCS tracks both
stateless tasks and stateful actors during initial
execution
32
Ray: A Distributed Framework for Emerging AI Applications

33. Handling Dependencies
When your script running on the distributed system……
- need some specific environment variables
- import/depend on some Python packages
- read some files outside of the script
- ModuleNotFoundError, FileNotFoundError
33
Ray: Handling dependencies

34. Handling Dependencies
34

35. 35
An out-of-core dataset without rewriting the ETL script
HOW
to handle

36. CASE II
36

37. CASE II
37

38. You are already know
how to use Modin!
38

39. Architecture
39
Modin: Architecture

40. pandas API coverage
40
Modin vs. Dask DataFrame vs. Koalas

41. - Dask DataFrame and Koalas
- Lazy execution
- Support row-oriented partitioning and parallelism
- Modin
- Eager execution
- Support row, column, and cell-oriented partitioning
and parallelism
Modin vs. Dask DataFrame vs. Koalas
41
Modin vs. Dask DataFrame vs. Koalas

42. Decomposition
42
Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System

43. - Dask DataFrame and Koalas
- Lazy execution
- Support row-oriented partitioning and parallelism
- Modin
- Eager execution
- Support row, column, and cell-oriented partitioning
and parallelism
- If the API is not supported yet, it is being executed
in the default to pandas mode
Modin vs. Dask DataFrame vs. Koalas
43
Modin vs. Dask DataFrame vs. Koalas

44. default to pandas
44
Defaulting to pandas

45. Supported APIs
- pd.DataFrame
- Y: iloc, T, all, any, quantile, apply, applymap……
- D: plot, to_parquet, to_pickle, to_json……
- pd.Series
- Y: iloc, T, all, any, quantile, apply, value_counts, to_frame……
- D: plot, to_parquet, to_pickle, to_json……
- pd.read_<file>
- Y: read_csv, read_parquet……
- D: read_pickle, read_html……
- Utilities
- Y: pd.concat, pd.unique, pd.get_dummies……
- D: pd.cut, pd.to_datetime, pd.to_numeric……
45
Supported APIs

46. RECAP
46

47. Ray
47
- Scalable Dataframe
- High-level API
- out-of-memory
- High pandas API coverage
- Running on Ray/Dask/……
- Distributed framework
- Low-level API
- fit-in-memory(w/ pandas)
- Rich native/3rd libraries
- Deploy on Cloud/K8s/……
Modin

48. Thank you for your time
48