Keynote from OmniSci Converge conference 10/22/2019 in Mountain View, CA

2. Leveling up the Analytics Stack
Wes McKinney
@wesmckinn

3. Apache
ibis

4. The Need for Speed

5. 21 Years Ago...

7. A semi-revisionist
history of Big Data

8. Cloudera’s Jeff Hammerbacher
“Data first, ask questions later”
Enable everyone to
“party on the data”
(paraphrasing!)

9. Decoupling Storage from Processing

10. Simplified MapReduce Arch
Storage
Step 1 Step 2 Step 3 ...

11. “Scalability! But at what COST?”
McSherry, Isard, Murray 2015
https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf

12. Configuration that
Outperforms a
Single
Thread

13. Brute Force Scalability

14. Storage Feasibility
Computational Feasibility
Resource Utilization
Interactivity
Up the hierarchy of needs...

15. Rise of “End-to-end” Execution Engines
Storage
Step 1 Step 2 Step 3 ...
INPUT OUTPUT

16. End-to-end engines: drawbacks
• Example: SQL on Hadoop systems, Apache Spark,
others
• Serve some use cases well, others less well
• Fall short in ML/AI domain

17. The Interoperability Conundrum
Engine A Engine B
Data Handoff

18. Some Hardware Trends
• Manycore processor architectures
• Much faster disk
• Much faster networking
• Beyond CPUs

19. “Why Modern CPUs Are Starving
and What Can Be Done about It”
Francesc Alted, IEEE 2010

20. Recognizing Serialization as
an Enemy

21. Serialization
Translation of data into a form that can
be stored or transmitted, and
reconstructed later

22. How to Eliminate Serialization
“Serialized” and In-Memory Format
must be the same (or nearly so)

23. A Collective Realization in 2015
Many open source developers had noted the
absence of an in-memory standard for
structured data analytics

24. ● Language-agnostic in-memory format for
analytical query processing on modern
hardware
● Low-overhead data sharing and transport
● A cross-language development platform to
build Arrow-powered applications
Mission

25. Why Column-oriented?
• Reduce unnecessary IO
• Increase memory throughput
• Better parallelism
• Leverage SIMD instructions

26. Apache Arrow “meta” goals
• Forge collaborations between database
systems and data science / ML / AI
communities
• Eliminate barriers to code sharing between
application ecosystems and programming
languages

27. Community over Code
• ASF open governance model
• ~400 unique contributors
• 49 committers, 28 PMC members
• 11 programming languages
represented

28. Arrow Development in Practice
• “Core” format and protocol implementations
• “Batteries-included” standard libraries
• Common build / test / package infrastructure and
compatibility testing

29. Language Relationships
C++
Java
Go
Rust
C#
JavaScript
C Ruby
Python
R
MATLAB

30. Some Arrow Success Stories
Apache

31. • gRPC-based framework for custom data
services
• High-speed network dataset transfer
• Now available for C++, Java, Python
Arrow Flight: Fast Data Services
Development Partners

32. Flight key ideas
• Zero-serialization
• Bidirectional streaming transfers
• Parallel transfers + horizontal scalability
designed into the protocol
• Reap benefits of Google’s work on gRPC

33. Flight use cases
• Replacing slow database protocols like
JDBC / ODBC
• General network data movement
• Retrofit legacy systems with fast Arrow IO

34. Notable Arrow subcomponents
Rust DataFusion
Arrow-native
Rust query
engine
Gandiva
LLVM analytical
expression
compiler
Plasma
Shared memory
object store

35. Funding Arrow Development
• Apache projects are technically communities of
volunteers
• Much development contributed by direct users of Arrow
• Ursa Labs: not-for-profit group I founded in 2018 with
initial support of RStudio and Two Sigma

36. Ursa Labs Sponsors

37. Thank you!
https://arrow.apache.org
https://ursalabs.org