2. DREMIO
Who
Wes McKinney
• Engineer at Cloudera, formerly
DataPad CEO/founder
• Wrote bestseller Python for
Data Analysis 2012
• Open source projects
– Python {pandas, Ibis,
statsmodels}
– Apache {Arrow, Parquet, Kudu
(incubating)}
• Mostly work in Python and
Cython/C/C++
Jacques Nadeau
• CTO & Co-Founder at
Dremio, formerly Architect
at MapR
• Open Source projects
– Apache {Arrow, Parquet,
Calcite, Drill, HBase,
Phoenix}
• Mostly work in Java
3. DREMIO
Arrow in a Slide
• New Top-level Apache Software Foundation project
– Announced Feb 17, 2016
• Focused on Columnar In-Memory Analytics
1. 10-100x speedup on many workloads
2. Common data layer enables companies to choose best of breed
systems
3. Designed to work with any programming language
4. Support for both relational and complex data as-is
• Developers from 13+ major open source projects involved
– A significant % of the world’s data will be processed through Arrow!
Calcite
Cassandra
Deeplearning4j
Drill
Hadoop
HBase
Ibis
Impala
Kudu
Pandas
Parquet
Phoenix
Spark
Storm
R
6. DREMIO
Overview
• A high speed in-memory representation
• Well-documented and cross language
compatible
• Designed to take advantage of modern
CPU characteristics
• Embeddable in execution engines, storage
layers, etc.
8. DREMIO
High Performance Sharing & Interchange
Today With Arrow
• Each system has its own internal
memory format
• 70-80% CPU wasted on serialization
and deserialization
• Similar functionality implemented in
multiple projects
• All systems utilize the same memory
format
• No overhead for cross-system
communication
• Projects can share functionality (eg,
Parquet-to-Arrow reader)
Pandas Drill
Impala
HBase
KuduCassandra
Parquet
Spark
Arrow Memory
Pandas Drill
Impala
HBase
KuduCassandra
Parquet
Spark
Copy & Convert
Copy & Convert
Copy & Convert
Copy & Convert
Copy & Convert
9. DREMIO
Shared Need > Open Source Opportunity
• Columnar is Complex
• Shredded Columnar is even
more complex
• We all need to go to same
place
• Take Advantage of Open
Source approach
• Once we pick a shared
solution, we get interchange
for “free”
“We
are
also
considering
switching
to
a
columnar
canonical
in-‐memory
format
for
data
that
needs
to
be
materialized
during
query
processing,
in
order
to
take
advantage
of
SIMD
instrucBons”
-‐Impala
Team
“A
large
fracBon
of
the
CPU
Bme
is
spent
waiBng
for
data
to
be
fetched
from
main
memory…we
are
designing
cache-‐friendly
algorithms
and
data
structures
so
Spark
applicaBons
will
spend
less
Bme
waiBng
to
fetch
data
from
memory
and
more
Bme
doing
useful
work
–
Spark
Team
18. DREMIO
Java: Memory Management (& NVMe)
• Chunk-based managed allocator
– Built on top of Netty’s JEMalloc implementation
• Create a tree of allocators
– Limit and transfer semantics across allocators
– Leak detection and location accounting
• Wrap native memory from other applications
• New support for integration with Intel’s Persistent
Memory library via Apache Mnemonic
20. DREMIO
Common Message Pattern
• Schema Negotiation
– Logical Description of structure
– Identification of dictionary
encoded Nodes
• Dictionary Batch
– Dictionary ID, Values
• Record Batch
– Batches of records up to 64K
– Leaf nodes up to 2B values
Schema
NegoBaBon
DicBonary
Batch
Record
Batch
Record
Batch
Record
Batch
1..N
Batches
0..N
Batches
21. DREMIO
Record Batch Construction
Schema
NegoBaBon
DicBonary
Batch
Record
Batch
Record
Batch
Record
Batch
name
(offset)
name
(data)
iq
(data)
addresses
(list
offset)
addresses.number
addresses.street
(offset)
addresses.street
(data)
data
header
(describes
offsets
into
data)
name
(bitmap)
iq
(bitmap)
addresses
(bitmap)
addresses.number
(bitmap)
addresses.street
(bitmap)
{
name:
'wes',
iq:
180,
addresses:
[
{number:
2,
street
'a'},
{number:
3,
street
'bb'}
]
}
Each
box
is
conBguous
memory,
enBrely
conBguous
on
wire
22. DREMIO
RPC & IPC: Moving Data Between Systems
RPC
• Avoid Serialization & Deserialization
• Layer TBD: Focused on supporting vectored io
– Scatter/gather reads/writes against socket
IPC
• Alpha implementation using memory mapped files
– Moving data between Python and Drill
• Working on shared allocation approach
– Shared reference counting and well-defined ownership
semantics
24. DREMIO
Real World Example: Python With Spark or Drill
in partition 0
…
in partition
n - 1
SQL Engine
Python
function
input
Python
function
input
User-supplied
Python code
output
output
out partition 0
…
out partition
n - 1
SQL Engine
25. DREMIO
Real World Example: Feather File Format for
Python and R
• Problem: fast, language-
agnostic binary data
frame file format
• Written by Wes
McKinney (Python)
Hadley Wickham (R)
• Read speeds close to
disk IO performance
Arrow array 0
Arrow array 1
…
Arrow array n
Feather
metadata
Feather file
Apache Arrow
memory
Google
flatbuffers
26. DREMIO
Real World Example: Feather File Format for
Python and R
library(feather)
path
<-‐
"my_data.feather"
write_feather(df,
path)
df
<-‐
read_feather(path)
import
feather
path
=
'my_data.feather'
feather.write_dataframe(df,
path)
df
=
feather.read_dataframe(path)
R
Python
27. DREMIO
What’s Next
• Parquet for Python & C++
– Using Arrow Representation
• Available IPC Implementation
• Spark, Drill Integration
– Faster UDFs, Storage interfaces
28. DREMIO
Get Involved
• Join the community
– dev@arrow.apache.org
– Slack:
https://apachearrowslackin.herokuapp.com/
– http://arrow.apache.org
– @ApacheArrow, @wesmckinn, @intjesus