Lightning talk presented at HPTS 2015: http://hpts.ws/ Apache Parquet is the de facto standard columnar storage for big data. Open source and proprietary SQL engines already integrate with it as their users don’t want to load and duplicate their data in every tool. Users want an open, interoperable, efficient format to experiment with the many options they have. The format is defined by the open source community integrating feedback from many teams working on query engines (including but not limited to Impala, Drill, Hawq, SparkSQL, Presto, Hive, etc) or on infrastructure at scale (Twitter, Netflix, Stripe, Criteo, ...). Building on its initial success, the Parquet community is defining new features for the next iteration of the format. For example: improved metadata layout, type system completude or mergeable statistics used for planning.

1. © 2015 Dremio Corporation
If you have your own Columnar format,
stop now and use Parquet
😛
Julien Le Dem
Principal Architect, Dremio
VP Apache Parquet

2. © 2015 Dremio Corporation
About Dremio
Jacques Nadeau
Founder & CTO
•Apache Drill PMC Chair
•Recognized SQL & NoSQL expert
•Quigo (AOL); Offermatica (ADBE);
aQuantive (MSFT)
Tomer Shiran
Founder & CEO
•Apache Drill Founder
•MapR (VP Product); Microsoft; IBM
Research
•Carnegie Mellon, Technion
Julien Le Dem
Architect
•Apache Parquet Founder
•Apache Pig PMC Member
•Twitter (Lead, Analytics Data Pipeline);
Yahoo! (Architect)
Top Silicon Valley VCs• Enabling self-service data discovery, exploration and
analysis on modern data
• Founded in June 2015
• Building on open source technologies including Drill,
Parquet, Spark

3. © 2015 Dremio Corporation
Background of Parquet
• Twitter’s data
• Lots of data: Instrumentation, User graph, Derived data, ...
• Complex: deeply nested structures
• Analytics infrastructure:
• Several 1000s nodes Hadoop clusters
• Log collection to HDFS in Thrift
• Parquet
• Columnar: space and query efficient
• Inspired from the Google Dremel Paper
• supports complex data
• interoperable
Caillebotte: The Parquet Planers

4. © 2015 Dremio Corporation
Parquet timeline
• Fall 2012: Twitter & Cloudera’s Impala team
merge efforts to develop columnar formats.
• March 2013: OSS announcement; Criteo
signs on for Hive integration.
• July 2013: 1.0 release. 18 contributors from
more than 5 organizations.
• August 2013: Drill chose Parquet as its
primary storage format.
• May 2014: Apache Incubator. 40+
contributors, 18 with 1000+ LOC. 26
incremental releases.
• Apr 2015: Parquet graduates from the
Apache Incubator.

5. © 2015 Dremio Corporation
What does Parquet do?
Interoperability
Space efficiency
Query efficiency
@EmrgencyKittens

6. © 2015 Dremio Corporation
Columnar storage
Logical table
representation
Row layout
Column layout
Nested schema
a b c
a b c
a1 b1 c1
a2 b2 c2
a3 b3 c3
a4 b4 c4
a5 b5 c5
a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5
a1 b1 c1a2 b2 c2a3 b3 c3a4 b4 c4a5 b5 c5
encoded chunk encoded chunk encoded chunk
On Disk:
Encodings: Dictionary, RLE, Delta, Prefix

7. © 2015 Dremio Corporation
Nested representation
Document
DocId Links Name
Backward Forward Language Url
Code Country
Columns:
docid
links.backward
links.forward
name.language.code
name.language.country
name.url
Schema:
Borrowed from the Google Dremel paper
https://blog.twitter.com/2013/dremel-made-simple-with-parquet

8. © 2015 Dremio Corporation
Statistics
Vertical partitioning
(projection push down)
Horizontal partitioning
(predicate push down)
Read only the data
you need!
+ =
a b c
a1 b1 c1
a2 b2 c2
a3 b3 c3
a4 b4 c4
a5 b5 c5
a b c
a1 b1 c1
a2 b2 c2
a3 b3 c3
a4 b4 c4
a5 b5 c5
a b c
a1 b1 c1
a2 b2 c2
a3 b3 c3
a4 b4 c4
a5 b5 c5
+ =

9. © 2015 Dremio Corporation
Interoperability
Library level integration Query engine integration
Avro Thrift
Protocol
Buffer
Pig Tuple Hive SerDe
Assembly/striping (model agnostic)
Parquet file format (language agnostic)
Object model
parquet-avroConverters parquet-thrift parquet-proto parquet-pig parquet-hive
Column encodings
Impala
...
...
Encodings (C)
PrestoDrill …

10. © 2015 Dremio Corporation
Query engines, frameworks and libraries
integrated with Parquet (non exhaustive)
Query engines:
Hive, Impala, HAWQ,
IBM Big SQL, Drill, Tajo,
Pig, Presto, SparkSQL
Frameworks:
Spark, MapReduce, Cascading,
Crunch, Scalding, Kite
Data Models:
Avro, Thrift, ProtocolBuffers,
POJOs

11. © 2015 Dremio Corporation
Loose coupling
• Users don’t want to load
their data into every tool.
• Many tools are available
and show up every day.
• The cost of trying a new
tool should be minimal
Storage (HDFS/S3/…)
Interactive
queries
(Drill, Impala,
Presto, …)
automated
dashboard
machine
learning
Query-efficient
format
Parquet
Graph
Processing
(Giraph, …)
Batch
computation
(Pig, Cascading,
Scalding, Spark,
…)

12. © 2015 Dremio Corporation
Get involved
Twitter:
- @ApacheParquet
Mailing list:
- dev@parquet.apache.org
Github repo:
- https://github.com/apache/parquet-mr
Parquet sync ups:
- Regular meetings on google hangout