The document describes an in-memory data pipeline and warehouse using Spark, Spark SQL, Tachyon and Parquet. It involves ingesting financial transaction data from S3, transforming the data through cleaning and joining steps, and building a data warehouse using Spark SQL and Parquet for querying. Key aspects covered include distributing metadata lookups, balancing data partitions, broadcasting joins to avoid skew, caching data in Tachyon and Jaws for a RESTful interface to Spark SQL.

1. Radu Chilom
radu.chilom@gmail.com
In-memory data pipeline
and warehouse at scale
using Spark, Spark SQL,
Tachyon and Parquet
Buzzwords Berlin - 2015
Ema Iancuta
iorhian@gmail.com

2. ‹#›
• Big data analytics / machine learning
• 6+ years with Hadoop ecosystem
• 2 years with Spark
• http://atigeo.com/
• A research group that focuses on the technical
problems that exist in the big data industry and
provides open source solutions
• http://bigdataresearch.io/

3. ‹#›
• Intro
• Use Case
• Data pipeline with Spark
• Spark Job Rest Service
• Spark SQL Rest Service (Jaws)
• Parquet
• Tachyon
• Demo
Agenda

4. ‹#›
• Build an in memory data pipeline for millions
financial transactions used downstream by
data scientists for detecting fraud
• Ingestion from S3 to our Tachyon/HDFS
cluster
• Data transformation
• Data warehouse
Use Case

5. ‹#›
• “fast and general engine for large-scale
data processing”
• Built around the concept of RDD
• API for Java/Scala/Python (80 operators)
• powers a stack of high level tools including
Spark SQL, MLlib, Spark Streaming.
Apache Spark

6. ‹#›
Public S3 Bucket: public-financial-transactions
public-financial-
transactions
(s3-bucket)
scheme scheme.csv
data input-0.csv
data2
input-1.csv
. . .
. . .

7. ‹#›
• Download from S3
1. Ingestion
• Resolving the wildcards means listing files
metadata
• Listing the metadata for a large number
of files from external sources can take a
long time

8. ‹#›
Listing the metadata (distributed)
Driver
Worker Worker Worker
folder1
folder2
folder3
folder4
folder5
folder6
folder1
folder2
folder3
folder4
folder5
folder6
file-11
file-12
file-21
file-22
file-23
file-31
file-32
file-41
file-42
file-43
file-44
file-51
file-52
file-61

9. ‹#›
Listing the metadata (distributed)
• For fine tuning, specify the number of partitions

10. ‹#›
• Unbalanced partitions
Download Files

11. ‹#›
Unbalanced partitions
Partition 0
transactions.csv
Partition 1
input.csv
data.csv
values.csv
buzzwords.csv
buzzwords.txt

12. ‹#›
Balancing partitions
Partition 0
(0, transactions.csv)
(2, data.csv)
(4, buzzwords.csv)
Partition 1
(1, input.csv)
(3, values.csv)
(5, buzzwords.txt)

13. ‹#›
• Balancing partitions
Keep in mind that repartitioning your data is a
fairly expensive operation.
Balancing partitions

14. ‹#›
• Data cleaning is the first step in any data
science project
• For this use-case:
- Remove lines that don't match the structure
- Remove “useless” columns
- Transform data to be in a consistent format
2. Data Transformation

15. ‹#›
• Join
Find Country char code
Numeric Format Alpha 2 Format
276 DE
Name
Germany
• Problem with skew in the key distribution

16. ‹#›
Metrics for Join

17. ‹#›
• Broadcast Country Codes Map
Find Country char code

18. ‹#›
Metrics

19. ‹#›
Transformation with
Join vs Broadcasted Map
(skewed key)
Seconds
0
60
120
180
240
300
Rows
1 Million 2 Million 3 Million
Join Broadcasted Map

20. ‹#›
• Supports multiple contexts
• Launches a new process for each Spark context
• Inter-process communication with Akka actors
• Easy context creation & job runs
• Supports Java and Scala code
• Friendly UI
Spark-Job-Rest
https://github.com/Atigeo/spark-job-rest

21. ‹#›
• Hive
• Apache Pig
• Impala
• Presto
• Stinger (Hive on Tez)
• Spark SQL
Build a data warehouse

22. ‹#›
Spark SQL
• Support for multiple
input formats
• Rich language interfaces
• RDD-aware optimizer
RDD
DataFrame / SchemaRDD
JDBC
HIVE QL SQL

23. ‹#›
Creating a data frame

24. ‹#›
Perform a simple query:
Explore data
> Directly on the data frame
> Registering a temporary table
- select
- filter
- join
- groupBy
- agg
- join
- count
- sort
- where ..etc.

25. ‹#›
Creating a data warehouse
https://github.com/Atigeo/xpatterns-spark-parquet

26. ‹#›
• TextFile
• SequenceFile
• RCFile (RowColumnar)
• ORCFile (OptimizedRowColumnar)
• Avro
• Parquet
File Formats
> columnar format
> good for aggregation queries
> only the required columns are read from disk
> nested data structures
> schema with the data
> spark sql supports schema evolution
> efficient compression

27. ‹#›
Tachyon
• memory-centric distributed file system
enabling reliable file sharing at memory-speed
across cluster frameworks
• Pluggable underlayer file system: hdfs, S3,…

28. ‹#›
Caching in Spark SQL
• Cache data in columnar format
• Automatically compression tune

29. ‹#›
• spark context might crash
Spark cache vs Tachyon
• GC kicks in
• share data between different applications

30. ‹#›
- Highly scalable and resilient data warehouse
- Submit queries concurrently and asynchronously
- Restful alternative to Spark SQL JDBC having a
interactive UI
- Since Spark 091 with Shark
- Support for Spark SQL and Hive - MR (and more to
come)
https://github.com/Atigeo/jaws-spark-sql-rest
Jaws spark sql rest

31. ‹#›
- Akka actors to communicate through instances
- Support cancel queries
- Supports large results retrieval
- Parquet in memory warehouse
- returns persisted logs, results, query history
- provides a metadata browser
- configuration file to fine tune spark
Jaws main features

32. ‹#›
https://github.com/big-data-research/in-memory-data-pipeline
Code available at

33. ‹#›
Q & A

34. © 2013 Atigeo, LLC. All rights reserved. Atigeo and the xPatterns logo are trademarks of Atigeo. The information herein is for informational purposes only and represents the current view of Atigeo as of
the date of this presentation. Because Atigeo must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Atigeo, and Atigeo cannot guarantee the
accuracy of any information provided after the date of this presentation. ATIGEO MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.