Hadoop Performance Optimization at Scale, Lessons Learned at Twitter

L E S S O N S L E A R N E D AT T W I T T E R
H A D O O P P E R F O R M A N C E
O P T I M I Z AT I O N AT S C A L E
A L E X L E V E N S O N |
I A N O ' C O N N E L L |
@ T H I S W I L LW O R K
@ 0 X 1 3 8

DATA PLATFORM @TWITTER
Develop, maintain, and support the core data processing libraries used
at Twitter
In a good position to make system-wide performance improvements
Core Data Libraries Team

Idiomatic functional Scala library for writing Hadoop map reduce
Functional programming is a natural ﬁt for map reduce
Compile time type checked
github.com/twitter/scalding

Columnar storage format for the Hadoop ecosystem
Uses the Google Dremel column shredding and assembly algorithm
APACHE PARQUET
github.com/apache/parquet-mr

Streaming map reduce for hybrid realtime / batch topologies
Write once, execute in parallel on Storm / Heron (online) and Scalding
(ofﬂine)
SUMMINGBIRD
github.com/twitter/summingbird

Hadoop at Twitter Scale
H A D O O P AT T W I T T E R

100k MAP REDUCE
JOBS DAILY
MULTIPLES OF

1000+
MACHINE
HADOOP
CLUSTERS
MULTIPLE

LARGEST
HADOOP
CLUSTERS IN THE
WORLD
AMONG THE

At this scale, even small system-wide
improvements can save signiﬁcant
amounts of compute resources
C O S T AT S C A L E

What does your Hadoop cluster spend
most of its time doing?
W H AT T O I M P R O V E ?

Proﬁle your cluster, you might be
surprised by what you ﬁnd
M E A S U R E - D O N ' T G U E S S

ENABLE JVM PROFILING WITH -XPROF
Built into the JVM (HotSpot), so there's nothing to install
Xprof: a low overhead proﬁler built into the jvm
mapreduce.task.profile='true'
mapreduce.task.profile.maps='0-'
mapreduce.task.profile.reduces='0-'
mapreduce.task.profile.params='-Xprof'

ENABLE JVM PROFILING WITH -XPROF
Low overhead (uses stack sampling)
Surfaces the most expensive methods
Prints directly to task logs (stdout)
Xprof: a low overhead proﬁler built into the jvm

Flat profile of 412.48 secs (38743 total ticks): SpillThread
Interpreted + native Method
12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect
4.6% 0 + 822 java.io.FileOutputStream.writeBytes
...
19.4% 352 + 3082 Total interpreted (including elided)
Compiled + native Method
50.0% 8549 + 299 java.lang.StringCoding.decode
16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement
4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode
2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write
2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare
1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare
...
79.0% 13514 + 467 Total compiled
Thread-local ticks:
54.3% 21053 Blocked (of total)

HADOOP CONFIGURATION OBJECT
Looks and behaves a lot like a HashMap
Surprisingly expensive
Configuration conf = new Configuration()
conf.set("myKey", "myValue")
String value = conf.get("myKey")

Constructor reads + unzips + parses an XML ﬁle from disk
public class KryoSerialization {
public KryoSerialization() {
this(new Configuration())
}
}

get() method involves regular expressions, variable substitution
String value = conf.get("myKey")

Calling these methods in a loop, or once per record, is expensive
Some (non trivial) jobs were spending 30% of their time in
Conﬁguration methods

It's hard to predict what needs to be
optimized without a proﬁler
L E S S O N L E A R N E D

If you don't proﬁle, you could be missing
easy wins

Measure whether IO or CPU is your
biggest cost

INTERMEDIATE COMPRESSION
Xprof surfaced that compression + decompression in the spill thread
was taking a lot of time
Intermediate outputs are temporary
We now use lz4 instead of lzo level 3, which produces 30% larger
intermediate data that's faster to read
Made some large jobs 1.5X faster
Find the right balance

Record Serialization + Deserialization can
be the most expensive part of your job

Record Serialization is CPU intensive, and
may overshadow IO

How to reduce costs due to record
serialization?

USE HADOOP'S RAW COMPARATOR API
Hadoop MR deserializes the map output keys in order to sort them
between the map and reduce phases
Don't make sorting more expensive than it already is
deserialize(keyBytes1).compare(deserialize(keyBytes2))

This can cost a lot, especially for complex non-primitive keys, which is
fairly common
requests.groupBy { req => (req.country, req.client) }

This can cost a lot, especially for complex non-primitive keys, which is
fairly common
Complex object
that requires sorting
requests.groupBy { req => (req.country, req.client) }

Hadoop comes with a RawComparator API for comparing records in
their serialized (raw) form
deserialize(keyBytes1).compare(deserialize(keyBytes2))
compare(keyBytes1, keyBytes2)

Hadoop comes with a RawComparator API for comparing records in
their serialized (raw) form
public interface RawComparator<T> {
public int compare(byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2);
}

Unfortunately, this requires you to write a custom comparator by hand
And assumes that your data is actually easy to compare in its
serialized form
public interface RawComparator<T> {
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2);
}

SCALA MACROS FOR RAW COMPARATORS
Macros to the rescue!
A slightly more hipster API for Raw Comparators in Scala
And a handful of macros to generate implementations of this API for
tuples, case classes, thrift objects, primitives, Strings, etc.

1 3 f o o 0 1 17 1 88 ...
First, creates a custom dense serialization format that's easy to
compare
1 3 f o o 0 1 22 0 ... ...
non-null String
null value
non-null int non-null int
null value

1 3 f o o 0 1 17 1 88 ...
Then, creates a compare method that takes advantage of this format
1 3 f 0 o 0 1 22 0 ... ...

TotalComputeTime
Default Raw Comparators
1.5X
FASTER

COLUMN PROJECTION
Don't read or deserialize data that you don't need
struct User {
1: i64 id
2: Address address
3: string name
4 list<Interest> interests
}

COLUMN PROJECTION
Columnar ﬁle formats like Apache Parquet support this directly
Specialized record deserializers can skip over unwanted ﬁelds in row
oriented storage
Don't read or deserialize data that you don't need

APACHE PARQUET
Columnar storage for the people
In traditional row-oriented storage layout, an entire record is stored
sequentially
R1.A R1.B R1.C R2.A R2.B R2.C R3.A R3.B R3.C

APACHE PARQUET
In traditional row-oriented storage layout, an entire record is stored
sequentially
9903489083
"123 elm street"
"alice"
"columnar file formats"
9903489084
"333 oak street"
"bob"
"Hadoop"
Compressed with lzo / gzip / snappy

APACHE PARQUET
In columnar storage layout, an entire column is stored sequentially
R1.A R2.A R3.A R1.B R2.B R3.B R1.C R2.C R3.C

APACHE PARQUET
All user ids stored together
In columnar storage layout, an entire column is stored sequentially
9903489083
9903489084
9903489085
9903489075
9903489088
9903489087
"123 elm street"
"333 oak street"
"827 maple drive"

APACHE PARQUET
Schema aware storage can use specialized encodings
9903489083
9903489084
9903489085
9903489075
9903489088
9903489087
9903489083
+1
+1
-10
+3
-1
delta
"twitter.com/foo/bar"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com/123"
"twitter.com/foo/bar": 0
"blog.twitter.com": 1
"blog.twitter.com/123": 2
0
1
0
0
1
1
1
2
dictionary

FILE SIZE COMPARISON
SizeinGB
B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
2X
SMALLER
B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet

APACHE PARQUET
Collocating entire columns allows for efﬁcient
column projection
Read off disk only the columns you need
Possibly more importantly: deserialize only the
columns you need

TotalComputeTime
1 column 10 columns 40 columns
Parquet Lzo Thrift
COLUMN PROJECTION WITH PARQUET
3X
FASTER
1.5X
FASTER
1.15X
FASTER

APACHE PARQUET
Parquet is often slower to read all columns than row oriented
storage
Parquet is a dense format, read performance scales with the number
of columns in the schema -- nulls take time read
Sparse, row oriented formats (thrift) scale with the number of columns
present in the data -- nulls take no time read

COLUMN PROJECTION FOR ROW ORIENTED DATA
Row oriented is a very common way to store Thrift, Avro, Protocol
Buffers, etc.
Specialized record deserializers can skip over unwanted ﬁelds in these
row oriented storage formats
Prototype implemented as a Scala macro that creates a custom
deserializer at compile time
Don't deserialize data that you don't need

198 111 121 054 e l m _ s t r ... a l i c e ...
Decode User Id to Long
Skip over unwanted address ﬁeld
Decode Name to String

No IO savings
But only decodes the ﬁelds you care about into objects
CPU time spent decoding Strings can be huge compared to time it
takes to load + ignore the encoded bytes

TotalComputeTime
Number of Columns Selected
1 7 10 13 48
Parquet Thrift
Parquet Pig
Lzo Thrift + Projection
COLUMN PROJECTION: THRIFT VS. PARQUET
Parquet Thrift has a lot of
room for improvement
Parquet faster than row
oriented until 13 columns
This schema is relatively
ﬂat, and most columns
populated

APACHE PARQUET
Predicate push-down also allows parquet to skip
over records that don't match your ﬁlter criteria
Parquet stores statistics about chunks of records,
so in some cases entire chunks of data can be
skipped after examining these statistics

APACHE PARQUET
Combining both column projection and predicate push down is a
powerful combination

TotalComputeTime
Lzo Thrift Parquet + Filter Parquet + Filter + Project
FILTER PUSH DOWN WITH PARQUET
4.3X
FASTER

APACHE PARQUET
Predicate push-down performance depends on the nature of the
ﬁlter
Searching for rare records is the best case, entire chunks of
records are likely to not contain the records you are looking for

Key take aways
I N S U M M A R Y

IN SUMMARY
Key takeaways
Proﬁle!

IN SUMMARY
Key takeaways
Proﬁle!
Serialization is expensive, and Hadoop does a lot of it

IN SUMMARY
Key takeaways
Proﬁle!
Choose a storage format that ﬁts your access patterns

IN SUMMARY
Key takeaways
Proﬁle!
Use column projection

IN SUMMARY
Key takeaways
Proﬁle!
Sorting is expensive -- use Raw Comparators

IN SUMMARY
Key takeaways
Proﬁle!
Sorting is expensive -- use Raw Comparators
IO may not be your bottleneck -- more IO for less CPU may be a good
tradeoff

ACKNOWLEDGEMENTS
Thanks to everyone involved!
Dmitriy Ryaboy @squarecog
Gera Shegalov @gerashegalov
Julien Le Dem @J_
Katya Gonina @katyagonina
Mansur Ashraf @mansur_ashraf
Oscar Boykin @posco
Sriram Krishnan @krishnansriram
Tianshuo Deng @tsdeng
Zak Taylor @zakattacktaylor
And many more!

GET INVOLVED
Contributions always welcome!
github.com/twitter/scalding
github.com/twitter/algebird
github.com/twitter/chill
github.com/apache/parquet-mr

JOIN THE FLOCK
We're Hiring!
Work on data processing challenges at scale
Strong commitment to open source
jobs.twitter.com
Data Platform: (https://about.twitter.com/careers/positions?jvi=oipMYfwb,Job)

Q U E S T I O N S ?
A L E X L E V E N S O N |
I A N O ' C O N N E L L |
@ T H I S W I L LW O R K
@ 0 X 1 3 8

Hadoop Performance Optimization at Scale, Lessons Learned at Twitter

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Hadoop Performance Optimization at Scale, Lessons Learned at Twitter

Similar to Hadoop Performance Optimization at Scale, Lessons Learned at Twitter (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Hadoop Performance Optimization at Scale, Lessons Learned at Twitter