Hive 0.13 ORC Tez with vectorization provides significant performance improvements over earlier versions of Hive for queries on large datasets. On a 1TB TPC-H dataset:
- Queries ran on average 6 times faster than with Hive 0.10 RCFiles
- 61% of queries completed in under 2 minutes (compared to over 7 minutes previously)
- Vectorized processing provided additional speedups, with some queries running over 1.5 times faster than without vectorization.
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
June 2014 HUG : Hive On Tez - Benchmarked at Yahoo Scale
1. Benchmarking Hive at Yahoo Scale
P R E S E N T E D B Y M i t h u n R a d h a k r i s h n a n ⎪ J u n e 1 8 , 2 0 1 4
H a d o o p U s e r G r o u p
2. About myself
2
HCatalog Committer, Hive
contributor
› Metastore, Notifications, HCatalog APIs
› Integration with Oozie, Data Ingestion
Other odds and ends
› DistCp
mithun@apache.org
Hadoop User Group, 201406181830, Yahoo Sunnyvale
3. About this talk
3
Introduction to “Yahoo Scale”
The use-case in Yahoo
The Benchmark
The Setup
The Observations (and, possibly, lessons)
Fisticuffs
Hadoop User Group, 201406181830, Yahoo Sunnyvale
4. The Y!Grid
4
16 Hadoop Clusters in YGrid
› 32500 Nodes
› 750K jobs a day
Hadoop 0.23.10.x, 2.4.x
Large Datasets
› Daily, hourly, minute-level frequencies
› Terabytes of data, 1000s of files, per dataset instance
Pig 0.11
Hive 0.10 / HCatalog 0.5
› => Hive 0.12
Hadoop User Group, 201406181830, Yahoo Sunnyvale
5. Data Processing Use cases
5 Hadoop User Group, 201406181830, Yahoo Sunnyvale
Pig for Data Pipelines
› Imperative paradigm
› ~45% Hadoop Jobs on Production Clusters
• M/R + Oozie = 41%
Hive for Ad hoc queries
› SQL
› Relatively smaller number of jobs
• *Major* Uptick
Use HCatalog for Inter-op
6. 6 Yahoo Confidential & Proprietary
Hive is Currently the Fastest Growing Product on the Grid
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
9.0%
10.0%
0
5
10
15
20
25
30
Mar-13 Apr-13 May-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14
HiveJobs(%ofAllJobs)
AllGridJobs(inMillions)
All Jobs Hive (% of all jobs)
2.4 million
Hive jobs
7. Business Intelligence Tools
7
{Tableau, MicroStrategy, Excel, … }
Challenges:
› Security
• ACLs, Authentication, Encryption over the wire, Full-disk Encryption
› Bandwidth
• Transporting results over ODBC
› Query Latency
• Query execution time
• Cost of query “optimizations”
• “Bad” queries
Hadoop User Group, 201406181830, Yahoo Sunnyvale
8. The Benchmark
8
TPC-h
› Industry standard (tpc.org/tpch)
› 22 queries
› dbgen –s 1000 –S 3
• Parallelizable
Reynold Xin’s excellent work:
› https://github.com/rxin
› Transliterated queries to suit Hive 0.9
Hadoop User Group, 201406181830, Yahoo Sunnyvale
16. 1 TB
16
› 6.2x speedup over Hive 0.10 (RCFile)
• Between 2.5-17x
› Average query time: 172 seconds
• Between 5-947 seconds
• Down from 729 seconds (Hive 0.10 RCFile)
› 61% queries completed in under 2 minutes
› 81% queries completed in under 4 minutes
Hadoop User Group, 201406181830, Yahoo Sunnyvale
21. 21 Hadoop User Group, 201406181830, Yahoo Sunnyvale
ORC File Layout
Data is composed of multiple streams
per column
Index allows for skipping rows (default to
every 10,000 rows), keeping position in
each stream, and min-max for each
column
Footer contains directory of stream
locations, and the encoding for each
column
Integer columns are serialized using run-
length encoding
String columns are serialized using
dictionary for column values, and the
same run length encoding
Stripe footer is used to find the requested
column’s data streams and adjacent
stream reads are merged File Footer
Postscript
Index Data
Row Data
Stripe Footer
256MBStripe
Index Data
Row Data
Stripe Footer
256MBStripe
Index Data
Row Data
Stripe Footer
256MBStripe
Column 1
Column 2
Column 7
Column 8
Column 3
Column 6
Column 4
Column 5
Column 1
Column 2
Column 7
Column 8
Column 3
Column 6
Column 4
Column 5
Stream 2.1
Stream 2.2
Stream 2.3
Stream 2.4
22. 22 Hadoop User Group, 201406181830, Yahoo Sunnyvale
ORC Usage
CREATE TABLE addresses (
name string,
street string,
city string,
state string,
zip int
)
STORED AS orc TBLPROPERTIES ("orc.compress"= "ZLIB");
LOCATION ‘/path/to/addresses’;
ALTER TABLE ... [PARTITION partition_spec] SET FILEFORMAT orc
SET hive.default.fileformat = orc
SET hive.exec.orc.memory.pool = 0.50 (ORC writer is allowed 50% of JVM heap size by default)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde’
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat’
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';
Key Default Comments
orc.compress ZLIB high-level compression (one of NONE, ZLIB, Snappy)
orc.compress.size 262,144 (256 KB) number of bytes in each compression chunk
orc.stripe.size 67,108,864 (64 MB) number of bytes in each stripe. Each ORC stripe is processed in one map task (try 32
MB to cut down on disk I/O)
orc.row.index.stride 10,000 number of rows between index entries (must be >= 1,000). A larger stride-size
increases the probability of not being able to skip the stride, for a predicate.
orc.create.index true whether to create row indexes. This is for predicate push-down. If data is frequently
accessed/filtered on a certain column, then sorting on the column and using index-filters
makes column filters work faster
25. Configuring ORC
25
set hive.merge.mapredfiles=true
set hive.merge.mapfiles=true
set orc.stripe.size=67,108,864
› Half the HDFS block-size
• Tangent: nStripes vs nBlocks
• Tangent: DistCp
set orc.compress=???
› Depends on size and distribution
› Snappy compression hasn’t been explored
YMMV
› Experiment
Hadoop User Group, 201406181830, Yahoo Sunnyvale
33. Sharky comments
33
Testing with Shark 0.7.x and Shark 0.8
› Compatible with Hive Metastore 0.9
› 100GB datasets : Admirable performance
› 1TB/10TB: Tests did not run completely
• Failures, especially in 10TB cases
• Hangs while shuffling data
• Scaled back to 100 nodes -> More tests ran through, but not completely
› nReducers: Not inferred
Miscellany
› Security
› Multi-tenancy
› Compatibility
Hadoop User Group, 201406181830, Yahoo Sunnyvale
Notes de l'éditeur
Gopal was supposed to be presenting this with me, to talk about Tez. Point to Gopal/Jitendra’s talk on Hive/Tez for details on things I’ll have to skim over.
Also, acknowledge Thomas Graves, who’s talking today about the excellent work he’s doing on driving Spark on Yarn.
There are several sides to query latency:
Query execution time : Addressed in the physical query-execution layer.
Query optimizations: The first step while optimizing the query plan seems to be to query for all partition instances. Very expensive for “Project Benzene”.
Bad queries : Tableau, I’m looking at you.
The Transaction Processing Performance Council (inexplicably abbreviated to TPC) suggests a set of benchmarks for query processing. Many have adopted TPC-DS to showcase performance. We chose TPC-h to complement. (Also, 22 much smaller number to deal with than… 90?)
Transliteration: Evita and Kylie Minogue
Lineitem and Orders are extremely large Fact tables. Nation and Region are the smallest dimension tables.
Tangent: Funny story:
1. About hard-drives: Can set up MR intermediate directories and HDFS data-node directories to be on different disks. Traffic from one doesn’t affect the other. But on the other hand, total read bandwidth might be reduced.
Line-item: Partitioned on Ship-date.
Orders: Order-date
Customers: By market-segment
Suppliers: On their region-key.
Q5 and q21 are anomalous.
Q21: Hit a trailing reducer across all versions of Hive tested. Perhaps this can be improved with a better plan.
Q5: Slow reducer that hit only Hive 13. Could be a bad plan. Could be a difference in data distribution when data was regenerated for Hadoop 2 cluster.
Tez : Scheduling. Playing the gaps, like Beethoven’s Fifth.
Vectorization: On average: 1.2x.
Except for a few outliers, ZLIB compression actually reduced performance for a 1TB dataset. Uncompressed was 1.3x faster than Compressed.
The situation reverses at the 10 TB level. The gains from decompression are actually offset by the disk-read time.
The long-tail in 10TB/q21 threw the scale of the graph off, so I’ve excluded it in the results.
Talk about file-coalesce, small-file generation, Namenode pressure and parallelism.
You don’t want to read an ORC stripe from a different node.
Talk about distcp –pgrub, for ORC files.
Mention that SNAPPY’s license is not Apache.
Also, Yoda.
At 100 nodes, it performs at 0.9x the 350 node performance.
We’ve seen Hive and Tez scale down for latency, scale up for data-size, and scale out across larger clusters.
Familiarity : We have an existing ecosystem with Hive, HCatalog, Pig and Oozie that delivers revenue to Yahoo today. It’s hard to rock the boat.
Community: The Apache Hive community is large, active and thriving. They’ve been solving issues with query latency for ages now. The switch to using the Tez execution engine was a solution within the Apache Hive project. This wasn’t a fork of Hive. This is Hive, proper.
Scale: We’ve seen Hive and Tez perform at scale. Heck, we’ve seen Pig perform on Tez.
Multitenant: Yahoo’s use-case is unique, and not just because of data-scale. There’s hundreds of active users and genuine multitenancy and security concerns.
Design: We think the Hive community has tackled the right problems first, rather than throw RAM at the problem.
Bucky Lasek at the X-Games in 2001. Notice where he’s looking… Not at the camera, but setting up his next trick.
Security: Kerberos support was patched in, after the benchmarks were run.
Multi-tenancy:
Data needs to be explicitly pinned into memory as RDDs.
In a multi-tenant system, how would pinning work? Eviction policy for data.
Compatibility:
Needs to work with Metastore versions 12 and 13. Shark’s gone to 0.11 just recently.
Integration with the rest of the stack: Oozie and Pig.
Overall, we wanted a solution that works with high-dynamic range. i.e. works well with small datasets (100s of GBs), as well as scale to multi-terabyte datasets. We have a familiar system that seems to fit that bill. It doesn’t quite rock the boat. It’s not perfect yet. There are bugs that we’re working on. And we still haven’t solved the problem of data-volume/BI.
By the way, I really like the idea of BlinkDB. I saw the JIRA.