11. Overview
impalad daemon runs on HDFS nodes
Queries run on "relevant" nodes
Supports common HDFS file formats
statestored, uses Hive metastore
(for database metadata)
12. Overview (cont'd)
Does not use Map/Reduce
Not fault tolerant !
(query fails if any query on any node fails)
Submit queries via Hue/Beeswax
Thrift API, CLI, ODBC, JDBC (future)
16. 9 queries, run in Impala Demo VM
Macbook Pro Retina, mid 2012
16GB RAM,
4GB for VM (VMWare 5),
Intel i7 2.6GHz quad-core processor
Hardware
No other load on system during queries
Pseudo-cluster + Impala daemons
17. Benchmarks (cont'd)
(from simple projection queries to
multiple joins, aggregation, multiple
predicates, and order by)
Impala vs. Hive performance
"TPC-DS" sample dataset
(http://www.tpc.org/tpcds/)
24. Query "G"
select
count(c.c_customer_sk)
from customer c
join customer_address ca
on c.c_current_addr_sk = ca.ca_address_sk
join customer_demographics cd
on c.c_current_cdemo_sk = cd.cd_demo_sk
where
ca.ca_zip in ('20191', '20194') and
cd.cd_credit_rating in ('Unknown', 'High Risk');
26. select
i_item_id,
s_state,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
from store_sales
join date_dim
on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
join item
on (store_sales.ss_item_sk = item.i_item_sk)
join customer_demographics
on (store_sales.ss_cdemo_sk = customer_demographics.cd_demo_sk)
join store
on (store_sales.ss_store_sk = store.s_store_sk)
where
cd_gender = 'M' and
cd_marital_status = 'S' and
cd_education_status = 'College' and
d_year = 2002 and
s_state in ('TN','SD', 'SD', 'SD', 'SD', 'SD')
group by
i_item_id,
s_state
order by
i_item_id,
s_state
limit 100;
Query "TPC-DS"
27. Query Hive (sec) # M/R jobs Impala (sec) x Hive perf.
A 12.4 1 0.21 59
B 30.9 1 0.37 84
C 29.6 1 0.33 91
D 22.8 1 0.60 38
E 22.5 1 0.52 44
F 66.4 2 1.56 43
G 83.0 3 1.33 62
H 66.1 2 1.50 44
TPC-DS 248.3 6 3.05 82
(remember, unscientific...)
34. Queries performed in-memory
Intermediate data never hits disk!
Data streamed to clients
C++
runtime code generation
intrinsics for optimization
Execution engine:
36. Metadata
Shares Hive metastore
Daemons cache metadata
Push to cluster via statestored
(scheduled for GA release)
Create tables in Hive
(then REFRESH impalad)
41. Current Limitations
(as of beta version 0.6)
No join order optimization
No custom file formats or SerDes or UDFs
Limit required when using ORDER BY
Joins limited by memory of single node
(at GA, aggregate memory of cluster)
42. Current Limitations
(as of beta version 0.6)
No advanced data structures
(arrays, maps, json, etc.)
No DDL (do in Hive)
Limited file formats (text, sequence
w/ snappy/gzip compression)
43. Future - GA & beyond...
Structure types (structs,
arrays, maps, json, etc.)
DDL support
Additional file formats &
compression support
Columnar format
(Parquet?)
"Performance"
Metadata
(via statestore)
JDBC
Join optimization
(e.g. cost-based)
UDFs
45. Dremel is a scalable, interactive ad-hoc
query system for analysis of read-only
nested data. By combining multi-level
execution trees and columnar data layout, it
is capable of running aggregation queries
over trillion-row tables in seconds. The
system scales to thousands of CPUs and
petabytes of data, and has thousands of
users at Google.
Comparing Impala to Dremel
- http://research.google.com/pubs/pub36632.html
46. Comparing Impala to Dremel
Impala = Dremel features circa 2010 + join
support, assuming columnar data format
(but, Google doesn't stand still...)
Dremel is production, mature
Basis for Google's BigQuery
47. Comparing Impala to Hive
Hive uses Map/Reduce -> high latency
Impala is in-memory, low-
latency query engine
Sacrifices fault tolerance for
performance
48. Comparing Impala to Others
Stinger
Apache Drill
Improve Hive performance (e.g. optimize execution plan)
Based on Dremel
In very early stages...
Support for analytics (e.g. OVER clause, window functions)
TEZ framework to optimize execution
Columnar file format
52. My Info
scott dot leberknight at nearinfinity dot com
twitter.com/sleberknight www.sleberknight.com/blog
www.nearinfinity.com/blogs/scott_leberknight/all/
scott dot leberknight at gmail dot com