Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years.
Inspired by the increasingly complex SQL queries run by the Presto user community, engineers at Facebook and Starburst have recently focused on cost-based query optimization. In this talk we will present the initial design and implementation of the CBO, support for connector-provided statistics, estimating selectivity, and choosing efficient query plans. Then, our detailed experimental evaluation will illustrate the performance gains for several classes of queries achieved thanks to the optimizer. Finally, we will discuss our future work enhancing the initial CBO and present the general Presto roadmap for 2018 and beyond.
Speakers
Kamil Bajda-Pawlikowski, Starburst Data, CTO & Co-Founder
Martin Traverso
6. Presto in Production
Facebook: 1000s of nodes, HDFS (ORC, RCFile), sharded MySQL, 1000s of users
Uber: 800+ nodes (2 clusters on premises) with 200K+ queries daily over HDFS (Parquet/ORC)
Twitter: 800+ nodes (several clusters on premises) for HDFS (Parquet)
LinkedIn: 350+ nodes (2 clusters on premises), 40K+ queries daily over HDFS (ORC), 600+ users
Netflix: 250+ nodes in AWS, 40+ PB in S3 (Parquet)
Lyft: 200+ nodes in AWS, 20K+ queries daily, 20+ PBs in Parquet
Yahoo! Japan: 200+ nodes (4 clusters on premises) for HDFS (ORC), ObjectStore, and Cassandra
FINRA: 120+ nodes in AWS, 4PB in S3 (ORC), 200+ users
7. Built for Performance
Query Execution Engine:
● MPP-style pipelined in-memory execution
● Columnar and vectorized data processing
● Runtime query bytecode compilation
● Memory efficient data structures
● Multi-threaded multi-core execution
● Optimized readers for columnar formats (ORC and Parquet)
● Now also Cost-Based Optimizer
8. Evolving the optimizer - Challenges
● Diverse and widespread production workloads
● Fast-changing codebase
● Many developers
● Large surface area and usage of plan IR
9. Before
● Monolithic visitor-based plan transformations
● Visitors responsible for walking and transforming plan tree
● Problems
○ Hard to add new operations (IR node types)
○ Hard to add new optimizations
○ Hard to test optimizers
class LimitPushdown {
Plan optimize(Plan) { return plan.root.accept(this) }
Node visitLimit(LimitNode) { ... }
Node visitProject(ProjectNode) { ... }
Node visitFilter(FilterNode) { ... }
...
}
10.
11. Now
● Granular rule-based transformations
● Rules responsible for transforming localized subplan structure
● Optimizer loop responsible for walking plan and driving rule application
● Benefits
○ Decouples traversal from rule application
○ Decouples adding new optimizations from adding new operations (IR node types)
○ Easier to reason about and test individual rule behavior
class PushLimitThroughProjectRule {
Pattern getPattern() { Patterns.limit().with(source().matching(project())) }
Node apply(Node) { ... }
}
12.
13.
14.
15.
16. Migrating from monolithic optimizers to rules
● Fallback behavior
● Controlled via config option or
per-query session property
● Removed after a few releases
optimizers = [
RuleBasedOptimizer(
legacy = LimitPushdown,
rules = [
PushLimitThroughProject,
PushLimitThroughUnion,
PushLimitThroughJoin
]
),
PredicatePushdown,
PruneUnusedColumns,
AddExchanges,
EliminateCrossJoins,
...
]
17. Adding cost-aware optimizers
● Just another rule
● Can reason about cost
optimizers = [
RuleBasedOptimizer(
rules = [
PushLimitThroughProject,
PushLimitThroughUnion,
PushLimitThroughJoin,
ReorderJoins
]
),
...
]
class ReorderJoins {
ReorderJoins(CostComparator) { ... }
Pattern getPattern() { ... }
Node apply(Node) { ... }
}
18. CBO in a nutshell
Cost-Based Optimizer v1 includes:
● support for statistics stored in Hive Metastore
● join reordering based on selectivity estimates and cost
● automatic join type selection (repartitioned vs broadcast)
● automatic left/right side selection for joined tables
https://www.starburstdata.com/technical-blog/
19. Statistics & Cost
Hive Metastore statistics:
● number of rows in a table
● number of distinct values in a column
● fraction of NULL values in a column
● minimum/maximum value in a column
● average data size for a column
Cost calculation includes:
● CPU
● Memory
● Network I/O
Presto is known for raw performance, but…
… there’s only so much we can get without reducing algorithmic complexity of a query plan. Cost-aware optimizations are a step towards being able to handle a broader range of queries efficiently.
New behavior needs to be gated and there has to be a fallback to old behaviors
Changes have to be made as incrementally as possible
Big-bang changes cause developer churn (resolving conflicts, redoing work) and increase risk for production systems
Quick look at the history of the presto optimizer
Results from evaluating CBO on an internal FB query corpus drawn from production workloads.