Arm Treasure Data utilizes Presto as the query engine processing over 1 million queries per day to support the data business of 500+ companies in three regions; US, EU, and Asia. Arm Treasure Data had been using Presto 0.205 and in 2019 started a big migration project to Presto 317. Although we performed extensive query simulations to check any incompatibilities, we faced many unexpected challenges during the migration at production.

1. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Taro L. Saito
Arm Treasure Data
July 16th, 2020
Starburst Presto Webinar
Presto At Arm Treasure Data:
Journey of Migrating
1 Million Presto Queries
1

2. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
About Me: Taro L. Saito (Leo)
2
● Principal Software Engineer of
Arm Treasure Data
● Ph.D. of Computer Science
● Active OSS developer
● Created Presto As A Service
● 6 years since 2014

3. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017
Thank You!
Danke!
Merci!
谢谢!
ありがとう!
Gracias!
Kiitos!400+
Customers
Founded in
2011
Raised
$54M
Security
Acquired by Arm / Softbank
2018
Arm Treasure Data

4. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Customer Data Platform (CDP) Solution
4

5. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Arm Treasure Data: Ready-to-Use Cloud Data Platform From Day 1
5
Logs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS
Data

6. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Data Connectors And Integrations
6

7. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Our Contributions to Presto
7

8. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Presto Usage Stats (2020): 1 Million Queries / Day
● November 2013
● Presto was open-sourced
● July 2014
● Presto service launch
● 2015
● 20,000 queries / day
● 2019
● 3,500 users
● 600,000 queries / day
● 2020
● 5,000+ users
● 1,000,000 queries / day
● 100 trillion rows processed /
day
■ 1.2 billion rows / sec.
8

9. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
A Challenge: Every Presto Upgrade Is A Big Migration
● 2014
● Presto 0.64 ~
● 2015
● Presto 0.113 ~
● 2016
● Presto 0.152 ~
● 2017
● Presto 0.178 ~
● 2018
● Presto 0.205
■ Used only this version
● 2019-2020
● Presto 0.205 to 317
● Only 1 customer is remaining
at Presto 0.205 (July 2020)
● Migration took almost 1 year
9
Very Frequent
Every 1-2 weeks
Less Frequent
Presto Upgrade
Storage Upgrade
(e.g., format change)
Customer
Queries
Customer
Queries
Migration requires human actions
(e.g., Fixing incompatible queries,
scheduling migrations,
performance tuning, etc.)
Limiting Factor

10. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Migration: Why Presto 317?
● 2018
● We were using Presto 0.205
● 2019
● Presto Software Foundation (prestosql) was founded => Presto 300
● Tested Presto 311, 316, and 317 (The latest version on Aug 2019)
● Presto 318 had an incorrect query result bug, so we chose 317
■ The bug was fixed in 319
● Criteria to choose a Presto version
● Less query correctness issues
● Stability
■ Less major GCs and memory related bugs
● Less number of incompatible changes
■ SQL syntax changes, query behavior changes.
10

11. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Presto Conference Tokyo 2019 (July 11, 2019)
● Held At Arm Treasure Data Tokyo Office (6F)
● Invited 3 Presto Creators (Martin, Dain, and David)
● Conference Report: https://prestosql.io/blog/2019/07/11/report-for-presto-conference-tokyo.html
● Presto: Current And Future
● How to Read the Source Code of Presto, etc.
● After this conference, we started Presto migration
● Presto 317 was released on Aug 2019
11

12. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Presto Queries Form Complex Dependency Graphs
● A real customer example
● 2000 query patterns, 5000 queries in a day
● Boxes: input tables, edges: queries, red circles: external targets
●
12

13. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Collecting Query Logs
● Collecting all Presto query logs since 2015
● Query statements, performance statistics, logs, etc.
■ QueryCreation, QueryCompletion, SplitCompletion events
● TD is one of the biggest users of TD itself
● Using Presto for optimizing Presto
13
Logs
User
Query
Collect Query Logs
Analyze Query Logs
Machine
Learning
Query
Optimization
Optimize System

14. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Usages of Query Logs
● Estimate Query Service-Level Objectives (SLOs)
● Query Running Time
● Query Resource Usage
● Replay Queries for Simulation
● Creating Customer-Specific Benchmark
● Problem investigation
● Finding heavy queries, performance issues, job failures, etc.
14

15. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Monitoring Service Level Objectives (SLOs)
● Estimating query finishing time using the historical query stats of last 7 days
● Using p50 and MAD (median absolute deviation)
● SLO Violation: Query running time > p50 + 3MAD
15

16. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Query Clustering
● Query Signature
16

17. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Queries With The Same Query Signature
17

18. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Query Clusters
● Using query signatures as keys for clustering queries
● For creating better clusters:
● Include not only query structures, but also input and output table names
● Replace (table_name)_(date strings) patterns into (table_name)_N
18

19. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Query Simulator: Testing Customer-Specific Queries
● Testing all 1M query / day = 30M queries / month is impractical
● Standard benchmarks like TPC-H/TPC-DS don’t have sufficient query coverage.
● Clustering queries
● Reduced the number of queries into 100,000 query patterns
● Reducing query size by narrowing input table data ranges
● Running production-scale queries can be expensive
19
User
QueryUser
QueryUser
QueryUser
QueryUser
QueryUser
Query
30,000,000
queries
clustering
Query
SigQuery
SigQuery
SigQuery
Sig
minimize
Small
QuerySmall
QuerySmall
QuerySmall
Query
100,000 query
patterns
100,000 small
queries
simulate
queries
simulation
results and
stats

20. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Query Coverage
● Query Pattern Coverage
● SQL syntax compatibility test coverage: Almost 100%
● Reporting necessary SQL fixes in advance to customers
20

21. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Testing SELECT Queries
● Problems:
● Slow query result transfer in JSON from Presto Coordinator to Client (e.g.,
presto-jdbc)
● Employees can’t see customer data
● Our workaround
● Run SELECT queries in two steps:
■ Check the result schema with:
○ SELECT * from (original query) limit 0
■ Rewrite the original SQL to compute checksum (order-insensitive)
○ SELECT checksum(col_1), checksum(col_2), … from (original query)
● Report only the final checksum
■ Presto consumes all result rows (no row transfer)
● Compare only the checksum between Presto versions
21

22. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Scheduling Migration
● Reporting All Breaking Changes In Advance
● Incompatible SQL queries
● Customers needed to modify their queries and
workflows
● Providing Grace Period for Migration
● Supported early testing of the next Presto version
● Using a magic comment within SQL for specifying a
Presto version to use
● Not only our customers, but also their partner
companies are involved
● Partner companies might be writing SQL queries on
behalf of customers
● Consider scheduling migrations of 400+ customers
and their partners!
22

23. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
No Migration At Holiday and Event Seasons
● US
● July 4th
● November
■ Thanksgiving, Black Friday, Cyber Monday
● Japan
● May 3 - May 5
■ Golden Week
● Week of August 15th
■ Obon: Japanese tradition to visit home town
● Long winter holiday
■ Weeks around the year-end and new-year start
● We also need to avoid changes at the end of month
● Most of the companies have important monthly jobs
■ KPI, monthly revenue reporting, etc.
23

24. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Queries Difficult To Simulate
● Our query simulation coverage is still around 80%
● Queries involving temporary tables
● The data might be missing
● Queries updating production tables
● DELETE (TD supports record-level deletion)
■ Need to create a copy of the data in a secure manner
● CREATE TABLE AS (CTAS), INSERT INTO
■ Using a hidden database inside customer’s account so as not to leak any
data outside the customer account.
● Long running queries
● Some queries keep running for more than 1 day
■ 2 days are the current maximum
24

25. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Testing Workflow Queries
● Often uses a lot of temporary tables
● Need to maintain snapshots of partitions and table schema
● Workflow involves varieties of tasks
● Presto, Hive queries, Python, Spark codes, etc.
● Importing data from external sources, etc.
● There is almost no way to test the entire workflows
25

26. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Integration Testing
● Model class changes required presto client upgrades
● Query stats model class changes
● Query plan model class changes
● Job queue and reporting system can be broken
● presto-client-ruby
● Embedding multiple model classes of both Presto 0.205 and 317
● Query Error Retry Pattern Changes
● Not only Presto’s error code, we are retrying queries using error message patterns
■ e.g., OOM, Java exception message patterns, system, JVM crashes, etc.
● If the error message changes, we need to add more error patterns.
26

27. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Production Scale Query Testing Is Also Necessary
● Example: UDF Performance Regression
● Example: TD_TIME_PARSE(time_string)
■ Parse date time strings and convert them into unixtime
● We introduced a date string cache for efficiency, but the cache size was too small
for a query calling this UDF billions of times for wide-range of data
● The query slowed down from 3 minutes -> 40 minutes
■ Because of too many cache misses
● Fixed this issue by increasing the cache size
● A Lesson
● Even if 99.9999% of queries are ok, it’s not 100% yet
■ 1 query out of 1,000,000 queries = 0.0001%
■ We still need to check data-specific performance characteristics
27

28. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Testing Optimization Rules
● After migrating to Presto 317, we noticed some optimization was not working
● Time-index pushdown (TD-Specific)
● Extract condition related to time column, and use this tuple domain for removing
unnecessary partition scans
● Eager evaluation of nested UDFs (CURRENT_DATE, CURRENT_TIMESTAMP) are
necessary to extract time ranges
● However, Presto 317 does not evaluate cast wraps in advance
■ => Failed to extract the time range and resulted in the full table scans
● We added check rules to see how many partitions are scanned
28

29. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Testing Presto-PostgreSQL connector
● DataTank service
● Hosted PostgreSQL inside Treasure Data
● Accessing TD and PostgreSQL’s data using Presto
● Problem
● There is no way to run any simulation because PostgreSQL doesn’t isolate
compute and storage. Any test query affects the other queries.
● Some customers are using this connector a lot
● An issue: Connecting to PostgreSQL more than hundreds of times inside a single
Presto query
● Caused connection failures in Presto 317
■ We need to introduce retry with Jittering for acquiring JDBC connections
○ https://github.com/prestosql/presto/pull/3125
● Workaround: Increased the max connection parameter of PostgreSQL
29

30. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Operating Multiple Presto Versions
● We should reduce the version differences as much as possible
● For cost-efficiency (Multi-tenancy)
● And also, because no cloud has unlimited scale.
● Our Deployment: Blue-Green Deployment
● Preparing two versions of clusters, and gradually migrating customers from the
old version to the new version
● If we have 100-node cluster, we need 200 instances for blue-green deployment
● We encountered a lot of AWS insufficient capacity errors
● We are using hundreds of r4.8xlarge (240GB, 32 core) instances
● Depending on the time in the day, we can’t create 100+ instances
● We also wanted to upgrade to r5.8xlarge (next generation hypervisor), but failed
because AWS doesn’t have so many r5 instances in US/JP/EU regions...
30

31. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Configuration Changes
● Good
● Presto 317 no longer uses the reserved memory pool
■ If the reserved pool is used, only a single query can use large memory and
blocks the other queries
● General performance improvement and new UDFs (e.g., format function)
● Several unexpected bugs and behaviors
● Cannot list/kill QUEUED queries. We noticed it after the upgrade.
● Backported fixes from Presto 319
● More strict queuing and resource control
● Some customer experienced longer queued time of queries
● We needed to perform peak-time simulation to see the number of concurrent
queries (CQs) and their queued time changes
31

32. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Engineering Challenges
● Debugging Asynchronous Calls
● S3 read and write
● Transaction processing
● Async call is necessary for maximizing the I/O throughput, but reduce the code
readability
● A Race Condition Bug
● Regular updates
■ Read S3 -> uploading new partitions to S3 -> registering partition entries to
PlazmaDB (PostgreSQL) -> committing new partitions
● We had a race condition issue around Java’s Future task management involving
RDBMS transaction, and the above update order was not ensured
■ Failed to upload some partitions (partially missing update results)
● Hard to find such a bug with peer-review, and it was hard to reproduce
■ We need to add random-delays to such critical paths
32

33. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Next Presto Migration Plans
● Service limitation changes
● Max query running time: unlimited -> 2 days -> 4 hours (ideal)
■ For enabling quick JVM/cluster refresh and load balancing
● Changing query max memory size for reducing the number of query_max_memory
errors
● Such limitation changes require query log analysis and simulations
● Next Migration Plans
● Upgrading from Java 8 to Java 11
● Using Gravition2 instances (Arm64 CPU)
● Presto 338 or later
● Active Development
● Presto Query Simulator
■ Registering query sets, simulation, and reporting
● Customer-specific benchmark generator from query logs
33

34. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Arm Graviton 2
● Arm64 instances on AWS (r6g, m6g, etc.), available since June 2020
● Experimental Results By Kai Sasaki
○ https://blog.treasuredata.com/blog/2020/03/27/high-performance-sql-aws-gravito
n2-benchmarks-with-presto-and-arm-treasure-data-cdp/
30 Percent Faster,
20 Percent Cheaper,
Up to 50 Percent
Better ROI
34

35. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Summary
● Migration of 1,000,000 queries revealed many challenges including:
● Compatibility tests
● Performance regression tests
■ Optimizer, production-scale and peak-time tests are necessary
■ How to find 0.0001% of regression?
● Difficult types of queries to test
■ Temporary tables, workflows, presto-postgresql connector
● Collecting query logs is important
● Without data, no migration can be supported
● Query clustering, SLO management, etc.
35