Optimising Your Amazon Redshift Cluster for Peak Performance In this session we take an in-depth look at the latest features in Amazon Redshift, including analysing data store in and outside of your cluster with Amazon Redshift Spectrum, query and platform enhancements, and more. We will dive deep into best practices on how to design optimal schemas, load data efficiently, and optimise your queries to deliver high throughput and performance. Eric Ferreira , Principal Database Engineer, Amazon Web Services

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Eric Ferreira
Principal Engineer, Amazon Redshift
Optimising Your Amazon Redshift
Cluster For Peak Performance

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Recent Features
Timeless Best Practices
Additional Resources

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Are You An Amazon Redshift User?

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PostgreSQL
Columnar
MPP
OLAP
AWS IAMAmazon VPCAmazon SWF
Amazon S3 AWS KMS
Amazon
Route 53
Amazon
CloudWatch
Amazon EC2
Amazon Redshift

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
February 2013
April 2018
> 125 Significant Patches
> 165 Significant Features
Significant Features & Patches

6. Load
Unload
Backup
Restore
Massively parallel, shared nothing columnar
architecture
Leader node
• SQL endpoint
• Stores metadata
• Coordinates parallel SQL processing
Compute nodes
• Local, columnar storage
• Executes queries in parallel
• Load, unload, backup, restore
Amazon Redshift Spectrum nodes
• Execute queries directly against
Amazon Simple Storage Service
(Amazon S3)
SQL Clients/BI Tools
128GB RAM
16TB disk
16 cores
JDBC/ODBC
128GB RAM
16TB disk
16 coresCompute
Node
128GB RAM
16TB disk
16 coresCompute
Node
128GB RAM
16TB disk
16 coresCompute
Node
Leader
Node
Amazon S3
...
1 2 3 4 N
Amazon
Redshift
Spectrum
Load
Query
Amazon Redshift Architecture

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Think: Toaster
• You submit your workload
• Choose a few options
• It runs fast and cheap

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recent Features

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Short Query Acceleration
• New queue (#14) can be enabled via the console or API
• Total Concurrency should be less than 15
• Adds 6 new slots and equivalent query processors
• Only active if there are queries waiting in queue
• Machine learning algorithm picks queries that are eligible
• 3x throughput improvement on short query workload with
minimal effect on long running queries.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results Caching
• Sub-second response time for repeat queries
• Fully ACID (we take care of invalidating the cache if data
changes)
• No queueing
• Size proportional of your node type
• New View SVL_QLOG to get information of both regular
and cached queries.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Late Materialisation
• Further reduces I/O on scanning tables when using
multiple predicates
• In some cases allows for compressing Sort Keys without
performance degradation on scans.
• Fully automatic
• New column (is_rlf_scan) on STL_SCAN

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Improved Commit Speeds
• Read-Only transactions batch checkpoints
• Selects and CTAS/DML on Temporary objects
• Greatest improvement on complex but fast executing
queries (volt_tt temporary tables)
• We are constantly working to improve transaction speeds

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Timeless Best Practices

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration
• Lift-and-Shift is NOT an ideal approach
• Depending where you are coming from, it is
sure to fail
• AWS has a rich ecosystem of solutions
• Your final solution will use other AWS services
• AWS Solution Architects, ProServ, and Partners
can help

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Distribution
• Distribution Styles
• KEY: Value is hashed, same value goes to same location (slice)
• ALL: Full table data goes to first slice of every node
• EVEN: Round robin
• Goals
• Distribute data evenly for parallel processing
• Minimise data movement during query processing
KEY
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
EVEN
ALL
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table Design Summary
• Materialise often filtered columns from dimension tables
into fact tables
• Materialise often calculated values into tables
• Avoid DIST KEYS on temporal columns
• Keep data types as wide as necessary (but no longer than
necessary)
• VARCHAR, CHAR and NUMERIC
• Add compression to columns
• Optimal compression can be found using ANALYSE COMPRESSION
• Add SORT KEYS on the primary columns that are filtered on

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Copy & Unload
• Delimited files are recommend
• Split files so there is a multiple of the number of slices
• Files sizes should be 1MB – 1GB after compression
• Use UNLOAD to extract large amounts of data from the
cluster
• Non-parallel UNLOAD only for very small amounts of data
S3

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Extract, Load & Transform (ELT)
Wrap workflow/statements in an explicit transaction
Consider using DROP TABLE or TRUNCATE instead of DELETE
Staging Tables
• Use temporary table or permanent table with the “BACKUP NO” option
• If possible use DISTSTYLE KEY on both the staging table and production table to speed
up the INSERT AS SELECT statement
• Turn off automatic compression - COMPUPDATE OFF
• Copy compression settings from production table or use ANALYSE COMPRESSION
statement
• Use CREATE TABLE LIKE or write encodings into the DDL
• For copying a large number of rows (> hundreds of millions) consider using ALTER
TABLE APPEND instead of INSERT AS SELECT
SQL

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vacuum & Analyse
• VACUUM should be run as necessary
• Typically nightly or weekly
• Consider “Deep Copy” for larger or wide tables
• ANALYSE can be run periodically after ingestion on just predicate
columns
• Utility to VACUUM and ANALYSE all the tables in the cluster:
https://github.com/awslabs/amazon-redshift utils/tree/master/src/AnalyzeVacuumUtility

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
WLM & QMR
• Keep the number of WLM queues to a minimum, typically just 3 queues
to avoid having unused queues
• https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminScripts/wlm_apex_hourly.sql
• Use WLM to limit ingestion/ELT concurrency to 2-3
• To maximise query throughput use WLM to throttle number of
concurrent queries to 15 or less
• Use QMR rather than WLM to set query timeouts
• Use QMR to log long running queries
• Save the superuser queue for administration tasks and canceling
queries

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cluster Sizing
Use at least two computes nodes (multi-node cluster) in production for data mirroring
• Leader Node is given for no additional cost
Amazon Redshift is significantly faster in a VPC compared to EC2 Classic
Maintain at least 20% free space or 3x the size of the largest table
• Scratch space for re-writing tables
• Free space is required for vacuum to resort table
• Temporary tables used for intermediate query results
The maximum number of available Amazon Redshift Spectrum nodes is a function of the
number of slices in the Amazon Redshift cluster
If you’re using DC1 instances, upgrade to the DC2 instance type
• Same price as DC1, significantly faster
• Reserved Instances do not automatically transfer over

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional Resources

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Labs On Github – Amazon Redshift
https://github.com/awslabs/amazon-redshift-utils
https://github.com/awslabs/amazon-redshift-monitoring
https://github.com/awslabs/amazon-redshift-udfs
Admin Scripts
• Collection of utilities for running diagnostics on your cluster
Admin Views
• Collection of utilities for managing your cluster, generating schema DDL, etc.
Analyse Vacuum Utility
• Utility that can be scheduled to vacuum and analyse the tables within your Amazon Redshift cluster
Column Encoding Utility
• Utility that will apply optimal column encoding to an established schema with data already loaded

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Big Data Blog – Amazon Redshift
Amazon Redshift Engineering’s Advanced Table Design Playbook
https://aws.amazon.com/blogs/big-data/amazon-redshift-engineerings-advanced-table-design-
playbook-preamble-prerequisites-and-prioritization/
- Zach Christopherson
Top 10 Performance Tuning Techniques for Amazon Redshift
https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-techniques-for-amazon-redshift/
- Ian Meyers and Zach Christopherson
10 Best Practices for Amazon Redshift Spectrum
https://aws.amazon.com/blogs/big-data/10-best-practices-for-amazon-redshift-spectrum/
- Po Hong and Peter Dalton

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Back To Our Toaster…

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank You