AWS Data Engineer Associate (DEA-C01) Exam Dumps 2024.pdf
Best Practices for Migrating your Data Warehouse to Amazon Redshift
1. Best Practices for Migrating your
Data Warehouse to Amazon
Redshift
Tony Gibbs, Data Warehousing Solutions Architect
Feb 2017
2. Migrating your Data Warehouse Overview
• Why Migrate
• Customer Success Stories
• Amazon Redshift History and Development
• Cluster Architecture
• Migration Best Practices
• Migration Tools
• Open Q&A
5. If you forget everything else…
• Lift-and-Shift is NOT an ideal approach
• Depending where you are coming from, it is sure to fail
• AWS has a rich ecosystem of solutions
• Your final solution will use other AWS services
• AWS Solution Architects, ProServ, and Partners can help
6. 100x faster
Scales from GBs to PBs
Analyze data without storage
constraints
10x cheaper
Easy to provision and operate
Higher productivity
10x faster
No programming
Standard interfaces and
integration to leverage BI tools,
machine learning, streaming
Transactional database MPP database Hadoop
Why Migrate to Amazon Redshift?
10. Migration from Greenplum @ NTT Docomo
68 million customers
10s of TBs per day of data across
mobile network
6PB of total data (uncompressed)
Data science for marketing
operations, logistics etc.
Legacy DW: Greenplum on-premises
After migration:
125 node DS2.8XL cluster
4,500 vCPUs, 30TB RAM
6 PB uncompressed
10x faster analytic queries
50% reduction in time for new BI
app. deployment
Significantly less ops. overhead
11. Migration from SQL on Hadoop @ Yahoo
Analytics for website/mobile events
across multiple Yahoo properties
On an average day
2B events
25M devices
Before migration: Hive – Found it to be
slow, hard to use, share and repeat
After migration:
21 node DC1.8XL (SSD)
50TB compressed data
100x performance improvement
Real-time insights
Easier deployment and
maintenance
12. Migration from SQL on Hadoop @ Yahoo
1
10
100
1000
10000
Count
Distinct
Devices
Count All
Events
Filter
Clauses
Joins
Seconds
Amazon Redshift
Impala
20. Designed for I/O Reduction
Columnar storage
Data compression
Zone maps
aid loc dt
1 SFO 2016-09-01
2 JFK 2016-09-14
3 SFO 2017-04-01
4 JFK 2017-05-14
• Accessing dt with row storage:
– Need to read everything
– Unnecessary I/O
aid loc dt
CREATE TABLE loft_migration (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
);
21. Designed for I/O Reduction
Columnar storage
Data compression
Zone maps
aid loc dt
1 SFO 2016-09-01
2 JFK 2016-09-14
3 SFO 2017-04-01
4 JFK 2017-05-14
• Accessing dt with columnar storage:
– Only scan blocks for relevant
column
aid loc dt
CREATE TABLE loft_migration (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
);
22. Designed for I/O Reduction
Columnar storage
Data compression
Zone maps
aid loc dt
1 SFO 2016-09-01
2 JFK 2016-09-14
3 SFO 2017-04-01
4 JFK 2017-05-14
• Columns grow and shrink independently
• Effective compression ratios due to like data
• Reduces storage requirements
• Reduces I/O
aid loc dt
CREATE TABLE loft_migration (
aid INT ENCODE LZO
,loc CHAR(3) ENCODE BYTEDICT
,dt DATE ENCODE RUNLENGTH
);
23. Designed for I/O Reduction
Columnar storage
Data compression
Zone maps
aid loc dt
1 SFO 2016-09-01
2 JFK 2016-09-14
3 SFO 2017-04-01
4 JFK 2017-05-14
aid loc dt
CREATE TABLE loft_migration (
aid INT --audience_id
,loc CHAR(3) --location
,dt DATE --date
);
• In-memory block metadata
• Contains per-block MIN and MAX value
• Effectively prunes blocks which cannot
contain data for a given query
• Eliminates unnecessary I/O
25. Terminology and Concepts: Slices
A slice can be thought of like a “virtual compute node”
– Unit of data partitioning
– Parallel query processing
Facts about slices:
– Each compute node has either 2, 16, or 32 slices
– Table rows are distributed to slices
– A slice processes only its own data
26. Terminology and Concepts: Data Distribution
• Distribution style is a table property which dictates how that table’s data is
distributed throughout the cluster:
• KEY: Value is hashed, same value goes to same location (slice)
• ALL: Full table data goes to first slice of every node
• EVEN: Round robin
• Goals:
• Distribute data evenly for parallel processing
• Minimize data movement during query processing
KEY
ALL
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
Node 1
Slice
1
Slice
2
Node 2
Slice
3
Slice
4
EVEN
27. DS2.8XL Compute Node
Ingestion Throughput:
– Each slice’s query processors can load one file at a time:
Streaming decompression
Parse
Distribute
Write
Realizing only partial node usage as 6.25% of slices are active
0 2 4 6 8 10 12 141 3 5 7 9 11 13 15
Data Loading Best Practices
28. Use at least as many input
files as there are slices in the
cluster
With 16 input files, all slices
are working so you maximize
throughput
COPY continues to scale
linearly as you add nodes
16 Input Files
DS2.8XL Compute Node
0 2 4 6 8 10 12 141 3 5 7 9 11 13 15
Data Loading Best Practices Continued
29. Data Preparation
Export Data from Source System
– CSV Recommend (Delimiter '|')
Be aware of UTF-8 varchar columns (UTF-8 take 4 bytes per char)
Be aware of your NULL character (N)
– GZIP Compress Files
– Split Files (1MB – 1GB after gzip compression)
Useful COPY Options for PoC Data
– MAXERRORS
– ACCEPTINVCHARS
– NULL AS
30. Keep Columns as Narrow as Possible
• Buffers allocated based on declared
column width
• Wider than needed columns mean
memory is wasted
• Fewer rows fit into memory; increased
likelihood of queries spilling to disk
• Check
SVV_TABLE_INFO(max_varchar)
• SELECT max(len(col)) FROM table
31. Amazon Redshift is a Data Warehouse
Optimized for batch inserts
– The time to insert a single row in Redshift is roughly the same as inserting
100,000 rows
Updates are delete + insert of the row
• Deletes mark rows for deletion
Blocks are immutable
– Minimum space used is one block per column, per slice
32. Auto Compression
– Samples data automatically when COPY into an empty table
Samples up to 100,000 rows and picks optimal encoding
– Turn off Auto Compression for Staging Tables
Bake encodings into your DDL or use CREATE TABLE (LIKE …)
Analyze Compression
– Data profile has changed
– Run after changing sort key
Column Compression
34. Primary/Unique/Foreign Key Constraints
Primary/Unique/Foreign Key constraints are NOT enforced
– If you load data multiple times, Amazon Redshift won’t complain
– If you declare primary keys in your DDL, the optimizer will expect the data to
be unique
Redshift optimizer uses declared constraints to pick optimal plan
– In certain cases it can result in performance improvements
39. Start your first migration in few minutes
Sources include: Aurora, Oracle, SQL
Server, MySQL and PostgreSQL
Bulk load and continuous replication
Migrate a TB for $3
Fault tolerant
(AWS DMS)
Our view of data warehousing and why we built Amazon Redshift.
We want to make data warehousing fast, inexpensive and easy to use for companies of all sizes
For Enterprises, benefits are: 1/10th cost, easy to provision and manage, productive DBAs
For Big data companies: 10x faster, use standard SQL-no need to write custom code, use existing BI tools
For SaaS: analysis in-line with process flows, pay as you go, grow as you need, managed availability & DR
Nasdaq security – Ingests 5B rows/trading day, analyzes orders, trades and quotes
Yelp – 10s of millions of ads/day, 250M mobile events/day, analyzes ad campaign performance and new feature usage
Desk.com – Ingests 200K case related events/hour and runs a user facing portal on Redshift
Maybe a slide showing cost of rs vs. vertica vs. impala
Secondly, the tool can automatically re-write code which is stored inside the database; views, stored procedures, triggers and functions are automatically evaluated and where possible, we’ll go ahead and convert them to their equivalent in the new database; where we can’t do this accurately, we’ll highlight it and make smart recommendations (providing links to the docs) to guide you through the process manually. So when coupled with the Database Migration Service, the Schema Conversion tool will help you go from whatever database you’re on now, to a more open database in the Cloud with remarkably low effort in terms of skilled DBA time and cost, giving you an ‘out’ for your current database relationship woes.
The Database Migration Service preview starts today, and the Schema Conversion tool is available for you to start using right away.
Define what goes in/out of S3
SLOW !
CPU Trade-off
Explain the limitation of min/max
Not as good as bloom filter…
Redshift value proposition: fast, simple, cost-effective data warehousing. Massively parallel and columnar technology. Easily scalable to petabytes or more. Makes administration simple, no DBA needed. You can choose from hard disk drive or solid state drive platform depending on your needs. All this for $1,000/TB/year, which is one-tenth of the cost of traditional data warehousing solutions.
You can set up a migration task within minutes in the AWS Management Console.
This includes setting up connections to the source and target databases, as well as choosing the replication instance used to run the migration process.
Oracle supplemental logging
MySQL row-level bin logging
SQL Server bulk logged/full recovery
Postgres WAL
No agent
Uses recovery log
Native change data capture API