Amazon Aurora is a fully-managed relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. The initial launch of Amazon Aurora delivered these benefits for MySQL. We have now added PostgreSQL compatibility to Amazon Aurora. In this session, Amazon Aurora experts discuss best practices to maximize the benefits of the Amazon Aurora PostgreSQL-compatible edition in your environment.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Deep Dive on Amazon Aurora PostgreSQL -
Compatible Edition
G r a n t M c A l i s t e r — S e n i o r P r i n c i p a l E n g i n e e r
D A T 4 0 2
N o v e m b e r 2 9 , 2 0 1 7
D A T 4 0 2

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RDS PostgreSQL Universe

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RDS PostgreSQL Universe
CLIENTS
RDS
PostgreSQL
Aurora
PostgreSQL
EBS
Aurora
Storage
Postgres 9.6—same extensions
Backup/Recovery - PITR
High Availability & Durability
Secure
Read Replicas
Cross Region Snapshots
Scale Compute – Online Scale Storage
Cross Region Replication
Outbound Logical Replication

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Base Architecture

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RO
Application
AZ-1 AZ-2 AZ-3
Aurora Storage and Replicas
RW
Application
RO
Application
Async
Invalidation
& Update
Async
Invalidation
& Update
Write log
records
Read
blocks
RW
Automatic Scalable Storage to 64TB

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Log-Based Storage

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concurrency—Remove Log Buffer
Queued Work
Log Buffer
PostgreSQL Aurora PostgreSQL
Storage
A Queued Work
Storage
B C D E
0 0 0 0 0
A B C D E
2 2 1 0 1
A B C D E
4 3 4 2 4
A B C D E
6 5 6 3 5
A B C D E
Durability
Tracking

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Checkpoints & Full-Page Writes in PostgreSQL
Block in
Memory
PostgreSQL
update t set y = 6;
Checkpoint
Datafile
Full
Block
WAL
Archive
4K
4K
8K
During crash
recovery,
PostgreSQL
uses the FPW
block in the
WAL to replace
the bad
checkpointed
block

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Aurora PostgreSQL—Writing Less
Block in
Memory
PostgreSQL Aurora
update t set y = 6; update t set y = 6;
Checkpoint
Datafile
Full
Block
WAL
Archive
Block in
Memory
Aurora
Storage
no
checkpoint
=
no FPW
Amazon S3

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1-
200
1-
100
101-
200
Insert
124
1-50
51-
100
1-25
26-
50
51-
75
76-
100
101-
150
151-
200
101-
125
126-
150
151-
175
176-
200
Insert 99
1-
200
101-
200
101-
150
101-
125
4 blocks
loaded
1-
100
51-
100
76-
100
3 blocks
loaded
Insert a Random Value into a B-Tree
151-
200
151-
200
2 blocks
loaded
Insert 161
At least 3 FPW

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Insert Test
Test Table
• UUID PK—Random
• ID int—Right Lean Sequence
• VARCHAR(100)—Random
• VARCHAR(50)—Small Set of Words
• INT—Random
• INT—Random (smaller set)
• BOOLEAN—Random (50/50)
• BOOLEAN—Somewhat Random (75/25)
• Timestamp—Right Lean

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
5,000
10,000
15,000
20,000
25,000
30,000
1 31 61 91 121 151 181 211 241 271
InsertsPerSecond
Minutes
Insert Workload—PostgreSQL 9.6
BASE 16GB Max WAL Aurora PostgreSQL

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Update Test
Test Table
• UUID PK—Random
• ID int—Right Lean Sequence
• VARCHAR(100)—Random
• VARCHAR(50)—Small Set of Words
• INT—Random
• INT—Random (smaller set)
• BOOLEAN—Random (50/50)
• BOOLEAN—Somewhat Random (75/25)
• Timestamp—Right Lean
UPDATE #1
UPDATE #2

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3,729
4,871
17,158
-
5,000
10,000
15,000
20,000
TPS(2UpdatesperTransaction)
Update Workload—PostgreSQL 9.6
BASE 16GB Max WAL Aurora PostgreSQL

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora Recovers Up to 97%Faster
3 GiB Redo
Recovered in 19
seconds
10 GiB Redo
Recovered in 50
seconds
30 GiB Redo
Recovered in 123
seconds
0
20
40
60
80
100
120
140
160
0 20,000 40,000 60,000 80,000 100,000 120,000 140,000
RecoveryTimeinSeconds(lessisbetter)
Writes / Second (more is better)
RECOVERY TIME FROM CRASH UNDER LOAD
Bubble size represents redo log, which must be recovered
As PostgreSQL
throughput goes up,
so does log size and
crash recovery time
Amazon Aurora has no redo.
Recovered in 3 seconds while
maintaining significantly greater
throughput.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Durability—4 of 6 Quorum

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Issues with 6 of 6 Replication
Location 1
Location 2
Location 3
Start Finish

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
6
10
21
31
7
12
28
123
0
20
40
60
80
100
120
140
50 90 99.9 99.99
Latency(ms)
Percentile
High Concurrency Sync Write Test
2 Node (4 copy) 3 Node (6 Copy)
Cost of Additional Synchronous Replicas

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora Timing Example
Location 1
Location 2
Location 3
Start Finish
Only need 4/6
sync writes

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Aurora Gives >2x Lower Response Times
0.00
100.00
200.00
300.00
400.00
500.00
600.00
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
responsetime,ms
minute
sysbench response time (p95), 30 GiB, 1024 clients
PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas and Clones

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—PostgreSQL
PostgreSQL
RW
EBS Snapshot
PostgreSQL
RO
EBS
update
Async Replication
Catchup

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Amazon Aurora
Aurora
RW
PostgreSQL
RO
update
Async Replication
Aurora Storage
update in
memory

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pgbench Benchmark
Async Replication
and apply
PostgreSQL
Aurora
PostgreSQL
RW RO
accounts
tellers
branches
history
accounts
tellers
branches
history
Async Replication
and memory update
RW RO
accounts
tellers
branches
history
accounts

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Backfill on PostgreSQL
pgbench RW 8K tps on Primary – RO 200k tps on replica
backfill
pgbench_history

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replicas—Backfill on Amazon Aurora
pgbench RW 8K tps on Primary – RO 200k tps on replica
backfill
pgbench_history

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Aurora Storage
RO
Application
AZ-1 AZ-2 AZ-3
Fast Clones
RW
Application
RW
Developers
Reporting
Async
Invalidation
& Update
Clone
update
Primary Storage
Clone Storage

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Caching

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Caching Changes—No Double Buffering
488GB RAM
PG + OS
processes
shared_buffers
Linux
pagecache
select of data –
check for block in
shared_buffers
if not in
shared_buffers
load from
pagecache/disk
EBS
1/4
duplicate
buffers
PG + OS
processes
shared_buffers
PostgreSQL Aurora
PostgreSQL
Aurora
Storage
select of data –
check for block
in
shared_buffers
or load from
Aurora storage
3/4
Survivable
Cache

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Caching Changes—No Double Buffering
689,068
417,496
334,691
682,931
-
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
TransactionsPerSecond(tps)
pgbench read only - scale 22,000 - r4.16xlarge
Aurora 75% Cache PostgreSQL 25% Cache PostgreSQL 10% Cache PostgreSQL 75% Cache
1.6x 2.0x
18K read iops
no reads
heavy double
buffering
no double
buffering
no survivable
cache
Approx 350GB working set

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vacuuming

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why Vacuuming Matters
tuple1
tuple2
tuple3
tuple4
tuple5
tuple6
tuple1tuple1
tuple2 tuple2
tuple7
tuple8
tuple9
tuple10
Run Vacuumtuple3
tuple4
tuple3
tuple4
tuple1
tuple2
tuple3
tuple4
tuple5
tuple6
tuple1tuple1
tuple2 tuple2
tuple7
tuple8
tuple9
tuple10
No Vacuumtuple3
tuple4
tuple3
tuple4
more blocks = more Cache misses, non HOT updates and more FPW

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4,000
5,000
6,000
7,000
8,000
9,000
10,000
1 61 121 181 241 301 361 421 481 541 601 661 721 781 841 901 961 1021 1081 1141 1201 1261
TPS
Minutes
Updates—No Vacuum Running
transaction id wrap around

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Intelligent Vacuum Prefetch
PostgreSQL
Aurora PostgreSQL
Submit
Batch I/O

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vacuuming in Memory—Insert Only Workload
HEAP
PAGES
1-
200
1-
100
101-
200
1-50
51-
100
1-25
26-
50
51-
75
76-
100
101-
150
151-
200
101-
125
126-
150
151-
175
176-
200
1-
201
101-
201
151-
201
176-
201
1-
202
101-
202
151-
202
176-
202
No long running transactions

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vacuum Freeze Not in Memory—PostgreSQL
Block in
Memory
Checkpoint
Datafile
Full
Block
WAL
Archive
Not
Frozen
VACUUM

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vacuum Freeze in Memory—PostgreSQL
Block in
Memory
Checkpoint
Datafile
WAL
Archive
Frozen

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vacuum in Memory, Cont.
• alter table X set (vacuum settings)
• Increase checkpoint_timeout
• Manual Test
• Vacuum in Memory before checkpoint – 72 seconds
• Vacuum in Memory after checkpoint – 152 seconds
• Vacuum not in Memory – 402 seconds – 163 seconds

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring Performance

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enhanced Monitoring—Process List

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance Insights

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance Insights—Zoom

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance Insights—Bad Query

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration to Aurora PostgreSQL
Methods
• AWS Data Migration Service (DMS)
• PostgreSQL - pg_dump / pg_restore
• RDS PostgreSQL - Snapshot Import

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer
Premises,
EC2, RDS
Application Users
Aurora
PostgreSQLVPN
AWS DMS—Logical Replication
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
Let the AWS Database Migration
Service create tables and load data
Uses change data capture to keep
them in sync
Switch applications over to the target
at your convenience
AWS Database
Migration Service

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration—Snapshot Import
Snapshot

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
D e e p D i v e o n A m a z o n A u r o r a P o s t g r e S Q L — D A T 4 0 2
G r a n t M c A l i s t e r — S e n i o r P r i n c i p a l E n g i n e e r