Database tuning is an ongoing exercise. This presentation (from Postgres Conf 2018 NJ) shows you how to continually tune your database. What metrics do you need to track and how can you feed them back into your application, data lifecycle and database config.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Postgres Conf US - April, 2018

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Tuning
• Application
• Data Lifecycle
• Database configuration

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tuning Methodology
1. Run unit tests
2. Track database statistics
3. Baseline Config
4. Run Load
5. Track database statistics
6. Update config
7. GOTO 4

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What should we track?
• Activity
• Database Utilization
• Table Utilization
• Index Utilization
• Locking

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do we track it?
• PG Catalog
• Schema in every postgres db (pg_catalog)
• Appended to search_path
• “Data Dictionary”

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog
SELECT pn.nspname, pc.relkind, count(1)
FROM pg_namespace pn, pg_class pc
WHERE pn.oid = pc.relnamespace
AND pn.nspname = 'pg_catalog'
GROUP BY pn.nspname, pc.relkind
ORDER BY 1 DESC;
nspname | relkind | count
------------+---------+-------
pg_catalog | i | 115
pg_catalog | r | 62
pg_catalog | v | 59
i = indexes
r = tables
v = views

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog functions
select count(1)
FROM (
select pn.nspname, pp.proname
FROM pg_namespace pn, pg_proc pp
WHERE pn.oid = pp.pronamespace
AND pn.nspname = 'pg_catalog'
) foo;
count
-----
2882

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog functions
SELECT pn.nspname, pp.proname
FROM pg_namespace pn, pg_proc pp
WHERE pn.oid = pp.pronamespace
AND pn.nspname = 'pg_catalog’;
nspname | proname
------------+--------------
pg_catalog | boolin
pg_catalog | boolout
pg_catalog | byteain
pg_catalog | byteaout
pg_catalog | charin
pg_catalog | charout
pg_catalog | namein
pg_catalog | nameout
...

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog
• Structural
• Performance
• Informational

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog – translation!
• Relation = table, index, sequence, etc…
• Attribute = column
• Namespace = schema

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog - oid
• Object Identifier datatype
• 4 byte unsigned int
• Hidden column on catalog tables
• Primary / Foreign keys throughout pg_catalog schema

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog - oid
postgres=# select * from pg_namespace;
nspname | nspowner | nspacl
--------------------+----------+----------------------------
pg_toast | 10 |
pg_temp_1 | 10 |
pg_toast_temp_1 | 10 |
pg_catalog | 10 | {meads=UC/meads,=U/meads}
public | 10 | {meads=UC/meads,=UC/meads}
information_schema | 10 | {meads=UC/meads,=U/meads}
(6 rows)

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog - oid
postgres=# select oid, * from pg_namespace;
oid | nspname | nspowner | nspacl
-------+--------------------+----------+----------------------------
99 | pg_toast | 10 |
11816 | pg_temp_1 | 10 |
11817 | pg_toast_temp_1 | 10 |
11 | pg_catalog | 10 | {meads=UC/meads,=U/meads}
2200 | public | 10 | {meads=UC/meads,=UC/meads}
12349 | information_schema | 10 | {meads=UC/meads,=U/meads}
(6 rows)

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
pg_catalog – List all tables in a schema
SELECT pn.nspname, pc.relkind, count(1)
FROM pg_namespace pn, pg_class pc
WHERE pn.oid = pc.relnamespace
AND pn.nspname = 'pg_catalog'
GROUP BY pn.nspname, pc.relkind
ORDER BY 1 DESC;

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s this have to do with tuning??

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tuning Methodology
1. Run unit tests
2. Track database statistics
3. Baseline Config
4. Run Load
5. Track database statistics
6. Update config
7. GOTO 4

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What should we track?
• Activity
• Database Utilization
• Table Utilization
• Index Utilization
• Locking

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where is the data?
• pg_stat_activity
• pg_stat_database
• pg_stat_all_tables / pg_stat_user_tables
• pg_stat_all_indexes / pg_stat_user_indexes
• pg_locks

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Activity
• pg_stat_activity
• Current activity in the database
• Snapshot of that instant
• Starting point for tuning
• State
• Idle
• Active
• Idle in transaction
select datname, usename,
client_addr,
wait_event,
now() - state_change as runtime,
state, query
from pg_stat_activity ;

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Activity - Tuning
• Application
• Connections
• Concurrency
• Data Lifecycle
• Query runtimes
• WAL

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Activity - Tuning
• Server configuration
• max_connections
• shared_buffers
• work_mem

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Statistics
• pg_stat_database
• Utilization stats for each DB in an instance
• Global view of all DBs
select datname, numbackends, xact_commit,
tup_returned, tup_fetched, tup_inserted,
tup_updated, tup_deleted,
pg_database_size(datname),
pg_size_pretty(pg_database_size(datname))
FROM pg_stat_database;

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table Statistics
• pg_stat_all_tables / pg_stat_user_tables
• Cumulative view of table-level stats
• Basis for workload profile
• pg_stat_all_indexes / pg_stat_user_indexes
• Cumulative view of index-level stats
• Index utilization stats
select schemaname, relname,
n_tup_ins,
n_tup_upd, n_tup_del,
idx_scan, seq_scan
FROM pg_stat_user_tables;

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where are my hot-spots?
• What’s the most modified (INS / UPD / DEL)?
• What’s the most read ?
• What’s the largest?

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table Stats -Tuning
• Application
• Partitioning
• SQL
• Stored Procedures

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table Stats - Tuning
• Data Lifecycle
• Datamodel
• Indexing
• Workflow (Truncate vs Delete)
• Database configuration
• Vacuum
• Shared_buffers

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Locking
• pg_locks
• Status of all locks in a given database
• Look for conflicting locks WHERE granted = ‘f’

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Locking - Tuning
• Application
• Workflow
• Data Lifecycle
• Data Model
• Normalization
• ETL
• Maintenance

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tracking vs Storing
• Cloud Watch
• Performance Insights
• POWA
• PGWatch2
• Snapshot scripts
• Grafana
• ELK
Max CPU

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn more..
aws.amazon.com/rds
aws.amazon.com/rds/aurora

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Scott Mead
meads@amazon.com