The pg_stat_monitor is the statistics collection tool based on PostgreSQL’s contrib module pg_stat_statements. PostgreSQL’s pg_stat_statements provides only basic statistics, which is sometimes not enough. The major shortcoming in pg_stat_statements is that it accumulates all the queries and statistics, but does not provide aggregated statistics or histogram information. In this case, a user needs to calculate the aggregate, which is quite expensive. Pg_stat_monitor provides the pre-calculated aggregates. pg_stat_monitor collects and aggregates data on a bucket basis. The size and number of buckets should be configured using GUC (Grand Unified Configuration). The buckets are used to collect the statistics and aggregate them in a bucket. The talk will cover the usage of pg_stat_monitor and how it is better than pg_stat_statements.

1. pg_stat_monitor
A cool extension for better database (PostgreSQL) monitoring

2. © 2021 Percona
Ibrar Ahmed - The Author
● Software industries since 1998
● Open Source Contribution other than PostgreSQL
○ Cross platform implementation of FTP for Google Chrome
○ Testing Framework for Google Chromium Project
○ Cross Platform Telnet application
● Working on PostgreSQL Since 2006
● EnterpriseDB
○ Associate Software Architect core Database Engine 2006-2009
○ Software Architect core Database Engine 2011 - 2016
○ Senior Software Architect core Database Engine 2016 – 2018
● Percona
○ Senior Software Architect core Database Engine 2018 – Present
2
Twitter: @ibrar_ahmad
LinkedIn: ibrarahmed74
pgelephant.com

3. © 2021 Percona
Peter Zaitsev - The Mentor
● CEO, Percona
● Database Performance
Geek
● Know enough PostgreSQL
to be dangerous
3
Twitter: @PeterZaitsev
LinkedIn: peterzaitsev

4. © 2021 Percona
Denys Kondratenko
● Engineering Manager in Percona
○ Percona Monitoring and Management
● Past experience around distributed storage as eng mgr
○ SUSE - Ceph
○ Seagate - Lustre
● Monitoring experience in Massive Solutions
● Embedded engineer in Mindspeed before that
● love linux, run full Wayland, openSUSEway (sway)
4
github: denisok
LinkedIn: kondratenko
https://per.co.na/discord

5. © 2021 Percona | Confidential
Database
Performance
Basics
● Percona - 15+ Years Open Source Database Performance
Passion
● Most Common Reason of Poor Performance “Bad Queries”
● Users often do not even know they have such queries
● When they may not know source of such queries
● Or why they are bad and how to fix them
5

6. © 2021 Percona | Confidential
Sourcing the
Data
● System State Sampling
○ Limited Overhead
○ Limited Accuracy
○ Can have measurement Artifacts
● Event Counting
○ Risk of high overhead
○ Very Good accuracy
● pg_stat_monitor is Event Counting Approach
6

7. © 2021 Percona | Confidential
Why Some Data
Points are
Important ?
● Actual parameters in the statements
○ So you can run explain or modify the query and see if it improves
● Time Bucketing
○ High Fidelity stats, Better Network Quality Tolerance
● Multi-Dimensional grouping, like Client IP
○ Security Analyses. Spot where workload which should not be
running runs
● Capture Failed Queries
○ Some poorly Performing Queries never complete (ERROR,
WARNING and LOG error levels). Catch errors in your application
● Capture Touched Relation (tables accessed by the statement)
○ Easily spot all queries touching given relation
7

8. © 2021 Percona | Confidential
Why Some Data
Points are
Important ?
● Response Time Distribution (histograms)
○ “Average” query execution time does not tell the whole story
● Bucketing by Different Query Plans
○ Because Different Query Plans can have drastically difference
performance behavior
● Plan Information
○ Having Plan for actual query executions is helpful with unstable
plans
● Query Hierarchy
○ When Query is called from Stored Routine both query and caller
are very meaningful
8

9. © 2021 Percona
Download pg_stat_monitor
Source code
○ git clone https://github.com/percona/pg_stat_monitor.git
○ cd pg_stat_monitor
○ make USE_PGXS=1 install
Packages
○ Percona
■ https://www.percona.com/downloads/postgresql-distribution-13
■ https://repo.percona.com/ppg-13/yum/release/7/RPMS/x86_64/
○ PostgreSQL yum repositories
■ https://yum.postgresql.org/packages/
○ PostgreSQL Extensions Network (PGXN)
■ pgxn install pg_stat_monitor
9

10. © 2021 Percona
Install pg_stat_monitor
● pg_stat_monitor library can be loaded by setting shared_preload_libraries in
postgresql.conf file.
shared_preload_libraries = 'pg_stat_monitor' # (change requires restart)
● The same parameter shared_preload_libraries can be changed using ALTER SYSTEM
command.
postgres=# ALTER SYSTEM SET shared_preload_libraries=pg_stat_monitor;
ALTER SYSTEM
● Restart the server
# sudo systemctl restart postgresql*
● Create the extension using the create extension command.
postgres=# CREATE EXTENSION pg_stat_monitor;
CREATE EXTENSION
10

11. © 2021 Percona
SQL version for pg_stat_monitor
# pg_stat_monitor extension
comment = 'The pg_stat_monitor is a PostgreSQL Query Performance
Monitoring tool, based on PostgreSQL contrib module pg_stat_
statements. pg_stat_monitor provides aggregated statistics, client
information, plan details including plan, and histogram in
formation.'
default_version = '1.0'
module_pathname = '$libdir/pg_stat_monitor'
relocatable = true
$ ls -lrt *.sql
-rw-r--r-- 1 vagrant vagrant 6756 Oct 28 19:18 pg_stat_monitor--1.0.sql
11

12. © 2021 Percona
Build version for pg_stat_monitor
postgres=# SELECT pg_stat_monitor_version();
pg_stat_monitor_version
-------------------------
1.0.0-beta-3
(1 row)
12

13. © 2021 Percona
pg_stat_monitor additional columns
13
Column Type Description
bucket integer Data collection unit. The number shows what bucket in a chain a record belongs to
bucket_start_time timestamp
with time zone
The start time of the bucket
client_ip inet The IP address of a client that run the query
planid text An internally generated ID of a query plan
query_plan text The sequence of steps used to execute a query. This parameter is available only when the
pgsm_enable_query_plan is enabled.
top_query text Shows the top query used in a statement
top_queryid text Shows the top query id used in a statement
application_name text Shows the name of the application connected to the database
relations text[] The list of tables involved in the query

14. © 2021 Percona
pg_stat_monitor additional columns
14
Column Type Description
cmd_type integer Type of the query executed
cmd_type_text text
elevel integer Shows the error level of a query (WARNING, ERROR, LOG)
sqlcode integer The IP address of a client that run the query
message text The error message
resp_calls text[] Call histogram
cpu_user_time double precision The time (in ms) the CPU spent on running the query
cpu_sys_time double precision The time (in ms) the CPU spent on executing the kernel code
state_code bigint Shows the state code of a query
state text The state message

15. © 2021 Percona
postgres=# SELECT * FROM pg_stat_monitor_settings;
name | value | default_value | description | minimum | maximum | restart
------------------------------------------+--------+---------------+----------------------------------------------------+---------+------------+---------
pg_stat_monitor.pgsm_max | 100 | 100 | Sets the maximum size of shared memory in (MB).... | 1 | 1000 | 1
pg_stat_monitor.pgsm_query_max_len | 1024 | 1024 | Sets the maximum length of query. | 1024 | 2147483647 | 1
pg_stat_monitor.pgsm_enable | 1 | 1 | Enable/Disable statistics collector. | 0 | 0 | 0
pg_stat_monitor.pgsm_track_utility | 1 | 1 | Selects whether utility commands are tracked. | 0 | 0 | 0
pg_stat_monitor.pgsm_normalized_query | 0 | 1 | Selects whether save query in normalized format. | 0 | 0 | 0
pg_stat_monitor.pgsm_max_buckets | 10 | 10 | Sets the maximum number of buckets. | 1 | 10 | 1
pg_stat_monitor.pgsm_bucket_time | 60 | 60 | Sets the time in seconds per bucket. | 1 | 2147483647 | 1
pg_stat_monitor.pgsm_histogram_min | 0 | 0 | Sets the time in millisecond. | 0 | 2147483647 | 1
pg_stat_monitor.pgsm_histogram_max | 100000 | 100000 | Sets the time in millisecond. | 10 | 2147483647 | 1
pg_stat_monitor.pgsm_histogram_buckets | 10 | 10 | Sets the maximum number of histogram buckets | 2 | 2147483647 | 1
pg_stat_monitor.pgsm_query_shared_buffer | 20 | 20 | Sets the maximum size of shared memory in (MB) | 1 | 10000 | 1
pg_stat_monitor.pgsm_overflow_target | 0 | 1 | Sets the overflow target for pg_stat_monitor | 0 | 1 | 1
pg_stat_monitor.pgsm_enable_query_plan | 0 | 0 | Enable/Disable query plan monitoring | 0 | 0 | 0
pg_stat_monitor.pgsm_track_planning | 1 | 1 | Selects whether planning statistics are tracked. | 0 | 0 | 0
(14 rows)
15

16. © 2021 Percona
name | value | default_value | description | minimum | maximum | restart
------------------------------------------+--------+---------------+----------------------------------------------------+---------+------------+---------
pg_stat_monitor.pgsm_max | 100 | 100 | Sets the maximum size of shared memory in (MB).... | 1 | 1000 | 1
● Name
○ pg_stat_monitor.pgsm_max
● Value
○ 100
● Default Value
○ 100
● Description
○ Sets the maximum size of shared memory in (MB) used for statement's metadata tracked by
pg_stat_monitor.
● Minimum
○ 1
● Maximum
○ 1000
● Restart
○ 1
16

17. © 2021 Percona
Buckets
SELECT bucket,
bucket_start_time,
substr(query,0,50)|| '...' AS query,
calls
FROM pg_stat_monitor;
bucket | bucket_start_time | query | calls
-------+---------------------+--------------------------- +-----
0 | 2020-10-14 09:30:00 | INSERT INTO pgbench_tell...| 2
1 | 2020-10-14 09:30:30 | INSERT INTO pgbench_bran...| 1
1 | 2020-10-14 09:30:30 | SELECT relname, relkind ...| 1
1 | 2020-10-14 09:30:30 | SELECT queryid, bucket, ...| 1
2 | 2020-10-14 09:31:00 | CREATE table pgbench_his...| 1
2 | 2020-10-14 09:31:00 | SELECT queryid, bucket, ...| 1
2 | 2020-10-14 09:31:00 | SELECT queryid, bucket ...| 1
(7 rows)
17
b
u
c
k
e
t
4
Bucket 3 Bucket 2
Bucket 0
B
u
c
k
e
t
1
Bucket Start Time
B
u
c
k
e
t
T
i
m
e
Aggregates

18. © 2021 Percona
Query information
postgres=# SELECT userid, datname, queryid, substr(query,0, 50) AS query, calls FROM
pg_stat_monitor;
userid | datname | queryid | query | calls
---------+----------+------------------+---------------------------------------------------+-------
vagrant | postgres | 939C2F56E1F6A174 | END | 561
vagrant | postgres | 2A4437C4905E0E23 | SELECT abalance FROM pgbench_accounts WHERE aid = | 561
vagrant | postgres | 4EE9ED0CDF143477 | SELECT userid, datname, queryid, substr(query,$1 | 1
vagrant | postgres | 8867FEEB8A5388AC | vacuum pgbench_branches | 1
vagrant | postgres | 41D1168FB0733CAB | select count(*) from pgbench_branches | 1
vagrant | postgres | E5A889A8FF37C2B1 | UPDATE pgbench_accounts SET abalance = abalance + | 561
vagrant | postgres | 4876BBA9A8FCFCF9 | truncate pgbench_history | 1
vagrant | postgres | 22B76AE84689E4DC | INSERT INTO pgbench_history (tid, bid, aid, delta | 561
vagrant | postgres | F6DA9838660825CA | vacuum pgbench_tellers | 1
vagrant | postgres | 214646CE6F9B1A85 | BEGIN | 561
vagrant | postgres | 27462943E814C5B5 | UPDATE pgbench_tellers SET tbalance = tbalance + | 561
vagrant | postgres | 4F66D46F3D4151E | SELECT userid, dbid, queryid, substr(query,0, 50 | 1
vagrant | postgres | 6A02C123488B95DB | UPDATE pgbench_branches SET bbalance = bbalance + | 561
(13 rows)
18

19. © 2021 Percona
Query information
postgres=# SELECT application_name, client_ip, substr(query,0,100) AS query FROM
pg_stat_monitor;
application_name | client_ip | query
------------------+-----------+-----------------------------------------------------------------------------
pgbench | 127.0.0.1 | truncate pgbench_history
pgbench | 127.0.0.1 | SELECT abalance FROM pgbench_accounts WHERE aid = $1
pgbench | 127.0.0.1 | UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2
pgbench | 127.0.0.1 | BEGIN;
pgbench | 127.0.0.1 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3
pgbench | 127.0.0.1 | END;
pgbench | 127.0.0.1 | vacuum pgbench_branches
pgbench | 127.0.0.1 | UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2
pgbench | 127.0.0.1 | vacuum pgbench_tellers
pgbench | 127.0.0.1 | UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2
pgbench | 127.0.0.1 | select o.n, p.partstrat, pg_catalog.count(i.inhparent) from pg_catalog.pg_
psql | 127.0.0.1 | SELECT application_name, client_ip, substr(query,$1,$2) as query FROM pg_s
pgbench | 127.0.0.1 | select count(*) from pgbench_branches
(13 rows)
19

20. © 2021 Percona
Query Monitoring
postgres=# SET pg_stat_monitor.pgsm_normalized_query = true;
postgres=# SELECT a FROM foo where a = 10;
a
---
(0 rows)
postgres=# SELECT queryid, substr(query,0,50)|| '...' AS query,calls FROM pg_stat_monitor;
queryid | query | calls
------------------+------------------------------------------------------+-------
55F88A754A1BC5FF | SELECT a from foo where a = $1... | 1
postgres=# SET pg_stat_monitor.pgsm_normalized_query = false;
postgres=# SELECT a,2 FROM foo where a = 10;
a | ?column?
---+----------
(0 rows)
postgres=# SELECT queryid, substr(query,0,50)|| '...' AS query,calls FROM pg_stat_monitor;
queryid | query | calls
------------------+------------------------------------------------------+-------
EF380BA0410F35EC | SELECT a,2 from foo where a = 10;... | 1
20

21. © 2021 Percona
Error Logging
SELECT decode_error_level(elevel) AS elevel,
sqlcode,
query,
message
FROM pg_stat_monitor
WHERE elevel != 0;
elevel | sqlcode | query | message
--------+----------+-----------------------+----------------------------------
ERROR | 16908420 | SELECT * FROM pg_foo; | relation "pg_foo" does not exist
ERROR | 33816706 | SELECT 1/0; | division by zero
21

22. © 2021 Percona
Query Timing Histogram
postgres=# SELECT substr(query,0,40) || '... ' AS query,
calls,
resp_calls
FROM pg_stat_monitor
LIMIT 5;
query | calls | resp_calls
---------------------------------------------+-------+------------------------
SELECT * FROM pg_stat_database;... | 24 | {23,0,1,0,0,0,0,0,0,0}
select extract(epoch from current_times... | 24 | {24,0,0,0,0,0,0,0,0,0}
SELECT /* pmm-agent:pgstatmonitor */ pg... | 1 | {1,0,0,0,0,0,0,0,0,0}
SELECT pg_database.datname,tmp.mode,COA... | 24 | {24,0,0,0,0,0,0,0,0,0}
SELECT /* pmm-agent:pgstatmonitor */ "p... | 1 | {1,0,0,0,0,0,0,0,0,0}
(5 rows)
22

23. © 2021 Percona
Query Timing Histogram
SELECT resp_calls, query FROM pg_stat_monitor;
resp_calls | query
--------------------------------------------------+----------------------------------------------
{1," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"} | select client_ip, query from pg_stat_monitor
{3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | select * from pg_stat_monitor_reset()
{3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | SELECT * FROM foo
SELECT * FROM histogram(0, 'F44CD1B4B33A47AF') AS a(range TEXT, freq INT, bar TEXT);
range | freq | bar
--------------------+------+--------------------------------
(0 - 3)} | 2 | ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
(3 - 10)} | 0 |
(10 - 31)} | 1 | ■■■■■■■■■■■■■■■
(31 - 100)} | 0 |
(100 - 316)} | 0 |
(316 - 1000)} | 0 |
(1000 - 3162)} | 0 |
(3162 - 10000)} | 0 |
(10000 - 31622)} | 0 |
(31622 - 100000)} | 0 |
(10 rows)
23

24. © 2021 Percona
Planning Statistics
SELECT queryid,
plans_calls,
rows_retrieved,
substr(query,0,40) || '... ' AS query,
calls
FROM pg_stat_monitor;
24

25. © 2021 Percona
Planning Statistics
queryid | plans_calls | rows_retrieved | query | calls
------------------+-------------+----------------+-----------------------------+-------
BA6EC88C00347CF7 | 0 | 0 | truncate table pgbench_... | 2
1B12EDE3C70B8F88 | 100 | 100 | insert into pgbench_tel... | 100
3FD5C490B83F760D | 0 | 0 | create table pgbench_br... | 2
B1B991EE89D61CB6 | 0 | 0 | create table pgbench_ac... | 2
29A99577F1695D28 | 0 | 11000000 | copy pgbench_accounts ... | 2
772222CE7E8765 | 0 | 0 | alter table pgbench_bra... | 2
D59D5F4391AA6B3 | 10 | 10 | insert into pgbench_bra... | 10
25

26. © 2021 Percona
Query Plan: SELECT substr(query,0,50) AS query FROM
pg_stat_monitor LIMIT 10;
query |
-------------------------------------------------- +
SELECT abalance FROM pgbench_accounts WHERE aid = |
|
SELECT substr(query,$1,$2),calls, planid,query_pl |
26

27. © 2021 Percona
Query Plan: SELECT substr(query,0,50) AS query_plan
FROM pg_stat_monitor LIMIT 10;
query_plan
---------------------------------------------------------------------------+
Index Scan using pgbench_accounts_pkey on pgbench_accounts |
Index Cond: (aid = 102232) |
Limit |
-> Subquery Scan on pg_stat_monitor |
-> Result |
-> Sort |
Sort Key: p.bucket_start_time |
-> Hash Join |
Hash Cond: (p.dbid = d.oid) |
-> Function Scan on pg_stat_monitor_internal p |
...
27

28. © 2021 Percona
Top Query
CREATE OR REPLACE function add2(int, int) RETURNS INTEGER AS
$$
BEGIN
return (select $1 + $2);
END;
$$ language plpgsql;
SELECT add2(1,2);
add2
------
3
(1 row)
28

29. © 2021 Percona
Top Query
SELECT queryid, top_queryid, query, top_query FROM pg_stat_monitor;
queryid | top_queryid | query | top_query
------------------+------------------+--------------------------+--------------
762B99349F6C7F31 | 7209043A6BB0BCD5 | SELECT (select $1 + $2) | SELECT add2($1,$2)
7209043A6BB0BCD5 | | SELECT add2($1,$2) |
(3 rows)
29

30. © 2021 Percona
Query / Command Type
SELECT bucket, substr(query,0, 50) AS query, cmd_type_text FROM pg_stat_monitor WHERE elevel
= 0;
bucket | query | cmd_type_text
--------+---------------------------------------------------+----------
4 | END |
4 | SELECT abalance FROM pgbench_accounts WHERE aid = | SELECT
4 | vacuum pgbench_branches |
4 | select count(*) from pgbench_branches | SELECT
4 | UPDATE pgbench_accounts SET abalance = abalance + | UPDATE
4 | truncate pgbench_history |
4 | INSERT INTO pgbench_history (tid, bid, aid, delta | INSERT
5 | SELECT relations query FROM pg_stat_monitor | SELECT
9 | SELECT bucket, substr(query,$1, $2) AS query, cmd |
30

31. © 2021 Percona
pg_stat_monitor: Others
postgres=# d pg_stat_monitor
View "public.pg_stat_monitor"
Column | Type |
---------------------+------------------+
bucket | bigint |
bucket_start_time | text |
userid | regrole |
datname | name |
client_ip | inet |
queryid | text |
top_queryid | text |
query | text |
comments | text |
31
planid | text |
query_plan | text |
top_query | text |
application_name | text |
relations | text[] |
cmd_type | integer |
cmd_type_text | text |
elevel | integer |
sqlcode | text |
message | text |
calls | bigint |
total_time | double precision |
min_time | double precision |

32. © 2021 Percona
pg_stat_monitor: Others
max_time | double precision |
mean_time | double precision |
stddev_time | double precision |
rows_retrieved | bigint |
plans_calls | bigint |
plan_total_time | double precision |
plan_min_time | double precision |
plan_max_time | double precision |
plan_mean_time | double precision |
shared_blks_hit | bigint |
shared_blks_read | bigint |
shared_blks_dirtied | bigint |
shared_blks_written | bigint |
local_blks_hit | bigint |
32
local_blks_read | bigint |
local_blks_dirtied | bigint |
local_blks_written | bigint |
temp_blks_read | bigint |
temp_blks_written | bigint |
blk_read_time | double precision |
blk_write_time | double precision |
resp_calls | text[] |
cpu_user_time | double precision |
cpu_sys_time | double precision |
wal_records | bigint |
wal_fpi | bigint |
wal_bytes | numeric |
state_code | bigint |
state | text |

33. © 2021 Percona
33
Vanila PostgreSQL pg_stat_statment pg_stat_monitor
Database Size TPS
1.6GB 3256
16GB 840
160GB 608
Database Size TPS
1.6GB 3268
16GB 843
160GB 610
Database Size TPS
1.6GB 3262
16GB 844
160GB 611
Performance Metrics

34. © 2021 Percona | Confidential
pg_stat_monitor
in PMM
34

35. © 2021 Percona | Confidential
pg_stat_monitor
in PMM
35

36. © 2021 Percona | Confidential
pg_stat_monitor
in PMM
36

37. © 2021 Percona | Confidential
pg_stat_monitor
in PMM
37

38. © 2021 Percona
pg_stat_monitor add service to PMM
# pmm-admin add postgresql --query-source=pgstatmonitor
38

39. © 2021 Percona
Feedback
● If you found a bug or have a feature request in pg_stat_monitor, please submit the report to the
https://jira.percona.com/projects/PG/issues
● If there is no existing report, submit your report following these steps:
○ Sign in to Jira issue tracker. You will need to create an account if you do not have one.
○ In the Summary, Description, Steps To Reproduce, Affects Version fields describe the problem you have
detected.
As a general rule of thumb, try to create bug reports that are:
● Reproducible: describe the steps to reproduce the problem.
● Unique: check if there already exists a JIRA ticket to describe the problem.
● Scoped to a Single Bug: only report one bug in one JIRA ticket.
39

40. Question