Looking ahead at PostgreSQL 15

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Looking ahead at PostgreSQL 15
Jonathan Katz, Principal Product Manager Technical
Jim Mlodgenski, Principal Database Engineer
AWS RDS Open Source

Amazon Relational Database Service (Amazon RDS)
Fully managed relational database service
Spend time innovating and building new apps,
not managing infrastructure
• Schema design
• Query construction
• Query optimization
Automatic failover
Backup and recovery
Isolation and security
Industry compliance
Push-button scaling
Automated patching and
upgrades
Advanced monitoring
Routine maintenance
You AWS

Amazon RDS
Set up, operate, and scale a relational database in the cloud
with just a few clicks
Available and
durable
Automatic Multi-AZ
data replication, with
automated backup,
snapshots, and failover
Easy to
administer
Easily deploy and
maintain hardware, OS,
and DB software,
with built-in monitoring
Performant and
scalable
Scale compute
and storage with a few clicks,
plus minimal downtime for
your application
Secure and
compliant
Data encryption at rest
and in transit,
with industry compliance
and assurance programs
PostgreSQL-Compatible
Edition
MySQL-Compatible
Edition

• A PostgreSQL “major release” is an annual feature release
• PostgreSQL 15 release cycle:
• Release cycle begins: July 2021
• Feature freeze: March/April 2022
• Beta: May 2022
• General availability: Late Q3 / Early Q4 2022
• During beta:
• Very unlikely that new functionality is added
• Some functionality may be removed
4
PostgreSQL major release process

• PostgreSQL releases average ~175 new features
• Review beta release notes:
https://www.postgresql.org/docs/15/release-15.html
• As we go through the new PostgreSQL features in this talk, we will:
• Look at use cases for each feature
• Review previous work
• Describe the new functionality
• See examples
How to explore PostgreSQL 15

Evolution of Conditional SQL

What is “conditional SQL?”
• “Conditional SQL” is when a
statement takes an additional
action based on the outcome of
a previous statement
• Prior to PostgreSQL 9.5, two
ways of doing this both prone
to race conditions
• Application layer
• Procedural Language (e.g.
PL/pgSQL)
• Additional syntax + overhead
• Ease of use with ORMs
“Add a new credit card to this
account.
However, if the credit card
number already exists, update the
expiration date.”

PostgreSQL 9.5: INSERT … ON CONFLICT
CREATE TABLE active_accounts (
account_id int PRIMARY KEY,
last_active_at timestamptz NOT NULL
);
INSERT INTO active_accounts
VALUES (1, CURRENT_TIMESTAMP)
ON CONFLICT (account_id) DO
UPDATE SET last_active_at = CURRENT_TIMESTAMP;
TABLE active_accounts;
account_id | last_active_at
------------+-------------------------------
1 | 2022-04-29 12:12:25.626644-04

PostgreSQL 9.5: INSERT … ON CONFLICT
VALUES (1, CURRENT_TIMESTAMP)
UPDATE SET last_active_at = CURRENT_TIMESTAMP;
------------+-------------------------------
1 | 2022-05-02 11:45:06.317415-04
(1 row)

PostgreSQL 15: MERGE
TRUNCATE TABLE active_accounts; -- clear out the data for this demo
-- store each account activity in a log table
CREATE TABLE activity_log (
account_id int NOT NULL,
active_at timestamptz NOT NULL
);
INSERT INTO activity_log
VALUES
(1, CURRENT_TIMESTAMP - '15 days'::interval),
(1, CURRENT_TIMESTAMP - '10 days'::interval),
(1, CURRENT_TIMESTAMP - '8 days'::interval);

MERGE INTO active_accounts
USING (
SELECT account_id, max(active_at) AS last_active_at
FROM activity_log
GROUP BY account_id
) alog
ON active_accounts.account_id = alog.account_id
WHEN NOT MATCHED AND alog.last_active_at >= CURRENT_TIMESTAMP - '14 days'::interval THEN
INSERT VALUES (alog.account_id, alog.last_active_at)
WHEN MATCHED AND alog.last_active_at < CURRENT_TIMESTAMP - '14 days'::interval THEN
DELETE
WHEN MATCHED AND active_accounts.last_active_at < alog.last_active_at THEN
UPDATE SET last_active_at = alog.last_active_at;

MERGE INTO active_accounts Specifies target table that is acted on by the MERGE conditions

USING (
SELECT account_id, max(active_at) AS last_active_at
FROM activity_log
GROUP BY account_id
) alog
ON active_accounts.account_id = alog.account_id
Source data set and its relation
to target.
This example finds the most
recent account activity.
Specifies target table that is acted
on by the MERGE conditions

WHEN NOT MATCHED AND alog.last_active_at >= CURRENT_TIMESTAMP - '14 days'::interval THEN
INSERT VALUES (alog.account_id, alog.last_active_at)
WHEN MATCHED AND alog.last_active_at < CURRENT_TIMESTAMP - '14 days'::interval THEN
DELETE
WHEN MATCHED AND active_accounts.last_active_at < alog.last_active_at THEN
UPDATE SET last_active_at = alog.last_active_at;
Conditions and actions.
If no row is found and the most recent activity was within 14 days: INSERT
If a row is found but most recent activity older than 14 days: DELETE
If a row is found and most recent activity is newer: UPDATE

-- execute MERGE command
------------+-------------------------------
1 | 2022-04-24 11:20:59.445867-04
INSERT INTO activity_log VALUES (1, CURRENT_TIMESTAMP);
------------+-------------------------------
1 | 2022-05-02 11:21:36.55613-04

DELETE FROM activity_log
WHERE active_at >= CURRENT_TIMESTAMP - '14 days'::interval;
------------+----------------
(0 rows)

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
INSERT … ON CONFLICT vs Merge
SELECT g, CURRENT_TIMESTAMP
FROM generate_series(1, 10000000) g
UPDATE SET
last_active_at = CURRENT_TIMESTAMP;
INSERT 0 10000000
Time: 36295.022 ms (00:36.295)
INSERT 0 10000000
Time: 66868.616 ms (01:06.869)
USING (SELECT g, CURRENT_TIMESTAMP as t
FROM generate_series(1, 10000000) g) AS a
ON active_accounts.account_id = a.g
WHEN NOT MATCHED THEN
INSERT VALUES (a.g, a.t)
WHEN MATCHED THEN
UPDATE SET last_active_at = a.t;
MERGE 10000000
Time: 17938.168 ms (00:17.938)
MERGE 10000000
Time: 495383.872 ms (08:15.384)
SET work_mem = '256MB’;
MERGE 10000000
Time: 54970.733 ms (00:54.971)

JSON

• 9.2 (2012): Support for JSON stored as text type
• 9.3: Introspection / extraction functionality
• 9.4: Support for JSON as binary type (JSONB) and indexing
• 9.5: JSONB building functions
(2017: SQL/JSON standard published)
• 10: Fulltext search for JSON(B) documents
• 11: JSON(B) transforms from PL/Python / PL/Perl
• 12: SQL/JSON path language
• 13: jsonpath.datetime
• 14 (2021): JSON subscripting syntax (e.g. a[‘b’][‘c’])
A brief history of PostgreSQL and JSON

• Published as SQL standard extension with SQL:2016
• SQL standard for interfacing with JSON
• Provide better interoperability between database engines
• Previous PostgreSQL releases have supported functionality similar to
SQL/JSON
20
SQL/JSON

• Constructors
• JSON(), JSON_SCALAR(), JSON_ARRAY(), JSON_ARRAYAGG(), JSON_OBJECT(),
JSON_OBJECTAGG()
• Return "json" type by default; for "jsonb" use "RETURNING jsonb"
• Query functions
• JSON_EXISTS, JSON_VALUE, JSON_QUERY
• Helpful for introspection and writing constraints
• JSON table
• Converts JSON into a PostgreSQL table
• IS JSON
• "SELECT value IS JSON;"
21
SQL/JSON and PostgreSQL 15

SQL/JSON constructors
=$ SELECT pg_typeof(JSON('{"a": 1}')); -- returns "json"
=$ SELECT pg_typeof(JSON('{"a": 1}' RETURNING jsonb)); -- returns "jsonb"
=$ SELECT JSON(1);
ERROR: cannot cast type integer to json
=$ SELECT JSON(JSON_SCALAR(1));
json
------
1

SQL/JSON constructors
=$ SELECT JSON_ARRAYAGG(x ORDER BY x DESC RETURNING jsonb)
FROM generate_series(1,5) x;
json_arrayagg
-----------------
[5, 4, 3, 2, 1]
=$ SELECT JSON_OBJECT(x: x+1, x*2: x^2 RETURNING jsonb) FROM generate_series(1,5);
json_object
--------------------
{"1": 2, "2": 1}
{"2": 3, "4": 4}
{"3": 4, "6": 9}
{"4": 5, "8": 16}
{"5": 6, "10": 25}

SQL/JSON query functions
=$ SELECT JSON_EXISTS(JSON('{"a": { "aa": 1 } }'), '$.a.aa');
ERROR: JSON_EXISTS() is not yet implemented for json type
=$ SELECT JSON_EXISTS(JSON('{"a": { "aa": 1 } }' RETURNING jsonb), '$.a.aa');
json_exists
-------------
t
=$ SELECT JSON_EXISTS(JSON('{"a": { "aa": 1 } }' RETURNING jsonb), '$.b');
json_exists
-------------
f

SQL/JSON query functions
=$ SELECT JSON_QUERY(
JSON_ARRAY(
JSON_OBJECT('id': 1), JSON_OBJECT('id': 2), JSON_OBJECT('id': 3)
RETURNING jsonb
),
'$[$i].id'
PASSING x AS i
)
FROM generate_series(0, 2) x;
json_query
------------
1
2
3

SQL/JSON table
CREATE TABLE users AS
SELECT *
FROM JSON_TABLE(
JSON_ARRAY(
JSON_OBJECT('id': 1, 'name': 'abc', 'created_on': '2022-04-30'),
JSON_OBJECT('id': 2, 'name': 'def', 'created_on': '2022-05-01'),
JSON_OBJECT('id': 3, 'name': 'ghi', 'created_on': '2022-05-02')
RETURNING jsonb
),
'$[*]'
COLUMNS (
id int,
name text,
created_on date
)
);

SQL/JSON table
TABLE users;
id | name | created_on
----+------+------------
1 | abc | 2022-04-30
2 | def | 2022-05-01
3 | ghi | 2022-05-02

Backups, Recovery, Archiving

• "recovery_prefetch" parameter
• During recovery, enables prefetching of blocks that are referenced in the WAL
that are not yet in the buffer pool
• Works for crash recovery, replication, and PITR
• Extensibility
• Custom WAL resource managers
• Allows table access method extensions to participate in logical decoding
• Remove exclusive backup mode
Note: Some custom backup scripts may need to change
30
PostgreSQL 15 Backup and recovery improvements

pg_walinspect
=# SELECT pg_current_wal_lsn();
pg_current_wal_lsn
--------------------
16/F2392B60
(1 row)
=# INSERT INTO active_accounts VALUES (-1, now());
INSERT 0 1
=# SELECT start_lsn, xid, resource_manager, record_type, record_length
FROM pg_get_wal_records_info_till_end_of_wal('16/F2392B60');
start_lsn | xid | resource_manager | record_type | record_length
-------------+-----+------------------+-------------------+---------------
16/F2392B60 | 789 | Heap | INSERT | 71
16/F2392BA8 | 789 | Btree | INSERT_LEAF | 64
…
(7 rows)
• Provides SQL functions that allow view access to the contents of
write-ahead log

• Allows for WAL archiving to be handled by a custom library instead
of a shell command using “archive_command”
• Will often be considerably more robust and performant
• Example module provided by contrib/basic_archive
# postgresql.conf
archive_mode = 'on'
archive_library = 'basic_archive'
basic_archive.archive_directory = '/path/to/archive/directory'
32
Archive modules

• WAL compression (zstd + lz4)
• Server-side compression for pg_basebackup
• gzip, zstd, lz4
• Add zstd + lz4 for client-side
• pg_receivewal (lz4 only)
33
Compression

0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
1 2 4 8 16 32 64 128 256 512
transaction
per
second
connections
WAL compression
off
pglz
lz4
zstd
• PostgreSQL has the ability to compress full page writes
• lz4 and zstd added to the existing pglz algorithm
34
WAL compression - performance
31%
41%

0
20
40
60
80
100
120
140
160
1 2 4 8 16 32 64 128 256 512
size
(GB)
connections
WAL compression
off
pglz
lz4
zstd
• PostgreSQL has the ability to compress full page writes
• lz4 and zstd added to the existing pglz algorithm
35
WAL compression – size-on-disk
71%
79%

Logical Replication

• Logical replication allows for changes to be streamed to another
database independent of the physical filesystem
• Several use cases:
• Change-data capture (CDC)
• Streaming analytics / BI
• Major version upgrades
• Multi-writeable databases
• Logical replication currently limited to data changes
37
Logical replication overview

• 9.4: Logical decoding, output plugins, replication slots
• 10: Logical replication of INSERT/UPDATE/DELETE
• 13: Logical replication of partitions
• 14: Performance improvements; stream in-flight transactions
A brief history of PostgreSQL and logical replication

• ALTER SUBSCRPTION … SKIP
• Skip a transaction (e.g. on a conflict) and then resume replay
• Can specify log sequence number (LSN)
• Requires superuser privileges
• Temporarily disable streaming replication
• ALTER SUBSCRIPTION … SET disable_on_error = true;
• Fix conflict on publisher side
• ALTER SUBSCRIPTION ... ENABLE;
39
PostgreSQL 15 logical replication improvements

• Support for two-phase commit / prepared transactions
• Sends transaction to subscriber when PREPARED TRANSACTION called
• Publish all tables in schema
• Previous was all tables in a database
• Publish a subset of rows/columns in a table
40
PostgreSQL 15 logical replication improvements

PostgreSQL 15 Logical replication examples
-- publish all tables in schema "app"
=# CREATE SCHEMA app;
=# CREATE TABLE app.users (
id int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, username text);
=# CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA app;
-- publish only a subset of columns on a table
=# CREATE TABLE abc(x int, y int, z int);
=# ALTER TABLE abc REPLICA IDENTITY FULL;
=# CREATE PUBLICATION pub2 FOR TABLE abc (x);
-- publish only non-test data
=# CREATE TABLE logs (id uuid PRIMARY KEY, log jsonb, is_test bool);
=# CREATE PUBLICATION pub3 FOR TABLE logs WHERE (NOT is_test);

Looking ahead at PostgreSQL 15

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Looking ahead at PostgreSQL 15

Similaire à Looking ahead at PostgreSQL 15 (20)

Plus de Jonathan Katz

Plus de Jonathan Katz (14)

Dernier

Dernier (20)

Looking ahead at PostgreSQL 15