2. https://www.2ndQuadrant.com
PostgreSQL Now
● Features
○ Matured for very wide range of applications
● More production use case
○ Important business scenes
● More chance of database maintenance
○ Version up
○ Database design change
■ Physical
■ Logical
● Business Requirement (24/7 for example)
○ Less acceptance of long maintenance window
3. https://www.2ndQuadrant.com
Database Migration and Design Change
● Column data type change
○ INT -> BIGINT
○ INT -> NUMERIC
○ VARCHAR -> TEXT
Accomodate more records to a table
Business Growth
ALER TABLE?
COPY?
CREATE TABLE … SELECT?
6. https://www.2ndQuadrant.com
Most of these operations are blocking
Typically take days
Cannot simply perform in production environment
Many use case require minimum down time
(maintenance window), for example,
< 15min.
General Issue
8. https://www.2ndQuadrant.com
Partitioning an existing table
Usual process:
1. CREATE TABLE PARTITIONED …
2. COPY from old table to new table
Typical application/data characteristics:
● Log type: can use creation timestamp as partitioning
key
● Data are not updated (may be deleted though)
● Has another primary key which is used as foreign key
from other tables, but not declared explicitly.
9. https://www.2ndQuadrant.com
Apps can continue to run.
Partitioning Steps
1. Create new table with same column definition
a. Creation timestamp as the partitioning key
b. Exclude for PKEY/Unique constraint
2. Create partitions as needed
3. Copy row from original table
4. Lock original/new table
5. Do last copy
6. Rename original/new table
7. Unlock original/new table
Maintenance Window,
requres apps to stop.
15. https://www.2ndQuadrant.com
Maintain Unique Constraint
Partitioning key must be a part of unique constraints
Maintaining uniqueness is application’s responsibility
1. INSERT
○ Use of sequence/other default value
2. UPDATE
○ Use RULE/TRIGGER to prohibit unique value update
All these depends upon application/date characteristics and/or
requirements.
Unique constraint in each partition may reduce a risk.
21. https://www.2ndQuadrant.com
Migrate too many tables
● Some database have huge number of tables
> 500,000
● Migration by pg_dump is not practical
○ Takes too long
● Upgrade from very old version
< even 9.0
22. https://www.2ndQuadrant.com
Why FDW is good
● No specific feature/extension on the source
● Only libpq is needed
● Need postgres_fdw extension at the target
25. https://www.2ndQuadrant.com
Table migration (1)
● We can import all the remote schema
● No need to create each local table
Create Foreign Table only:
Actual data is still at remote
server.
Need to tweak
max_locks_per_transaction GUC
setting to import more tables in
single statement.
26. https://www.2ndQuadrant.com
Table Migration (2)
● Need to issue CREATE TABLE to import the data to
local server
To do this in single transaction for speedup, need
to tweak max_locks_per_transaction GUC setting.
31. https://www.2ndQuadrant.com
Result
pg_dump dump tables at PostgreSQL 9.3 1:51
pg_dump load tables to PostgreSQL 11.4 7:04
FDW Import table schema 1’18”
FDW Import all the data 2:29
We need to consider the duration needed to copy
pg_dump data from remote to local. In this case, the size
of pg_dump output was about 11Gig.
32. https://www.2ndQuadrant.com
Remarks/Summary
● No general steps to avoid long down time in
database migration
● Need to observe characteristics of application and
data itself
● Find what is dynamic and what is static
○ Beware DELETE and UPDATE
● Leverage static characteristics of the data
● Beware requirements/constraints
○ Find if you can ease requirements
■ May need to work with application
■ Plan carefully: schedule and resource
○ May need compromise
35. https://www.2ndQuadrant.com
Change enum data type
Enum type: customer_class
class1
class2
class2
priority0
priority1
normal0
normal1
internal0
internal1
Business Rule Change
->
Sometimes need both column for a while
