PGConf.ASIA 2019 Bali - 10 September 2019 Speaker: Lucky Haryadi Room: ACID Title: PostgreSQL: Business Best Practices

1. PostgreSQL: Business
Best Practices
PGConf.ASIA 2019 Bali

2. This document is owned by Equnix Business Solutions PTE LTD. This document
contains confidential information, which is protected by law. No part of this publication
could be photocopied, reproduced or translated into another language without
permitted or the express written consent of Equnix Business Solutions PTE LTD.
Data and information regarding the proposal and its offer is for limited use and are
not disclosed. The information contained in this document is subject to change at any
time without prior notice.
All rights reserved, © Copyright 2019 - Equnix Business Solutions, PT
Copyright Notice

3. A full-stack Software Solution Architect and
Developer. Proficient in Java/PHP/C/C++ also Linux
Administration. Open Source enthusiast, loves
PostgreSQL too much. Professional Open Source
Solution Architect.
Bringing Open Source Solutions to Indonesia, is
bringing a new hope for enterprise business. To save
Indonesia and grow our own resources is the total
achievement.
1. Speaker in JPUG PgConf 2014 Japan;
2. PostgreSQL trainers for basic and advanced;
3. Lecture: Open Source Technopreneurship in Campuses.
Who.M.I

4. Open Source Challenges
Inferior mentality
Of client in
perceive local
solutions
Bad rumours
of local
vendor.
A lot of time to
research, a lot of time
to test.
Keeping up the
spirits in
building the
winning team.

6. Great Disruptive
Solutions
(Innovation)
Business Model
Proprietary business model is using
Commercial Licenses, Open Source
business Model generally using Yearly
Subscription Services
Rebrand or Open Source Brand
Create our own brand and make
the product as ours, hide Open
Source thing, depends on the
license agreement. Marketing
Effort is higher.
Sell as an appliance
By rebranding from the original
and sells with the hardware and
become appliances, hides Open
Source and sells like selling
hardware product.
Creates X as a services
People don’t really need a
software, and today
everywhere is connected into
Internet. People needs a
purpose, and it can be provided
remotely (like: Software as
Services)
Maintenance/Managed Support Services
People use Open Source, we assist them in terms of
Maintenance Support or Managed Support. We become
their Principal.
Preparing the Innovation

7. Mastering Subject
Matters
(Deep Research) Understand Open Source License
(Open Source License Mastery)
Build a great Team
Support/Developer
(Solid Team Building)
Trusted
Open Source
Expert Solution
Provider
Mastering the architecture of
the system, including the
source code. Knowing what do
and don’t.
Understand the limitation of
the software, able to calculate
capacity planning.
Understand well about
licensing of Open Source
software. Doing business is
always dealing with law
and Tax.
Build a winning team, and also do continuous
improvement research. Always learning and practice.
Passionate in Software
Development
especially in
C/C++/Java
Understand about IT
System Infrastructure
Preparing the Solutions

8. Supreme Power Amongst the Other
How to become The “JEDI”

9. Manage your SLA!

10. Support Level
Vendor 1st Level Support (Reactive Support)
A. Installation, non-production and non-conversion migration
B. Vital Sign Report and Alert through Email
C. Script tools for daily operational
D. Preventive Maintenance every 3 (three) Month, and present Report
E. Schedule Standby Support for critical time (any action involving downtime)
F. Corrective Maintenance: On Phone, email, up to On Site. Follows Severity level escalation
G. Slow Query resolution

11. Support Level
Level 2: Expert Level Support
A. Configuration, Reconfiguration, Resource Adjustment
B. Performance Tuning (Hardware and Server Tuning)
C. Query Tuning (Client Side Tuning)
D. Capacity Planning, Sizing, and Scaling
E. Availability Design and implementation (HA, Hot Standby, etc)
F. Multi-tier Backup & Disaster Recovery Implementation
G. On request meeting consultation for RAS (Reliability, Availability and
Scalability)
H. PostgreSQL to PostgreSQL Data migration
I. Upgrade Feature and Version assessment and delivery decision
J. Consultation for Performance and Security

12. Support Level
Level 3: Principal Level Support
A. Bug Fixing, Performance patch
B. Consultation and advice to use PostgreSQL Efficiently and Investigate for
glitch if occurs
C. Customization if needed and charged separately.
D. Feature, Version, and Source Code operation and security Assurance.

13. Severity Levels (Maintenance)
Severity
Level
Lead Time to
Respond
Lead Time to
On-Site
Lead Time to
Resolve
1 1 hour 4 hours 4 hours
2 1 hour 12 hours 48 hours
3 1 hour Next Business Day 5 Business Days

14. Severity Levels (Managed)
No Uptime Level
Guarantee
Max Tolerable
Downtime (in
hours)
Supporting
Technology
1 95.00% 438 Cold Standby
2 98.00% 175 Warm Standby
3 99.00% 87,6 Hot Standby
4 99.90% 8,76 High Availability

15. What we shouldn’t do
1. Logic on Stored Procedures or embedded queries that are not related to the Operational
Database outside of the Managed / Maintenance Services scope. Being part of tuning
performance is a single query statement. Logic on Stored Procedures related to the business
process of the application, requiring separate services (mandays).
2. Service specific or case specific such as system recovery (system recovery).
3. Damage or Error to hardware, OS, equipment or programs not included in support.
4. Any case caused by "Force Majeure", such as Flood, Fire, Electricity Die or damage caused by
other utilities.
5. Customer refuses to comply with the rules requested by Vendor in terms of: Update Security
Patches, applying good / best practice behavior, system updates, patches, and bugfix.

16. Maintaining the Business Continuity

17. PostgreSQL Native Mirroring
Write-Ahead Log

18. 1. Cold Backup (Offline mode)
Stores data into a offline mode (file dump). There are 2 kinds of cold backup:
a. Full Backup,
Use SQL dump, or just copy the whole datadir
b. Incremental Backup.
Logs the archive and play back later.
2. Warm Backup
The backup machine is up and running, but the service is not ready yet, usually
caused by the disk synchronization (PPRC service or DRBD). So the backup is done by 3rd party other
than database itself.
3. Hot Backup
The backup is on-line, and the service is also ready to serve. It is already hot, as hot as the Master / Main
server. There are some kind of Hot backup:
a. Synchronous Replication,
b. Asynchronous Replication,
c. Multi Master Replication
Multi-tier Backup Features

19. A Backup is not a “backup” if cannot be restored
Point-in-Time Recovery mechanism, allows you to go “Going to the past” to timeline you
decide
Recovering an accidentally dropped data? Auditing a suspicious event on your database?
Can be used to replay transactions from last week until a specific day.
Point-in-Time Recovery

20. High Availability amongst PostgreSQL
Enterprise Level High Availability
configuration without hassle
Keep It Simple, SUCCESS!

21. Native Partitioning - Scalability

22. Migrating to PostgreSQL?

23. Migration STAGES
Migration Stages
A. Migration Assessment
a. Preliminary
b. Comprehensive
B. Schematic Migration
C. Data Migration (Test and
clean up)
D. Logic Migration
E. System Migration
~To win the battle, you
must understand your
enemy~ Sun Tzu.

24. Migration Assessment
- Preliminary Assessment
Given a couple of days (2 days) to
analyze all the database Objects briefly
to get some ideas on how long it take
to do such Comprehensive
Assessment.
Look for potential Stopper: System
Functions, Customization possibility,
Specific Function/Datatype, Allowable
Downtime, Any potential copyright
infrigment, etc. Develop Mitigation,
solution and possibility workaround.
- Comprehensive Assessment
Assess all the object mentioned in the
scope of work, very detailed. Produce
Comprehensive Assessment Report,
consist:
1. Level of Difficulties
2. How many Mandays Effort
3. Migration Schedules
4. Proposed Migration Rundown
5. Analysis Report

25. Schematic Migration
Most of the Structures are compatible with PostgreSQL (even not 100%).
Prepare the “bucket” first, then the “water”
❖ Dump all database object structures (tables, sequences, views, etc.)
❖ Check whether specific data types should be converted
❖ User Defined Types should be deeply examined
❖ Constraints (PK, FK, Uniqueness, Indices) will be recreated in
PostgreSQL, as long as Database Object Relationship is kept consistent

26. Data Migration - Analysis
Migration Data from Oracle to PostgreSQL is dealing with downtime and
ETL, and somehow it needs very fast.
❖ How much is the Downtime tolerance
❖ Housekeep data from Oracle Source Database
❖ Doing Stress Test to the pre-migrated data
❖ Using FDW/ETL Tools, it is possible to start synchronizing data without
interrupting current Transactional Process on Source Database

27. Logic Migration - Analysis
Logic conversion is the most PAIN of migration.
It requires a deep understanding on how the function works.
❖ Streamline the Logic
❖ How many total line of codes used on Stored
Procedures/Functions/Packages
❖ Is there any Oracle specific functions used?
❖ Create a mimic for called functions, as long as output is the same
❖ PostgreSQL is highly extensible platform, if not supported yet, we can
create it as long as we have the expertise
❖ Do System Integration and Regression Test with application

28. System Migration - Analysis
1. How long downtime can be tolerate by
Management?
2. Based on the finding from Data
Migration, we can understand how long
it take to migration the live data, this is
the main consideration to develop
Rundown System Migration
3. How many Instance Application involve
in the System Migration?
4. How many party involved in System
Migration?
Goal:
1. Develop Rundown System Migration
2. Develop fallback scenario
3. Rehearse Rundown System
Migration smoothly
4. Develop rock solid Rundown and its
fallback plan.
5. Ensure it can be run within
constraints
6. RUN SYSTEM MIGRATION

29. Example of Migration Plan (1 of 2)
NO TASKS MANDAYS
SOL/
ARCH
DB/
ARCH
DB/
ADM
DEV TW
A. Client’s DB
1 TEST DATA MIGRATION AND ANALYSIS FOR SYSTEM MIGRATION PLANNING
(RUNDOWN) 2TB
2 3 5 0 2
2 CONVERTS STORED PROCEDURES, FUNCTIONS, TRIGGERS, PACKAGES, SEQUENCES;
AND SYNTAX TESTS (FROM PL/SQL INTO PL/PGSQL) [1XX,xxx LOC]. FOR
APPS: SMARTXXX, LANXXXXXSIK, XXXXXX, MUSXXXXX
10 25 25 132 15
3 DEVELOPS TOOLS TO DO END-TO-END STRESS TESTS; AND REGRESSION TESTS; 3 3 5 10 3
4 BRAINSTORM RUNDOWN PLAN WITH EXACT DATE AND TIME 1 1 2 0 2
5 MIGRATION RUNDOWN REHEARSAL 1; RUN TESTS (STRESSED AND REGRESSION) 0 2 2 0 2

30. NO TASKS MANDAYS
SOL/
ARCH
DB/
ARCH
DB/
ADM
DEV TW
A. Client’s DB
6 MIGRATION REHEARSAL EVALUATION 1, ADD SOME DAYS FOR FIXING SPARE
1 2 2 2 2
7 MIGRATION RUNDOWN REHEARSAL 2; RUN TESTS (STRESSED AND
REGRESSION)
0 2 2 0 2
8 MIGRATION REHEARSAL EVALUATION 2, ADD SOME DAYS FOR FIXING SPARE
1 2 2 2 2
9 EXECUTE DATA AND SYSTEM MIGRATION RUNDOWN 2 3 3 0 2
10 MONITORING 0 3 5 0 2
TOTAL MANDAYS 20 46 53 146 34
Example of Migration Plan (2 of 2)

31. Deliverables
★ System Migration
★ Stabilization Monitoring
★ Documentations:
○ Data, Structure and System
○ Comprehensive Migration
Report
○ HoC
★ Comprehensive Assessment
Report
★ Data and Structure Migrated
★ DB Code Converted:
○ Stored Procedure,
○ Functions, Triggers,
○ Packages, etc.
★ Functional Tests
★ Stressed Test
★ Regression Tests

32. Deliverables
➔ Managed Services
◆ All mentioned in Premium Maintenance
services.
◆ Proactively to support Database
operational, helps administrator to
maintain the database time to time.
◆ On site every event necessary needed by
clients.
➔ Premium Maintenance Services:
◆ Preventive Maintenance every 3 months,
healthy checks, correction report, etc.
On site standby support by appointment.
◆ Corrective Maintenance: On phone, email,
chat, up to On Site based on the severity of
the case. 24 hours per day dan 7 days per
week, along the year.
◆ Installation, Configuration, Tuning, Fine
Tuning, SQL Tuning, Hardware
Consultation, Performance Consultation,
etc.

33. Make the “Recipe” (SOP)
(sop = soup (bahasa))

34. Preventive Maintenance SOP
● Every beginning of a new year, Support Administration must create a schedule of Preventive
Maintenance for 1 year and assign a PIC for every Preventive Maintenance client.
● Preventive Maintenance will be scheduled at the last working day of the month, and then will be
moved 1 day forward if the last working day of the month is unavailable, and so on.
● In 1 working day, Preventive Maintenance will be scheduled for 2 client except if we have an odd
number of client.
● If possible, do not schedule for 2 clients with a long distance from each other.
● Before creating the schedule for Preventive Maintenance, check the distance of the client from each
other, holiday date, PIC Client leave date, and specific rules from clients
● In 1 month, the same PIC will handle all the client in that month, and the PIC will be changed every
month
● If the contract of the client is ending before their schedule of Preventive Maintenance, then the
Preventive Maintenance for that client will be scheduled at the same month as their end of contract.
● If Client extends their contract then the next Preventive Maintenance will follow the schedule that have
been created in the beginning of year and will not be affected by the end of contract.
● If client have a specific rule about the schedule of Preventive Maintenance then we will follow those
rules from client

35. Tighten Escalation Procedure
1. Client can contact any support media, such as:
- Create support request to email
- Ticket Portal
2. No response on the email, in emergency condition, Client may call directly to
Support Contact Phone.
3. No response on Support Contact Phone, Client may call directly to 2nd Level
Support (Supervisor).
4. No response on 2nd Level Support, Client may call directly to 3rd Level
Support (Management).

36. How we implemented it?

37. Winning Stories - EDII
Oracle Migration
From 4 Servers Oracle RAC
System
Into 1 Server Intel
PostgreSQL

38. EDII - Current Topology

39. We Are EQUNIX.ASIA, dare to be:
info@equnix.asia | http://equnix.asia