SlideShare a Scribd company logo
1 of 22
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 1
Migration Journey: Amazon RDS to
PPCD
Vibhor Kumar | 2015.05.19
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 2
Migration Journey
Amazon RDS to PPCD
Vibhor Kumar | 2015.05.19
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 3
• A distributed relational database service.
• Running as a web service "in the cloud”
− First released on 22 October 2009 supporting MySQL
databases
− Oracle Database in June 2011
− Microsoft SQL Server in May 2012
− PostgreSQL in November 2013
About Amazon RDS
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 4
• Limited Storage Capacity – 3 TB
− RDS size limit reached
• Slow Performance
− Concurrency Issues
− Latency
• Cost
• Less control over System.
Business Challenges in RDS
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 5
• Easy, fast start-up
in minutes
• Self-healing clusters
• Scheduled backups
• Automatic failover
• Elastic storage
Adds disk space when needed
• Elastic scale out
Clusters scale automatically
• Easy to use monitoring
EASY
• Set your own scale-out and back-up parameters
• Access to set your own tuning configurations
MAXIMUM CONTROL
• Run PostgreSQL or Postgres Plus Advanced Server
• Same DB in Cloud as in data center--Portability between
environments
• AWS, OpenStack, or bare metal environments
PORTABILITY
© 2014 EnterpriseDB Corporation. All rights reserved. 5
CLOUD DATABASE
POSTGRES
PLUS
Enterprise Control
& Portability
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 6
DATABASE CLUSTER
Master
Database
Writes Reads
Streaming
Replication
Master
Replicas
Master
Replicas
Master
Replicas
AUTOMATICALLY CREATED
Connection Pooler
& Load Balancer
Postgres Plus Cloud Database
Cluster Manager
Auto-provisioning, Health
Check, Auto: Failover,
Scaling, Backup
GUI Cloud Console
Admin App
or Terminal
Cloud Resources
Network, Elastic IP, Elastic
Storage, VMs, Security,
Hardware
Client Apps/Users
Auto
Elastic
Scale-Out
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 7
Postgres Plus Cloud Database
ssh
1
2
3
4
7
6
5
8
1 Scale replicas up / down
2 Clone cluster
3 Upgrade Postgres version
4 Scale server class
5 Server DNS / Port
6 Failover modes
7 User selectable scaling thresholds
8 DB Configuration, monitors, log
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 8
• Activities performed by
customer:
− Initially started with
pg_dump and pg_restore
for data movement.
− Tried with Bucardo
Replication from Amazon
RDS to PPCD
− Bucardo Crashed
Initial engagement
• Initial expectations with
sales:
− Review their strategies
− Review the replication
− Work with them for Slony
replication review
Resulted
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 9
• Limitation learned from Amazon RDS team
• Slony replication limitation.
• Streaming Replication limitation.
Key Findings
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 10
• Proper plan required:
− Understanding of Business Constraints (Maintenance
Windows, Downtime window etc).
− Information about objects
− Active Data/objects information
− Inactive data/objects information
− Business Critical Objects.
Planning
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 11
• Factors considered
− Active tables:
− Customer facing tables (direct access from application)
− Table Sizes
− Maintenance Windows
− Inactive tables:
− Archived Tables.
− Internal reporting tables.
• Solution recommended:
− xDB replication for :
− Large Business Critical objects.
− Parallel pg_dump and pg_restore for small active tables.
− Others can be done during Migration plan implementations
Migration plan
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 12
• EDB tool for replication
− Has CLI options
• customer requirements
− Replication Monitoring.
− Data Validation.
xDB replication and Customer
requirements
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 13
• xDB limitations
− No command line monitoring option.
− Prepared statements
− No default xDB connection monitoring.
− Architecture issues.
− NaN issue
xDB replication
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 14
• Issue: Monitoring Replication.
• Solution:
− Define scripts based on xDB Meta tables
− xdb_print_last_replog.sh
− xdb_row_lag.sh
xDB limitation and Solution
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 15
• Issue: prepared statements
− Generic plan.
• Solution.
− Update catalog xDB catalog table:
− erep_pub_database.url_options=prepareThreshold=0
xDB limitation and Solution
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 16
• Issue: xDB connections Monitoring
• Solution.
− Update catalog xDB catalog table:
− erep_pub_database.url_options=‘ApplicationName=xdb&prepa
reThreshold=0’
- pg_stat_activity.application_name = ‘xdb’, shows connections
xDB limitation and Solution
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 17
• Issue: Architecture issue
− Source connection lost before updating replicated
transactions:
− Duplicate key issue
• Solution.
− xDB synchronization script:
− xdb_synch_subscription.sh
− Before perfoming sync operation, updates
erep_publication_subscriptions=‘O’
xDB limitation and Solution
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 18
• Issue: NaN (Not a Number issue)
− xDB doesn’t NaN value properly
• Solution.
− Modified shadow table to have varchar data type for specific
shadow column:
− Created a trigger on shadow table, so that NaN value stored in
shadow table should have ‘NaN’
xDB limitation and Solution
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 19
• Customers Trigger on Replicated tables
− Resulted in Data loss
• Solution :
− Ask when they have created trigger
− Verifed with Customer.
− Verified in postgresql log (proper log_statements)
− Made transactions pending which was created after Trigger in
xDB catalog rrep_txset.status = ‘P’
Customer’s Mistake
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 20
• After fixing all issues of xDB, perfored
− Final sync with xDB commands
• Performed parallel dump and restore on target
database.
• Application was pointed to
Project Wrap up
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 21
• Amazon RDS has limitations
• Online Migration from Amazon RDS to PPCD is
possible!
• Normal Solution for Migration will not work.
• Make Proper plan by Interviewing Customer.
• Be ready with before and after Migration scripts
according to Plan.
• Be ready for surprises and validate setup before
Migration activity starts.
• Have deep knowledge of tools used for Migration.
Lessons Learned
© Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 22

More Related Content

What's hot

MariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB plc
 
Introducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorIntroducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorMariaDB plc
 
How to Design for Database High Availability
How to Design for Database High AvailabilityHow to Design for Database High Availability
How to Design for Database High AvailabilityEDB
 
Deploying MariaDB databases with containers at Nokia Networks
Deploying MariaDB databases with containers at Nokia NetworksDeploying MariaDB databases with containers at Nokia Networks
Deploying MariaDB databases with containers at Nokia NetworksMariaDB plc
 
Building Apps with Distributed In-Memory Computing Using Apache Geode
Building Apps with Distributed In-Memory Computing Using Apache GeodeBuilding Apps with Distributed In-Memory Computing Using Apache Geode
Building Apps with Distributed In-Memory Computing Using Apache GeodePivotalOpenSourceHub
 
5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres DatabaseEDB
 
Introducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQLIntroducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQLMariaDB plc
 
How Orwell built a geo-distributed Bank-as-a-Service with microservices
How Orwell built a geo-distributed Bank-as-a-Service with microservicesHow Orwell built a geo-distributed Bank-as-a-Service with microservices
How Orwell built a geo-distributed Bank-as-a-Service with microservicesMariaDB plc
 
An Expert Guide to Migrating Legacy Databases to PostgreSQL
An Expert Guide to Migrating Legacy Databases to PostgreSQLAn Expert Guide to Migrating Legacy Databases to PostgreSQL
An Expert Guide to Migrating Legacy Databases to PostgreSQLEDB
 
Active/Active Database Solutions with Log Based Replication in xDB 6.0
Active/Active Database Solutions with Log Based Replication in xDB 6.0Active/Active Database Solutions with Log Based Replication in xDB 6.0
Active/Active Database Solutions with Log Based Replication in xDB 6.0EDB
 
Transform your DBMS to drive engagement innovation with Big Data
Transform your DBMS to drive engagement innovation with Big DataTransform your DBMS to drive engagement innovation with Big Data
Transform your DBMS to drive engagement innovation with Big DataAshnikbiz
 
DBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWSDBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWSEDB
 
How we switched to columnar at SpendHQ
How we switched to columnar at SpendHQHow we switched to columnar at SpendHQ
How we switched to columnar at SpendHQMariaDB plc
 
eBay Cloud CMS based on NOSQL
eBay Cloud CMS based on NOSQLeBay Cloud CMS based on NOSQL
eBay Cloud CMS based on NOSQLXu Jiang
 
Under the hood: SkySQL monitoring
Under the hood: SkySQL monitoringUnder the hood: SkySQL monitoring
Under the hood: SkySQL monitoringMariaDB plc
 
Automating Postgres Deployments on AWS and VMware, with Terraform and Ansible
Automating Postgres Deployments on AWS and VMware, with Terraform and AnsibleAutomating Postgres Deployments on AWS and VMware, with Terraform and Ansible
Automating Postgres Deployments on AWS and VMware, with Terraform and AnsibleEDB
 
Why you should use native packages to install PostgreSQL on Linux
Why you should use native packages to install PostgreSQL on LinuxWhy you should use native packages to install PostgreSQL on Linux
Why you should use native packages to install PostgreSQL on LinuxEDB
 
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...Severalnines
 
How to Monitor Postgres Like a Pro!
How to Monitor Postgres Like a Pro!How to Monitor Postgres Like a Pro!
How to Monitor Postgres Like a Pro!EDB
 
The Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data CenterThe Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data CenterEDB
 

What's hot (20)

MariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introduction
 
Introducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorIntroducing the R2DBC async Java connector
Introducing the R2DBC async Java connector
 
How to Design for Database High Availability
How to Design for Database High AvailabilityHow to Design for Database High Availability
How to Design for Database High Availability
 
Deploying MariaDB databases with containers at Nokia Networks
Deploying MariaDB databases with containers at Nokia NetworksDeploying MariaDB databases with containers at Nokia Networks
Deploying MariaDB databases with containers at Nokia Networks
 
Building Apps with Distributed In-Memory Computing Using Apache Geode
Building Apps with Distributed In-Memory Computing Using Apache GeodeBuilding Apps with Distributed In-Memory Computing Using Apache Geode
Building Apps with Distributed In-Memory Computing Using Apache Geode
 
5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database
 
Introducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQLIntroducing the ultimate MariaDB cloud, SkySQL
Introducing the ultimate MariaDB cloud, SkySQL
 
How Orwell built a geo-distributed Bank-as-a-Service with microservices
How Orwell built a geo-distributed Bank-as-a-Service with microservicesHow Orwell built a geo-distributed Bank-as-a-Service with microservices
How Orwell built a geo-distributed Bank-as-a-Service with microservices
 
An Expert Guide to Migrating Legacy Databases to PostgreSQL
An Expert Guide to Migrating Legacy Databases to PostgreSQLAn Expert Guide to Migrating Legacy Databases to PostgreSQL
An Expert Guide to Migrating Legacy Databases to PostgreSQL
 
Active/Active Database Solutions with Log Based Replication in xDB 6.0
Active/Active Database Solutions with Log Based Replication in xDB 6.0Active/Active Database Solutions with Log Based Replication in xDB 6.0
Active/Active Database Solutions with Log Based Replication in xDB 6.0
 
Transform your DBMS to drive engagement innovation with Big Data
Transform your DBMS to drive engagement innovation with Big DataTransform your DBMS to drive engagement innovation with Big Data
Transform your DBMS to drive engagement innovation with Big Data
 
DBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWSDBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWS
 
How we switched to columnar at SpendHQ
How we switched to columnar at SpendHQHow we switched to columnar at SpendHQ
How we switched to columnar at SpendHQ
 
eBay Cloud CMS based on NOSQL
eBay Cloud CMS based on NOSQLeBay Cloud CMS based on NOSQL
eBay Cloud CMS based on NOSQL
 
Under the hood: SkySQL monitoring
Under the hood: SkySQL monitoringUnder the hood: SkySQL monitoring
Under the hood: SkySQL monitoring
 
Automating Postgres Deployments on AWS and VMware, with Terraform and Ansible
Automating Postgres Deployments on AWS and VMware, with Terraform and AnsibleAutomating Postgres Deployments on AWS and VMware, with Terraform and Ansible
Automating Postgres Deployments on AWS and VMware, with Terraform and Ansible
 
Why you should use native packages to install PostgreSQL on Linux
Why you should use native packages to install PostgreSQL on LinuxWhy you should use native packages to install PostgreSQL on Linux
Why you should use native packages to install PostgreSQL on Linux
 
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
 
How to Monitor Postgres Like a Pro!
How to Monitor Postgres Like a Pro!How to Monitor Postgres Like a Pro!
How to Monitor Postgres Like a Pro!
 
The Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data CenterThe Need For Speed - Strategies to Modernize Your Data Center
The Need For Speed - Strategies to Modernize Your Data Center
 

Viewers also liked

Postgres Plus Cloud Database
Postgres Plus Cloud DatabasePostgres Plus Cloud Database
Postgres Plus Cloud DatabaseAshnikbiz
 
How To Reach Your Goals with Postgres Plus Cloud Database
How To Reach Your Goals with Postgres Plus Cloud DatabaseHow To Reach Your Goals with Postgres Plus Cloud Database
How To Reach Your Goals with Postgres Plus Cloud DatabaseEDB
 
Best Practices: Migrating a Postgres Production Database to the Cloud
Best Practices: Migrating a Postgres Production Database to the CloudBest Practices: Migrating a Postgres Production Database to the Cloud
Best Practices: Migrating a Postgres Production Database to the CloudEDB
 
Postgres Integrates Effectively in the "Enterprise Sandbox"
Postgres Integrates Effectively in the "Enterprise Sandbox"Postgres Integrates Effectively in the "Enterprise Sandbox"
Postgres Integrates Effectively in the "Enterprise Sandbox"EDB
 
Expanding with EDB Postgres Advanced Server 9.5
Expanding with EDB Postgres Advanced Server 9.5Expanding with EDB Postgres Advanced Server 9.5
Expanding with EDB Postgres Advanced Server 9.5EDB
 
PostgreSQL 9.5 - Major Features
PostgreSQL 9.5 - Major FeaturesPostgreSQL 9.5 - Major Features
PostgreSQL 9.5 - Major FeaturesInMobi Technology
 
Overview of Postgres 9.5
Overview of Postgres 9.5 Overview of Postgres 9.5
Overview of Postgres 9.5 EDB
 

Viewers also liked (7)

Postgres Plus Cloud Database
Postgres Plus Cloud DatabasePostgres Plus Cloud Database
Postgres Plus Cloud Database
 
How To Reach Your Goals with Postgres Plus Cloud Database
How To Reach Your Goals with Postgres Plus Cloud DatabaseHow To Reach Your Goals with Postgres Plus Cloud Database
How To Reach Your Goals with Postgres Plus Cloud Database
 
Best Practices: Migrating a Postgres Production Database to the Cloud
Best Practices: Migrating a Postgres Production Database to the CloudBest Practices: Migrating a Postgres Production Database to the Cloud
Best Practices: Migrating a Postgres Production Database to the Cloud
 
Postgres Integrates Effectively in the "Enterprise Sandbox"
Postgres Integrates Effectively in the "Enterprise Sandbox"Postgres Integrates Effectively in the "Enterprise Sandbox"
Postgres Integrates Effectively in the "Enterprise Sandbox"
 
Expanding with EDB Postgres Advanced Server 9.5
Expanding with EDB Postgres Advanced Server 9.5Expanding with EDB Postgres Advanced Server 9.5
Expanding with EDB Postgres Advanced Server 9.5
 
PostgreSQL 9.5 - Major Features
PostgreSQL 9.5 - Major FeaturesPostgreSQL 9.5 - Major Features
PostgreSQL 9.5 - Major Features
 
Overview of Postgres 9.5
Overview of Postgres 9.5 Overview of Postgres 9.5
Overview of Postgres 9.5
 

Similar to PPCD_And_AmazonRDS

EDB Postgres Platform 11 Webinar
EDB Postgres Platform 11 WebinarEDB Postgres Platform 11 Webinar
EDB Postgres Platform 11 WebinarEDB
 
The Central View of your Data with Postgres
The Central View of your Data with PostgresThe Central View of your Data with Postgres
The Central View of your Data with PostgresEDB
 
IMS05 IMS V14 8gb osam for haldb
IMS05   IMS V14 8gb osam for haldbIMS05   IMS V14 8gb osam for haldb
IMS05 IMS V14 8gb osam for haldbRobert Hain
 
The New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and MoreThe New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and MoreMariaDB Corporation
 
The New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and moreThe New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and moreMariaDB Corporation
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13EDB
 
Top 10 Tips for an Effective Postgres Deployment
Top 10 Tips for an Effective Postgres DeploymentTop 10 Tips for an Effective Postgres Deployment
Top 10 Tips for an Effective Postgres DeploymentEDB
 
Neuerungen in EDB Postgres 11
Neuerungen in EDB Postgres 11Neuerungen in EDB Postgres 11
Neuerungen in EDB Postgres 11EDB
 
Les nouveautés d'EDB Postgres 11
Les nouveautés d'EDB Postgres 11Les nouveautés d'EDB Postgres 11
Les nouveautés d'EDB Postgres 11EDB
 
9.6_Course Material-Postgresql_002.pdf
9.6_Course Material-Postgresql_002.pdf9.6_Course Material-Postgresql_002.pdf
9.6_Course Material-Postgresql_002.pdfsreedb2
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to PostgresEDB
 
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...Mydbops
 
Big data and polyglot solutions
Big data and polyglot solutionsBig data and polyglot solutions
Big data and polyglot solutionsKumaran Ramanujam
 
An overview of reference architectures for Postgres
An overview of reference architectures for PostgresAn overview of reference architectures for Postgres
An overview of reference architectures for PostgresEDB
 
LDAP at Lightning Speed
 LDAP at Lightning Speed LDAP at Lightning Speed
LDAP at Lightning SpeedC4Media
 
Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...
Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...
Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...avanttic Consultoría Tecnológica
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13EDB
 
[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...
[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...
[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...Insight Technology, Inc.
 
Oracle big data appliance and solutions
Oracle big data appliance and solutionsOracle big data appliance and solutions
Oracle big data appliance and solutionssolarisyougood
 
Optimize with Open Source
Optimize with Open SourceOptimize with Open Source
Optimize with Open SourceEDB
 

Similar to PPCD_And_AmazonRDS (20)

EDB Postgres Platform 11 Webinar
EDB Postgres Platform 11 WebinarEDB Postgres Platform 11 Webinar
EDB Postgres Platform 11 Webinar
 
The Central View of your Data with Postgres
The Central View of your Data with PostgresThe Central View of your Data with Postgres
The Central View of your Data with Postgres
 
IMS05 IMS V14 8gb osam for haldb
IMS05   IMS V14 8gb osam for haldbIMS05   IMS V14 8gb osam for haldb
IMS05 IMS V14 8gb osam for haldb
 
The New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and MoreThe New MariaDB Offering: MariaDB 10, MaxScale and More
The New MariaDB Offering: MariaDB 10, MaxScale and More
 
The New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and moreThe New MariaDB Offering - MariaDB 10, MaxScale and more
The New MariaDB Offering - MariaDB 10, MaxScale and more
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13
 
Top 10 Tips for an Effective Postgres Deployment
Top 10 Tips for an Effective Postgres DeploymentTop 10 Tips for an Effective Postgres Deployment
Top 10 Tips for an Effective Postgres Deployment
 
Neuerungen in EDB Postgres 11
Neuerungen in EDB Postgres 11Neuerungen in EDB Postgres 11
Neuerungen in EDB Postgres 11
 
Les nouveautés d'EDB Postgres 11
Les nouveautés d'EDB Postgres 11Les nouveautés d'EDB Postgres 11
Les nouveautés d'EDB Postgres 11
 
9.6_Course Material-Postgresql_002.pdf
9.6_Course Material-Postgresql_002.pdf9.6_Course Material-Postgresql_002.pdf
9.6_Course Material-Postgresql_002.pdf
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
 
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availabilit...
 
Big data and polyglot solutions
Big data and polyglot solutionsBig data and polyglot solutions
Big data and polyglot solutions
 
An overview of reference architectures for Postgres
An overview of reference architectures for PostgresAn overview of reference architectures for Postgres
An overview of reference architectures for Postgres
 
LDAP at Lightning Speed
 LDAP at Lightning Speed LDAP at Lightning Speed
LDAP at Lightning Speed
 
Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...
Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...
Meetup Oracle Database: 3 Analizar, Aconsejar, Automatizar… las nuevas funcio...
 
New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13New enhancements for security and usability in EDB 13
New enhancements for security and usability in EDB 13
 
[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...
[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...
[db tech showcase Tokyo 2016] E34: Oracle SE - RAC, HA and Standby are Still ...
 
Oracle big data appliance and solutions
Oracle big data appliance and solutionsOracle big data appliance and solutions
Oracle big data appliance and solutions
 
Optimize with Open Source
Optimize with Open SourceOptimize with Open Source
Optimize with Open Source
 

PPCD_And_AmazonRDS

  • 1. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 1 Migration Journey: Amazon RDS to PPCD Vibhor Kumar | 2015.05.19
  • 2. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 2 Migration Journey Amazon RDS to PPCD Vibhor Kumar | 2015.05.19
  • 3. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 3 • A distributed relational database service. • Running as a web service "in the cloud” − First released on 22 October 2009 supporting MySQL databases − Oracle Database in June 2011 − Microsoft SQL Server in May 2012 − PostgreSQL in November 2013 About Amazon RDS
  • 4. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 4 • Limited Storage Capacity – 3 TB − RDS size limit reached • Slow Performance − Concurrency Issues − Latency • Cost • Less control over System. Business Challenges in RDS
  • 5. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 5 • Easy, fast start-up in minutes • Self-healing clusters • Scheduled backups • Automatic failover • Elastic storage Adds disk space when needed • Elastic scale out Clusters scale automatically • Easy to use monitoring EASY • Set your own scale-out and back-up parameters • Access to set your own tuning configurations MAXIMUM CONTROL • Run PostgreSQL or Postgres Plus Advanced Server • Same DB in Cloud as in data center--Portability between environments • AWS, OpenStack, or bare metal environments PORTABILITY © 2014 EnterpriseDB Corporation. All rights reserved. 5 CLOUD DATABASE POSTGRES PLUS Enterprise Control & Portability
  • 6. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 6 DATABASE CLUSTER Master Database Writes Reads Streaming Replication Master Replicas Master Replicas Master Replicas AUTOMATICALLY CREATED Connection Pooler & Load Balancer Postgres Plus Cloud Database Cluster Manager Auto-provisioning, Health Check, Auto: Failover, Scaling, Backup GUI Cloud Console Admin App or Terminal Cloud Resources Network, Elastic IP, Elastic Storage, VMs, Security, Hardware Client Apps/Users Auto Elastic Scale-Out
  • 7. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 7 Postgres Plus Cloud Database ssh 1 2 3 4 7 6 5 8 1 Scale replicas up / down 2 Clone cluster 3 Upgrade Postgres version 4 Scale server class 5 Server DNS / Port 6 Failover modes 7 User selectable scaling thresholds 8 DB Configuration, monitors, log
  • 8. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 8 • Activities performed by customer: − Initially started with pg_dump and pg_restore for data movement. − Tried with Bucardo Replication from Amazon RDS to PPCD − Bucardo Crashed Initial engagement • Initial expectations with sales: − Review their strategies − Review the replication − Work with them for Slony replication review Resulted
  • 9. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 9 • Limitation learned from Amazon RDS team • Slony replication limitation. • Streaming Replication limitation. Key Findings
  • 10. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 10 • Proper plan required: − Understanding of Business Constraints (Maintenance Windows, Downtime window etc). − Information about objects − Active Data/objects information − Inactive data/objects information − Business Critical Objects. Planning
  • 11. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 11 • Factors considered − Active tables: − Customer facing tables (direct access from application) − Table Sizes − Maintenance Windows − Inactive tables: − Archived Tables. − Internal reporting tables. • Solution recommended: − xDB replication for : − Large Business Critical objects. − Parallel pg_dump and pg_restore for small active tables. − Others can be done during Migration plan implementations Migration plan
  • 12. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 12 • EDB tool for replication − Has CLI options • customer requirements − Replication Monitoring. − Data Validation. xDB replication and Customer requirements
  • 13. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 13 • xDB limitations − No command line monitoring option. − Prepared statements − No default xDB connection monitoring. − Architecture issues. − NaN issue xDB replication
  • 14. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 14 • Issue: Monitoring Replication. • Solution: − Define scripts based on xDB Meta tables − xdb_print_last_replog.sh − xdb_row_lag.sh xDB limitation and Solution
  • 15. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 15 • Issue: prepared statements − Generic plan. • Solution. − Update catalog xDB catalog table: − erep_pub_database.url_options=prepareThreshold=0 xDB limitation and Solution
  • 16. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 16 • Issue: xDB connections Monitoring • Solution. − Update catalog xDB catalog table: − erep_pub_database.url_options=‘ApplicationName=xdb&prepa reThreshold=0’ - pg_stat_activity.application_name = ‘xdb’, shows connections xDB limitation and Solution
  • 17. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 17 • Issue: Architecture issue − Source connection lost before updating replicated transactions: − Duplicate key issue • Solution. − xDB synchronization script: − xdb_synch_subscription.sh − Before perfoming sync operation, updates erep_publication_subscriptions=‘O’ xDB limitation and Solution
  • 18. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 18 • Issue: NaN (Not a Number issue) − xDB doesn’t NaN value properly • Solution. − Modified shadow table to have varchar data type for specific shadow column: − Created a trigger on shadow table, so that NaN value stored in shadow table should have ‘NaN’ xDB limitation and Solution
  • 19. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 19 • Customers Trigger on Replicated tables − Resulted in Data loss • Solution : − Ask when they have created trigger − Verifed with Customer. − Verified in postgresql log (proper log_statements) − Made transactions pending which was created after Trigger in xDB catalog rrep_txset.status = ‘P’ Customer’s Mistake
  • 20. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 20 • After fixing all issues of xDB, perfored − Final sync with xDB commands • Performed parallel dump and restore on target database. • Application was pointed to Project Wrap up
  • 21. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 21 • Amazon RDS has limitations • Online Migration from Amazon RDS to PPCD is possible! • Normal Solution for Migration will not work. • Make Proper plan by Interviewing Customer. • Be ready with before and after Migration scripts according to Plan. • Be ready for surprises and validate setup before Migration activity starts. • Have deep knowledge of tools used for Migration. Lessons Learned
  • 22. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 22

Editor's Notes

  1. This PowerPoint Template includes a series of slide masters with predefined layouts and color schemes for formatting slides Slide Masters are displayed when you right click on a slide and select Layout from menu
  2. This PowerPoint Template includes a series of slide masters with predefined layouts and color schemes for formatting slides Slide Masters are displayed when you right click on a slide and select Layout from menu
  3. KEY TAKEAWAY: PPCD combines on-premise database features with the scalability, elasticity, ease of use, and low cost of virtual cloud infrastructures. Easy: Easy, fast start-up in minutes Elastic storage—adds disk space when needed Elastic scale out—adds clusters automatically based on usage levels Self-healing clusters Scheduled backups Automatic failover Easy to use monitoring Maximum control: Private instances—not shared Set your own scale-out and back-up parameters Access to set your own tuning configurations Portability: Run PostgreSQL or Postgres Plus Advanced Server Same DB in Cloud as in DC—portability between environments AWS, OpenStack, CloudStack or bare metal environments Sample script: “Cloud computing can be beneficial both from a cost savings perspective and from a flexibility perspective. Many of our customers are moving or at least considering the move to cloud solutions for IT. You pay only for what you use, you save costs on the reduction of physical environments, including energy costs. EDB’s PPCD is arguably the most flexible, easy to use database service available today. Not only can one set up a database cluster in a matter of minutes, PPCD automates so many of the admin tasks that developers can spend more time on their app and less time worrying about the database….and DBAs have virtually nothing to do—unless they want to. What I mean by that is PPCD automates scale up, adding storage, failover, back up, etc. But we also give the ‘super user’ the ability to make changes to the architecture in a way that other cloud db services do not. And its flexibility and portability is second to none. All of the cloud features of PPCD reside in the cluster manager console….not in the database. This means our customers have the ability to move workloads to and from the cloud seamlessly because the db is exactly the same whether it’s a physical db instance, a virtualized db or in the cloud.”
  4. KEY TAKEAWAY: The PPCD console is easy to use by DBAs and Developers for self-service deployments. A smart GUI console makes quick work for DBAs. Developer self service is also an option.