Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features

17 005 vues

Publié le

Learn the specifics of Amazon RDS for PostgreSQL’s capabilities and extensions that make it powerful. This session begins with a brief overview of the RDS PostgreSQL service, how it provides High Availability & Durability and will then deep dive into the new features that we have released since re:Invent 2014, including major version upgrade and newly added PostgreSQL extensions to RDS PostgreSQL. During the session, we will also discuss lessons learned running a large fleet of PostgreSQL instances, including specific recommendations. In addition we will present benchmarking results looking at differences between the 9.3, 9.4 and 9.5 releases.

Publié dans : Technologie
  • Soyez le premier à commenter

(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Grant McAlister – Senior Principal Engineer - RDS October 2015 DAT402 Amazon RDS for PostgreSQL Lessons Learned and Deep Dive on New Features
  2. 2. Major version upgrade Coming Soon Prod 9.3 Prod 9.4 pg_upgrade Backup Backup No PITR Test 9.3 Test 9.4 pg_upgrade Restore to a test instance Application Testing
  3. 3. What’s new in storage 6TB storage • PIOPS has 30K IOPS max • GP2 increase storage above 3TB = increase throughput & IOPS Encryption at rest • Uses the AWS Key Management Service (KMS) part of AWS Identity and Access Management (IAM) • Your own key • Use a default one • Includes all data files, log files, log backups, and snapshots
  4. 4. 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000 2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads TransactionsPerSecond(TPS) PG Bench - Read Only - In Memory Regular Encrypted Encryption at rest overhead No measureable overhead
  5. 5. 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 2 Threads 4 Threads 8 Threads 16 Threads 32 Threads 64 Threads TransactionsPerSecond(TPS) PG Bench - Read & Write Regular Encrypted Encryption at rest overhead 5 to 10% Overhead on heavy write
  6. 6. Version updates RDS now supports • 9.3.6 – Fix for RDS Bug – RESET ALL • 9.3.9 (Default) • 9.4.1 and 9.4.4 (Default) • JSONB • GIN Index Improvements • pg_prewarm extension • New PLV8 & PostGIS versions
  7. 7. Operating System (OS) metrics 5 second granularity Coming SooncpuUtilization • guest • irq • system • wait • idl: • user • total • steal • nice diskIO • writeKbPS • readIOsPS • await • readKbPS • rrqmPS • util • avgQueueLen • tps • readKb • writeKb • avgReqSz • wrqmPS • writeIOsPS memory • writeback • cached • free • inactive • dirty • mapped • active • total • slab • buffers • pageTable swap • cached • total • free tasks • sleeping • zombie • running • stopped • total • blocked fileSys • used • usedFiles • usedFilePercent • maxFiles • total • usedPercent loadAverageMinute • fifteen • five • one uptime processList • name • cpuTime • parentID • memoryUsedPct • cpuUsedPct • id • rss • vss
  8. 8. OS metrics
  9. 9. Data movement
  10. 10. Move data to the same or different database engine Keep your apps running during the migration Start your first migration in 10 minutes or less Replicate within, to, or from AWS EC2 or RDS AWS Database Migration Service
  11. 11. Customer Premises Application Users EC2 or RDS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync Switch applications over to the target at your convenience Keep your apps running during the migration AWS Database Migration Service
  12. 12. AWS Database Migration Service - PostgreSQL • Source - on premises or Amazon EC2 PostgreSQL (9.4) • Destination can be EC2 or RDS • Initial bulk copy via consistent select • Uses PostgreSQL logical replication support to provide change data capture http://aws.amazon.com/rds/DatabaseMigrationService/preview
  13. 13. Loading data • Disable backups – backup_retention=0 • Disable Multi-AZ & autovacuum • pg_dump –Fc (compressed) pg_restore –j (parallel) • Increase maintenance_work_mem • Increase checkpoint_segments & checkpoint_timeout • Disable FSYNC • Disable synchronous_commit
  14. 14. 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 both on fsync=0 sync commit=0 fsync=0 & sync commit=0 TransactionsperSecond 32 thread insert- fsync vs sync commit 16 segments 256 segments
  15. 15. 0 20 40 60 80 100 120 140 160 both on fsync=0 sync commit=0 fsync=0 & sync commit=0 Time-Seconds Bulk load 2GB of data -fsync vs sync commit 16 segments 256 segments
  16. 16. 29.1 28.8 26.1 25.223.9 0 5 10 15 20 25 30 35 fsync=1 & sync commit=0 fsync=0 & sync commit=0 Time-Minutes Index build on 20GB table maintenance_work_mem=16MB & checkpoint_segments=16 maintenance_work_mem=1024MB & checkpoint_segments=16 maintenance_work_mem=1024MB & checkpoint_segments=1024
  17. 17. Vacuuming – 100% read-only workload
  18. 18. Vacuum parameters Will auto vacuum when • autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor * pgclass.reltuples How hard auto vacuum works • autovacuum_max_workers • autovacuum_nap_time • autovacuum_cost_limit • autovacuum_cost_delay
  19. 19. postgres_fdw + Amazon Redshift
  20. 20. session_replication_role Table Foo Trigger Table Foo Trigger DB1 DB2 insert
  21. 21. Scale and availability
  22. 22. shared_buffers parameter 244GB RAM PG processes shared_buffers Linux pagecache select of data – check for buffer in shared_buffers if not in shared_buffers load from pagecache/disk EBS 1/4 shared_buffers = working set size
  23. 23. 0 2,000 4,000 6,000 8,000 10,000 12,000 3% 6% 13% 25% 50% 75% transactionspersecond(TPS) shared_buffers as a percentage of system memory pgbench write workload on r3.8xlarge working set = 10% of memory 25 threads 50 threads 100 threads 200 threads 400 threads 800 threads
  24. 24. 0 2,000 4,000 6,000 8,000 10,000 12,000 13% 25% 50% 75% transactionspersecond(TPS) shared_buffers as a percentage of system memory pgbench write workload on r3.8xlarge working set = 50% of memory 25 threads 50 threads 100 threads 200 threads 400 threads 800 threads
  25. 25. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update Primary Update
  26. 26. Read Replicas = Availability Sync Replication Multi-AZ Async Replication
  27. 27. Read Replica promotion AZ1 AZ2 AZ3
  28. 28. Read Replicas = Scale AZ1 AZ2 AZ3
  29. 29. Replication parameters wal_keep_segments xlog1 xlog2 xlog3 xlog99 xlog1 xlog1
  30. 30. pg_stat_replication benchdb=> select * from pg_stat_replication; -[ RECORD 1 ]----+-------------------------------------------- pid | 40385 usesysid | 16388 usename | rdsrepladmin application_name | walreceiver client_addr | 10.22.132.253 client_hostname | ip-10-22-132-253.us-west-2.compute.internal client_port | 22825 backend_start | 2014-10-29 21:44:58.080324+00 state | streaming sent_location | 98/7A000900 write_location | 98/7A000900 flush_location | 98/7A000900 replay_location | 98/7A000900 sync_priority | 0 sync_state | async
  31. 31. Replication parameters – continued vacuum_defer_cleanup_age max_standby_archive_delay max_standby_streaming_delay hot_standby_feedback A - Foo A- Bar Source A - Foo A- Bar Replica
  32. 32. vacuum_defer_cleanup_age on primary default is 0 # of transactions Table T1 t1 – foo, bar t2 – foo, car t3 – foo, dar t4 – foo, ear t5 – foo, far t6 – foo, gar t1 – foo, bar t2 – foo, car t3 – foo, dar t4 – foo, ear t5 – foo, far
  33. 33. max_standby_archive/streaming_delay xlog1 Not all sessions will see the max delay
  34. 34. hot_standby_feedback xlog1 select * from t1select * from t1
  35. 35. pg_stat_database_conflicts benchdb=> select * from pg_stat_database_conflicts; datid | datname | confl_tablespace | confl_lock | confl_snapshot | confl_bufferpin | confl_deadlock -------+-----------+------------------+------------+----------------+-----------------+---------------- 12891 | template0 | 0 | 0 | 0 | 0 | 0 16384 | rdsadmin | 0 | 0 | 0 | 0 | 0 1 | template1 | 0 | 0 | 0 | 0 | 0 12896 | postgres | 0 | 0 | 0 | 0 | 0 16394 | benchdb | 0 | 0 | 0 | 0 | 0 32810 | bench2 | 0 | 0 | 1 | 0 | 0
  36. 36. pg_stat_statements Change parameter shared_preload_libraries=pg_stat_statements =>create extenstion pg_stats_statements =>select query, calls, total_time, rows, shared_blks_read from pg_stat_statements where total_time > 100 and query like '%usertable%'; query | calls | total_time | rows | shared_blks_read -------------------------------------------------------------------------------------------+----------+------------------+------------+----------------- SELECT * FROM usertable WHERE YCSB_KEY = $1 | 71356782 | 8629119.24887683 | 71356780 | 28779668 SELECT * FROM usertable WHERE YCSB_KEY >= $1 LIMIT ? | 12068394 | 62530609.930002 | 1206839246 | 171093346 UPDATE usertable SET FIELD1=$1 WHERE YCSB_KEY = $2 | 7048967 | 35813107.3580354 | 7048967 | 3825857 analyze usertable; | 1 | 2129.84 | 0 | 15679 SELECT * FROM usertable WHERE YCSB_KEY >= $1 AND md5(YCSB_KEY) = md5(YCSB_KEY) LIMIT ? | 15441280 | 39356905.8080029 | 1544127640 | 230668106
  37. 37. Burst mode: GP2 and T2 T2 – Amazon EC2 instance with burst capability • Base performance + burst • Earn credits per hour when below base performance • Can store up to 24 hours worth of credits • Amazon CloudWatch metrics to see credits and usage GP2 – SSD-based Amazon EBS storage • 3 IOPS per GB base performance • Earn credits when usage below base • Burst to 3000+ IOPS
  38. 38. T2 – CPU credits
  39. 39. Burst mode: what’s new db.t2.large • 60 CPU Initial Credit • 36 CPU Credit earned per hour • Base Performance – 60% • 8 GB RAM • Increased IO bandwidth • Encryption at rest support
  40. 40. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard Burst mode vs. classic vs. Provisioned IOPS $0.58 per hour
  41. 41. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS Burst mode vs. classic vs. Provisioned IOPS $0.58 per hour $0.40 per hour
  42. 42. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS Burst mode vs. classic vs. Provisioned IOPS $0.58 per hour $0.40 per hour $0.50 per hour
  43. 43. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.40 per hour $0.50 per hour
  44. 44. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 db.t2.medium + 1TB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.23 per hour $0.40 per hour $0.50 per hour
  45. 45. 0 2000 4000 6000 8000 10000 12000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 TransactionsperSecond(TPS) Hours 100% Read - 20GB data db.m1.medium + 200GB standard db.m3.medium + 200G + 2000 IOPS db.m3.large + 200G + 2000 IOPS db.t2.medium + 200GB gp2 db.t2.medium + 1TB gp2 db.t2.large + 1TB gp2 Burst mode vs. Classic vs. Provisioned IOPS $0.10 per hour $0.58 per hour $0.23 per hour $0.40 per hour $0.50 per hour $0.30 per hour
  46. 46. Thank you!
  47. 47. Remember to complete your evaluations!

×