Submit Search
Upload
Feed Burner Scalability
•
23 likes
•
3,915 views
didip
Follow
See more scalability tales at: http://rapd.wordpress.com
Read less
Read more
Business
Technology
Report
Share
Report
Share
1 of 33
Download now
Download to read offline
Recommended
Scaling Twitter 12758
Scaling Twitter 12758
davidblum
Spark / Mesos Cluster Optimization
Spark / Mesos Cluster Optimization
ebiznext
JahiaOne - Performance Tuning
JahiaOne - Performance Tuning
Jahia Solutions Group
PGConf.ASIA 2019 Bali - Mission Critical Production High Availability Postgre...
PGConf.ASIA 2019 Bali - Mission Critical Production High Availability Postgre...
Equnix Business Solutions
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
Equnix Business Solutions
vSphere vStorage: Troubleshooting Performance
vSphere vStorage: Troubleshooting Performance
ProfessionalVMware
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backup
Saewoong Lee
Introduction to Stacki at Atlanta Meetup February 2016
Introduction to Stacki at Atlanta Meetup February 2016
StackIQ
Recommended
Scaling Twitter 12758
Scaling Twitter 12758
davidblum
Spark / Mesos Cluster Optimization
Spark / Mesos Cluster Optimization
ebiznext
JahiaOne - Performance Tuning
JahiaOne - Performance Tuning
Jahia Solutions Group
PGConf.ASIA 2019 Bali - Mission Critical Production High Availability Postgre...
PGConf.ASIA 2019 Bali - Mission Critical Production High Availability Postgre...
Equnix Business Solutions
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
PGConf.ASIA 2019 Bali - Building PostgreSQL as a Service with Kubernetes - Ta...
Equnix Business Solutions
vSphere vStorage: Troubleshooting Performance
vSphere vStorage: Troubleshooting Performance
ProfessionalVMware
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backup
Saewoong Lee
Introduction to Stacki at Atlanta Meetup February 2016
Introduction to Stacki at Atlanta Meetup February 2016
StackIQ
How Prometheus Store the Data
How Prometheus Store the Data
Hao Chen
MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527
Saewoong Lee
Wido den Hollander - 10 ways to break your Ceph cluster
Wido den Hollander - 10 ways to break your Ceph cluster
ShapeBlue
Salesforce at Stacki Atlanta Meetup February 2016
Salesforce at Stacki Atlanta Meetup February 2016
StackIQ
Varnish Configuration Step by Step
Varnish Configuration Step by Step
Kim Stefan Lindholm
Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2
Denish Patel
Webinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera Cluster
Severalnines
Logical replication with pglogical
Logical replication with pglogical
Umair Shahid
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
DataStax Academy
Spark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting Guide
IBM
Oracle: Binding versus caging
Oracle: Binding versus caging
BertrandDrouvot
Build Automation 101
Build Automation 101
Martin Jackson
Postgres in Amazon RDS
Postgres in Amazon RDS
Denish Patel
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Severalnines
IT Infrastructure Through The Public Network Challenges And Solutions
IT Infrastructure Through The Public Network Challenges And Solutions
Martin Jackson
Understanding DSE Search by Matt Stump
Understanding DSE Search by Matt Stump
DataStax
JavaOne 2014: Taming the Cloud Database with jclouds
JavaOne 2014: Taming the Cloud Database with jclouds
zshoylev
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
DataStax
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Apache Traffic Server & Lua
Apache Traffic Server & Lua
Kit Chan
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
Lenz Grimmer
[Roblek] Distributed computing in practice
[Roblek] Distributed computing in practice
javablend
More Related Content
What's hot
How Prometheus Store the Data
How Prometheus Store the Data
Hao Chen
MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527
Saewoong Lee
Wido den Hollander - 10 ways to break your Ceph cluster
Wido den Hollander - 10 ways to break your Ceph cluster
ShapeBlue
Salesforce at Stacki Atlanta Meetup February 2016
Salesforce at Stacki Atlanta Meetup February 2016
StackIQ
Varnish Configuration Step by Step
Varnish Configuration Step by Step
Kim Stefan Lindholm
Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2
Denish Patel
Webinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera Cluster
Severalnines
Logical replication with pglogical
Logical replication with pglogical
Umair Shahid
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
DataStax Academy
Spark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting Guide
IBM
Oracle: Binding versus caging
Oracle: Binding versus caging
BertrandDrouvot
Build Automation 101
Build Automation 101
Martin Jackson
Postgres in Amazon RDS
Postgres in Amazon RDS
Denish Patel
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Severalnines
IT Infrastructure Through The Public Network Challenges And Solutions
IT Infrastructure Through The Public Network Challenges And Solutions
Martin Jackson
Understanding DSE Search by Matt Stump
Understanding DSE Search by Matt Stump
DataStax
JavaOne 2014: Taming the Cloud Database with jclouds
JavaOne 2014: Taming the Cloud Database with jclouds
zshoylev
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
DataStax
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Apache Traffic Server & Lua
Apache Traffic Server & Lua
Kit Chan
What's hot
(20)
How Prometheus Store the Data
How Prometheus Store the Data
MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527
Wido den Hollander - 10 ways to break your Ceph cluster
Wido den Hollander - 10 ways to break your Ceph cluster
Salesforce at Stacki Atlanta Meetup February 2016
Salesforce at Stacki Atlanta Meetup February 2016
Varnish Configuration Step by Step
Varnish Configuration Step by Step
Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2
Webinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera Cluster
Logical replication with pglogical
Logical replication with pglogical
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Spark 2.x Troubleshooting Guide
Spark 2.x Troubleshooting Guide
Oracle: Binding versus caging
Oracle: Binding versus caging
Build Automation 101
Build Automation 101
Postgres in Amazon RDS
Postgres in Amazon RDS
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
IT Infrastructure Through The Public Network Challenges And Solutions
IT Infrastructure Through The Public Network Challenges And Solutions
Understanding DSE Search by Matt Stump
Understanding DSE Search by Matt Stump
JavaOne 2014: Taming the Cloud Database with jclouds
JavaOne 2014: Taming the Cloud Database with jclouds
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Apache Traffic Server & Lua
Apache Traffic Server & Lua
Similar to Feed Burner Scalability
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
Lenz Grimmer
[Roblek] Distributed computing in practice
[Roblek] Distributed computing in practice
javablend
MySQL Aquarium Paris
MySQL Aquarium Paris
Alexis Moussine-Pouchkine
Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012
Wim Godden
MySQL Tuning
MySQL Tuning
Ford AntiTrust
MySQL Backup and Security Best Practices
MySQL Backup and Security Best Practices
Lenz Grimmer
Advanced Deployment
Advanced Deployment
Jonathan Weiss
2007-05-23 Cecchet_PGCon2007.ppt
2007-05-23 Cecchet_PGCon2007.ppt
nadirpervez2
From One to a Cluster
From One to a Cluster
guestd34230
Magee Dday2 Fixing App Performance Italiano
Magee Dday2 Fixing App Performance Italiano
Dominopoint - Italian Lotus User Group
Acus08 Advanced Load Balancing Apache2.2
Acus08 Advanced Load Balancing Apache2.2
Jim Jagielski
Nevmug Lighthouse Automation7.1
Nevmug Lighthouse Automation7.1
csharney
phptek13 - Caching and tuning fun tutorial
phptek13 - Caching and tuning fun tutorial
Wim Godden
How We Made Scylla Maintenance Easier, Safer and Faster
How We Made Scylla Maintenance Easier, Safer and Faster
ScyllaDB
Upgrade to MySQL 5.6 without downtime
Upgrade to MySQL 5.6 without downtime
Olivier DASINI
HBase tales from the trenches
HBase tales from the trenches
wchevreuil
Running Stateful Apps on Kubernetes
Running Stateful Apps on Kubernetes
Yugabyte
Zend Server Data Caching
Zend Server Data Caching
El Taller Web
High Availability with MySQL
High Availability with MySQL
Thava Alagu
Scalabe MySQL Infrastructure
Scalabe MySQL Infrastructure
Balazs Pocze
Similar to Feed Burner Scalability
(20)
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
[Roblek] Distributed computing in practice
[Roblek] Distributed computing in practice
MySQL Aquarium Paris
MySQL Aquarium Paris
Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012
MySQL Tuning
MySQL Tuning
MySQL Backup and Security Best Practices
MySQL Backup and Security Best Practices
Advanced Deployment
Advanced Deployment
2007-05-23 Cecchet_PGCon2007.ppt
2007-05-23 Cecchet_PGCon2007.ppt
From One to a Cluster
From One to a Cluster
Magee Dday2 Fixing App Performance Italiano
Magee Dday2 Fixing App Performance Italiano
Acus08 Advanced Load Balancing Apache2.2
Acus08 Advanced Load Balancing Apache2.2
Nevmug Lighthouse Automation7.1
Nevmug Lighthouse Automation7.1
phptek13 - Caching and tuning fun tutorial
phptek13 - Caching and tuning fun tutorial
How We Made Scylla Maintenance Easier, Safer and Faster
How We Made Scylla Maintenance Easier, Safer and Faster
Upgrade to MySQL 5.6 without downtime
Upgrade to MySQL 5.6 without downtime
HBase tales from the trenches
HBase tales from the trenches
Running Stateful Apps on Kubernetes
Running Stateful Apps on Kubernetes
Zend Server Data Caching
Zend Server Data Caching
High Availability with MySQL
High Availability with MySQL
Scalabe MySQL Infrastructure
Scalabe MySQL Infrastructure
More from didip
Oregon Measure 66
Oregon Measure 66
didip
High Performance Erlang
High Performance Erlang
didip
Why I Love Python
Why I Love Python
didip
Turbogears Presentation
Turbogears Presentation
didip
Ecma 262
Ecma 262
didip
Socket Programming In Python
Socket Programming In Python
didip
Google Research Paper
Google Research Paper
didip
Theory Psyco
Theory Psyco
didip
Super Sizing Youtube with Python
Super Sizing Youtube with Python
didip
More from didip
(9)
Oregon Measure 66
Oregon Measure 66
High Performance Erlang
High Performance Erlang
Why I Love Python
Why I Love Python
Turbogears Presentation
Turbogears Presentation
Ecma 262
Ecma 262
Socket Programming In Python
Socket Programming In Python
Google Research Paper
Google Research Paper
Theory Psyco
Theory Psyco
Super Sizing Youtube with Python
Super Sizing Youtube with Python
Recently uploaded
Memorándum de Entendimiento (MoU) entre Codelco y SQM
Memorándum de Entendimiento (MoU) entre Codelco y SQM
Voces Mineras
Organizational Structure Running A Successful Business
Organizational Structure Running A Successful Business
Seta Wicaksana
Call Girls in DELHI Cantt, ( Call Me )-8377877756-Female Escort- In Delhi / Ncr
Call Girls in DELHI Cantt, ( Call Me )-8377877756-Female Escort- In Delhi / Ncr
dollysharma2066
Digital Transformation in the PLM domain - distrib.pdf
Digital Transformation in the PLM domain - distrib.pdf
Jos Voskuil
Corporate Profile 47Billion Information Technology
Corporate Profile 47Billion Information Technology
Data Analytics Company - 47Billion Inc.
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
pollardmorgan
Kenya Coconut Production Presentation by Dr. Lalith Perera
Kenya Coconut Production Presentation by Dr. Lalith Perera
ictsugar
Kenya’s Coconut Value Chain by Gatsby Africa
Kenya’s Coconut Value Chain by Gatsby Africa
ictsugar
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
HajeJanKamps
Annual General Meeting Presentation Slides
Annual General Meeting Presentation Slides
KeppelCorporation
Youth Involvement in an Innovative Coconut Value Chain by Mwalimu Menza
Youth Involvement in an Innovative Coconut Value Chain by Mwalimu Menza
ictsugar
Ten Organizational Design Models to align structure and operations to busines...
Ten Organizational Design Models to align structure and operations to busines...
Seta Wicaksana
Cybersecurity Awareness Training Presentation v2024.03
Cybersecurity Awareness Training Presentation v2024.03
DallasHaselhorst
Market Sizes Sample Report - 2024 Edition
Market Sizes Sample Report - 2024 Edition
Mintel Group
Islamabad Escorts | Call 03070433345 | Escort Service in Islamabad
Islamabad Escorts | Call 03070433345 | Escort Service in Islamabad
Ayesha Khan
Marketplace and Quality Assurance Presentation - Vincent Chirchir
Marketplace and Quality Assurance Presentation - Vincent Chirchir
ictsugar
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
ashishs7044
Contemporary Economic Issues Facing the Filipino Entrepreneur (1).pptx
Contemporary Economic Issues Facing the Filipino Entrepreneur (1).pptx
MarkAnthonyAurellano
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
ashishs7044
(Best) ENJOY Call Girls in Faridabad Ex | 8377087607
(Best) ENJOY Call Girls in Faridabad Ex | 8377087607
dollysharma2066
Recently uploaded
(20)
Memorándum de Entendimiento (MoU) entre Codelco y SQM
Memorándum de Entendimiento (MoU) entre Codelco y SQM
Organizational Structure Running A Successful Business
Organizational Structure Running A Successful Business
Call Girls in DELHI Cantt, ( Call Me )-8377877756-Female Escort- In Delhi / Ncr
Call Girls in DELHI Cantt, ( Call Me )-8377877756-Female Escort- In Delhi / Ncr
Digital Transformation in the PLM domain - distrib.pdf
Digital Transformation in the PLM domain - distrib.pdf
Corporate Profile 47Billion Information Technology
Corporate Profile 47Billion Information Technology
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
Kenya Coconut Production Presentation by Dr. Lalith Perera
Kenya Coconut Production Presentation by Dr. Lalith Perera
Kenya’s Coconut Value Chain by Gatsby Africa
Kenya’s Coconut Value Chain by Gatsby Africa
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
Annual General Meeting Presentation Slides
Annual General Meeting Presentation Slides
Youth Involvement in an Innovative Coconut Value Chain by Mwalimu Menza
Youth Involvement in an Innovative Coconut Value Chain by Mwalimu Menza
Ten Organizational Design Models to align structure and operations to busines...
Ten Organizational Design Models to align structure and operations to busines...
Cybersecurity Awareness Training Presentation v2024.03
Cybersecurity Awareness Training Presentation v2024.03
Market Sizes Sample Report - 2024 Edition
Market Sizes Sample Report - 2024 Edition
Islamabad Escorts | Call 03070433345 | Escort Service in Islamabad
Islamabad Escorts | Call 03070433345 | Escort Service in Islamabad
Marketplace and Quality Assurance Presentation - Vincent Chirchir
Marketplace and Quality Assurance Presentation - Vincent Chirchir
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
Contemporary Economic Issues Facing the Filipino Entrepreneur (1).pptx
Contemporary Economic Issues Facing the Filipino Entrepreneur (1).pptx
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
(Best) ENJOY Call Girls in Faridabad Ex | 8377087607
(Best) ENJOY Call Girls in Faridabad Ex | 8377087607
Feed Burner Scalability
1.
FeedBurner: Scalable Web Applications using MySQL
and Java Joe Kottke, Director of Network Operations
2.
What is FeedBurner?
2 • Market-leading feed management provider • 170,000 bloggers, podcasters and commercial publishers including Reuters, USA TODAY, Newsweek, Ars Technica, BoingBoing… • 11 million subscribers in 190 countries. • Web-based services help publishers expand their reach online, attract subscribers and make money from their content • The largest advertising network for feeds © 2006 F eedBurner
3.
Scaling history
3 • July 2004 – 300Kbps, 5,600 feeds – 3 app servers, 3 web servers 2 DB servers • April 2005 – 5Mbps, 47,700 feeds – My first MySQL Users Conference – 6 app servers, 6 web servers (same machines) • September 2005 – 20Mbps, 109,200 feeds • Currently – 115 Mbps, 270,000 feeds, 100 Million hits per day © 2006 F eedBurner
4.
Scalability Problem 1:
Plain old reliability 4 • August 2004 • 3 web servers, 3 app servers, 2 DB servers. Round Robin DNS • Single-server failure, seen by 1/3 of all users © 2006 F eedBurner
5.
Solution: Load Balancers,
Monitoring 5 • Health Check pages – Round trip all the way back to the database – Same page monitored by load balancers and monitoring • Monitoring – Cacti (http://www.cacti.net/) – Nagios (http://www.nagios.org) © 2006 F eedBurner
6.
Health Check
6 UserComponent uc = UserComponentFactory.getUserComponent(); User user = uc.getUser(”monitor-userquot;); // If first load, mark as down. // Let FeedServlet mark things as up in init method. load-on-startup String healthcheck = (String) application.getAttribute(quot;healthcheckquot;); if(healthcheck == null || healthcheck.length() < 1) { healthcheck = new String(”DOWNquot;); application.setAttribute(quot;healthcheckquot;,healthcheck); } // We return null in case of problem, or if user doesn’t exist if( user == null ) { healthcheck = new String(quot;DOWNquot;); application.setAttribute(quot;healthcheckquot;,healthcheck); } System.out.print(healthcheck); © 2006 F eedBurner
7.
Cacti
7 © 2006 F eedBurner
8.
Start/Stop scripts
8 #!/bin/bash # Source the environment . ${HOME}/fb.env # Start TOMCAT cd ${FB_APPHOME} # Remove stale temp files find ~/rsspp/catalina/temp/ -type f -exec rm -rf {} ; # Remove the work directory #rm -rf ~/rsspp/catalina/work/* ${CATALINA_HOME}/bin/startup.sh © 2006 F eedBurner
9.
Start/Stop scripts
9 #!/bin/bash FB_APPHOME=/opt/fb/fb-app JAVA_HOME=/usr CATALINA_HOME=/opt/tomcat CATALINA_BASE=${FB_APPHOME}/catalina CATALINA_OPTS=quot;-Xmx768m -Xms7688m -Dnetworkaddress.cache.ttl=0quot; WEBROOT=/opt/fb/webroot export JAVA_HOME CATALINA_HOME CATALINA_BASE CATALINA_OPTS WEBROOT © 2006 F eedBurner
10.
Scalability Problem 2:
Stats recording/mgmt 10 • Every hit is recorded • Certain hits mean more than others • Flight recorder • Any table management locks • Inserts slow way down (90GB table) © 2006 F eedBurner
11.
Solution: Executor Pool
11 • Executor Pool – Doug Lea’s concurrency library – Use a PooledExecutor so stats inserts happen in a separate thread – Spring bean definition: <bean id=quot;StatsExecutorquot; class=quot;EDU.oswego.cs.dl.util.concurrent.PooledExecutorquot;> <constructor-arg> <bean class=quot;EDU.oswego.cs.dl.util.concurrent.LinkedQueuequot;/> </constructor-arg> <property name=quot;minimumPoolSizequot; value=quot;10quot; /> <property name=quot;keepAliveTimequot; value=quot;5000quot; /> </bean> © 2006 F eedBurner
12.
Solution: Lazy rollup
12 • Only today’s detailed stats need to go against real-time table • Roll up previous days into sparse summary tables on-demand • First access for stats for a day is slow, subsequent request are fast © 2006 F eedBurner
13.
Scalability Problem 3:
Primary DB overload 13 • Mostly used master DB server for everything • Read vs. Read/Write load didn’t matter in the beginnning • Slow inserts would block reads, when using MyISAM © 2006 F eedBurner
14.
Solution: Balance read
and read/write load 14 • Looked at workload – Found where we could break up read vs. read/write – Created Spring ExtendedDaoObjects – Tomcat-managed DataSources • Balanced master vs. slave load (Duh) – Slave becomes perfect place for snapshot backups • Watch for replication problems – Merge table problems (later) – Slow queries slow down replication © 2006 F eedBurner
15.
Example: Cacti graph
of MySQL handlers 15 © 2006 F eedBurner
16.
ExtendedDaoObject
16 • Application code extends this class and uses getHibernateTemplate() or getReadOnlyHibernateTemplate() depending upon requirements • Similar class for JDBC public class ExtendedHibernateDaoSupport extends HibernateDaoSupport { private HibernateTemplate readOnlyHibernateTemplate; public void setReadOnlySessionFactory(SessionFactory sessionFactory) { this.readOnlyHibernateTemplate = new HibernateTemplate(sessionFactory); readOnlyHibernateTemplate.setFlushMode(HibernateTemplate.FLUSH_NEVER); } protected HibernateTemplate getReadOnlyHibernateTemplate() { return (readOnlyHibernateTemplate == null) ? getHibernateTemplate() : readOnlyHibernateTemplate; } } © 2006 F eedBurner
17.
Scalability Problem 4:
Total DB overload 17 • Everything slowing down • Using DB as cache • Database is the ‘shared’ part of all app servers • Ran into table size limit defaults on MyISAM (4GB). We were lazy. – Had to use Merge tables as a bridge to newer larger tables © 2006 F eedBurner
18.
Solution: Stop using
the database 18 • Where possible :) • Multi-level caching – Local VM caching (EHCache, memory only) – Memcached (http://www.danga.com/memcached/) – And finally, database. • Memcached – Fault-tolerant, but client handles that. – Shared nothing – Data is transient, can be recreated © 2006 F eedBurner
19.
Scalability Problem 5:
Lazy initialization 19 • Our stats get rolled up on demand – Popular feeds slowed down the whole system • FeedCount chicklet calculation – Every feed gets its circulation calculated at the same time – Contention on the table © 2006 F eedBurner
20.
Solution: BATCH PROCESSING
20 • For FeedCount, we staggered the calculation – Still would run into contention – Stats stuff again slowed down at 1AM Chicago time. • We now process the rolled-up data every night – Delay showing the previous circulation in the FeedCount until roll-up is done. • Still wasn’t enough © 2006 F eedBurner
21.
Scalability Problem 6:
Stats writes, again 21 • Too much writing to master DB • More and more data stored associated with each feed • More stats tracking – Ad Stats – Item Stats – Circulation Stats © 2006 F eedBurner
22.
Solution: Merge Tables
22 • After the nightly rollup, we truncate the subtable from 2 days ago • Gotcha with truncating a subtable: – FLUSH TABLES; TRUNCATE TABLE ad_stats0; – Could succeed on master, but fail on slave • The right way to truncate a subtable: – ALTER TABLE ad_stats TYPE=MERGE UNION=(ad_stats1,ad_stats2); – TRUNCATE TABLE ad_stats0; – ALTER TABLE ad_stats TYPE=MERGE UNION=(ad_stats0,ad_stats1,ad_stats2); © 2006 F eedBurner
23.
Solution: Horizontal Partitioning
23 • Constantly identifying hot spots in the database – Ad serving – Flare serving – Circulation (constant writes, occasional reads) • Move hottest tables/queries off to own clusters – Hibernate and certain lazy patterns allow this – Keeps the driving tables from slowing down © 2006 F eedBurner
24.
Scalability Problem 7:
Master DB Failure 24 • Still using just a primary and slave • Master crash: Single point of failure • No easy way to promote a slave to a master © 2006 F eedBurner
25.
Solution: No easy
answer 25 • Still using auto_increment – Multi-master replication is out • Tried DRBD + HeartBeat – Disk is replicated block-by-block – Hot primary, cold secondary • Didn’t work as we hoped – Myisamchk takes too long after failure – I/O + CPU overhead • InnoDB is supposedly better © 2006 F eedBurner
26.
Our multi-master solution
26 • Low-volume master cluster – Uses DRBD + HeartBeat – Works well under smaller load – Does mapping to feed data clusters • Feed Data Cluster – Standard Master + Slave(s) structure – Can be added as needed © 2006 F eedBurner
27.
Mapping / Marshalling
Database Cluster 27 © 2006 F eedBurner
28.
Scalability Problem 8:
Power Failure 28 • Chicago has ‘questionable’ infrastructure. • Battery backup, generators can be problematic • Colo techs have been known to hit the Big Red Switch • Needed a disaster recovery/secondary site – Active/Active not possible for us. Yet. – Would have to keep fast connection to redundant site – Would require 100% of current hardware, but would lie quiet © 2006 F eedBurner
29.
Code Name: Panic
App 29 • Product Name: Feed Insurance • Elegant, simple solution • Not Java (sorry) • Perl-based feed fetcher – Downloads copies of feeds, saved as flat XML files – Synchronized out to local and remote servers – Special rules for click tracking, dynamic GIFs, etc © 2006 F eedBurner
30.
General guidelines
30 • Know your DB workload – Cacti really helps with this • ‘EXPLAIN’ all of your queries – Helps keep crushing queries out of the system • Cache everything that you can • Profile your code – Usually only needed on hard-to-find leaks © 2006 F eedBurner
31.
Our settings /
what we use 31 • Don’t always need the latest and greatest – Hibernate 2.1 – Spring – DBCP – MySQL 4.1 – Tomcat 5.0.x • Let the container manage DataSources © 2006 F eedBurner
32.
JDBC
32 • Hibernate/iBatis/Name-Your-ORM-Here – Use ORM when appropriate – Watch the queries that your ORM generates – Don't be afraid to drop to JDBC • Driver parameters we use: # For Internationalization of Ads, multi-byte characters in general useUnicode=true characterEncoding=UTF-8 # Biggest performance bits cacheServerConfiguration=true useLocalSessionState=true # Some other settings that we've needed as things have evolved useServerPrepStmts=false jdbcCompliantTruncation=false © 2006 F eedBurner
33.
Thank You
33 Questions? joek@feedburner.com © 2006 F eedBurner
Download now