SlideShare une entreprise Scribd logo
1  sur  56
Télécharger pour lire hors ligne
Big Bird.
(scaling twitter)
Rails Scales.
(but not out of the box)
First, Some Facts
• 600 requests per second. Growing fast.
• 180 Rails Instances (Mongrel). Growing fast.
• 1 Database Server (MySQL) + 1 Slave.
• 30-odd Processes for Misc. Jobs
• 8 Sun X4100s
• Many users, many updates.
Joy          Pain




Oct   Nov   Dec    Jan   Feb     March   Apr
IM IN UR RAILZ




     MAKIN EM GO FAST
It’s Easy, Really.
1. Realize Your Site is Slow
2. Optimize the Database
3. Cache the Hell out of Everything
4. Scale Messaging
5. Deal With Abuse
It’s Easy, Really.
1. Realize Your Site is Slow
2. Optimize the Database
3. Cache the Hell out of Everything
4. Scale Messaging
5. Deal With Abuse
6. Profit
the
     more
      you
        know

{ Part the First }
We Failed at This.
Don’t Be Like Us

• Munin
• Nagios
• AWStats & Google Analytics
• Exception Notifier / Exception Logger
• Immediately add reporting to track problems.
Test Everything

•   Start Before You Start

•   No Need To Be Fancy

•   Tests Will Save Your Life

•   Agile Becomes
    Important When Your
    Site Is Down
<!-- served to you through a copper wire by sampaati at 22 Apr
    15:02 in 343 ms (d 102 / r 217). thank you, come again. -->
 <!-- served to you through a copper wire by kolea.twitter.com at
22 Apr 15:02 in 235 ms (d 87 / r 130). thank you, come again. -->
 <!-- served to you through a copper wire by raven.twitter.com at
22 Apr 15:01 in 450 ms (d 96 / r 337). thank you, come again. -->



                  Benchmarks?
                       let your users do it.
 <!-- served to you through a copper wire by kolea.twitter.com at
22 Apr 15:00 in 409 ms (d 88 / r 307). thank you, come again. -->
  <!-- served to you through a copper wire by firebird at 22 Apr
   15:03 in 2094 ms (d 643 / r 1445). thank you, come again. -->
   <!-- served to you through a copper wire by quetzal at 22 Apr
     15:01 in 384 ms (d 70 / r 297). thank you, come again. -->
The Database
  { Part the Second }
“The Next Application I Build is Going
to Be Easily Partitionable” - S. Butterfield
“The Next Application I Build is Going
to Be Easily Partitionable” - S. Butterfield
“The Next Application I Build is Going
to Be Easily Partitionable” - S. Butterfield
Too Late.
Index Everything
class AddIndex < ActiveRecord::Migration
     def self.up
       add_index :users, :email
     end

     def self.down
       remove_index :users, :email
     end
   end


Repeat for any column that appears in a WHERE clause

             Rails won’t do this for you.
Denormalize A Lot
class DenormalizeFriendsIds < ActiveRecord::Migration
  def self.up
    add_column quot;usersquot;, quot;friends_idsquot;, :text
  end

  def self.down
    remove_column quot;usersquot;, quot;friends_idsquot;
  end
end
class Friendship < ActiveRecord::Base
  belongs_to :user
  belongs_to :friend

 after_create :add_to_denormalized_friends
 after_destroy :remove_from_denormalized_friends

  def add_to_denormalized_friends
    user.friends_ids << friend.id
    user.friends_ids.uniq!
    user.save_without_validation
  end

  def remove_from_denormalized_friends
    user.friends_ids.delete(friend.id)
    user.save_without_validation
  end
end
Don’t be Stupid
bob.friends.map(&:email)
     Status.count()
“email like ‘%#{search}%’”
That’s where we are.
                  Seriously.
  If your Rails application is doing anything more
complex than that, you’re doing something wrong*.



        * or you observed the First Rule of Butterfield.
Partitioning Comes Later.
   (we’ll let you know how it goes)
The Cache
 { Part the Third }
MemCache
MemCache
MemCache
!
class Status < ActiveRecord::Base
  class << self
    def count_with_memcache(*args)
      return count_without_memcache unless args.empty?
      count = CACHE.get(“status_count”)
      if count.nil?
        count = count_without_memcache
        CACHE.set(“status_count”, count)
      end
      count
    end
    alias_method_chain :count, :memcache
  end
  after_create :increment_memcache_count
  after_destroy :decrement_memcache_count
  ...
end
class User < ActiveRecord::Base
  def friends_statuses
    ids = CACHE.get(“friends_statuses:#{id}”)
    Status.find(:all, :conditions => [“id IN (?)”, ids])
  end
end

class Status < ActiveRecord::Base
  after_create :update_caches
  def update_caches
    user.friends_ids.each do |friend_id|
      ids = CACHE.get(“friends_statuses:#{friend_id}”)
      ids.pop
      ids.unshift(id)
      CACHE.set(“friends_statuses:#{friend_id}”, ids)
    end
  end
end
The Future


            ve d
          ti r
         co
         Ac
           e
         R
90% API Requests
     Cache Them!
“There are only two hard things in CS:
 cache invalidation and naming things.”

             – Phil Karlton, via Tim Bray
Messaging
{ Part the Fourth }
You Already Knew All
That Other Stuff, Right?
Producer             Consumer
           Message
Producer             Consumer
           Queue
Producer             Consumer
DRb
• The Good:
 • Stupid Easy
 • Reasonably Fast
• The Bad:
 • Kinda Flaky
 • Zero Redundancy
 • Tightly Coupled
ejabberd


            Jabber Client
                (drb)




           Incoming         Outgoing
Presence
           Messages         Messages


              MySQL
Server
     DRb.start_service ‘druby://localhost:10000’, myobject




                         Client
myobject = DRbObject.new_with_uri(‘druby://localhost:10000’)
Rinda

• Shared Queue (TupleSpace)
• Built with DRb
• RingyDingy makes it stupid easy
• See Eric Hodel’s documentation
• O(N) for take(). Sigh.
Timestamp: 12/22/06 01:53:14 (4 months ago)
      Author: lattice
      Message: Fugly. Seriously. Fugly.




        SELECT * FROM messages WHERE
substring(truncate(id,0),-2,1) = #{@fugly_dist_idx}
It Scales.
(except it stopped on Tuesday)
Options

• ActiveMQ (Java)
• RabbitMQ (erlang)
• MySQL + Lightweight Locking
• Something Else?
erlang?


What are you doing?
 Stabbing my eyes out with a fork.
Starling

• Ruby, will be ported to something faster
• 4000 transactional msgs/s
• First pass written in 4 hours
• Speaks MemCache (set, get)
Use Messages to
Invalidate Cache
   (it’s really not that hard)
Abuse
{ Part the Fifth }
The Italians
9000 friends in 24 hours
        (doesn’t scale)
http://flickr.com/photos/heather/464504545/
http://flickr.com/photos/curiouskiwi/165229284/
http://flickr.com/photo_zoom.gne?id=42914103&size=l
http://flickr.com/photos/madstillz/354596905/
http://flickr.com/photos/laughingsquid/382242677/
http://flickr.com/photos/bng/46678227/

Contenu connexe

Tendances

MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527Saewoong Lee
 
Plny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesPlny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesDimas Prasetyo
 
Multi Source Replication With MySQL 5.7 @ Verisure
Multi Source Replication With MySQL 5.7 @ VerisureMulti Source Replication With MySQL 5.7 @ Verisure
Multi Source Replication With MySQL 5.7 @ VerisureKenny Gryp
 
Galera Cluster: Synchronous Multi-Master Replication for MySQL HA
Galera Cluster: Synchronous Multi-Master Replication for MySQL HAGalera Cluster: Synchronous Multi-Master Replication for MySQL HA
Galera Cluster: Synchronous Multi-Master Replication for MySQL HALudovico Caldara
 
合并到 XtraDB 存储引擎集群
合并到 XtraDB 存储引擎集群合并到 XtraDB 存储引擎集群
合并到 XtraDB 存储引擎集群YUCHENG HU
 
Plmce2k15 15 tips galera cluster
Plmce2k15   15 tips galera clusterPlmce2k15   15 tips galera cluster
Plmce2k15 15 tips galera clusterFrederic Descamps
 
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...Severalnines
 
MySQL Backup and Security Best Practices
MySQL Backup and Security Best PracticesMySQL Backup and Security Best Practices
MySQL Backup and Security Best PracticesLenz Grimmer
 
Intro ProxySQL
Intro ProxySQLIntro ProxySQL
Intro ProxySQLI Goo Lee
 
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
 
Online MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackupOnline MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackupKenny Gryp
 
Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...
Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...
Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...DataStax
 
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Replication in 10  Minutes - SCALEPostgreSQL Replication in 10  Minutes - SCALE
PostgreSQL Replication in 10 Minutes - SCALEPostgreSQL Experts, Inc.
 
PGPool-II Load testing
PGPool-II Load testingPGPool-II Load testing
PGPool-II Load testingEDB
 
Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Marco Tusa
 
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practiceAlexey Lesovsky
 
MySQL Replication Overview -- PHPTek 2016
MySQL Replication Overview -- PHPTek 2016MySQL Replication Overview -- PHPTek 2016
MySQL Replication Overview -- PHPTek 2016Dave Stokes
 

Tendances (20)

MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527
 
Plny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesPlny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practices
 
Multi Source Replication With MySQL 5.7 @ Verisure
Multi Source Replication With MySQL 5.7 @ VerisureMulti Source Replication With MySQL 5.7 @ Verisure
Multi Source Replication With MySQL 5.7 @ Verisure
 
Galera Cluster: Synchronous Multi-Master Replication for MySQL HA
Galera Cluster: Synchronous Multi-Master Replication for MySQL HAGalera Cluster: Synchronous Multi-Master Replication for MySQL HA
Galera Cluster: Synchronous Multi-Master Replication for MySQL HA
 
合并到 XtraDB 存储引擎集群
合并到 XtraDB 存储引擎集群合并到 XtraDB 存储引擎集群
合并到 XtraDB 存储引擎集群
 
Plmce2k15 15 tips galera cluster
Plmce2k15   15 tips galera clusterPlmce2k15   15 tips galera cluster
Plmce2k15 15 tips galera cluster
 
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
Webinar slides: 9 DevOps Tips for Going in Production with Galera Cluster for...
 
Scaling symfony apps
Scaling symfony appsScaling symfony apps
Scaling symfony apps
 
MySQL Backup and Security Best Practices
MySQL Backup and Security Best PracticesMySQL Backup and Security Best Practices
MySQL Backup and Security Best Practices
 
Intro ProxySQL
Intro ProxySQLIntro ProxySQL
Intro ProxySQL
 
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
 
Online MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackupOnline MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackup
 
Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...
Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...
Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski...
 
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Replication in 10  Minutes - SCALEPostgreSQL Replication in 10  Minutes - SCALE
PostgreSQL Replication in 10 Minutes - SCALE
 
PGPool-II Load testing
PGPool-II Load testingPGPool-II Load testing
PGPool-II Load testing
 
Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2
 
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
 
Cassandra as Memcache
Cassandra as MemcacheCassandra as Memcache
Cassandra as Memcache
 
MySQL Replication Overview -- PHPTek 2016
MySQL Replication Overview -- PHPTek 2016MySQL Replication Overview -- PHPTek 2016
MySQL Replication Overview -- PHPTek 2016
 

En vedette

Oh&S In The It Workplace
Oh&S In The It WorkplaceOh&S In The It Workplace
Oh&S In The It Workplacejorge_llama_guy
 
Grade 6 Travel the World Assignment
Grade 6 Travel the World AssignmentGrade 6 Travel the World Assignment
Grade 6 Travel the World Assignmentlauramclean
 
Webquest Pp
Webquest PpWebquest Pp
Webquest Pptiffz
 
introduccion
introduccionintroduccion
introducciongrupo4
 
คนไทยในอังกฤษชุมนุมครั้งที่ ๒
คนไทยในอังกฤษชุมนุมครั้งที่ ๒คนไทยในอังกฤษชุมนุมครั้งที่ ๒
คนไทยในอังกฤษชุมนุมครั้งที่ ๒konthaiuk
 
Medical Information Workshop (23 Jan 2007 )
Medical Information Workshop (23 Jan 2007 )Medical Information Workshop (23 Jan 2007 )
Medical Information Workshop (23 Jan 2007 )rwakefor
 

En vedette (7)

Oh&S In The It Workplace
Oh&S In The It WorkplaceOh&S In The It Workplace
Oh&S In The It Workplace
 
Grade 6 Travel the World Assignment
Grade 6 Travel the World AssignmentGrade 6 Travel the World Assignment
Grade 6 Travel the World Assignment
 
Webquest Pp
Webquest PpWebquest Pp
Webquest Pp
 
1mais1=2
1mais1=21mais1=2
1mais1=2
 
introduccion
introduccionintroduccion
introduccion
 
คนไทยในอังกฤษชุมนุมครั้งที่ ๒
คนไทยในอังกฤษชุมนุมครั้งที่ ๒คนไทยในอังกฤษชุมนุมครั้งที่ ๒
คนไทยในอังกฤษชุมนุมครั้งที่ ๒
 
Medical Information Workshop (23 Jan 2007 )
Medical Information Workshop (23 Jan 2007 )Medical Information Workshop (23 Jan 2007 )
Medical Information Workshop (23 Jan 2007 )
 

Similaire à Scaling Twitter 12758

Hiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceHiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceJesse Vincent
 
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceBeijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceJesse Vincent
 
Microblogging via XMPP
Microblogging via XMPPMicroblogging via XMPP
Microblogging via XMPPStoyan Zhekov
 
Socket applications
Socket applicationsSocket applications
Socket applicationsJoão Moura
 
Aprendendo solid com exemplos
Aprendendo solid com exemplosAprendendo solid com exemplos
Aprendendo solid com exemplosvinibaggio
 
Dynomite at Erlang Factory
Dynomite at Erlang FactoryDynomite at Erlang Factory
Dynomite at Erlang Factorymoonpolysoft
 
Fisl - Deployment
Fisl - DeploymentFisl - Deployment
Fisl - DeploymentFabio Akita
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsSerge Smetana
 
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...MongoDB
 
NPW2009 - my.opera.com scalability v2.0
NPW2009 - my.opera.com scalability v2.0NPW2009 - my.opera.com scalability v2.0
NPW2009 - my.opera.com scalability v2.0Cosimo Streppone
 
RubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - KeynoteRubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - KeynoteDr Nic Williams
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerElixir Club
 
How to avoid hanging yourself with Rails
How to avoid hanging yourself with RailsHow to avoid hanging yourself with Rails
How to avoid hanging yourself with RailsRowan Hick
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking systemJesse Vincent
 
Web 2.0 Performance and Reliability: How to Run Large Web Apps
Web 2.0 Performance and Reliability: How to Run Large Web AppsWeb 2.0 Performance and Reliability: How to Run Large Web Apps
Web 2.0 Performance and Reliability: How to Run Large Web Appsadunne
 
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & AnalyticsMongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & AnalyticsServer Density
 
JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...
JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...
JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...PROIDEA
 
When To Use Ruby On Rails
When To Use Ruby On RailsWhen To Use Ruby On Rails
When To Use Ruby On Railsdosire
 

Similaire à Scaling Twitter 12758 (20)

Hiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceHiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret Sauce
 
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceBeijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
 
Microblogging via XMPP
Microblogging via XMPPMicroblogging via XMPP
Microblogging via XMPP
 
From crash to testcase
From crash to testcaseFrom crash to testcase
From crash to testcase
 
Socket applications
Socket applicationsSocket applications
Socket applications
 
Aprendendo solid com exemplos
Aprendendo solid com exemplosAprendendo solid com exemplos
Aprendendo solid com exemplos
 
Dynomite at Erlang Factory
Dynomite at Erlang FactoryDynomite at Erlang Factory
Dynomite at Erlang Factory
 
Fisl - Deployment
Fisl - DeploymentFisl - Deployment
Fisl - Deployment
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
 
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
 
NPW2009 - my.opera.com scalability v2.0
NPW2009 - my.opera.com scalability v2.0NPW2009 - my.opera.com scalability v2.0
NPW2009 - my.opera.com scalability v2.0
 
RubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - KeynoteRubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - Keynote
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan Wintermeyer
 
How to avoid hanging yourself with Rails
How to avoid hanging yourself with RailsHow to avoid hanging yourself with Rails
How to avoid hanging yourself with Rails
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking system
 
Web 2.0 Performance and Reliability: How to Run Large Web Apps
Web 2.0 Performance and Reliability: How to Run Large Web AppsWeb 2.0 Performance and Reliability: How to Run Large Web Apps
Web 2.0 Performance and Reliability: How to Run Large Web Apps
 
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & AnalyticsMongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
 
Sinatra for REST services
Sinatra for REST servicesSinatra for REST services
Sinatra for REST services
 
JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...
JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...
JDD2015: Sharding with Akka Cluster: From Theory to Production - Krzysztof Ot...
 
When To Use Ruby On Rails
When To Use Ruby On RailsWhen To Use Ruby On Rails
When To Use Ruby On Rails
 

Dernier

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embeddingZilliz
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostZilliz
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 

Dernier (20)

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embedding
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector Databases
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 

Scaling Twitter 12758

  • 2. Rails Scales. (but not out of the box)
  • 3. First, Some Facts • 600 requests per second. Growing fast. • 180 Rails Instances (Mongrel). Growing fast. • 1 Database Server (MySQL) + 1 Slave. • 30-odd Processes for Misc. Jobs • 8 Sun X4100s • Many users, many updates.
  • 4.
  • 5.
  • 6.
  • 7. Joy Pain Oct Nov Dec Jan Feb March Apr
  • 8. IM IN UR RAILZ MAKIN EM GO FAST
  • 9. It’s Easy, Really. 1. Realize Your Site is Slow 2. Optimize the Database 3. Cache the Hell out of Everything 4. Scale Messaging 5. Deal With Abuse
  • 10. It’s Easy, Really. 1. Realize Your Site is Slow 2. Optimize the Database 3. Cache the Hell out of Everything 4. Scale Messaging 5. Deal With Abuse 6. Profit
  • 11. the more you know { Part the First }
  • 12. We Failed at This.
  • 13. Don’t Be Like Us • Munin • Nagios • AWStats & Google Analytics • Exception Notifier / Exception Logger • Immediately add reporting to track problems.
  • 14. Test Everything • Start Before You Start • No Need To Be Fancy • Tests Will Save Your Life • Agile Becomes Important When Your Site Is Down
  • 15. <!-- served to you through a copper wire by sampaati at 22 Apr 15:02 in 343 ms (d 102 / r 217). thank you, come again. --> <!-- served to you through a copper wire by kolea.twitter.com at 22 Apr 15:02 in 235 ms (d 87 / r 130). thank you, come again. --> <!-- served to you through a copper wire by raven.twitter.com at 22 Apr 15:01 in 450 ms (d 96 / r 337). thank you, come again. --> Benchmarks? let your users do it. <!-- served to you through a copper wire by kolea.twitter.com at 22 Apr 15:00 in 409 ms (d 88 / r 307). thank you, come again. --> <!-- served to you through a copper wire by firebird at 22 Apr 15:03 in 2094 ms (d 643 / r 1445). thank you, come again. --> <!-- served to you through a copper wire by quetzal at 22 Apr 15:01 in 384 ms (d 70 / r 297). thank you, come again. -->
  • 16. The Database { Part the Second }
  • 17. “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield
  • 18. “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield
  • 19. “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield
  • 22. class AddIndex < ActiveRecord::Migration def self.up add_index :users, :email end def self.down remove_index :users, :email end end Repeat for any column that appears in a WHERE clause Rails won’t do this for you.
  • 24. class DenormalizeFriendsIds < ActiveRecord::Migration def self.up add_column quot;usersquot;, quot;friends_idsquot;, :text end def self.down remove_column quot;usersquot;, quot;friends_idsquot; end end
  • 25. class Friendship < ActiveRecord::Base belongs_to :user belongs_to :friend after_create :add_to_denormalized_friends after_destroy :remove_from_denormalized_friends def add_to_denormalized_friends user.friends_ids << friend.id user.friends_ids.uniq! user.save_without_validation end def remove_from_denormalized_friends user.friends_ids.delete(friend.id) user.save_without_validation end end
  • 27. bob.friends.map(&:email) Status.count() “email like ‘%#{search}%’”
  • 28. That’s where we are. Seriously. If your Rails application is doing anything more complex than that, you’re doing something wrong*. * or you observed the First Rule of Butterfield.
  • 29. Partitioning Comes Later. (we’ll let you know how it goes)
  • 30. The Cache { Part the Third }
  • 34. !
  • 35. class Status < ActiveRecord::Base class << self def count_with_memcache(*args) return count_without_memcache unless args.empty? count = CACHE.get(“status_count”) if count.nil? count = count_without_memcache CACHE.set(“status_count”, count) end count end alias_method_chain :count, :memcache end after_create :increment_memcache_count after_destroy :decrement_memcache_count ... end
  • 36. class User < ActiveRecord::Base def friends_statuses ids = CACHE.get(“friends_statuses:#{id}”) Status.find(:all, :conditions => [“id IN (?)”, ids]) end end class Status < ActiveRecord::Base after_create :update_caches def update_caches user.friends_ids.each do |friend_id| ids = CACHE.get(“friends_statuses:#{friend_id}”) ids.pop ids.unshift(id) CACHE.set(“friends_statuses:#{friend_id}”, ids) end end end
  • 37. The Future ve d ti r co Ac e R
  • 38. 90% API Requests Cache Them!
  • 39. “There are only two hard things in CS: cache invalidation and naming things.” – Phil Karlton, via Tim Bray
  • 41. You Already Knew All That Other Stuff, Right?
  • 42. Producer Consumer Message Producer Consumer Queue Producer Consumer
  • 43. DRb • The Good: • Stupid Easy • Reasonably Fast • The Bad: • Kinda Flaky • Zero Redundancy • Tightly Coupled
  • 44. ejabberd Jabber Client (drb) Incoming Outgoing Presence Messages Messages MySQL
  • 45. Server DRb.start_service ‘druby://localhost:10000’, myobject Client myobject = DRbObject.new_with_uri(‘druby://localhost:10000’)
  • 46. Rinda • Shared Queue (TupleSpace) • Built with DRb • RingyDingy makes it stupid easy • See Eric Hodel’s documentation • O(N) for take(). Sigh.
  • 47. Timestamp: 12/22/06 01:53:14 (4 months ago) Author: lattice Message: Fugly. Seriously. Fugly. SELECT * FROM messages WHERE substring(truncate(id,0),-2,1) = #{@fugly_dist_idx}
  • 48. It Scales. (except it stopped on Tuesday)
  • 49. Options • ActiveMQ (Java) • RabbitMQ (erlang) • MySQL + Lightweight Locking • Something Else?
  • 50. erlang? What are you doing? Stabbing my eyes out with a fork.
  • 51. Starling • Ruby, will be ported to something faster • 4000 transactional msgs/s • First pass written in 4 hours • Speaks MemCache (set, get)
  • 52. Use Messages to Invalidate Cache (it’s really not that hard)
  • 55. 9000 friends in 24 hours (doesn’t scale)