Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
2009.4

hongqn@douban.com
•   2005 3

•

•

•
• 2.9M            1/4
•
• 23M        /        500~600/sec
• 23      PC      (1U*15/2U*8)
  • 12
  • 38G memcached
•        1U         (frodo)
    •        AMD Athlon 64 1.8GHz
    •   1G       160G SATA*2
•   Gentoo Linux
•   MySQL 5
• ...
Gentoo Linux
•
    • emerge mysql
    • ebuild          patch
•
•
    • GLSA(Gentoo Linux Security Advisories)
MySQL

•   The world’s most popular open source database

•            /             ==> MyISAM

•                ==> Inno...
Python

•
•   Battery Included

•
•
    •   CPUG: http://python.cn/
Quixote
•                           REST         URL
•              Django, TurboGears, Pylons
              ZOPE
• http:/...
Lighttpd
•
•       SCGI
    • SCGI:               FastCGI
      Quixote
•                  80        lighttpd
            ...
Memcache
•                                           MySQL
•     libmemcache   python                          Pyrex
     ...
Internet




                  Lighttpd

                SCGI



          App

                       FS




MySQL   Memc...
• 1.2M   /
• IO
•
•                1U
    •   pippin       meriadoc (      merry)
    •         , 4G          250G SATA*3
•
•               ...
Internet




        Lighttpd (#$)                               DNS
                                        HTTP
        ...
•
    • innodb_buffer_pool_size
•
•              IP
• 1.5M   /
•
• IP
•                            IP(BGP)
•                         (arwen)
    • 74G 1w   SATA * 3
    •
•
Internet




                                                               Data
               Lighttpd                  ...
• 2M              /
•                     IO
  •
       , etc...
•
•                               4G   250G SATA*3
•                                10000
    •   mod_rewrite       URL
•   ...
Internet

                                                                store.farm
                                     ...
•   store          farmr
•                replicate delay
    •
    •

    •                          cache
        •
•

 ...
replicate delay
def get_subject(sid):
   sbj = mc.get(‘s:’+sid)
   if sbj is None:
      sbj = flush_subject(sid, store.fa...
• 2.5M       /
•
  •      /
• SATA
•
•   Scale Up               1U
    •   16G        147G SCSI *2 + 500G SATA
    •   SCSI     RAID-0
•       MySQL Slave
•   ...
Internet
                                                     Sphinx


                                                   ...
• 5.2M   /
•
• Web        IO


•
•
•                             :)
    •
    •
    •
•        3 1U             4        32G        1T SATA * 3
•            ...
Internet


                          www.douban.com          otho.douban.com


                                           ...
write




                                      !quot;#$%
                replicate
                                      ...
• 6.4M       /   (5M PV)
•
  •
 • CPU   memcache          /
•
    •   lighttpd     mod_scgi     round-robin
    •   lighttpd 1.5
    •   mod_proxy
        •   proxy.balance = fair (l...
Internet


                                                     Lighttpd

                                                ...
• 11M      /      3
•
• Sphinx
•
•
           load
•
    •
    •
• Sphinx -> Xapian
• MogileFS
!quot;
             Master

                       replicate

replicate
                                                !q...
•
•   store.farm[r] -> store.get_cursor(table=‘xxx’,
    ro=True/False)
    def flush_subject(sid, ro=False):
        curs...
Internet

                         upload.douban.com otho.douban.com




                                                 ...
•   libmemcache -> libmemcached              consistent
    hash    memcache
    •       libmemcached consistent hash     ...
Consistent Hash




                                charlee http://
                  tech.idv2.com/2008/07/24/
          ...
Internet


                                                                  App
                                         ...
• 13M   /
•                 MogileFS
  •         Tracker DB


•
•       8
    • 32G            CPU
    • (300G SCSI×2 + 1T SATA) × 3
    • (1T SATA × 3) × 5
    •6          2
•       Dou...
DoubanFS
•                            hash


•      hash                              hash


•                          Me...
Merkle Tree
Internet

                         upload.douban.com otho.douban.com




                                                 ...
• 16M        /
•
• DoubanFS       IO
•
•       DoubanDB
    •
    •                 MySQL


• MySQL Master
  • failover
  • replicate delay
DoubanDB
•         Key-Value
•     Amazon Dynamo
•              set(key, value), get(key), delete(key)
•   memcache
•     ...
charlee http://
tech.idv2.com/2008/07/24/
memcached-004/
!quot;
             Master1

                        replicate

replicate
                                                ...
•       DoubanFS
    •
    •
• ngnix
• www.douban.com       LVS
• RabbitMQ         spread
•
•              profile
•   memcache            cache


•                join
•
•
•
豆瓣技术架构的发展历程 @ QCon Beijing 2009
Prochain SlideShare
Chargement dans…5
×

豆瓣技术架构的发展历程 @ QCon Beijing 2009

38 882 vues

Publié le

Publié dans : Technologie

豆瓣技术架构的发展历程 @ QCon Beijing 2009

  1. 1. 2009.4 hongqn@douban.com
  2. 2. • 2005 3 • • •
  3. 3. • 2.9M 1/4 • • 23M / 500~600/sec • 23 PC (1U*15/2U*8) • 12 • 38G memcached
  4. 4. • 1U (frodo) • AMD Athlon 64 1.8GHz • 1G 160G SATA*2 • Gentoo Linux • MySQL 5 • Quixote (a Python web framework) • Lighttpd + SCGI (shire) • Memcached (!)
  5. 5. Gentoo Linux • • emerge mysql • ebuild patch • • • GLSA(Gentoo Linux Security Advisories)
  6. 6. MySQL • The world’s most popular open source database • / ==> MyISAM • ==> InnoDB • Replicate for backup
  7. 7. Python • • Battery Included • • • CPUG: http://python.cn/
  8. 8. Quixote • REST URL • Django, TurboGears, Pylons ZOPE • http://www.douban.com/subject/1000001 # luz/subject/__init__.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) “<h1>%s</h1>” % subject.title site_footer(request)
  9. 9. Lighttpd • • SCGI • SCGI: FastCGI Quixote • 80 lighttpd SCGI localhost Quixote
  10. 10. Memcache • MySQL • libmemcache python Pyrex python 3x+ def get_subject(subject_id): subject = mc.get(‘s:’+subject_id) if subject is None: store.farm.execute(“select xxx, xxx from subject where id=%s”, subject_id) subject = Subject(*store.farm.fetchone()) mc.set(‘s:’+subject_id, subject) return subject
  11. 11. Internet Lighttpd SCGI App FS MySQL Memcache Static Files
  12. 12. • 1.2M / • IO •
  13. 13. • 1U • pippin meriadoc ( merry) • , 4G 250G SATA*3 • • IP DNS IP -_-b • frodo (subversion, trac, etc...)
  14. 14. Internet Lighttpd (#$) DNS HTTP Proxy SCGI App Lighttpd (!quot;) FS Static Files Memcache MySQL
  15. 15. • • innodb_buffer_pool_size • • IP
  16. 16. • 1.5M / • • IP
  17. 17. • IP(BGP) • (arwen) • 74G 1w SATA * 3 • •
  18. 18. Internet Data Lighttpd Mining read SCGI write App Replicate MySQL MySQL Master Slave Memcache Static Files
  19. 19. • 2M / • IO • , etc... •
  20. 20. • 4G 250G SATA*3 • 10000 • mod_rewrite URL • lighttpd mod_memcache • IO • web • •
  21. 21. Internet store.farm App MySQL SCGI Master Lighttpd Memcache Replicate store.farmr HTTP Proxy WebDAV Web Service Replicate Lighttpd (w/ mod_memcache) MySQL write Slave Spiders Lighttpd WebDAV Memcache Static Files !quot;#$% read Data Mining MySQL Slave
  22. 22. • store farmr • replicate delay • • • cache • • • ...... but it works
  23. 23. replicate delay def get_subject(sid): sbj = mc.get(‘s:’+sid) if sbj is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=None): cursor = cursor or store.farm cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject def update_subject(subject, props): store.farm.execute(“update subject ...”) store.farm.connection.commit() flush_subject(subject.id, store.farm)
  24. 24. • 2.5M / • • / • SATA •
  25. 25. • Scale Up 1U • 16G 147G SCSI *2 + 500G SATA • SCSI RAID-0 • MySQL Slave • memcached • MyISAM InnoDB • • Sphinx
  26. 26. Internet Sphinx MySQL Master store.farm Web Service Replicate Lighttpd SCGI App HTTP Proxy Memcache Lighttpd (w/ mod_memcache) WebDAV store.farmr Memcache Lighttpd Web Service WebDAV MySQL Static Files Slave Spiders Memcache Memcache
  27. 27. • 5.2M / • • Web IO • •
  28. 28. • :) • • • • 3 1U 4 32G 1T SATA * 3 • otho.douban.com lotho.douban.com • lighttpd 1.5 with aio support • LVS • Scale Up: 4G -> 8G
  29. 29. Internet www.douban.com otho.douban.com LVS LB (Master) Lighttpd Lighttpd 1.5 (w/ mod_cache) HTTP Proxy Lighttpd Keepalived LVS LB (backup) Lighttpd Lighttpd 1.5 WebDAV (w/ mod_cache) Static Files
  30. 30. write !quot;#$% replicate read Data Mining MySQL MySQL Slave Master read Replicate write !quot;#$% Data Mining MySQL Slave
  31. 31. • 6.4M / (5M PV) • • • CPU memcache /
  32. 32. • • lighttpd mod_scgi round-robin • lighttpd 1.5 • mod_proxy • proxy.balance = fair (load based, passive balancing) • • spread
  33. 33. Internet Lighttpd SCGI HTTP Proxy App Memcache Lighttpd HTTP Proxy spread HTTP Proxy Lighttpd Lighttpd Log Aggregator SCGI spread Lighttpd Memcache App WebDAV Static Files
  34. 34. • 11M / 3 • • Sphinx • • load
  35. 35. • • • • Sphinx -> Xapian • MogileFS
  36. 36. !quot; Master replicate replicate !quot; %&'() Slave #$ Master read replicate Data #$ read Slave Mining write *+,- !quot; Master Slave replicate *+,- Slave write replicate Data %&'() Mining #$ Slave
  37. 37. • • store.farm[r] -> store.get_cursor(table=‘xxx’, ro=True/False) def flush_subject(sid, ro=False): cursor = store.get_cursor(table=‘subject’, ro=ro) cursor.execute(“select ... from subject”) subject = Subject(*cursor.fetchone()) mc.set(‘s:’+sid, subject) return subject •
  38. 38. Internet upload.douban.com otho.douban.com nginx signed POST form Uploader (w/ proxy_store) HTTP redirect FileStorage MogileFS Tracker Gateway App MogileFS MogileFS Node Node MogileFS Master
  39. 39. • libmemcache -> libmemcached consistent hash memcache • libmemcached consistent hash bug • CPU • libmemcached failover bug • nginx lighttpd load balance • spread • nginx
  40. 40. Consistent Hash charlee http:// tech.idv2.com/2008/07/24/ memcached-004/
  41. 41. Internet App SCGI Lighttpd HTTP Proxy Lighttpd HTTP Proxy App HTTP Proxy SCGI Lighttpd HTTP Proxy Nginx Lighttpd HTTP Proxy App Lighttpd SCGI Lighttpd WebDAV Static Files
  42. 42. • 13M / • MogileFS • Tracker DB •
  43. 43. • 8 • 32G CPU • (300G SCSI×2 + 1T SATA) × 3 • (1T SATA × 3) × 5 •6 2 • DoubanFS
  44. 44. DoubanFS • hash • hash hash • Merkle Tree • consistent hash • WebDAV • MogileFS 3 50
  45. 45. Merkle Tree
  46. 46. Internet upload.douban.com otho.douban.com nginx signed POST form Uploader (w/ proxy_store) HTTP redirect FileStorage Gateway App DoubanFS DoubanFS Node Node
  47. 47. • 16M / • • DoubanFS IO •
  48. 48. • DoubanDB • • MySQL • MySQL Master • failover • replicate delay
  49. 49. DoubanDB • Key-Value • Amazon Dynamo • set(key, value), get(key), delete(key) • memcache • Merkle Tree • Consistent Hash • TokyoCabinet • DoubanDB • DoubanFS 2.0 DoubanDB
  50. 50. charlee http:// tech.idv2.com/2008/07/24/ memcached-004/
  51. 51. !quot; Master1 replicate replicate !quot; %&'() Slave #$ Master1 read replicate Data #$ read Slave Mining write *+,- !quot; Master Master2 replicate *+,- Slave write replicate Data %&'() Mining #$ Master2
  52. 52. • DoubanFS • • • ngnix • www.douban.com LVS • RabbitMQ spread
  53. 53. • • profile • memcache cache • join • • •

×