Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Big Data Real Time Analytics - A Facebook Case Study

37 708 vues

Publié le

Building Your Own Facebook Real Time Analytics System with Cassandra and GigaSpaces.

Facebook's real time analytics system is a good reference for those looking to build their real time analytics system for big data.

The first part covers the lessons from Facebook's experience and the reason they chose HBase over Cassandra.

In the second part of the session, we learn how we can build our own Real Time Analytics system, achieve better performance, gain real business insights, and business analytics on our big data, and make the deployment and scaling significantly simpler using the new version of Cassandra and GigaSpaces Cloudify.

Publié dans : Technologie, Business
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Big Data Real Time Analytics - A Facebook Case Study

  1. 1. Real Time Analytics for Big Data Lessons from Facebook..
  2. 2. The Real Time Boom.. ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved Google Real Time Web Analytics Google Real Time Search Facebook Real Time Social Analytics Twitter paid tweet analytics SaaS Real Time User Tracking New Real Time Analytics Startups..
  3. 3. Analytics @ Twitter
  4. 4. Note the Time dimension
  5. 5. The data resolution & processing models
  6. 6. Traditional analytics applications <ul><li>Scale-up Database </li></ul><ul><ul><li>Use traditional SQL database </li></ul></ul><ul><ul><li>Use stored procedure for event driven reports </li></ul></ul><ul><ul><li>Use flash memory disks to reduce disk I/O </li></ul></ul><ul><ul><li>Use read only replica to scale-out read queries </li></ul></ul><ul><li>Limitations </li></ul><ul><ul><li>Doesn’t scale on write </li></ul></ul><ul><ul><li>Extremely expensive (HW + SW) </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  7. 7. CEP – Complex Event Processing <ul><li>Process the data as it comes </li></ul><ul><li>Maintain a window of the data in-memory </li></ul><ul><li>Pros: </li></ul><ul><ul><li>Extremely low-latency </li></ul></ul><ul><ul><li>Relatively low-cost </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Hard to scale (Mostly limited to scale-up) </li></ul></ul><ul><ul><li>Not agile - Queries must be pre-generated </li></ul></ul><ul><ul><li>Fairly complex </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  8. 8. In Memory Data Grid <ul><li>Distributed in-memory database </li></ul><ul><li>Scale out </li></ul><ul><li>Pros </li></ul><ul><ul><li>Scale on write/read </li></ul></ul><ul><ul><li>Fits to event driven (CEP style) , ad-hoc query model </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Cost of memory vs disk </li></ul></ul><ul><ul><li>Memory capacity is limited </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  9. 9. NoSQL <ul><li>Use distributed database </li></ul><ul><ul><li>Hbase, Cassandra, MongoDB </li></ul></ul><ul><li>Pros </li></ul><ul><ul><li>Scale on write/read </li></ul></ul><ul><ul><li>Elastic </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Read latency </li></ul></ul><ul><ul><li>Consistency tradeoffs are hard </li></ul></ul><ul><ul><li>Maturity – fairly young technology </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  10. 10. Hadoop MapReudce <ul><li>Distributed batch processing </li></ul><ul><li>Pros </li></ul><ul><ul><li>Designed to process massive amount of data </li></ul></ul><ul><ul><li>Mature </li></ul></ul><ul><ul><li>Low cost </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Not real-time </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  11. 11. Hadoop Map/Reduce – Reality check.. ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  12. 12. So what’s the bottom line? ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  13. 13. Facebook Real-time Analytics System ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  14. 14. Goals <ul><li>Show why plugins are valuable. </li></ul><ul><ul><li>What value is your business deriving from it? </li></ul></ul><ul><li>Make the data more actionable. </li></ul><ul><ul><li>Help users take action to make their content more valuable. </li></ul></ul><ul><ul><li>How many people see a plugin, how many people take action on it, and how many are converted to traffic back on your site.   </li></ul></ul><ul><li>Make the data more timely.  </li></ul><ul><ul><li>Went from a 48-hour turn around to 30 seconds. </li></ul></ul><ul><ul><li>Multiple points of failure were removed to make this goal.  </li></ul></ul><ul><li>Handle massive load </li></ul><ul><ul><li>20 billion events per day (200,000 events per second) </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  15. 15. The actual analytics.. <ul><li>Like button analytics </li></ul><ul><li>Comments box analytics </li></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  16. 16. Technology Evaluation <ul><li>MySQL DB Counters </li></ul><ul><li>In-Memory Counters </li></ul><ul><li>MapReduce </li></ul><ul><li>Cassandra </li></ul><ul><li>HBase </li></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  17. 17. The solution.. PTail Scribe Puma Hbase HDFS Real Time Long Term Batch 1.5 Sec 10,000 write/sec per server FACEBOOK Log FACEBOOK Log FACEBOOK Log
  18. 18. Checking the assumptions.. ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  19. 19. Facebook Analytics.Next.. <ul><li>What if.. </li></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved <ul><ul><li>We can rely on memory as a reliable store? </li></ul></ul><ul><ul><li>We can’t decide on a particular NoSQL database? </li></ul></ul><ul><ul><li>We need to package the solution as a product? </li></ul></ul>
  20. 20. Step 1: Use memory.. <ul><li>Instead of treating memory as a cache, why not treat it as a primary data store? </li></ul><ul><ul><li>Facebook keeps 80% of its data in Memory (Stanford research) </li></ul></ul><ul><ul><li>RAM is 100-1000x faster than Disk (Random seek) </li></ul></ul><ul><ul><ul><li>Disk - 5 -10ms </li></ul></ul></ul><ul><ul><ul><li>RAM – x0.001msec </li></ul></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved Events Memory Grid Data Grid Data Grid Data Grid FACEBOOK FACEBOOK FACEBOOK
  21. 21. Step 1: Use memory.. <ul><li>Reliability is achieved through redundancy and replication </li></ul><ul><li>One Data. Any API </li></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved Events Any API Data Grid FACEBOOK FACEBOOK FACEBOOK
  22. 22. Step 2 – Collocate <ul><li>Putting the code together with the data. </li></ul>Events Processing Grid Data Grid Data Grid Data Grid FACEBOOK FACEBOOK FACEBOOK
  23. 23. Step 2 – Collocate <ul><li>Putting the code together with the data. </li></ul>Events Processing Grid Data Grid Data Grid Data Grid FACEBOOK FACEBOOK FACEBOOK @EventDriven @Polling public class SimpleListener { @EventTemplate Data unprocessedData () { Data template = new Data (); template . setProcessed ( false ); return template ; } @SpaceDataEvent public Data eventListener ( Data event ) { //process Data here } }
  24. 24. Step 3 – Write behind to SQL/NoSQL Events Processing Grid Open Long Term persistency Write Behind FACEBOOK FACEBOOK FACEBOOK Data Grid Data Grid Data Grid
  25. 25. Economic Data Scaling <ul><li>Combine memory and disk </li></ul><ul><ul><li>Memory is x100, x1000 lower than disk for high data access rate (Stanford research) </li></ul></ul><ul><ul><li>Disk is lower at cost for high capacity lower access rate. </li></ul></ul><ul><ul><li>Solution: </li></ul></ul><ul><ul><ul><li>Memory - short-term data, </li></ul></ul></ul><ul><ul><ul><li>Disk - long term. data </li></ul></ul></ul><ul><ul><li>Only ~16G required to store the log in memory ( 500b messages at 10k/h ) at a cost of ~32$ month per server. </li></ul></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved Memory Disk
  26. 26. Economic Scaling <ul><li>Automation - reduce operational cost </li></ul><ul><li>Elastic Scaling – reduce over provisioning cost </li></ul><ul><li>Cloud portability (JClouds) – choose the right cloud for the job </li></ul><ul><li>Cloud bursting – scavenge extra capacity when needed </li></ul>® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  27. 27. Putting it all together Analytic Application Event Sources Write behind <ul><li>- In Memory Data Grid </li></ul><ul><li>- RT Processing Grid </li></ul><ul><li>Light Event Processing </li></ul><ul><li>Map-reduce </li></ul><ul><li>Event driven </li></ul><ul><li>Execute code with data </li></ul><ul><li>Transactional </li></ul><ul><li>Secured </li></ul><ul><li>Elastic </li></ul><ul><li>NoSQL DB </li></ul><ul><li>Low cost storage </li></ul><ul><li>Write/Read scalability </li></ul><ul><li>Dynamic scaling </li></ul><ul><li>Raw Data and aggregated Data </li></ul>Generate Patterns
  28. 28. Putting it all together Analytic Application Event Sources Write behind <ul><li>- In Memory Data Grid </li></ul><ul><li>- RT Processing Grid </li></ul><ul><li>Light Event Processing </li></ul><ul><li>Map-reduce </li></ul><ul><li>Event driven </li></ul><ul><li>Execute code with data </li></ul><ul><li>Transactional </li></ul><ul><li>Secured </li></ul><ul><li>Elastic </li></ul><ul><li>NoSQL DB </li></ul><ul><li>Low cost storage </li></ul><ul><li>Write/Read scalability </li></ul><ul><li>Dynamic scaling </li></ul><ul><li>Raw Data and aggregated Data </li></ul>Generate Patterns Real Time Map/Reduce R Script script = new StaticScritpt( “groovy”,”println hi; return 0”) Query q = em.createNativeQuery( “execute ?”); q.setParamter(1, script); Integer result = query.getSingleResult();
  29. 29. 5x better performance per server! <ul><li>Hardware – Linux </li></ul><ul><ul><li>HP DL380 G6 servers - each has: </li></ul></ul><ul><ul><li>2 Intel quad-core Xeon X5560 processors (2.8 Ghz Nehalem) </li></ul></ul><ul><ul><li>32 Gb RAM (4GB per core) </li></ul></ul><ul><ul><li>6 * 146 Gb 15K RPM SAS disks </li></ul></ul><ul><ul><li>Red Hat 5.2 </li></ul></ul>Event injector Up to 128 threads GigaSpaces/ (Other Msg Server) App Services Up to 128 threads Other Giga 50,000 write/sec per server
  30. 30. Live demo Inter Day Activity (Real Time) Monthly Trend Analysis
  31. 31. 5 Big Data Predictions ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  32. 32. Summary Big Data Development Made Simple: Focus on your business logic, Use Big Data platform for dealing scalability, performance, continues availability ,.. Its Open: Use Any Stack : Avoid Lockin Any database (RDBMS or NoSQL); Any Cloud, Use common API’s & Frameworks . All While Minimizing Cost Use Memory & Disk for optimum cost/performance . Built-in Automation and management - Reduces operational costs Elasticity – reduce over provisioning cost
  33. 33. Further reading.. ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
  34. 34. Thank YOU! @natishalom http://blog.gigaspaces.com

×