Solbase & Real-time Activity
•
1 j'aime•1,492 vues
Solbase, the real time open-source search engine, is now available on github. Solbase was developed by Photobucket.com and is built upon Lucene, Solr and HBase. Photobucket has also recently released a real time community activity stream capturing the 4 million daily uploads as well as all of your friends' comments and favorite photos. The foundation of the system is HBase and also employs Kestrel queues. This talk will cover the architecture, implementation details and share many of the lessons learned while developing this real time big data system.
![[object Object],[object Object],[object Object],[object Object],Who we are](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Solbase & Real-time Activity
Similaire à Solbase & Real-time Activity (20)
Dernier
Dernier (20)
Solbase & Real-time Activity
- 3. Photobucket Solbase Activity Stream Agenda
- 4. • Photobucket is the most-visited photo site with 23.4 Million UVs • Over 9 Billion photos stored! • Users upload 4 Million images per day! • Photobucket users spend more time than any other photo site with 3.8 Avg mins/visit • 2.0 Million avg daily visitors - more daily visits than Flickr and Picasa combined Sources: 1comScore May 2011, 2Internal data Photobucket Overview
- 5. 23.4M UVs 9.9M UVs 9.5M UVs 7.9M UVs 1.6M UVs 19.7M UVs 6.0M UVs
- 7. Solbase is an open-source, real-time search platform based on Lucene, Solr and HBase built at Photobucket What is Solbase?
- 12. https://github.com/Photobucket/Solbase https://github.com/Photobucket/Solbase-Solr https://github.com/Photobucket/Solbase-Lucene Solbase repos
- 13. Activity Stream is Social networking feature using HBase, Flume, Kestrel, Camel built at Photobucket What is Activity Stream?
- 26. Schema: {row key 1 { column family 1{ c olumn 1 {data1}, column 2 {data 2} … } ...} } {row key 2 {...}} Example: {dog:spotty {owner{matt{age 41}, linda{age 41}} vaccinations{rabies{july 2011}}} {cat:fluffy {owner{doug{age 41}, heather{age 41}} vaccinations{rabies{june2011}}} HBase Tables
- 32. Q&A
Notes de l'éditeur
- We should go over these agendas and introduce each of presenters
- First, Koh is going to talk about Solbase. That's our real time search engine that was built on top of Lucene, Solr, and HBase. We started presenting Solbase about 9 months ago, and at that time we reported that our standard implementation of lucene/solr was no longer scaling to meet our needs, and our initial tests of Solbase gave us hope that we were going to solve that problem AND dramatically improve performance. In addition we were updating our search index in real time. Great results, but possibly the bigger news at that time was that we were planning to open source all the code. Tonight Koh is here to deliver on that promise. The next topic we'll cover is another HBase feature developed at PB: our activity stream. It's what you'd probably expect. A social network feature that distributes events about photos and videos in near real time. We've seen a number of presentations on similar features, but rarely to you see any detail on the architecture or lessons learned that would help you build your own. Ron and Josh are going to do exactly that. But before we jump into all that... why do you care? who is PB?
- We're the biggest dedicated photo site on the web and we're right next door. We have millions of active users and billions of photos.
- Here's a quick slide on our size compared to our peers… its a little old, but you get the idea. We have millions of unique visitors.
- Over time those users have contributed half a billion public photos and videos to our search index, and we generate a boatload of social events around all that public media.
- Lucene's Field cache for sorting and filtering became very problematic for us Turn around time for building entire set of indices took us about a day Every 100 ms improvement in response time equates to approximatey 1 extra page views Impractical to add significatn number of new docs and data
- In a nutch shell, Solbase have basically replaced indices stored in local filesystem to database in HBase also overcame lucene's inherent limitations. and one major one we solved is sort/filter
- Ron Here
- Ron Here
- Ron Here
- Ron Here
- Kestrel is open source and developed at twitter.
- Talk about scale and real-time processing speed. Ops per second. 1 thread push 40/s all the way to hbase.
- Talk about scale and real-time processing speed. Ops per second. 1 thread push 40/s all the way to hbase.
- Josh Here HBase is a distributed big-table like database build upon Hadoop components leverages HDFS, Hadoop ’s distributed file system Built upon Hadoop, scales to a massive size, virtually limitless used by many large scale companies: Facebook, Yahoo, Google (through their big-table implementaiton) Ask who has used hbase
- Josh Here HBase is a distributed big-table like database build upon Hadoop components leverages HDFS, Hadoop ’s distributed file system Built upon Hadoop, scales to a massive size, virtually limitless used by many large scale companies: Facebook, Yahoo, Google (through their big-table implementaiton) Ask who has used hbase To fix: 1. Features column store key/value store witih semi-structured values. 2. Why use hbase? -horizontal scalability -high write throughput -millions of columns billion of rows
- consists of master nodes with a set of region servers to distribute the data The master is the gateway interface to direct clients to the proper region server for the requested data Data is replicated among several data nodes by Hadoop ’s file system, HDFS There is ‘locational affinity’ between the region server and the data served
- Each table consists of a row key, a set of defined column families, and an arbitrary number of qualified columns for each family Keys are store lexicographically so that range scans between two keys is extremely fast All data is binary interestingly, this is similar to the concept of the inverted index, where the ‘terms’ are lexicographically stored; this is something that we leverage in our implementation
- Mention using lexicographical key to pre-sort data.
- Get : single row access, similar to SQL like query by primary key Put: single row update/insert (can be done in batch) Scan: lexicographic range query between 2 specified keys
- Back to Ron HBase optimization: scans continue to be fast, large multi-gets have been an issue.
- HBase optimization: scans continue to be fast, large multi-gets have been an issue.