9. Use Cases
• Queue Processing
• Set Intersection
• Stats Tracking/Leaderboards
10. Keyword Filter
Geo Filter
Other Custom Filters
Resque Queue (redis)
Resque Workers
MongoDB Status Storage
External Data Tagging
- bit.ly, geo, klout
rowfeeder.com
MySQL User & Track Data
Slave MySQL
Geo Lookups
Memcache
Web Status Processing
Google
Spreadsheet
11. Queues
• Resque - Killer Ruby library for
background jobs
• RowFeeder - 428,811,359 items processed
a b c d
LPOP
RPUSH
12.
13.
14. Set Intersections
• Goal: Contact Ashton Kutcher on Twitter
• Approach: Is there anybody I can DM
that can DM Ashton, or a friend of his?
15. Step 1
• Are any of my followers his friends?
• SADD - Add specified member to the set
• SINTER - Intersection of N sets
16.
17. Step 2
• Can I DM somebody that can DM a
friend of his?
• ?t
?
23. Step 1
• Maintain counters of all matched terms
• Increment the “stream::kw::time” key by 1
24. Step 2
• 5 minute running average of the top 10
terms
25.
26.
27. Limitations
• Greedy with memory
• If it grows to XGB...it will stay there
• (Not optimal if a queue backs up)
• All in RAM / Atomicity
• 2.0 solves both of these!
• NoSQL