[2024]Digital Global Overview Report 2024 Meltwater.pdf
Twitter streamingapi rubymongodbv2
1. Consuming the Twitter Streaming API
with Ruby and MongoDB
Jeff Linwood
Lone Star Ruby Conference V
August 12, 2011
http://www.jefflinwood.com
@jefflinwood
Friday, August 12, 2011
2. Goals
• Watch for any
tweet that contains
certain keywords
• Store those tweets
into the MongoDB
database
Friday, August 12, 2011
4. Twitter Streaming API+
Mongo DB
• Twitter Streaming API
• MongoDB
• Not a web application
Friday, August 12, 2011
5. The many APIs of
Twitter
• Twitter Streaming API
• User Streams
• Site Streams (for the big boys)
• REST API
• Search API
Friday, August 12, 2011
6. Twitter Streaming API
• Keywords
• Users
• Locations
Great photo is by rachel_thecat
http://www.flickr.com/photos/23209605@N00/2786126623/
Friday, August 12, 2011
7. TweetStream gem
by Michael Bleigh
https://github.com/intridea/tweetstream
Friday, August 12, 2011
8. Connecting to the
Twitter Streaming API
• JSON responses
• HTTP Basic Authentication
• One stream per account (dev/prod)
• Leave it open!
• Don’t constantly reconnect, and if you do,
back off
Friday, August 12, 2011
9. Limitations of Twitter
Streaming API
• 400 Keywords
• 5,000 User Ids
• 25 Location Boxes
• Can ask Twitter for increased access
Friday, August 12, 2011
11. Intro to MongoDB
• NoSQL - what does that mean?
• Great fit for JSON-oriented applications
• If you don’t know your schema in advance
• Query language
• Map Reduce
Friday, August 12, 2011
12. Storing data in MongoDB
• Native format of MongoDB is BSON,
similar to JSON
• Connect to a database (similar to MySQL)
• Connect to a collection (created if it
doesn’t exist)
• Insert JSON (in our case, a tweet)
Friday, August 12, 2011
14. Considerations for
MongoDB
• Tweets - very verbose JSON
• Date format in Tweets not same as
MongoDB
• May want to pre-process Tweets
• Can use both MongoDB and MySQL in
same app if you want
Friday, August 12, 2011
15. Here’s the code
• Okay, the whole thing is really done in
about three lines.
Friday, August 12, 2011
16. Where do you run this?
• Command line
• Your own server
• Heroku + MongoLab
• Other cloud services
Friday, August 12, 2011
