Presented at Hadoop-DC on October 4, 2011.

1. Demand, Media, and Search
Analytics
Sean Timm
sean.timm@teamaol.com
Twitter: @timmsc
October 4, 2011

2. Introduction
• Who am I?
• What do we use Hadoop for?
• Our best practices
• Lessons learned
• The related searches, seasonality—example applications
Page 2

3. History
• Originated in Search Backend in 2007
• Create data driven products for search.aol.com from search
logs
• No Netezza experience, decided to try Hadoop
• Took 3 weeks to write simple aggregation
• Apache Pig 0.3—2 days
• First product, related searches, launched in 2008
• Search breaking trends product led to further demand work
• Now Pig 0.8.1 and Hadoop 0.20.2
Page 3

4. Data
Hourly search.aol.com logs
•5 M log lines of data per hour
•Logs include searches, clicks, and other data
•70% of queries we only see once
Hourly Wikipedia page view data
•public data set http://dammit.lt/wikistats
•7 M pages viewed per hour
•2.7 M English pages per hour
BeacoN logs
•Page view and click logs for AOL HuffingtonPost Media, Patch, and other AOL
properties
Page 4

5. We like Pig!
• Hourly, daily, and monthly search and click aggregation
• Related searches
• Auto complete dictionary
• Mining spelling correction click through
• Temporal pattern analysis
• Classifying adult queries and URLs
• Categorizing queries
• Identifying queries in the form of a question or superlative
• Identifying breaking trends in AOL Search and Wikipedia page views
• Identify queries of local interest
• Clustering queries using click graph, temporal distance, Carrot2, k-means
• AOL HPMG stats and trends for page views, authors, tags, etc.
Page 5

6. Pig Process in General
Script run time < 2 minutes to > 2 hours
Ad hoc…wild west
Complex shell scripts
1. load/copy/backup data
2. Launch multiple Pig scripts—some in parallel—some
with serial dependencies
3. Check for errors—e-mail and halt
4. Load data into MySQL, Vertica, or Solr
Page 6

7. Getting data out of Hadoop
First approach: special StoreFunc to write directly to MySQL/Solr
•Network: Required master be on the same network as the
cluster
•Speculative optimization: data would be written more than once
increasing contention as well as doing unnecessary writes
•Replication: writing to the master in parallel, serial replication
was slow (MySQL)
•Timeouts: occasionally a task failed and restarted (Solr)
Page 7

8. Getting Data out of Hadoop
MySQL/Vertica Now
•Write data to HDFS
•Copy from HDFS to local file system using CLI
•Load into database: LOAD DATA LOCAL INFILE from mysql client
Solr Now
•Custom StoreFunc writes Solr XML to HDFS
•Starting with Pig 0.7 fields are named using the Pig schema
•Copy from HDFS to local file system using CLI
•Load into Solr using remote streaming
Page 8

9. UDFs
• Use Piggy Bank and builtins when possible
• 89 custom UDFs packaged in a single jar
• Most are simple
• Validate a URL, URL decode a string, calculate a hash value,
date math, etc.
• Some are complex
• Spell check/correct, LOESS regression, Carrot2 clustering,
FFT, Euclidean distance, etc.
Page 9

10. Lessons learned
• Many small categorization scripts, better to use a larger single
one
• Set priority on large time sensitive jobs that fight for resources
with other jobs
• Fair scheduler
• Tuning the cluster for maps or reduces
• Don't write copious debug
• Use appropriate number of reducers (PARALLEL)
Page 10

11. Related Searches
Group by Query

12. Challenges
• Adult terms
• Misspellings
• Breadth of suggestions
• Coverage
• Timeliness of suggestions

13. Process Flow
• Filter and clean data
• Block adult terms, long queries, non-alpha, second+ pages,
operators, URL like queries, search spam
• Lower case
• Join to get query-related query groups
• Contextual spell correct within group
• Cluster related queries and pick the best from each
group
• Load into Solr

14. Related Searches Graph “The Eagles”
Hotel California
The band
NFL
Tribute
Boston College
Page 14

15. Classification
• Supervised learning
• Provide categorized set of queries and/or URLs
• Calculate a score based on the edge weights
• If the score exceeds a specified threshold the query or URL is
tagged with the category

16. Applications Outside of Search
• Author/citation bipartite graph
• Social network graphs
• User/Page view graphs

17. Temporal traffic correlation of Wikipedia Page Views
Page 17

18. Tomato Seasonality
May: planting tomatoes, tomato cages, types of tomatoes
June: pruning tomato plants
July: tomato diseases, tomato blight, tomato worm
August: tomato recipes, tomato soup, tomato sauce, tomato salsa
September: sun dried tomatoes, canning and freezing tomatoes
October: green tomato recipes
Page 18