Slides presented by Richard McCreadie at CSE 2010 Workshop SIGIR

1. Crowdsourcing a News Query Classification Dataset Richard McCreadie, Craig Macdonald & IadhOunis 0

4. Workers can easily `game` the task

5. News Queries change over time for query in task return Random(Yes,No) end loop 2 News-related? Sea Creature? Query: Octopus? Work Cup Predications? How can we overcome these difficulties to create a high quality dataset for news query classification?

7. Dataset Construction Methodology (4--14)

8. Research Questions and Setting (15--17)

9. Experiments and Results (18--20)

10. Conclusions (21)Talk Outline 3

15. Interfaces : Basic What the workers need to do Clarify news-relatedness 8 Query and Date Binary labelling

16. Interfaces : Headline 12 news headlines from the New York Times . . . Will the workers bother to read these? 9

17. Interfaces : HeadlineInline 12 news headlines from the New York Times . . . 10 Maybe headlines are not enough?

18. Interfaces : HeadlineSummary news headline news summary . . . Query: Tigers of Tamil? 11

19. Interfaces : LinkSupported Links to three major search engines . . . Triggers a search containing the query and its date Also get some additional feedback from workers 12

22. Dataset Construction Methodology (4--14)

23. Research Questions and Setting (15--17)

24. Experiments and Results (18--20)

25. Conclusions (21)Talk Outline 14

28. Portal

29. Judgments per query : 3

32. Dataset Construction Methodology (4--14)

33. Research Questions and Setting (15--17)

34. Experiments and Results (18--20)

35. Conclusions (21)Talk Outline 17

37. How is our Baseline?Validation is very important: 32% of judgments were rejected Basic Interface Accuracy : Combined Measure (assumes that the workers labelled non-news-related queries correctly) Recall : The % of all news-related queries that the workers labelled correctly As expected the baseline is fairly poor, i.e. Agreement between workers per query is low (25-50%) 20% of those were completed VERY quickly: Bots? Kfree : Kappa agreement assuming that workers would label randomly Precision : The % of queries labelled as news-related that agree with our ground truth . . . and new users Kfleiss : Kappa agreement assuming that workers will label according to class distribution Watch out for bursty judging

40. High recall and agreement indicate that the labels are of high qualityPrecision . Workers finding other queries to be news-related Recall Workers got all of the news-related queries right! Agreement . Workers maybe learning the task? Majority of work done by 3 users testset fullset

42. We are confident that our dataset is reliable since agreement is highBest Practices Online worker validation is paramount, catch out bots and lazy workers to improve agreement Provide workers with additional information to help improve labelling quality Workers can learn, running large single jobs may allow workers to become better at the task Questions? 21