Web Observatory Workshop at the International World Wide Web Conference, 2013.

1. Design and Prototyping of
a Social Media Observatory
Karissa McKelvey and Filippo Menczer
Center for Complex Networks and Systems Research
School of Informatics and Computing
Indiana University, Bloomington
1

2. Can we use social media as
laboratories for social
science?
truthy.indiana.edu 2

3. Political Polarization on Twitter Michael Conover, Jacob Ratkiewicz, Bruno Gonçalves, Alessandro Flammini
& Filippo Menczer International Conference on Weblogs and Social Media 2011
truthy.indiana.edu 3

4. Data “mine” ing
truthy.indiana.edu 4

5. Data “ours” ing
truthy.indiana.edu 5

6. truthy.indiana.edu
truthy.indiana.edu 6

7. Design and Prototyping
truthy.indiana.edu 7

8. Social Media
truthy.indiana.edu 8
streaming
sensitive
very large
structured
multiple sources

9. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 9

10. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 10

11. Reliability
• Spam and misinformation
truthy.indiana.edu 11

12. celebrities
spam
astroturf
politics
12

13. Reliability
• Spam and misinformation
• Cleansing and tagging by social, algorithmic,
or other means
truthy.indiana.edu 13

14. truthy.indiana.edu 14

15. Reliability
• Spam and misinformation
• Cleansing and tagging by social, algorithmic,
or other means
• Sampling bias
truthy.indiana.edu 15

16. Data Collection
• Twitter Streaming API, random sample
• August, 2010 – present
• 5TB Compressed
• Real-time access to data from last 9 months
related to 3 themes: US Politics, Social
Movements, News
truthy.indiana.edu 16

17. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 17

18. Reproducibility
• Standard ontology for the storage, reference,
and transfer of these datasets between users.
truthy.indiana.edu 18

19. Model
• Events
– Post on a social media site
• Users
– Actors in the post
– Sender, receiver, forwarder, etc
• Meme
– Discernable unit of information transfer
– Eg, hashtag, URL, user, phrase…
truthy.indiana.edu 19

20. Meme Diffusion Networks
truthy.indiana.edu 20
Retweet (forward) Mention (conversation)

21. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 21

22. truthy.indiana.edu 22

23. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 23

24. truthy.indiana.edu 24

25. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 25

26. Open Access
• A public and free -- or low-cost -- social
observatory enables access to large-scale
social media data analytics for non-profit
endeavors.
truthy.indiana.edu 26

27. API & Website
• Filterable and
searchable interface
to find memes of
interest
• Endpoints for users
to access data
programmatically
• Visualizations
truthy.indiana.edu/apidoc
truthy.indiana.edu 27

28. Design Considerations
Reliability
Reproducibility
Topic Filtering
Visualization
Open Access
Legal Compliance
truthy.indiana.edu 28

29. Legal Compliance
• Terms of Service
– Tweet text
– Derived data
truthy.indiana.edu 29

30. truthy.indiana.edu
truthy.indiana.edu 30

31. Thanks!
Papers at cnets.indiana.edu/groups/nan/truthy
S
andro Flammini
Bruno Conçalves
J
acob Ratkiewicz
LilianWeng
Mike Conover
J
ohan Bollen
KarissaMcKelvey
Przemek Grabowicz
Mark Meiss
AlexVespignani
Alex Rudnick
LucaAiello
Fil Menczer
Mohsen J
afari-Asbagh
Onur Varol
Emilio Ferrara
Wednesday, September 26, 12