Understanding Big Data with Examples
•
1 like•1,933 views
A guest lecture in the Master elective "The Blind Spot: Tracking Young Media Users" by Susanne Baumgartner
Recommended
Recommended
More Related Content
What's hot
Similar to Understanding Big Data with Examples
Similar to Understanding Big Data with Examples (20)
More from Department of Communication Science, University of Amsterdam
More from Department of Communication Science, University of Amsterdam (19)
Recently uploaded
Recently uploaded (20)
Understanding Big Data with Examples
- 1. What’s big data? Some examples Problems A glimpse in the kitchen Questions? #bigdata in Communication Science Some examples from research by me and my students Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net Afdeling Communicatiewetenschap Universiteit van Amsterdam October 2013 #bigdata Damian Trilling
- 2. What’s big data? Some examples Problems A glimpse in the kitchen Questions? 1 What’s big data? 2 Some examples Rare events Tone in tweets Counting words and n-grams Network analysis 3 Problems 4 A glimpse in the kitchen 5 Questions? #bigdata Damian Trilling
- 3. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What’s big data? #bigdata Damian Trilling
- 4. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What’s big data? No definition, but . . . • Existing data • Too big to code manually • Sometimes also too big to handle with normal tools • New research questions • Call to revisit the relationship between theory and empirical research #bigdata Damian Trilling
- 5. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What’s big data? Some sources • Social Network Sites • RSS-feeds • Databases • Scraping text from the web • ... #bigdata Damian Trilling
- 6. What’s big data? Some examples Problems A glimpse in the kitchen Questions? It’s out there! You only have to collect it. #bigdata Damian Trilling
- 7. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Some examples #bigdata Damian Trilling
- 8. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events A recent master thesis Rare events #bigdata Damian Trilling
- 9. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events A recent master thesis Rare events Imagine you want to analyze some very rare content. #bigdata Damian Trilling
- 10. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events A recent master thesis Rare events Imagine you want to analyze some very rare content. Normal sampling won’t work, that’s for sure. #bigdata Damian Trilling
- 11. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events So you’d better collect everything first Getting all news coverage from Dutch news sites Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 12. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events So you’d better collect everything first Getting all news coverage from Dutch news sites We collected all articles from nine news sites during a period of two months, resulting in a database with 74.000 articles. Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 13. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events So you’d better collect everything first Getting all news coverage from Dutch news sites We collected all articles from nine news sites during a period of two months, resulting in a database with 74.000 articles. In a second step, we filtered those articles containing specific keywords. Those 292 articles where then manually coded. Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 14. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events #bigdata Damian Trilling
- 15. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events It’s just one line of code! url.txt http://www.gmx.at/themen/wissen/mensch/108g5xi-baeuerlich-schiefe-zaehne http://www.gmx.at/themen/unterhaltung/klatsch-tratsch/408g740-fuermannbittet-um-verzeihung http://www.gmx.at/themen/nachrichten/aufruhr-arabien/268g70u-regierungwill-zuruecktreten http://www.gmx.at/themen/nachrichten/panorama/828g54y-neues-zur-klagegegen-republik http://www.gmx.at/themen/nachrichten/panorama/968g72s-millionstrafewegen-oelpest http://www.gmx.at/themen/unterhaltung/klatsch-tratsch/368g6yc-keinbabybauch-nur-fast-food ... ... ... #bigdata wget-commando wget -i urls.txt Damian Trilling
- 16. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets A recent bachelor thesis Tone in tweets #bigdata Damian Trilling
- 17. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets A recent bachelor thesis Tone in tweets Imagine you want to know something about someone’s behavior on twitter. Or how a specific topic is discussed on Twitter. #bigdata Damian Trilling
- 18. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets A recent bachelor thesis Tone in tweets Imagine you want to know something about someone’s behavior on twitter. Or how a specific topic is discussed on Twitter. Do you really want to go through thousands of tweets by hand? #bigdata Damian Trilling
- 19. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 20. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents We took lists with positive and negative words and with a politician’s opponents. Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 21. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents We took lists with positive and negative words and with a politician’s opponents. We used a Python-script to check which type of words were used to refer to opponents. Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 22. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents We took lists with positive and negative words and with a politician’s opponents. We used a Python-script to check which type of words were used to refer to opponents. For further analysis, the results where imported in SPSS. Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
- 23. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets #bigdata Damian Trilling
- 24. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets #bigdata Damian Trilling
- 25. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams How often are specific expressions used? Counting words and n-grams #bigdata Damian Trilling
- 26. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams How often are specific expressions used? Counting words and n-grams Imagine you want to know which words or expressions dominate a discourse . #bigdata Damian Trilling
- 27. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams How often are specific expressions used? Counting words and n-grams Imagine you want to know which words or expressions dominate a discourse . There are plenty of possibilities to get an answer within minutes, here’s one: #bigdata Damian Trilling
- 28. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams Again, just one or two lines of code! For example with STATA • Install the package wordscore (net install http://www.tcd.ie/Political_Science/wordscores/wordscores) • voor wordcounts: wordfreq /home/dami/texts/lab92.txt /home/dami/texts/lab97.txt • voor ngrams (trigrams in dit geval): phrasefreq 3 lab92.txt lab97.txt #bigdata Damian Trilling
- 29. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams trigrams in Obama-Tweets #bigdata Damian Trilling
- 30. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis Another approach Network analysis #bigdata Damian Trilling
- 31. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis Another approach Network analysis Imagine you want to know who talks to whom and how networks are interconnected . #bigdata Damian Trilling
- 32. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis Another approach Network analysis Imagine you want to know who talks to whom and how networks are interconnected . Use a tool like NodeXL or Gephi! #bigdata Damian Trilling
- 33. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis #bigdata Damian Trilling
- 34. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems #bigdata Damian Trilling
- 35. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties #bigdata Damian Trilling
- 36. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties • Services can shut down (GoogleReader) or change their API (Twitter) #bigdata Damian Trilling
- 37. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties • Services can shut down (GoogleReader) or change their API (Twitter) • It’s rather easy to get (up to 3200) tweets from a specific user (e.g., allmytweets.net), but if you want to capture a #hashtag, you have to record it live #bigdata Damian Trilling
- 38. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties • Services can shut down (GoogleReader) or change their API (Twitter) • It’s rather easy to get (up to 3200) tweets from a specific user (e.g., allmytweets.net), but if you want to capture a #hashtag, you have to record it live • Twitter doesn’t give you all tweets, but just about 1% (+ a bunch of other limits) #bigdata Damian Trilling
- 39. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems Not sure if this a problem or a great opportunity. . . You cannot rely (only) on ready-made software but shout get ready to use tools like bash-scripts, grep, python, . . . (Which can be fun!) #bigdata Damian Trilling
- 40. What’s big data? Some examples Problems A glimpse in the kitchen Questions? A glimpse in the kitchen #bigdata Damian Trilling
- 41. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What I’m doing right now Analyzing #tvduell • 570.000 tweets • Identifyig clusters of nouns, verbs and adjectives • Assigning positivity and negativity scores to tweets • See if they can be interpreted as frames ⇒How are Merkel and Steinbrück framed on the Second Secreen during the debate? #bigdata Damian Trilling
- 42. What’s big data? #bigdata Some examples Problems A glimpse in the kitchen Questions? Damian Trilling
- 43. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Something you can use? 1 What’s big data? 2 Some examples Rare events Tone in tweets Counting words and n-grams Network analysis 3 Problems 4 A glimpse in the kitchen 5 Questions? #bigdata Damian Trilling
- 44. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Vragen of opmerkingen? Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net #bigdata Damian Trilling