This document outlines a course on data design and creative research practices. The course covers topics like data mining, visualization, analysis techniques, and prototyping. It discusses methods for gathering meaningful data through surveys, crawling, and user experience design. Example analysis techniques covered include clustering, classification, regression, and association rules. Students are asked to consider what types of systems or products could be built by analyzing collective intelligence data related to topics like politics, entertainment, food, and more. For a homework assignment, students form groups to explore potential data sources and analysis techniques for a collective intelligence topic.

1. Data Design
2114.409: Creative Research Practice
HTTP://WWW.FLICKR.COM/PHOTOS/SERGIU_BACIOIU/4370021957/

2. Reflection
Status Check
Concerns
Programming
What can we build
HTTP://WWW.FLICKR.COM/PHOTOS/FLOWER87/76719859/

3. Course Outline
1. Foundations 3. Prototyping
Introduction Crawling
Survey Methods / Data Mining Text Mining
Visualization and Analysis To be determined (TBD)
Social Mechanics Project Update
2. Methods 4. Refinement
Creativity and Brainstorming TBD x3
Prototyping Project Presentations
Project Management Reflection

4. Last Week: Building Blocks
Clustering
Classification
& Regression
Association
Rules
Outlier
Detection
HTTP://WWW.FLICKR.COM/PHOTOS/OGIMOGI/2253657555/

5. This Week: Systems
HTTPS://WWW.FACEBOOK.COM/PHOTO.PHP?FBID=407391545956901&SET=A.407391429290246.110679.100000581776191&TYPE=3&THEATER

6. Data Mining Overview
How do I see and
Visualization, Storytelling
communicate answers?
What questions should
Design, Data Exploration
I ask of the data?
How do I clean and
Analysis Techniques
process the data?
How do I gather
Crawling, Surveys, UX Design
meaningful data?

7. Why might we prefer analysis?
LABOR ACCURACY
Too many pictures to look at. Can test for statistical
significance, etc.
Don’t know which are
interesting. Some patterns don’t
visualize easily.
HTTP://WWW.FLICKR.COM/PHOTOS/STRIATIC/2144933705/

8. Clustering
Find natural
groupings in
the data
Organize data into classes:
‣ high intra-class similarity
‣ low inter-class similarity

9. Clustering
Input Data Output Clusters
Points Hard
OR
OR
Soft
Similarities OR
[ # of clusters ] Hierarchical

10. Classification Regression
Learn to map objects to Learn map objects to
categories continuous variables

11. Classification
Observations X Learn f(x) = y
Labels Y
Y = gender
Male
Female
X = height

12. The Whole Process
Data Set
Featurization
Featurized
Random Split (e.g. 90/10)
Training Data Test Data
Training
Model
Evaluation
Results

13. Association Rules
Learn interesting
relations in the data
= proportion of events in which X occurs

14. Anomaly Detection
Detect strange
events in the data
Simplest measure:

15. What Can
We Build?
HTTP://WWW.FLICKR.COM/PHOTOS/BPENDE/6736531173/

16. Collective Intelligence
Clicks,) Likes,) Updates,) Ar,cles,)
Scrolls,) Links,) Reviews,) Images,)
Time) Checkins) Comments) Video)
Collec,ve) How can we harness the
Intelligence)
activities of the world’s digital
citizens to build new and
useful consumer services?
Community)

17. Politics
The Korean elections are coming. How
does the Internet tell us more than
traditional polling ever could?

18. Politics
What issues are important?
Who are the influencers?
How can we segment/characterize support groups?
How do we spread our opinions more widely?
Who will win the election?

19. How can we build this?
“Can social
media predict
election
outcomes?”
HTTP://WWW.USATODAY.COM/TECH/
NEWS/STORY/2012-03-05/SOCIAL-
SUPER-TUESDAY-PREDICTION/
53374536/1

20. Tweet Insert Magic
Author
Date Here?
Body
Retweets
Hashtags Prediction
Candidate
Location
Classification &
Author Clustering
Regression Score
Profile Confidence
Tweets
Favorites
Following
Followers Association Outlier
Location Rules Detection

21. Workshop

22. Sentiment +
Candidate System Overview
Tweet Inputs
Correction based
Scoring
on past elections
Refinements
Author Inputs
RMSE Evaluation

23. Sentiment Detail
Input Observation Feature Extractor
Classifier Output Label
Confusion Matrix
Evaluation
N-Gram Features
Training Process
Tweet + Label

24. Entertainment Food Movements
HTTP://WWW.FLICKR.COM/PHOTOS/STUCKINCUSTOMS/2786154526/ HTTP://WWW.FLICKR.COM/PHOTOS/WILLIA4/2504379334/ HTTP://WWW.FLICKR.COM/PHOTOS/GILSONROME/6247208325/
Collaboration Shopping Travel
HTTP://WWW.FLICKR.COM/PHOTOS/FIDELMAN/4640722483/ HTTP://WWW.FLICKR.COM/PHOTOS/ZOOBOING/4473219605/ HTTP://WWW.FLICKR.COM/PHOTOS/FELIPENEVES/5414239936/
Investing Medicine Trust
HTTP://WWW.FLICKR.COM/PHOTOS/STUCKINCUSTOMS/2786154526/
HTTP://WWW.FLICKR.COM/PHOTOS/TRAVEL_AFICIONADO/2396819536/ HTTP://WWW.FLICKR.COM/PHOTOS/AGECOMBAHIA/6425101047/ HTTP://WWW.FLICKR.COM/PHOTOS/MARKETINGFACTS/6758968163/

25. Homework: Data Mining
1. Form groups!
2. Choose a Collective Intelligence topic from
Lecture 1, or propose similar.
3. Make a list of data sources that might
provide insights to that topic.
4. Propose a set of meaningful questions about
the data based on your intuition.
5. How would you have to clean/process your
data to start answering those questions?
6. Consider clustering, association rules,
anomaly detection, classification. For each
technique, how might you apply it to the
data and what would it show?
7. Document your work and be prepared to
present.
HTTP://WWW.FLICKR.COM/PHOTOS/31907740@N00/4860840019/

26. Feedback