This was a talk given at Chulalongkorn University in Bangkok, Thailand on January 16, 2015 as a part of CodeMania X2 "Data Science 101"

1. Krist Wongsuphasawat /@kristw
visualizationdata
A quick tour for data science enthusiasts

2. visualizationdata
What is it about?
What is it good for?
How is it related to data science?
Example projects
…

3. 1. What is it about?

4. “A picture is worth more than a thousand words.”
— ใครสักคนได้กล่าวไว้

6. Data Picture

7. Data Visual display

8. Help audience consume a lot of information rapidly
Data Visual display

9. 2. What is it good for?

10. Example / History

11. data

13. location (lat,lon => x,y), quantity of troops (width), direction (color)
time (x), temperature (y)

14. Example / Cholera epidemic

15. List of deceased patients
!
Mr. Smith, who lived at 11 Sunny St.
Miss White, who lived at 23 Cloudy Rd.
Mr. Jones, who lived at 30 Rainy St.
Mrs. Robinson, who lived at 34 Windy Rd.
…
data

16. John Snow

19. What is it good for?
Storytelling
Communicate known information
Exploratory data analysis
Explore data to reveal insights

20. More powerful
Visualization = Visual display + Interaction

21. 3. How is it related
to data science?

22. Turn data into
valuable insights
data product
interesting stories

23. data
wrangling
output
insights, products, stories
exploratory
data analysis
report
results
raw data
in-depth
analysis

24. data
wrangling
output
insights, products, stories
exploratory
data analysis
report
results
in-depth
analysis
communication,
storytelling
raw data

25. 4. Example projects

26. 4.1 Ballon d’Or

29. FIFA released voting data

30. • 3 voters / country
• National team captain
• National team coach
• Journalist (media)
• Each voter select 3 players for 1st, 2nd and 3rd place
Rules

32. data
wrangling
output
insights, products, stories
exploratory
data analysis
report
results
in-depth
analysis
communication,
storytelling
raw data

33. • Given data are tables in PDF.
• Extract to csv
• Format data to desired format.
Data Wrangling

34. Demo / Ballon d’Or
https://medium.com/@kristw/who-voted-for-who-diving-into-ballon-dor-voting-data-e09138ba9712

35. 4.2 Public-facing vis
& New year 2013

36. interactive.twitter.com

37. Geo
Heatmap
Low density
High density

38. Geo
San Francisco
flickr.com/photos/twitteroffice/8798020541
Low density
High density

39. Geo
San Francisco
Rebuild the world
based on
tweet volumes
twitter.github.io/interactive/andes/

40. How are these phrases used in Tweets?
Is there any pattern?

41. data
wrangling
output
insights, products, stories
exploratory
data analysis
report
results
in-depth
analysis
communication,
storytelling
raw data

42. Big data wrangling

43. Having all Tweets
How people think I feel.

44. How people think I feel. How I really feel.
Having all Tweets

45. • Too much data, want only relevant Tweets
• contain “สวัสดีปีใหม่”
• variations: หวัดดีปีใหม่, หวัดดีปีหม่ายยย
• typos: หวัดตีปีใหม่
• Need to aggregate & reduce size
• Long processing time (hours)
Challenges

46. Hadoop Cluster
Data Storage
Workflow

47. Hadoop Cluster
Pig / Hive / Scalding (slow)
Data Storage
Tool
Workflow

48. Hadoop Cluster
Pig / Hive / Scalding (slow)
Data Storage
Tool
Workflow

49. Hadoop Cluster
Pig / Hive / Scalding (slow)
Data Storage
Tool
Smaller datasetYour laptop
Workflow

50. Hadoop Cluster
Pig / Hive / Scalding (slow)
Data Storage
Tool
Final dataset
Tool node.js / python / etc. (fast)
Your laptop
Workflow
Smaller dataset

51. Exploratory Data Analysis

52. Improve design
for releasing to public

58. Demo / New Year 2013
twitter.github.io/interactive/newyear2014/

61. Another fun fact:
Developed using 2012 data
Then update data on Jan 2, 2013

62. 4.3 Data Analysis Tool

63. data
wrangling
output
insights, products, stories
exploratory
data analysis
report
results
in-depth
analysis
communication,
storytelling
raw data

64. Logging user activities

65. UsersUseTwitter

66. UsersUse
Product Managers
Curious
Twitter

67. UsersUse
Curious
Engineers
Log data
in Hadoop
Write Twitter
Instrument
Product Managers

68. What are being logged?
tweet
activities

69. What are being logged?
tweet from home timeline on twitter.com
tweet from search page on iPhone
activities

70. What are being logged?
tweet from home timeline on twitter.com
tweet from search page on iPhone
sign up
log in
retweet
etc.
activities

71. Organize?

72. log event a.k.a. “client event”
[Lee et al. 2012]

73. log event a.k.a. “client event”
client : page : section : component : element : action
web : home : timeline : tweet_box : button : tweet
1) User ID
2) Timestamp
3) Event name
4) Event detail
[Lee et al. 2012]

74. Twitter for Banana

75. Count page visits
banana : home : - : - : - : impression
home page

76. User sessions
Session#1
A
B
start
end
Session#4
start
end
A
Session#2
B
start
end
A
Session#3
C
start
end
Aclient event
client event

77. Funnel
home page
profile page

78. Funnel analysis
banana : home : - : - : - : impression
banana : profile : - : - : - : impression
1 jobhome page
profile page
1 hour

79. Funnel analysis
banana : home : - : - : - : impression
banana : profile : - : - : - : impression banana : search : - : - : - : impression
home page
profile page search page
2 jobs
2 hours

80. Funnel analysis
banana : home : - : - : - : impression
banana : profile : - : - : - : impression banana : search : - : - : - : impression
home page
profile page search page
Specify all funnels manually!
n jobs
n hours

81. Goal
banana : home : - : - : - : impression
… ……
1 job => all funnels, visualized
home page

82. User sessions
Session#1
A
B
start
end
Session#4
start
end
A
Session#2
B
start
end
A
Session#3
C
start
end
A

83. Aggregate
4 sessions
A
BB C
start
end endend
A A
end
A

84. Aggregate
A
BB C
start
end endend
end
4 sessions

85. Aggregate
C
start
end endend
end
A
B
4 sessions

86. Aggregate
C
start
end endend
end
A
B
4 sessions

87. Aggregate
C
start
end endend
A
B end
4 sessions

88. Aggregate
C
start
endend
A
B end
4 sessions

89. Aggregate
C
start
endend
A
B end
4 sessions

90. Aggregate
start
endend
A
CB end
4 sessions

91. Aggregate
4,000,000 sessions
endend
A
CB end
start

92. Demo / Flying Sessions
Using Visualizations to Monitor Changes and Harvest Insights from a Global-Scale Logging
Infrastructure at Twitter by Krist Wongsuphasawat and Jimmy Lin. in Proc. IEEE Conference on Visual
Analytics Science and Technology (VAST), Paris, France, 13 November, 2014

93. visualizationdata
What is it about?
Data => Visual display + Interaction
What is it good for?
Exploratory data analysis & storytelling
How is it related to data science?
It is one of the skills often utilized in the process.
Example projects
interactive.twitter.com @kristw / kristw.yellowpigz.com

94. Thank you

95. Questions?