Introductory course to Open Data

1. Introduction to
OPEN DATA
and other hypes
J. Minguillón
EIMT / UOC

2. course goals

3. understand the meaning of open,
data and other related concepts
know where to find and how to
reuse open data
create a proof-of-concept using
open data (project)
become open data supporters

4. what is Open Data?

5. what is Open?

6. what is Data?

7. plural of "datum" (thing given)
data is / data are
idea: the measure / amount / ...
of something
definition

8. 42

9. 42 what?
https://en.wikipedia.org/wiki/42_(disambiguation)

10. forty-two
quaranta-dos
amane nambili

11. representation

12. integer?
base / radix?
units?

13. D-I-K-W pyramid

14. D: 42
I: Patient's body temperature (t) is 42
degrees
K: Fever with t > 42 can cause severe brain
damage
W: never let t reach 42 degrees!

15. t = 42 degrees?
Celsius: strong fever
Fahrenheit: cold body
Kelvin: cold body floating in outer space

16. data is not just numbers
source: https://flic.kr/p/5A9X6P

17. tables, documents
wikipedia: pages / articles
flickr: images
twitter: tweets
metadata (data about data)

18. source: https://flic.kr/p/87P3sc

19. Locals and Tourists
Eric Fischer
metadata from flickr

20. data
=
internal structure
x
possible values

21. basic types
structured
semi-structured

22. basic types
integer, real, complex
vectors (RGB, ...)
characters, strings

23. structured data
flat: 1D, 2D, 3D, ...
hierarchical: tweets
relations: graphs

24. semi-structured data
documents
web: HTML pages

25. summary
knowing data format and
structure facilitates its
manipulation

26. what is Open?

27. definition

28. openness as freedom
source: https://flic.kr/p/6p2kFa

29. 5 Rs model
Reuse
Revise
Remix
Redistribute
Retain

31. open vs free
https://theodi.org/blog/when-data-is-free-but-not-open

32. open is a combination of
no technological barriers
no legal barriers

33. technological
barriers
source: https://flic.kr/p/ad8i3

34. technological barriers
data must be accessible
downloadable
manipulable

35. the 5 star model
* no manipulable: pdf, tiff
** proprietary: doc, ppt, xls
*** open formats: txt, csv, json
**** accessible (link): xml, rdf
***** provide context: xml, rdf
http://5stardata.info/en/

36. open data needs at least 3 star
open formats
open software

37. linked data

38. linked data
use URIs to name things
use HTTP to provide access
describe data using metadata
link to related data sources
readable by machines

39. example
<profile id="jminguillona">
<website>
https://en.wikipedia.org/wiki/User:Julià_Minguillón
</website>
<twitter>
https://twitter.com/jminguillona
</twitter>
<orcid>
https://orcid.org/0000-0002-0080-846X
</orcid>
<institution>
http://www.uoc.edu
</institution>
...
</profile>

40. why linked data?
automatic web data extraction
data exchange / enrichment
construction of knowledge
semantic searches

41. example: wikidata
municipalities surrounding
Barcelona?
https://en.wikipedia.org/wiki/Barcelona
https://www.wikidata.org/wiki/Q1492

42. "static" access
data is downloaded as a file
files are "pictures of the past"
not defined by final users
typical of data repositories
human oriented
http://dadesobertes.gencat.cat/en/cercador/detall-cataleg/?id=5

43. "dynamic" access
data is downloaded as a stream
streams are "pictures of the present"
parametrized by final users (API)
typical of online services
machine oriented

44. Application
Programming
Interface
https://www.programmableweb.com/category/all/apis

45. example: UAB campus
equipments (Generalitat de Catalunya)
↓
geolocalization
↓
flickr API

46. legal barriers
source: https://flic.kr/p/dQeTEq

47. legal barriers
reachable through Internet does
not mean open
licenses
terms and conditions
EULAs

48. licenses for open data
for datasets / databases
facts cannot be restricted...
...but collections can!
http://opendatacommons.org/licenses/

49. terms and conditions
for web data
legal language
http://www.coca-colacompany.com/our-company/the-coca-cola-company-terms-of-use

50. EULA
End-User License Agreement
for apps and online services
legal language
absurd!
https://www.eff.org/wp/dangerous-terms-users-guide-eulas

51. ethic issues
privacy
security
transparency

52. some bad practices
AOL's searcher 4417749
Ashley Madison leaked
AEMET paywall
...

53. other issues
quality
traceability
availability

54. summary
do not forget there might be
some limitations unless is
proper open data

55. why open data?

56. why not?

57. data belongs to their producers
in most cases, users!
it promotes participation
and discovers additional value
"data is the new oil" (C. Humby)
"data is the new soil" (D. McCandless)

58. data life-cycle

59. data is ...
generated
stored / published
gathered / captured
preprocessed
analyzed
visualized

60. data generation
by humans / sensors / services
anytime / anywhere
persistent / volatile
stored / published

61. data gathering
from repositories
APIs
social networks
databases / logs
web scraping
humans (captcha)

62. data preprocessing
filtering / selection
join (enrichment)
feature extraction
conversion
summarize / aggregate

63. data analysis
statistical descriptors
inference
unsupervised (clustering)
supervised (classification)
variable relevance
...

64. data visualization
visual analysis
summarization
reporting
dashboards
maps / graphs
interactivity

65. tools
...

66. big data

67. big data
3 Vs
volume
variety
velocity

68. volume is
the number of elements
sample / population size

69. variety is
the number of different forms
dimensionality

70. velocity is
how fast data is produced or
changes
longitudinal

71. other Vs
veracity
value
variability
visibility
...

72. example: Wal-Mart
(2017) 260 million people
shop at Wal-Mart every week
from a list of 140,000 items
who buys what when?
why?

73. example
include context data
customer loyalty cards
product interestingness (RFID)
CCTV cameras
social networks
...

74. other big huge data players
amazon
VISA
telcos
facebook, twitter, ...
google

75. still waiting for
health
education
SMEs

76. big data also
uses multiple sources
deals with population, not samples
makes traditional methods obsolete
requires supercomputing / cloud

77. tools
(examples)

78. "engineering" approach
solve this problem now with the
available tools
no tool solves all problems
problems change, tools too
tools related to data life-cycle

79. data gathering
tabula
scrapy
twitteR, TAGS, flocker
instagram, flickr
wikipedia dumps
URL manipulation

80. example: URL manipulation
IDESCAT
names of newborn children
parameters: year, sex, place
other: position, sort

81. example: URL manipulation
use scrapy for data gathering
define desired fields
create list of URLs
identify XPATH (inspect)

82. data preprocessing
Mr. Data Converter
JSON online editor
OpenRefine
bash+awk, perl, python

83. data analysis
R, R Studio
python pandas, scikit
anaconda
gephi
...

84. data visualization
R: ggplot2, ggmap, ...
python: Bokeh, plotly, ...
processing
D3
openstreetmap
other: tagxedo, infogr.am, ...

85. example
visualizing co-authorship at UOC

86. data gathered from SCOPUS
unify author names, build graph
no analysis
visualize graph

87. what knowledge can we extract
from the visualization?
most profilic authors/departments
interdisciplinarity, connectors
internal publication policies
"lone rangers"

88. what other tools can we use to
analyze the graph?
discover communities
centrality, reputation
R, gephi, ...

89. what open data can we use to
enrich the visualization?
from authors/departments
from papers/journals
...

90. open data
initiatives

91. agenda oberta
civio
15mpedia
wheredoesmymoneygo?
...

92. data sources
social networks
open data repositories
scraped web data
...

93. other data sources
barcelona
catalonia
spain
eu
...

94. contact
jminguillona[at]uoc[dot]edu
@jminguillona
webpage
This work is licensed under a
Creative Commons Attribution-ShareAlike 4.0
International License
.