Some insights about data curation processes @ SpazioDati. How we're using Big Data tools, Linked Data technologies, to build our products: Dandelion API (dandelion.eu) and Atoka (atoka.io).

1. Matteo Brunati
@dagoneye
22/07/2015
Data Curation
@SpazioDati
33° Nexa Lunch Seminar
http://nexa.polito.it/lunch-33

2. Big Data - Linked Data - Machine Learning
spaziodati.eu

3. a lot of European projects
http://www.spaziodati.eu/en/#research

4. Data Curation?
https://www.google.com/search?q=data+curation&ie=utf-8&oe=utf-8

5. !
!
Data curation is the process of turning independently
created data sources (structured and semi-structured data)
into unified data sets ready for analytics,
using domain experts to guide the process.
http://strataconf.com/stratany2014/public/schedule/detail/36021

6. a lot of things involved
!
ETL (Extract-Transform-Load) tools
Data Science tools
Linked Data tools
Big Data tools
Domain Knowledge

7. why we need a data curation process?
@

8. it’s our mantra: ALLYOU NEED IS DATA

9. :)
accessible
for everyone
lat 00° 00’ 00”
-> GPS -> Smartphones -> UI IPhone / Android
it’s our mantra: ALLYOU NEED IS DATA

10. we are building two products

11. Dandelion API
Text Analytics as a service
dandelion.eu

12. Sales Intelligence
atoka.io

13. <Powered by SpazioDati> codename
2014
2015
data platform

14. Why a knowledge graph?

15. Our Entity Extraction API is based on a graph
Brussels
Paris
Berlin
Eiffel Tower
2009 World Championships
in Athletics
King Baudouin Stadium
Champ de Mars
0.42
0.80
0.43
0.53
0.53
0.53
0.63
0.59
0.440.44
https://dandelion.eu/docs/api/datatxt/nex/v1/

16. CONTEXTUAL DATA

17. different sources;
different semantics;
companies, people,Wikipedia topics, POI…
simple to query on traversals
global statistics
why a knowledge graph

18. let’s start with some details on the
“Powered by SpazioDati” data platform…

19. http://blog.spaziodati.eu/en/2014/10/21/spaziodati-at-iswc-2014-visit-our-booth-research-plans-available/
“Powered by SpazioDati”
data platform backstage
PWR-BY-SD

20. OpenRefine
https://azkaban.github.io/
Azkaban
Open Source Workflow Manager
Apache Silk
Titan graph db
Apache Cassandra
The Linked Data Integration Framework
Tools involved

21. http://blog.spaziodati.eu/en/2014/07/24/using-openrefine-to-perform-text-mining-on-your-data-food-for-thoughts/
starting from OpenRefine to clean up
the data easily, for example
* reconcile and clean up the data
* align the data model to our internal ontologies, using
RDF skeletons
* export the RDF modelled using our rules

22. in other words…
Rexster: JSON-based REST interface to Titan

23. Our internal ontology: a sample

24. and now
atoka.io

25. ~5,9 ★ MLN companies
>10 ★ MLN persons
900k
updated weekly
★ Weekly web crawl of the Italian corporate

26. ★ Real-time data collection from company social accounts
★ ~1600 online & offline newspapers (updated daily)
updated weekly

27. atoka.io

28. Search: how it works
Direct search of one particular
company through its name or “partita
iva” (vat number)
Content search into company websites
Keyword search among extracted and
refined entities from company resources
!
Dandelion API is the extraction engine!
1.
2. [*]
3. [*]

29. Corporate page
atoka.io

30. Some details on
• Five main “types”:!
– Company!
– Person!
– Site!
– Administrative Division!
– Website

31. our infrastructure to crawl the Web for ATOKA

32. other details
Cerved
• Company
• People
• Site
• Position+Share
ISTAT
• AdminDiv
ES
DBPedia
• Company
cluster computing

33. something really interesting on OpenRefine

34. OpenRefine as usual

35. OpenRefine
on Spark

36. it rocks! :)
more background details on
http://blog.spaziodati.eu/wp-content/uploads/2015/07/RefineOnSpark.pdf

37. Thanks :)
@spaziodati
brunati@spaziodati.eu

38. References
1) From raw data to dataGEMs for developers - http://ceur-ws.org/Vol-1268/paper1.pdf
2) Knowledge Graph ovunque:
http://www.slideshare.net/dagoneye/knowledge-graphs-ovunque-un-quadro-di-insieme-e-le-implicazioni-per-uno-sviluppo-condiviso-del-web-of-data
3) Linking Enterprise Data - https://www.springer.com/it/book/9781441976642
4) Using OpenRefine - https://www.packtpub.com/big-data-and-business-intelligence/using-openrefine
5) WhyYour Business Needs A Customer Data Knowledge Graph -
http://www.dataversity.net/business-needs-customer-data-knowledge-graph/
6) Enabling parallel processing for OpenRefine: Spark vs Akka -
http://refinepro.com/blog/enabling-parallel-processing-for-openrefine-spark-vs-akka/