Big Data Curation And Its Application

Copyright © 2013 Hanmin Jung
Hanmin Jung
Head of Dept. of Computer Intelligence Research
KISTI
Big Data Curation & Its Application

Let Me Introduce Myself :-)
2

Recent Social Activities (2012~)
보건복지부 빅데이터 기술 전문가 패널
한국연구재단 융합연구우수성평가위원회 위원
국립전파연구원 방송통신표준 전문위원
기술표준원 빅데이터 표준화 프레임워크 표준연구회 위원
ITRC 과제기획위원회 빅데이터 분과, UI/UX 분과 기획위원/기술간사
교육과학기술부 과학기술 빅데이터 포럼 전문가위원회 간사
국가경쟁력강화위원회 ‘범부처 Giga KOREA 사업’ 전문가 위원회 위원
국가과학기술위원회 기술영향평가위원회 위원
방송통신위원회 빅데이터 포럼 인력양성분과 위원장/자문위원/운영위원
IBS 장비심의위원회, 지식경제부 중앙장비심의위원회 심의위원
한국과학창의재단 과학창의앰배서더
지식경제부 ‘SW+인문 포럼’ 멘토
IT VC 전문성 강화 프로그램 전문강사
ISO/IEC JTC1/SC32 전문위원회 위원, JTC 1/SC34 표준개발위원회 위원
ISO 20022 TC68 Advisory Expert
한국정보통신기술협회 정보통신표준화위원회 –
디지털콘텐츠/SW품질평가/메타데이타/사물지능통신 프로젝트 그룹 위원
교육과학기술부 차세대정보컴퓨팅분과 – 기획위원
국무총리실 정보지식인대회 출제/평가위원
문화체육관광부 공공문화정보 전문가 위원
Let Me Introduce Myself :-)
3

Weekly IT Trends (NIPA)
과학기술빅데이터동향및활용방안(Vol. 1593)
모바일비즈니스인텔리전스기술동향(Vol. 1573)
시맨틱소셜네트워크를구성하는온톨로지어휘기술현황(Vol. 1560)
정부빅데이터분석기술적용동향(Vol. 1556)
제품및기술수명주기활용동향(Vol. 1550)
개체식별관점에서바라본링크드데이터동향(Vol. 1524)
유망기술발굴모델및서비스동향(Vol. 1521)
포지셔닝의측면에서살펴본모바일태블릿컴퓨터동향(Vol. 1486)
테크놀로지인텔리전스동향(Vol. 1477)
추론기술연구동향(Vol. 1446)
트리플레파지토리벤치마킹(Vol. 1439)
차세대IT 기기와HCI 기술동향전망(Vol. 1435)
시맨틱검색기술동향(Vol. 1431)
시맨틱웹국내특허동향(Vol. 1420)
감성분석과브랜드모니터링기술동향(Vol. 1396)
시맨틱웹이경제⋅사회에미치는영향(Vol. 1372)
웹매핑서비스비교분석(Vol. 1352)
시맨틱웹2.0 기술동향(Vol. 1344)
국내포털검색시장및특허동향(Vol. 1341)
Open API 기술동향(Vol. 1296)
엔터프라이즈검색기술동향(Vol. 1276)
전자상거래검색기술동향(Vol. 1273)
시맨틱웹포털기술동향(Vol. 1264)
Trend Reports
4

Art Curator
5
http://www.chrysler.org/files/resources/gallery-talk-3.jpg

Copyright © 2013 Hanmin Jung6
Press

Definition
Active management of data over its lifecycle
Ensuring that data is trustworthy, discoverable, accessible, reusable, and
fit for use
E.g. data preparation for analytics
Curation Types
Manual curation
(Semi-)automatic curation
E.g. data cleansing, record duplication, classification
Sheer curation (curation at source)
Integrated in workflow for creating and managing data
Data Curation
7
E. Curry, A. Freitas, and S. O’Riain, “Data Curation at the New York Times”, 2011.

Steps to Setup the Process
Identify what data you need to curate
Identity who will curate the data
Define the curation workflow
Identify appropriate data-in & data-out formats
Identity the artifacts, tools, and processes needed to support the process
Data Curation
8
E. Curry, A. Freitas, and S. O’Riain, “Data Curation at the New York Times”, 2011.

Case Studies
9
1. Apple Maps
2. Strategic Foresight
3. Big Data

http://cdn0.sbnation.com/entry_photo_images/5667445/20120920-DSC_7192VERGE_large_verge_medium_landscape.jpg
Apple Maps
10

Google Maps vs. Apple Maps
11
http://www.gadgetreview.com/2013/01/google-maps-vs-apple-maps-ios-comparison.html

Apple Siri
http://www.apple.com/ios/siri/
12

We can see it as much about it
as we know
http://investor.google.com/financial/tables.html
Google’s Financial Table

Nokia’s Burning Ships Strategy
http://www.brightsideofnews.com/Data/2011_5_5/Steven-Elop-Burn-Boats-Strategy-is-a-Cortez-Move-for-Nokia/Hernando_Cortez_BurningBoat.jpg

Result of the Strategy
Worldwide Market Share
Worldwide mobile device sales to end users in 2008 ~ 2013
Gartner, IDC Worldwide Mobile Phone Tracker
38.6, 117.937.8, 108.531.6,110.427.1, 106.618.7, 82.914.8, 61.9Nokia
3.5, 12.14.9, 19.13.1, 13.73.2, 13.5ZTE
418.6
41.9, 175.4
3.7, 15.4
8.9, 37.4
27.5, 115.0
1Q2013
(%, M. Units)Company
3Q2012
(%, M. Units)
3Q2011
(%, M. Units)
3Q2010
(%, M. Units)
3Q2009
(%, M. Units)
3Q2008
(%, M. Units)
Samsung 23.7, 105.4 22.3, 87.8 20.5, 71.4 21.0, 60.2 17.0, 52.0
Apple 6.1, 26.9 4.3, 17.1 4.0, 14.1
LG 3.1, 14.0 5.4, 21.1 8.1, 28.4 11.0, 31.6 7.5, 23.0
Sony Ericsson 4.9, 14.1 8.4, 25.7
Motorola 4.7, 13.6 8.3, 25.4
Others 45.3, 201.6 36.1, 142 32.2, 112.5 20.6, 59.1 20.1, 61.5
Total 444.5 393.7 348.9 287.1 305.4
15

Sign of the Result
http://www.google.com/insights/search/
Google Insights for Search

Strategic Foresight
R. Rohrbeck, H. Arnold, and J. Heuer, “Strategic Foresight in Multimedia Enterprises”, 2007.
17

Hype Cycle
Emerging Technologies Hype Cycle 2012

Social Data
http://bynoy.files.wordpress.com/2011/08/united-noy-weblife-60-seconds.jpg

Machine Data
T. Baer, “What is Big Data? The Reality for Analytics”, OVUM, 2011.
Call data recordsCall data records
Sensory dataSensory data
Web log filesWeb log files
Financial Instrument TradeFinancial Instrument Trade

Crop Yield Estimation
http://www.geovar.com/data/satellite/rapideye/rapideye_sample_image_bands543.jpg

Crop Yield Estimation
https://www.agriskmanagementforum.org/content/improved-supply-intelligence-global-agriculture-markets
South American soybean production forecastsSouth American soybean production forecasts

Soybeans Futures (CME)

Data Mining Methods
A. Cheung, “Forecast Anything! The Seven Data Mining Models”
Decision Trees
Naive Bayes
Cluster Analysis
Sequence Clustering
Association Rules
Time Series
Neural Networks

Value Pyramid
Search
Clustering
Extracting
Decision
Support
Forecasting
Scenario
Planning
Advising
Modified from D. Bousfield & P. Fooladi, “STM Information: 2009 Final Market Size and Share Report”, 2010.
InSciTe Advanced (2011)
InSciTe Adaptive (2012)
InSciTe Advisory (2013~)
1st Target of Data Curation

Data Curation
for Getting Insights & Foresights
26
1. Crawling Data
2. Extracting Information (Text Mining)
3. Resolving Identities

Crawling Data
Crawling Full Texts from Google Scholar
27

Crawling Data
Crawling Web Data by RSS & Google API
28

Crawling Data
Web Data Statistics in InSciTe Adaptive
29
Source/Year 2001 … 2008 2009 2010 2011 2012 Total
IDC 0 … 1 13 21 313 250 598
Wikipedia 0 … 151,942 181,721 408,017 1,555,867 249,0209 4,975,178
InformationWeek 0 … 470 551 536 388 81 2,316
Gizmag 0 … 1,371 2,301 2,970 2,850 2,520 16,731
Technologyreview 89 … 646 693 758 873 405 5,330
IEEE spectrum 64 … 468 426 385 306 205 3,173
Technewsworld 31 … 696 742 1,396 2,332 2,485 9,843
DiscoverMagazine 2 … 104 51 79 57 49 606
NewYork Times 20,940 … 7,979 4,696 3,971 3,669 2,064 124,100
BBC 3,155 … 2,687 3,158 3,318 5,964 4,243 37,073
Fox News 0 … 1,577 1,965 1,123 2,015 1,872 10,749
CNN 3,594 … 1,154 1,499 1,071 1,308 814 19,769
Thomson Reuters 0 … 450 371 309 338 222 3,408
USA Today 458 … 4,616 3,878 4,630 6,579 4,276 38,750
EtnTws.com 447 … 1,964 2,241 2,099 1,585 844 14,259
Total 28,780 … 176,125 204,306 430,683 1,584,444 2,510,539 5,261,883

Text Mining – Concept
30

Mining Targets
Named entities: e.g. PLOs
Pattern-based entities: e.g. a-mail addresses, phone numbers
Concepts: abstractions of entities
Facts and relationships
Concrete and abstract attributes: e.g. 10-year, expensive, comfortable
Subjectivity in the forms of opinions, sentiments, and emotions: e.g.
positive/negative, angry
S. Grimes, “Text Analytics Overview: Technology, Solutions, Market”, 2011.
Text Mining – Concept

Text Mining – SINDI Architecture

Text Mining – Example
33

34

35

36

37

Text Mining – Application
38

Resolving Identities
Christian Bizer, Tom Heath, and Tim Berners-Lee, “Linked Data – The Story So Far”, 2009.
URI Aliases
URIs that refer to the same real-world objects
E.g. http://dbpedia.org/resource/Berlin (for Berlin in DBpedia)
E.g. http://sws.geonames.org/2950159 (for Berlin in Geonames)
Information providers can set owl:sameAs links to URI aliases they know
about
Resolution of Data Conflicts in Data Fusion
Choosing a value in situations where multiple sources provide different
values for the same property of an object
39

Example
소녀시대, 소시, 少女時代, しょうじょじだい, 少時, Sho Jo Ji Dai, So Nyeo
Shi Dae, SoShi, Girl’s Generation, SNSD, So Nyuh Shi Dae, …

http://sindice.com/search?q=Hanmin+Jung
41

OntoURIResolver
Demonstration

LOD Project
http://richard.cyganiak.de/2007/10/lod/lod-datasets_2011-09-19_colored.html
Linked Data
31 billion RDF triples, 504 RDF links (2011.9)
43

InSciTe Advanced (2011)
44

https://play.google.com/store/apps/details?id=net.xenix.inscite&feature=search_result#?t=W10.

Data Fact Sheet
Articles: 22.6 millions (9.8 millions for papers, 7.6 millions for patents, 5.3
millions for Web data)
All technical areas (2001~2011)
Named entities: 1.9 millions
Authority dictionary: 1.5 millions entries
LOD data: 290 GB (are being connected)
47

Press
48

InSciTe Architecture
Analytics Models
ETD Model
Emerging Technology Discovery Model
TLCD Model
Technology Life Cycle Discovery Model
TLC Model
Technology Life Cycle Model
OntoRelFinder®
Relationship Path Finder
OntoReasoner®
Reasoning Engine
OntoURI®
Semantic Knowledge Manager
OntoPipeliner®
Semantic Service Composer
SS&AE
Semantic Search & Analytics Engine
OntoURIResolver®
Identity Resolver
SINDI-CORE/LINK
Entity & Relationship Extractor
TUC Model
Terminology Use Cycle Model
Ontology
Linked Data
OntoFrame
OntoVerifier®
Reasoning Verifier
Web Data Crawler
RSS/Google API
Web Data
Literatures
49

Ontology
DB
Ontology Schema
Ontology Instances
RDF Triples
Portability & Connectibility
For Storing & Managing
For Service
Planning Services
Defining Concepts
Exploiting Relations
50

Big Data Processing in InSciTe
51

InSciTe Homepage
http://inscite.kisti.re.kr

Thank you
jhm@kisti.re.kr
“A lot of times, people don’t know what they want until you show it to them.”
by Steve Jobs
“Many people won’t be convinced until they’ve seen it for themselves.”
by Jakob Nielsen

Big Data Curation And Its Application

Recommandé

Recommandé

Contenu connexe

Similaire à Big Data Curation And Its Application

Similaire à Big Data Curation And Its Application (20)

Plus de Korea Institute of Science and Technology Information

Plus de Korea Institute of Science and Technology Information (12)

Dernier

Dernier (20)

Big Data Curation And Its Application