Data Curation at the New York Times

Digital Enterprise Research Institute www.deri.ie

Data Curation at the
New York Times
Edward Curry, Andre Freitas, Seán O'Riain

ed.curry@deri.org
http://www.deri.org/
http://www.EdwardCurry.org/
Copyright 2010 Digital Enterprise Research Institute. All rights reserved.

Speaker Profile

 Research Scientist at the Digital Enterprise Research
Institute (DERI)
 Leading international web science research organization
 Researching how web of data is changing way business
work and interact with information
 Projects include studies of enterprise linked data, community-
based data curation, semantic data analytics, and semantic
search
 Investigate utilization within the pharmaceutical, oil & gas,
financial, advertising, media, manufacturing, health care, ICT,
and automotive industries
 Invited speaker at the 2010 MIT Sloan CIO Symposium
to an audience of more than 600 CIOs

Overview

 Curation Background
 The Business Need for Curated Data
 What is Data Curation?
 Data Quality and Curation
 How to Curate Data

 New York Times Case Study

 Best Practices from Case Study Learning

The Business Need

 Knowledge workers need:
 Access to the right information
 Confidence in that information

 Working incomplete
inaccurate, or wrong
information can have
disastrous consequences

The Problems with Data

 Flawed Data
 Effects 25% of critical data in world‟s top companies
(Gartner)

 Data Quality
 Recent banking crisis (Economist Dec‟09)
 Inaccurate figures made it difficult to manage operations
(investments exposure and risk)
– “asset are defined differently in different programs”
– “numbers did not always add up”
– “departments do not trust each other‟s figures”
– “figures … not worth the pixels they were made of”

What is Data Curation?

 Digital Curation
 Selection, preservation, maintenance, collection, and
archiving of digital assets

 Data Curation
 Active management of data over its life-cycle

 Data Curators
 Ensure data is trustworthy, discoverable, accessible,
reusable, and fit for use
– Museum cataloguers of the Internet age

What is Data Curation?

 Data Governance
 Convergence of data quality, data management,
business process management, and risk
management

 Data Curation is a complimentary activity
 Part of overall data governance strategy for
organization

 Data Curator = Data Steward ??
 Overlapping terms between communities

Data Quality and Curation

 What is Data Quality?
 Desirable characteristics for information resource
 Described as a series of quality dimensions
– Discoverability, Accessibility, Timeliness, Completeness,
Interpretation, Accuracy, Consistency, Provenance &
Reputation

 Data curation can be used to improve these
quality dimensions


 Discoverability & Accessibility
 Curate to streamline search by storing and classifying
in appropriate and consistent manner

 Accuracy
 Curate to ensure data correctly represents the “real-
world” values it models

 Consistency
 Curate to ensure data created and maintained using
standardized definitions, calculations, terms, and
identifiers


 Provenance & Reputation
 Curate to track source of data and determine reputation
 Curate to include the objectivity of the source/producer
– Is the information unbiased, unprejudiced, and impartial?
– Or does it come from a reputable but partisan source?

Other dimensions discussed in chapter

How to Curate Data

 Data Curation is a large field with sophisticated
techniques and processes

 Section provides high-level overview on:
 Should you curate data?
 Types of Curation
 Setting up a curation process

Additional detail and references available in book
chapter

Should You Curate Data?

 Curation can have multiple motivations
 Improving accessibility, quality, consistency,…

 Will the data benefit from curation?
 Identify business case
 Determine if potential return support investment

 Not all enterprise data should be curated
 Suits knowledge-centric data rather than transactional
operations data

Types of Data Curation

 Multiple approaches to curate data, no single
correct way
 Who?
– Individual Curators
– Curation Departments
– Community-based Curation
 How?
– Manual Curation
– (Semi-)Automated
– Sheer Curation

Types of Data Curation – Who?

 Individual Data Curators
 Suitable for infrequently changing small quantity of
data
– (<1,000 records)
– Minimal curation effort (minutes per record)

Types of Data Curation – Who?

 Curation Departments
 Curation experts working with subject matter experts
to curate data within formal process
– Can deal with large curation effort (000‟s of records)

 Limitations
 Scalability: Can struggle with large quantities of
dynamic data (>million records)
 Availability: Post-hoc nature creates delay in curated
data availability

Types of Data Curation - Who?

 Community-Based Data Curation
 Decentralized approach to data curation
 Crowd-sourcing the curation process
– Leverages community of users to curate data
 Wisdom of the community (crowd)
 Can scale to millions of records

Types of Data Curation – How?

 Manual Curation
 Curators directly manipulate data
 Can tie users up with low-value add activities

 (Sem-)Automated Curation
 Algorithms can (semi-)automate curation activities
such as data cleansing, record duplication and
classification
 Can be supervised or approved by human curators

Types of Data Curation – How?

 Sheer curation, or Curation at Source
 Curation activities integrated in normal workflow of
those creating and managing data
 Can be as simple as vetting or “rating” the results of a
curation algorithm
 Results can be available immediately

 Blended Approaches: Best of Both
 Sheer curation + post hoc curation department
 Allows immediate access to curated data
 Ensures quality control with expert curation

Setting up a Curation Process

 5 Steps to setup a curation process:
1 - Identify what data you need to curate
2 - Identify who will curate the data
3 - Define the curation workflow
4 - Identity appropriate data-in & data-out formats
5 - Identify the artifacts, tools, and processes needed to
support the curation process

The New York Times

100 Years of Expert Data Curation

The New York Times

 Largest metropolitan and third largest
newspaper in the United States

 nytimes.com
 Most popular newspaper
website in US

 100 year old curated
repository defining its
participation in the
emerging Web of Data

The New York Times

 Data curation dates back to 1913
 Publisher/owner Adolph S. Ochs decided to provide a
set of additions to the newspaper
 New York Times Index
 Organized catalog of articles titles and summaries
– Containing issue, date and column of article
– Categorized by subject and names
– Introduced on quarterly then annual basis
 Transitory content of newspaper became
important source of searchable historical data
 Often used to settle historical debates

The New York Times

 Index Department was created in 1913
 Curation and cataloguing of NYT resources
– Since 1851 NYT had low quality index for internal use

 Developed a comprehensive catalog using a
controlled vocabulary
 Covering subjects, personal names, organizations,
geographic locations and titles of creative works
(books, movies, etc), linked to articles and their
summaries

 Current Index Dept. has ~15 people

The New York Times

 Challenges with consistently and accurately
classifying news articles over time
 Keywords expressing subjects may show some
variance due to cultural or legal constraints
 Identities of some entities, such as organizations and
places, changed over time

 Controlled vocabulary grew to hundreds of
thousands of categories
 Adding complexity to classification process

The New York Times

 Increased importance of Web drove need to
improve categorization of online content

 Curation carried out by Index Department
 Library-time (days to weeks)
 Print edition can handle next-day index

 Not suitable for real-time online publishing
 nytimes.com needed a same-day index

The New York Times

 Introduced two stage curation process
 Editorial staff performed best-effort semi-automated
sheer curation at point of online pub.
– Several hundreds journalists
 Index Department follow up with long-term accurate
classification and archiving

 Benefits:
 Non-expert journalist curators provide instant
accessibility to online users
 Index Department provides long-term high-quality
curation in a “trust but verify” approach

NYT Curation Workflow

 Curation starts with article getting out of the newsroom


 Member of editorial staff submits article to web-based rule
based information extraction system (SAS Teragram)


 Teragram uses linguistic extraction rules based on subset of
Index Dept‟s controlled vocab.


 Teragram suggests tags based on the Index vocabulary that
can potentially describe the content of article


 Editorial staff member selects terms that best describe the
contents and inserts new tags if necessary


 Reviewed by the taxonomy managers with feedback to
editorial staff on classification process


 Article is published online at nytimes.com


 At later stage article receives second level curation by Index
Dept. additional Index tags and a summary


 Article is submitted to NYT Index

The New York Times

 Early adopter of Linked Open Data (June „09)

The New York Times

 Linked Open Data @ data.nytimes.com
 Subset of 10,000 tags from index vocabulary
 Dataset of people, organizations & locations
– Complemented by search services to consume data
about articles, movies, best sellers, Congress votes,
real estate,…
 Benefits
 Improves traffic by third party data usage
 Lowers development cost of new applications for
different verticals inside the website
– E.g. movies, travel, sports, books

Overview

 Curation Background
 The Business Need for Curated Data
 What is Data Curation?
 Data Quality and Curation
 How to Curate Data

 Case Study New York Times

 Best Practices from Case Study Learning

Best Practices from Case Study
Learning

 Social Best Practices
 Participation
 Engagement
 Incentives
 Community Governance Models

 Technical Best Practices
 Data Representation
 Human- and AutomatedCuration
 Track Provenance

Social Best Practices

 Participation
 Stakeholders involvement for data producers and
consumers must occur early in project
– Provides insight into basic questions of what they want
to do, for whom, and what it will provide
 White papers are effective means to present these
ideas, and solicit opinion from community
– Can be used to establish informal „social contract‟ for
community


 Engagement
 Outreach activities essential for promotion and
feedback
 Typical consumers-to-contributors ratios of less than
5%
 Social communication and networking forums are
useful
– Majority of community may not communicate using
these media
– Communication by email still remains important


 Incentives
 Sheer curation needs line of sight from data curating
activity, to tangible exploitation benefits
 Lack of awareness of value proposition will slow
emergence of collaborative contributions
 Recognizing contributing curators through a formal
feedback mechanism
– Reinforces contribution culture
– Directly increases output quality


 Community Governance Models
 Effective governance structure is vital to ensure
success of community
 Internal communities and consortium perform well
when they leverage traditional corporate and
democratic governance models
 Open communities need to engage the community
within the governance process
– Follow less orthodox approaches using meritocratic
and autocratic principles

Technical Best Practices

 Data Representation
 Must be robust and standardized to encourage
community usage and tools development
 Support for legacy data formats and ability to
translate data forward to support new technology and
standards
 Human & Automated Curation
 Balancing will improve data quality
 Automated curation should always defer to, and never
override, human curation edits
– Automate validating data deposition and entry
– Target community at focused curation tasks

Technical Best Practices

 Track Provenance
 All curation activities should be recorded and
maintained as part data provenance effort
– Especially where human curators are involved
 Users can have different perspectives of provenance
– A scientist may need to evaluate the fine grained
experiment description behind the data
– For a business analyst the ‟brand‟ of data provider can
be sufficient for determining quality

Conclusions

 Data curation can ensure the quality of data and
its fitness for use
 Pre-competitive data can be shared without
conferring a commercial advantage
 Pre-competitive data communities
 Common curation tasks carried out once in public
domain
 Reduces cost, increase quantity and quality

Acknowledgements

 Collaborators Andre Freitas & Seán O'Riain

 Insight from Thought Leaders
 Evan Sandhaus (Semantic Technologist), Rob Larson (Vice President Product
Development and Management), and Gregg Fenton (Director Emerging Platforms)
from the New York Times
 Krista Thomas (Vice President, Marketing & Communications), Tom Tague
(OpenCalais initiative Lead) from Thomson Reuters
 Antony Williams (VP of Strategic Development ) from ChemSpider
 Helen Berman (Director), John Westbrook (Product Development) from the Protein
Data Bank
 Nick Lynch (Architect with AstraZeneca) from the Pistoia Alliance.

 The work presented has been funded by Science
Foundation Ireland under Grant No. SFI/08/CE/I1380 (Lion-
2).

Further Information

The Role of Community-Driven
Data Curation for Enterprises
Edward Curry, Andre Freitas, & Seán O'Riain

In David Wood (ed.),
Linking Enterprise Data Springer, 2010.
Available Free at:
http://3roundstones.com/led_book/led-curry-et-al.html

Data Curation at the New York Times

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Data Curation at the New York Times

Similaire à Data Curation at the New York Times (20)

Dernier

Dernier (20)

Data Curation at the New York Times