Digital Humanities Benelux 2017: Keynote Lora Aroyo

http://lora-aroyo.org @laroyo
Harnessing Human Semantics at Scale
Measurable, Reproducible, Engaging, Sustainable
Crowdsourcing & Nichesourcing
Lora Aroyo

20071998 2006 2009
from DVDs to data science

20071998 2006 2009
Team BellKor wins Netflix Prize

20061994 2003 2016 2017
from books to data science

data is at the centre of every process

data is essential to evolve with users

Ceci n'est pas … la mona lisa

Ceci n'est pas … la mona lisa
Louvre’s Mona Lisa
is only #14

the battle of two worlds
9,3 million
Louvre
visitors 2014
14 million
website visitors
2,3 million
social media

in the (very near) future
most visitors will be digital-born
not bound by time or location
native to new forms of co-makership
native to new media
Siebe Weide, Max Meijer and Marieke Krabshuis (2012).
Agenda 2026: Study on the Future of the Dutch Museum Sector

variety of meanings
multitude of perspectives
abundance of sources
endless contexts
know your data

crowdsourcing to know your data at scale

variety of types
multitude of platforms
abundance of interactions
endless characteristics
know your crowds

https://www.rijksmuseum.nl/en/rijksstudio
Engage with Co-creation

Engage with Co-creativity

Engage with Co-curation

Engage the Expert Niche
http://annotate.accurator.nl

expertise of Rijksmuseum professionals is
in annotating their collection
with art-historical information, e.g. when they
were created, by whom, etc.

detailed domain-specific information
about depicted objects, e.g. which species the
animal or plant belongs to,
is in most cases not available

use nichesourcing, i.e. niches of people with
the right expertise, to add more specific
information

Keep Reproducing
http://annotate.accurator.nl

Engage with Games
training the general crowd to be a niche:
game in which players can carry out an expert
annotation tasks with some assistance

http://waisda.nl
Engage with Games

http://spotvogel.vroegevogels.vara.nl
Keep Reproducing

CrowdTruth.org
Experiment with Paid Crowds

http://crowdtruth.org/

http://data.crowdtruth.org/

Challenges

Low reproducibility rates
Difficult to estimate & control the time to complete
Difficult to assess & compare quality
Demands continuous promotional effort
Active learning (human-in-the-loop) needs different expertise
Difficult to incorporate results into existing content infrastructure
Challenges
Crowdsourcing typically undertaken in isolation

Assess Impact of Task Design

Instructions
Layout
Sequence
Crowds
Payment
Campaign
Assess Impact of Task Design
experiment with different designs

for example
mapping music to mood

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Other
passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into
rousing, cheerful, fun, poignant, wistful, campy, quirky, tense, anxious, any of the 5
confident, sweet, amiable, bittersweet, whimsical, witty, intense, volatile, clusters
boisterous, good-natured autumnal, wry visceral
rowdy brooding
Choose one:
Which is the mood most appropriate
for each song?
Goal:
(Lee and Hu 2012)
1 song - 1 mood???

If “One Truth” & “No Disagreement”
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5
W1 1
W2 1
W3 1
W4 1
W5 1
W6 1
W7
W8
W9 1
W10 1
Totals 1 3 1 2 1

Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W1 1 1 1
W2 1 1 1
W3 1 1 1
W4 1 1
W5 1 1
W6 1 1 1
W7 1 1 1
W8 1 1 1
W9 1 1
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
If “Many Truths” & “Disagreement”

Web & Media Group
simplification of context
this all results in

Web & Media Group

● Identify Crowdsourcing Goals through user log analysis
○ # queries, #unique queries, #queries of specific type
○ ranked by popularity
○ ranked by popularity and with error, e.g.
■ # queries entered over 50 times with 0 results
■ # queries of specific type with 0 results
○ which will have biggest impact
○ which has biggest urgency
● … or through other user analysis
Assess Impact of Results

for example
in video search

people search for fragments
experts annotate full videos
35% of search queries result in not found
people search for fragments
experts annotate full videos
35% of search queries result in not found
for example
in video search

Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011

Measure Quality
time-based annotation
bernhard
88% of the tags useful
for specific genres
describe short segments
often not very specific
don’t describe program as a whole

for example
in video search
video annotation is time-consuming
5 times the video duration
experts use a specific vocabulary
that is unknown to general audiences
video annotation is time-consuming
5 times the video duration
experts use a specific vocabulary
that is unknown to general audiences

user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
engeland
persons (31%)
objects (57%)
Measure Quality

Web & Media Group
human subjectivity, ambiguity & uncertainty of expression
natural part of human semantics

measure quality
quality is not just about spam
quality is typically multi-dimensional
understand the diversity in crowd answers
do not ignore multitude of interpretations
understand the variety of contexts
identify cases with high ambiguity, similarity, …
experiment with explicit metrics
experiment with different designs

Measure Progress
6 months 2 years
340,551 tags 36,981 tags
137.421 matches
602 items 1.782 items
555 registered players 2,017 users (taggers)
thousands of anonymous players
12,279 visits (3+ min online)
44,362 pageviews
Riste Gligorov, Michiel Hildebrand, Jacco van Ossenbruggen, Guus Schreiber, Lora Aroyo
(2011). On the role of user-generated metadata in audio visual collections. International
conference on Knowledge capture K-CAP '11, Pages 145-152

campaign, campaign, campaign

Measurable quality
Reproducible results
Sustainable settings
Engaging interaction
Goals

Digital Humanities Benelux 2017: Keynote Lora Aroyo

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (15)

Similaire à Digital Humanities Benelux 2017: Keynote Lora Aroyo

Similaire à Digital Humanities Benelux 2017: Keynote Lora Aroyo (20)

Plus de Lora Aroyo

Plus de Lora Aroyo (18)

Dernier

Dernier (20)

Digital Humanities Benelux 2017: Keynote Lora Aroyo