Presented at the ISWC2014, RDBS Track

1. Crowd Truth
Machine-Human Computation Framework for Harnessing Disagreement in Semantic Interpretation
Goal: gather ground truth data to train, test, evaluate cognitive computing systems
Human
annotation
tasks:
use case #1: text annotation
concept: drug
relation:
treats
concept: disease
use case #2: image annotation use case #3: video annotation
Batavia
Army
Military
Indonesia
Triangle of
disagreement
crowdsourcing task
Oana Inel, Khalid Khamkham, Tatiana Cristea, Arne Rutjes, Jelle van der Ploeg, Lora Aroyo,
Robert-Jan Sips, Anca Dumitrache and Lukasz Romaszko
sentence vector
unit vectors for the same sentence
unit vector
sentence-annotation score
● sentence-annotation score:
measures how clearly the annotation is expressed in the sentence
● sentence clarity:
measures the maximum sentence-annotation score for the sentence
VS. The CrowdTruth approach
ask a large crowd
● allows for different interpretations
● minimal instructions
● large crowds of annotators
● harnessing disagreement
● continuously updated with new data
Traditional Ground Truth approach
ask few experts
● assumes one correct interpretation
● guidelines limit interpretations
● examples evaluated by single expert
● eliminating disagreement
● ground truth reused over time
Approach:
worker-sentence score
● worker-sentence score:
measures quality of worker for one sentence
● worker-worker disagreement:
● measures pairwise agreement between workers
● average worker agreement:
measures overall worker quality
Disagreement
metrics:
(use case #1:
text annotation)