The document presents a GATE evaluation framework for benchmarking named entity recognition (NER) systems that use Wikipedia as a knowledge base. It describes datasets annotated with entities from Wikipedia, an outline of the GATE plugin-based framework, and how NER systems can be integrated and evaluated for entity spotting, disambiguation and classification performance. Preliminary results using the framework to evaluate the Entityclassifier.eu NER system on tweets and news articles are also provided.

1. Datasets and GATE Evaluation Framework for
Benchmarking Wikipedia Based NER Systems
Milan Dojchinovski1,2, Tomáš Kliegr1
2 Faculty of Information Technology
Czech Technical University in Prague
1 Faculty of Informatics and Statistics
University of Economics, Prague
“NLP & DBpedia” ISWC 2013 workshop
October 22nd, 2013, Sydney, Australia
Milan Dojchinovski
milan.dojchinovski@vse.cz - @m1ci - http://dojchinovski.mk
Except where otherwise noted, the content of this presentation is licensed under
Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported
Czech Technical University
in Prague
University of Economics
Prague

2. Outline
‣ Introduction
‣ Prerequisites and challenges
‣ GATE framework for benchmarking NER
‣ Conclusion and future directions
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems 2

3. What is a Named Entity Recognition task?
3
‣ Main sub-tasks
- spotting of entities: tagging text fragment as an entity
- disambiguation of entities: unique identification of entities using URIs
- classification of entities: assignment of type to an entity
The Charles Bridge is a famous historic bridge that crosses the Vltava river in Prague,
Czech Republic. Its construction started in 1357 under the auspices of King Charles IV,
and finished in the beginning of the 15th century. The bridge replaced the old Judith
Bridge built 1158–1172 that had been badly damaged by a flood in 1342.
Entity Entity URI Type
Charles Bridge http://dbpedia.org/resource/Charles_Bridge http://dbpedia.org/ontology/Bridge
Vltava http://dbpedia.org/resource/Vltava http://dbpedia.org/ontology/River
Prague http://dbpedia.org/resource/Prague http://dbpedia.org/ontology/City
Czech Republic http://dbpedia.org/resource/Czech_Republic http://dbpedia.org/ontology/Country
King Charles IV http://dbpedia.org/resource/Charles_IV,_Holy_Roman_Emperor http://dbpedia.org/ontology/Person
Judith Bridge http://dbpedia.org/resource/Judith_Bridge http://dbpedia.org/ontology/Bridge
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

4. Existing NER tools
4
‣ DBpedia Spotlight, NERD, THD (EntityClassifier.eu), AlchemyAPI, Open
Calais, Evri, Lupedia, Wikimeta, Yahoo!, Zemata, and others.
‣ Differences
- types come from different taxonomies
- types with different granularity (Person, SoccerPlayer, Manager)
- types are plain text literals
‣ Similarities
- disambiguation with DBpedia or Wikipedia resources
- DBpedia Spotlight, Entityclassifier.eu, AlchemyAPI, Wikimeta
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

5. Outline
5
‣ Introduction
‣ Challenges and prerequisites
‣ GATE framework for benchmarking NER
‣ Conclusion and future directions
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

6. Challenges and Prerequisites
6
‣ Entity spotting
- spotted entity text fragments might not be exactly overlapping, but still correct
- entity start and end offset might be different
The Charles Bridge is a famous historic bridge ... - ground-truth annotations
The Charles Bridge is a famous historic bridge ... - annotations from a NER system
‣ Entity disambiguation
- using unique DBpedia/Wikipedia resource URIs or URIs from YAGO or Freebase.
The Charles Bridge is a famous historic bridge ...
“Charles Bridge” - http://dbpedia.org/resource/Charles_Bridge
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

7. Challenges and Prerequisites
7
‣ Entity classification
- different NER might return different types for same entity
- although different (in granularity), but still correct
- Person and SoccerManager are not same types, but they are correct
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

8. Outline
8
‣ Introduction
‣ Prerequisites and challenges
‣ GATE framework for benchmarking NER
‣ Conclusion and future directions
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

9. Architecture overview
9
‣ Unified evaluation framework
- any NER tool can be easily integrated and evaluated
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

10. Realization
10
‣ GATE text engineering framework
- open-source, strong community support
- easy to extend (plugins and new processing resources)
- several existing NER clients for GATE: THD, OpenCalais, ANNIE, etc.
- evaluation tools that can be reused
- Corpus Quality Assurance
- Annotation diff
‣ Developed tools
- plugins for import of News and Tweets datasets
- plugin for type alignment
- reference implementation of a NER client as a GATE plugin
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

11. Evaluation Workflow
11
• Steps to evaluate NER system
1. Import ground-truth dataset
- use provided plugins
- News and Tweets datasets
2. Run NER on the ground-truth corpus
- use a GATE client plugin for the NER system
- if not existent, should be implemented!
3. Align entity classes with the ground-truth classes
- use the provided OntologyAwareDiffPR plugin
4. Evaluate the performance of the NER tool
- use the Corpus Quality Assurance tool
- evaluate NE spotting, disambiguation, classification
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

12. News and Tweets datasets
12
• Tweets dataset - CC BY-NC-SA 3.0
- dataset from the Making Sense of Microposts (MSM) 2013 workshop challenge
- 1044 tweets, 1523 entities
• News dataset - CC BY-SA 3.0
- derivation from the datasets presented at the WEKEX 2011 workshop
- standard-length news articles, 10 articles, 588 entities
Fig. 1. Example of an entity annotation in GATE
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

13. Type Alignment
13
• Implemented as a GATE plugin
- OntologyAwareDiffPR
• Pre-requirements
- DBpedia Ontology
- typeURI feature in the ground-truth and NER annotation
typeURI: http://dbpedia.org/ontology/Person
aligned: http://dbpedia.org/Ontology/SoccerManager
before
alignment
typeURI: http://dbpedia.org/ontology/SoccerManager
after
alignment
NER annotation ground-truth annotation
typeURI: http://dbpedia.org/ontology/Person
typeURI: http://dbpedia.org/ontology/SoccerManager
aligned: http://dbpedia.org/Ontology/SoccerManager
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

14. Outline
14
‣ Introduction
‣ Prerequisites and challenges
‣ GATE framework for benchmarking NER
‣ Conclusion and future directions
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

15. Conclusion and Future Directions
15
• GATE Evaluation Framework
- two ground-truth datasets
- two plugins for import of News and Tweets datasets
- a plugin to perform basic type alignment
- reference implementation of Entityclassifier.eu NER as a GATE client
- plugins published under GPLv3.0
• Future Work
- integration of additional NER/NIF systems
- development of an advance type alignment
- improvement of existing datasets and creation of new
- additional ground-truth datasets
- additional NER evaluation statistics
- using NERD ontology to integrate tag sets of common wikifiers
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

16. 16
Thank you!
Questions, comments, ideas?
Milan Dojchinovski @m1ci
milan.dojchinovski@fit.cvut.cz http://dojchinovski.mk
Feedback
‣ Framework resources:
- general info: http://entityclassifier.eu/datasets/evaluation/
- datasets: http://entityclassifier.eu/datasets/evaluation/benchmark-datasets/
- tools: http://entityclassifier.eu/datasets/evaluation/tools/
Except where otherwise noted, the content of this presentation is licensed under
Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported

17. Datasets annotation process
17
‣ Inter-annotator agreement
- two annotators
- additional one/two annotators for spurious cases
Dataset Wikipedia URL Coarse grained
type
Fine grained
type
Most frequent
sense
News 0,61 0,65 0,7 0,77
Tweets 0,79 n/a 0,64 0,86
• Fields
- URL to English Wikipedia
- Fine-grained type
- Coarse-grained type
- MFS flag
- Common entity flag
- Full name
- Partial flag
- Incorrect capitalization flag (Tweets only)
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

18. News and Tweets datasets
18
• Tweets dataset
- dataset from the Making Sense of Microposts (MSM) 2013 Workshop challenge
- 1044 tweets, 1523 entities
- Creative Commons BY-NC-SA 3.0
• News dataset
- derivation from the datasets presented at the WEKEX 2011 workshop
- standard-length news articles, 10 articles, 588 entities)
- Creative Commons BY-SA 3.0
Documents
Entities
All With CoNNL type Ontology type Wikipedia URL
News 10 588 580 367 440
Tweets 1044 1523 1523 1379 1354
Fig. 1. Size metrics for the Tweets and News dataset
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems

19. Entityclassifier.eu preliminary results
19
• http://entityclassifier.eu
Task Precision
(strict/lenient)
Recall
(strict/lenient)
F1.0 score
(strict/lenient)
Entity spotting 0.45/0.56 0.67/0.84 0.54/0.67
Entity disambiguation 0.24/0.26 0.36/0.39 0.29/0.31
Entity classification 0.12/0.13 0.17/0.19 0.14/0.15
Fig. 1. Results for the Entityclassifier.eu NER on the Tweets dataset
Task Precision
(strict/lenient)
Recall
(strict/lenient)
F1.0 score
(strict/lenient)
Entity spotting 0.69/0.78 0.33/0.38 0.45/0.51
Entity disambiguation 0.37/0.41 0.18/0.20 0.24/0.27
Entity classification 0.69/0.78 0.33/0.38 0.45/0.51
Fig. 2. Results for the Entityclassifier.eu NER on the News dataset
GATE Evaluation Framework for Benchmarking Wikipedia-Based NER Systems