Ce diaporama a bien été signalé.
Le téléchargement de votre SlideShare est en cours. ×

Annotation of anaphora and coreference for automatic processing

Prochain SlideShare
Minimalist program
Minimalist program
Chargement dans…3

Consultez-les par la suite

1 sur 53 Publicité

Plus De Contenu Connexe

Similaire à Annotation of anaphora and coreference for automatic processing (20)


Annotation of anaphora and coreference for automatic processing

  1. 1. Annotation of anaphora and coreference for automatic processing Constantin Orasan Research Group in Computational Linguistics University of Wolverhampton, UK http://www.wlv.ac.uk/~in6093/
  2. 2. Why use corpora in anaphora/coreference resolution  In this talk corpora discussed for:  Training machine learning systems  Testing anaphora/coreference resolution algorithms  Annotation:  Linguistically motivated: tries to capture certain phenomena (usually focuses on anaphora)  Application motivated: limited relations are encoded (usually focuses on coreference)
  3. 3. Structure 1. Background information 2. The MUC annotation for coreference 3. The NP4E corpus 4. Event coreference and NP coreference 5. Conclusions
  4. 4. Anaphora and anaphora resolution  cohesion which points back to some previous item (Halliday and Hasan, 1976)  the pointing back word is called an anaphor, the entity to which it refers or for which it stands is its antecedent (Mitkov, 2002)  The process of determining the antecedent of an anaphor is called anaphora resolution (Mitkov, 2002)  Anaphora resolution can be seen as a process of filling empty or almost empty expressions with information from other expressions
  5. 5. Coreference and coreference resolution  When the anaphor refers to an antecedent and when both have the same referent in real world they are termed coreferential (Mitkov, 2002)  The process of establishing which referential NPs point to the same discourse entity is called coreference resolution
  6. 6. Examples of anaphoric expressions from Mitkov (2002) Sophia Loren says she will always be grateful to Bono. The actress revealed that the U2 singer helped her calm down when she became scared by a thunderstorm while travelling on a plane. Coreferential chains:  {Sophia Loren, she, the actress, her, she},  {Bono, the U2 singer},  {a thunderstorm},  {a plane}
  7. 7. Examples of anaphoric expressions from Mitkov (2002)  Indirect anaphora: Although the store had only just opened, the food hall was busy and there were long queues at the tills.  Identity-of-sense anaphora: The man who gave his paycheck to his wife was wiser that the man who gave it to his mistress  Verb and adverb anaphora: Stephanie sang, as did Mike  Bound anaphora: Every man has his own agenda  Cataphora: The elevator opened for him on the 14th floor, and Alec stepped out quickly.
  8. 8. Anaphora vs. coreference  There are many anaphoric expressions which are not coreferential  Most of the coreferential expressions are anaphoric (Sophia Loren, the actress)  Coreferential expressions that may be or may not be anaphoric  (Sophia Loren, the actress Sophia Loren) – not anaphoric?  (the actress Sophia Loren, Sophia Loren) – anaphoric  Coreferential expressions which are not anaphoric (Sophia Loren, Sophia Loren)  Cross-document coreference is not anaphora
  9. 9. Substitution test  To determine whether two entities are coreferential substitution test is used  Sophia Loren says she will always be grateful to Bono  Sophia Loren says Sophia Loren will always be grateful to Bono.  John has his own agenda  John has John’s own agenda  Every man has his own agenda.  Every man has every man’s own agenda. ??
  10. 10. Anaphora & coreference in computational linguistics  are important preprocessing steps for a wide range of applications such as machine translation, information extraction, automatic summarisation, etc.  From linguistic perspective the expressions processed are rather limited
  11. 11. Developing annotated corpora for computational linguistics A simple, reliable annotation task Producing an CL-oriented resource Capturing the most widespread and best-understood anaphoric relation identity-of-reference direct nominal anaphora Including identity, Referring expressions (pronouns, Elements synonymy, definite NPs, or proper names) corresponding to the generalisation and have non-pronominal NP same discourse entity specialisation antecedents in the preceding text / dialogue
  12. 12. Terminology  Entity = an object or set of objects in the world  Entities can have types (ACE requires to annotate only certain types e.g. person, location, organisation, etc.)  Mention = a textual reference to an entity (usually an NP)  Direct anaphora = identity of head, generalisation, specialisation or synonymy  Indirect anaphora = part-of, set-membership
  13. 13. Annotation of anaphora/ coreference  In general the process can be split into two stages:  Identification and annotation of elements involved in a relation (annotation of mentions)  Identification and annotation of relations between mentions  The two stages can be done together or separately
  14. 14. Annotation of mentions  Annotate everything?  Singletons should be annotated because they influence evaluation measures (except MUC score)  If everything is annotated it is easier if this annotation is done in the first instance  Syntactic annotation can be useful
  15. 15. Annotation of relations  Each annotation scheme defines a set of relations that should be covered  The relations normally happen between mentions/markables
  16. 16. MUC annotation (Hirchmann 1997)  Defined in the coreference resolution task at MUC  The criteria used to define the task were: 1. Support for the MUC information extraction tasks; 2. Ability to achieve good (ca. 95%) interannotator agreement; 3. Ability to mark text up quickly (and therefore, cheaply); 4. Desire to create a corpus for research on coreference and discourse phenomena, independent of the MUC extraction task.  These criteria are not necessarily consistent with each other
  17. 17. MUC annotation scheme  Marks only relations between noun phrases  Does not mark relations between verbs, clauses, etc.  Marks only IDENTITY which defines equivalence classes and is not directional  Values which are clearly distinct should not be allowed to be in the same class e.g. the stock price fell from $4.02 to $3.85
  18. 18. MUC annotation scheme (II)  SGML used <COREF ID="100">Lawson Mardon Group Ltd.</COREF> said <COREF ID="101" TYPE="IDENT" REF="100">it </COREF> ...  Attributes:  ID a unique identifier for a mention  REF indicates links between mentions  TYPE the type of link (only IDENT supported)  MIN the minimum span to be identified in order to be considered correct in automatic evaluation  STATUS=“OPT” to indicate optional elements to be resolved
  19. 19. MUC annotation scheme – markables (III)  NPs (including dates, percentages and currency expressions), personal and demonstrative pronouns  Interrogative “wh-” NPs are not marked (Which engine would you like to use?)  The extent of the markable is quite loosely defined (must include the head, but should really include the maximal NP and MIN attribute have the head as the value)
  20. 20. MUC annotation scheme – relations  Basic coreference  Bound anaphors  Apposition <COREF ID="1" MIN="Julius Caesar">Julius Caesar, <COREF ID="2" REF="1" MIN="emperor" TYPE="IDENT"> the/a well-known emperor,</COREF></COREF>  Predicate nominals <COREF ID="1" MIN="Julius Caesar">Julius Caesar</COREF> is <COREF ID="2" REF="1" MIN="emperor" TYPE="IDENT">the/a well-known emperor</COREF> who …  For appositions and predicate nominals there needs to be certainty (is not may be)
  21. 21. MUC annotation - criticism  Van Deemter and Kibble (1999) criticised the MUC scheme because it goes beyond annotation of coreference as it is commonly understood because:  It marks quantifying NPs (e.g. every man, most people)  Marks indefinite NPs Henry Higgins, who was formerly sales director of Sudsy Soaps, became president of Dreamy Detergents.  and one can argue not in a consistent manner the stock price fell from $4.02 to $3.85
  22. 22. MUC annotation & corpus  Despite criticism the MUC annotation provided a starting point for standardising anaphora/coreference annotation schemes  Designed to mark only a small set of expressions and relations which can be tackled by computers  Was proposed in the context of a competition  comparison of results and backing of an organisation  The corpus is available
  23. 23. Corpus of technical manuals (Mitkov et. al. 2000)  A corpus of technical manuals annotated with a MUC-7 like annotation scheme  Annotates only identity of reference between direct nominal referential expressions  Less interesting from linguistic perspective, but used to develop automatic methods
  24. 24. Corpus of technical manuals (Mitkov et. al. 2000)  Full coreferential chains are annotated  All the mentions are annotated regardless whether they are singletons or not  The relation of coreference is considered fully transitive  The MUC annotation scheme was used but the guidelines were not adapted completely  CLinkA (Orasan 2000) was used for annotation
  25. 25. Annotation guidelines  The starting point the MUC-7 annotation guidelines, but  More strict with what means identity of meaning (e.g. we do not consider indefinite appositional phrases coreferential with the phrases that contain them)  An indefinite NP cannot refer to anything  Not consider gerunds as mentions
  26. 26.  Add missing phenomena:  V [NP1] as [NP2] – not coreferential [use [a diagonal linear gradient] as [the map]] – is not coreferential [elect [John Prescott] as [Prime Minister]], – is not coreferential  …if [[ an NTSC Ø ]i or [ PAL monitor ]j]k is being used…[ The NTSC monitor ]l… - not coreferential …[[the pixels’ luminance]i or [Ø Ø saturation]j ]k is important… [The pixels’ saturation]j - coreferential
  27. 27. Annotation guidelines – short version Do: Do not: (i) annotate identity-of-reference direct nominal (i) annotate indefinite predicate nominals that are linked to anaphora other elements by perception verbs as coreferential with those elements (ii) annotate definite descriptions which stand in any of (ii) annotate identity-of-sense anaphora the identity, synonymy, generalisation, specialisation, or copula relationships with an antecedent (iii) annotate definite NPs in a copula relation as (iii) annotate indirect anaphora between markables coreferential (iv) annotate definite appositional and bracketed phrases (iv) annotate cross-document coreference as coreferential with the NP of which they are a part (v) annotate NPs at all levels from base to complex and (v) annotate indefinite NPs in copula relations with other co-ordinated NPs as coreferential (vi) familiarise yourself with the use of unfamiliar, (vi) annotate non-permanent or “potential” coreference highly specialised terminology by search through the between markables text (vii) annotate bound anaphors (viii) consider gerunds of any kind markable (ix) annotate anaphora over disjoined antecedents (x) consider first or second person pronouns markable
  28. 28. Speed of annotation (Mitkov et. al. 2000)  Speed of annotation in one hour:  At the beginning while the guidelines were being created: assign 288 mentions to 220 entities covering on average 2051 words in text  After the annotators became used to the task and the guidelines finalised: assign 315 mentions to 250 entities covering on average 1411 words in text  Fast track annotation for pronoun resolution in one hour: 113 pronouns, 944 candidates and 148 antecedents, covering 10862 words
  29. 29. Speed of annotation (II)  Most of the time during the annotation is spent identifying the mentions  … existing annotation levels can prove very beneficial
  30. 30. Reasons for disagreements  The process is tedious and requires high levels of concentration  Two main reasons for disagreement:  Unsteady references – mentions which may belong to different entities through the document (e.g. image, the window) – the automatic annotation option of the annotation tool may also mislead  Specialised terminology
  31. 31. Improving annotation strategies  Unsteady reference: Pre-annotation stage to clarify topic segments  Domain knowledge: Pre-annotation stage to disambiguate unknown technical terminology  ‘Master strategy’ combining individual approaches:  Printing text prior to annotation - increases familiarity  Two step process  Taking note of troublesome cases to discuss later with others  Annotating intensively vs sporadically
  32. 32. NP4E (Hasler et. al. 2006)  The goal was to develop annotation guidelines for NP and event coreference in newspaper articles about terrorism/security threats  A small corpus annotated with NP and event coreference was produced  An attempt to produce a more refined annotated resource than our previous corpus  5 clusters of related documents in the domain were built, about 50,000 words  http://clg.wlv.ac.uk/projects/NP4E/
  33. 33. NP coreference annotation  Used the guidelines developed by (Mitkov et. al. 2000) as the starting point,  but adapted them for our goals and texts  All the mentions need to be annotated, both definite and indefinte NPs  Introduced coref and ucoref tags to be able to deal with uncertainties The government] will argue that… [[McVeigh] and [Nichols]] were [the masterminds of [the bombing plot]]  Types of relations between an NP and its antecedent: identity, synonymy, generalisation, specialisation and other, but we do not annotate indirect anaphora
  34. 34. NP coreference annotation (II)  Types of (coreference) relations we identify NP, copular, apposition, bracketed text, speech pronoun and other  Link to the first element of the chain in most of the cases for type NP  For copular, apposition, bracketed text and speech pronouns (pronouns which occur in direct speech), the anaphor should be linked back to the nearest mention of the antecedent in the text  Do not annotate coreferential different readings of an NP as coreferential [A jobless Taiwanese journalist who commandeered [a Taiwan airliner] to [China]]… [China] ordered [[its] airports] to beef up [security]…
  35. 35. The user can override WordNet is consulted WordNet’s decision about the relation between the two NPs Annotation of NPs using PALinkA the plane is marked as coreferential with The aircraft
  36. 36. Issues arising during the NP annotation  The antecedent of pronoun we in direct speech can be linked to: the individual speaker, a group represented by the speaker or nothing  General concepts such as violence, terror, terrorism, police, etc are sometimes used in a general sense so it is difficult to know whether to annotate and how  Sometimes difficult to decide the best indefinite NP as an antecedent …the man detained for hijacking [a Taiwanese airliner]… Liu forced [a Far East Air Transport domestic plane]… Beijing returned [the Boeing 757]…
  37. 37. Issues arising during the NP annotation (II)  Mark relative pronouns/clauses and link them to the nearest mention Chinese officials were tightlipped whether [Liu Shan-chung, 45, [who] is in custody in China's southeastern city of Xiamen], would be prosecuted or repatriated to Taiwan.  The type of relation is sometimes difficult to establish without the help of WordNet (have ident, non-ident)
  38. 38. Annotation of event coreference  Event = a thing that happens or takes place, a single specific occurrence, either instantaneous or ongoing.  Used the ACE annotation guidelines as starting point  Events marked: ATTACK, DEFEND, INJURE, DIE, CONTACT  Identify the trigger = the best word to represent the event  Triggers: verbs, nouns, adjectives and pronouns {The blast} {killed} 168 people…and {injured} hundreds more… (ATTACK: noun, DIE: verb, INJURE: verb)
  39. 39. Event triggers  ATTACK: attack events are physical actions which aim to cause harm or damage tothings or people: attack, bomb, shoot, blast, war, fighting, clashes, throw, hit, hold, spent.  DEFEND: defend events are events where people or organisations defend something, usually against someone or something else: sheltering, reinforcing, running, prepared.  INJURE: injure events involve people experiencing physical harm: injure, hurt, maim, paralyse, wounded, ailing.  DIE: die events happen when a person’s life ends: kill, dead, suicide, fatal, assassinate, died, death.  CONTACT: contact events occur when two or more parties communicate in order to try and resolve something, reach an agreement or better relations between different sides etc. This category includes demands, threats and promises made by parties during negotiations: meeting, talks, summit, met, negotiations, conference, called, talked, phoned, discussed, promised, threatened, agree, reject, demand.
  40. 40. Annotation of event coreference  Two stage process: identify the triggers and then link them  Link arguments of an event to NP annotated in the previous stage  The arguments are event dependent (e.g. ATTACKER, MEANS, VICTIM, CAUSE, AGENT, TOPIC and MEDIUM)  The arguments should be linked to NPs from the same sentence or near by sentences if they are necessary to disambiguate the event  Also TENSE, MODALITY and POLARITY needs to be indicated
  41. 41. Annotation of an attack event using PALinkA the operation TYPE: attack TIME: Dec. 17 REF: stormed TARGET: the Japanese ambassador's residence in Lima (FACILITY) ATTACKER: MRTA rebels (PERSON) the operation PLACE: Lima (LOCATION)
  42. 42. Issues with event annotation  Very difficult annotation task  At times it is difficult to decide the tense of an event in direct speech  Whether to include demands, promises or threats in the CONTACT (or use them only as a signal of modality)  Whether to make a distinction between speaker/hearer in CONTACT events (especially in the case of demands, promises or threats)
  43. 43. What coreferential events indicate? (Hasler and Orasan 2009)  Starting point – do coreferential events have coreferential arguments?  We had a corpus of about 12,000 words annotated with event coreference  344 unique event mentions  106 coreferential chains with 2 to 10 triggers  238 events referred by only one trigger
  44. 44. Zaire planes bombs rebels as U.N. seeks war’s end. a293 TRIGGER: bombs ATTACKER: – MEANS: Zaire planes: ID=0: CHAIN=0: VEHICLE PLACE: – TARGET: rebels: ID=1: CHAIN=1: PERSON TIME: – Zaire said on Monday its warplanes were bombing three key rebel-held towns in its eastern border provinces and that the raids would increase in intensity. a333 TRIGGER: bombing ATTACKER: Zaire: ID=44: CHAIN=5: ORGANISATION MEANS: its warplanes: ID=46: CHAIN=46: VEHICLE PLACE: three key rebel-held towns in its eastern border provinces: ID=48: CHAIN=14: LOCATION TARGET: three key rebel-held towns in its eastern border provinces: ID=48: CHAIN=14: LOCATION TIME: Monday: ID=45: CHAIN=7 “Since this morning the FAZ (Zaire army) has been bombing Bukavu, Shabunda and Walikale”, said a defence ministry statement in the capital Kinshasa. a334 TRIGGER: bombing ATTACKER: the FAZ (Zaire army): ID=53: CHAIN=53: ORGANISATION MEANS: – PLACE: Bukavu, Shabunda and Walikale: ID=55: CHAIN=14: LOCATION TARGET: Bukavu, Shabunda and Walikale: ID=55: CHAIN=14: LOCATION TIME: this morning: ID=52: CHAIN=52
  45. 45. Referential relations between arguments  104 chains considered:  22 (21.15%) contained only coferential NPs  23 (22.12%) contained only non-coferential NPs  9 chains ignored  50 (48.07%) contain a mixture of coreferential and non-coreferential NPs  If indirect anaphora is not annotated, 70% of chains are affected
  46. 46. ID TRIGGER ARGUMENT: AGENT(S) c389 an emergency summit the leaders of both nations: ID=20: CHAIN=20: PERS c397 the two-hour closed meeting they: ID=24: CHAIN=20: PERS c408 the summit Fujimori: ID=60: CHAIN=32: PERS Hashimoto: ID=58:CHAIN=40:PERS c409 the summit Fujimori: ID=60: CHAIN=32: PERS Hashimoto: ID=58: CHAIN=40: PERS c418 the summit rebels: ID=110: CHAIN=14: PERS c432 the summit he: ID=170: CHAIN=40: PERS
  47. 47. Identity of sense  There are cases where even though the strings are the same we do not have identity of reference: at least nine people and nine confirmed dead  Hundred, at least 500 people, the first group of at least 500 people, but probably more than that and the 500  It can be argued that events of INJURE, DIE and DEFEND with such parameters are not coreferential, but the ATTACK events that causes them are.
  48. 48. at least nine people were killed and up to 37 wounded i343 TRIGGER: wounded AGENT: the FAZ (Zaire army): ID=53: CHAIN=53: ORG VICTIM: up to 37: ID=66: CHAIN=66: PERSON CAUSE: – PLACE: Bukavu: ID=70: CHAIN=17: LOCATION TIME: Monday: ID=69: CHAIN=7 there are nine confirmed dead and 37 wounded i346 TRIGGER: wounded AGENT: – VICTIM: 37 wounded: ID=86: CHAIN=78: PERSON CAUSE: – PLACE: – TIME: –
  49. 49. Missing slots  Coreference between events can be established even if many slots are not filled in:  Peru’s Fujimori says hostage talks still young.  ...the President said talks to free them were still in their preliminary phase.  ”We cannot predict how many more weeks these discussions will take.”  ”We are still at a preliminary stage in the conversations.”  Fujimori said he hoped Nestor Cerpa would personally take part in the talks when they resume on Monday at 11am.
  50. 50. Contact events  Involve 2 or more parties  The parties are usually introduced bit by bit and event coreference is necessary to establish all the participants  Cross-document event coreference is sometimes necessary collect all the participants
  51. 51. Conclusions  The guidelines should not be used directly and the characteristics of the texts should be considered  For automatic processing MUC-like may provide a good trade off between the linguistic detail encoded and the difficulty of annotation  However, quite often this annotation is not enough for more advanced processing  Have a more refined notion of “identity” Coreference is a scalar relation holding between two (or more) linguistic expressions that refer to DEs considered to be at the same granularity level relevant to the pragmatic purpose. (Recasens, Hovy and Marti, forthcoming)
  52. 52. Thank you!
  53. 53. References  van Deemter, Kees and Rodger Kibble, (1999). What is coreference and what should coreference annotation be? In Amit Bagga, Breck Baldwin, and S Shelton (eds.), Proceedings of ACL workshop on Coreference and Its Applications. Maryland.  Halliday, M. A. K., and Hasan, R. (1976).Cohesion in English. London: Longman.  Hasler, L. and Orasan. C (2009). Do coreferential arguments make event mentions coreferential? Proceedings of the 7th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2009), Goa, India, 5-6 November 2009, 151-163  Hasler, L., Orasan, C. and Naumann, K. (2006) NPs for Events: Experiments in coreference annotation. In Proceedings of the 5th Language Resources and Evaluation Conference (LREC2006). Genoa, Italy, 24-26 May, 1167-1172  Hirschman, L. (1997). MUC-7 coreference task definition. Version 3.0  Mitkov, R. (2002): Anaphora Resolution. Longman  Mitkov, R., Evans, R., Orasan, C., Barbu, C., Jones L. and Sotirova, V. (2000) Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies Proceedings of the Discourse Anaphora and Anaphora Resolution Colloquium (DAARC'2000)), 49-58. Lancaster, UK