Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Truth is a Lie - 7 Myths of Human Annotation

Watson & Natural Language Processing meet-up talk

  • Identifiez-vous pour voir les commentaires

Truth is a Lie - 7 Myths of Human Annotation

  1. 1. ★
  2. 2. • – … – • Anca Dumitrache
  3. 3. • • • • Anca Dumitrache
  4. 4. • • Anca Dumitrache
  5. 5. • • • • • • • Anca Dumitrache
  6. 6. Anca Dumitrache
  7. 7. Which is the mood most appropriate for each song? ★
  8. 8. METHYLERGOMETRINE is a blood vessel constrictor most commonly used to control excessive BLEEDING. What is the relation between the highlighted terms? ★
  9. 9. • • • • • Anca Dumitrache
  10. 10. ANTIBIOTICS are the first line treatment for indications of TYPHUS. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. Is the TREATS relation expressed between the highlighted terms? ★
  11. 11. • • • • Anca Dumitrache
  12. 12. Which mood cluster is most appropriate for a song? ★
  13. 13. agents are useful for patients with renal impairment, but in patients with severe renal failure requiring dialysis it presents a risk of nephrogenic systemic . ☒ ☑ What is the relation between the highlighted terms? ★
  14. 14. • • – – Anca Dumitrache
  15. 15. Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. → agreement 95% Patients with TYPHUS who were given ANTIBIOTICS exhibited side- effects. → agreement 80% With ANTIBIOTICS in short supply, DDT was used during WWII to control the insect vectors of TYPHUS. → agreement 50% ★
  16. 16. • • • Anca Dumitrache
  17. 17.
  18. 18. Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. is not a reliable cure for . ★
  19. 19. • • – • • Anca Dumitrache
  20. 20. What is the relation between the highlighted terms? He was the first physician to identify the relationship between and ★
  21. 21. What is the (medical) relation between the highlighted (medical) terms? • 91% of expert annotations covered by the crowd • expert annotators reach agreement only in 30% • most popular crowd vote covers 95% of this expert annotation agreement ★
  22. 22. • • – – Anca Dumitrache
  23. 23. Which are mentions of terrorists in this sentence? used money from his own construction company to support the in Afghanistan against Soviet forces. ★
  24. 24. • Annotator disagreement is signal, not noise. • It is indicative of the variation in human semantic interpretation of signs • It can indicate ambiguity, vagueness, similarity, over- generality, etc, as well as quality Anca Dumitrache
  25. 25. • methodology • disagreement-aware crowdsourcing to collect gold standard data • metrics to capture disagreement • software • online platform for crowdsourcing task workflows and data analytics • ground truth collection • medical relation extraction • salience in news and tweets • sound annotation http://CrowdTruth.org
  26. 26. CrowdTruth.org github.com/CrowdTruth data.CrowdTruth.org
  27. 27. • Goals: ○ collecting a relation extraction gold standard ○ improve the performance of a relation extraction classifier http://CrowdTruth.org
  28. 28. TASK →
  29. 29. 1 1 1
  30. 30. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0
  31. 31. 1 0 1 1 0 0 4 3 0 0 5 1 0 Unit vector for relation R6 Sentence Vector Cosine = .55
  32. 32. • methodology • disagreement-aware crowdsourcing to collect gold standard data • metrics to capture disagreement • software • online platform for crowdsourcing task workflows and data analytics • ground truth collection • medical relation extraction • salience in news and tweets • sound annotation http://CrowdTruth.org
  33. 33. http://crowdtruth.org http://github.com/CrowdTruth ★

×