Crowdsourcing Disagreement on Open-Domain Questions
β’
1 likeβ’3,603 views
Presentation on crowdsourcing disagreement on open-domain questions
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Crowdsourcing Disagreement on Open-Domain Questions
Similar to Crowdsourcing Disagreement on Open-Domain Questions (20)
Recently uploaded
Recently uploaded (20)
Crowdsourcing Disagreement on Open-Domain Questions
- 1. CrowdsourcingDisagreement on Open-Domain QuestionsJuly 18th 2014 Crowdsourcing Disagreement on Open-Domain Questions By Benjamin Timmermans, Lora Aroyo (VU) & Chris Welty (IBM)
- 2. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions β Natural Language β Generate and evaluate hypothesis β Learn from evidence Cognitive Systems
- 3. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Gold Standard β Distant supervision incomplete (Min et al., 2013) β Crowdsourcing ground-truth (Inel et al., 2013) β Large diverse volume of annotations β Crowd Truth
- 4. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Closed Questions Is rice a grain? (Yes) Are all snakes Brown? (No) Is it hard to get into Stanford? Is MO polar? Do isotopes of the same element have the same atonic weight?
- 5. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Do isotopes of the same element have the same atonic weight? 1. 1913: Henry Moseley, working fromVan den Broek's earlier idea, introduces concept of atomic number to fix inadequacies of Mendeleev's periodic table, which had been based on atomic weight,. ;1913:Frederick Soddyproposes theconcept of isotopes, that elements with thesamechemical properties may havediffering atomic weights. 2. A chemical element consists of atomswith a specific number of protonsin their nuclei, but different atomic weights owing to variationsin thenumber of neutrons. Atoms of thesameelement with differing atomic weights are called isotopes. 3. A primary isotopeshift is thechangein chemical shift observed between spectra of different isotopes of thesameelement, such as proton and deuterium. Primary isotopeshiftsbetween 1H and 2Dand areusually less than 0.1 ppm. 4. A radioisotopedoes not "rot";it decays by turning intoanother isotopeof thesameelement-or even into another element entirely. 5. Although receiving considerablesupport it was eventually rejected when it was found that many elementshavenon-integral weights (e.g. chlorine: 35.453). Frederick Soddy in 1913had introduced theidea of isotopes; that is, the samechemical element in different forms having differing weights. Aston established thatisotopes arenot restricted to radioactiveelements but arecommonthroughouttheperiodic table. 6. An alternateform of an element that has theusual number of protons but a nonstandard number of neutrons; thefewer or additional neutronsgivetheisotopea different atomic weight than theregular element and may make theisotoperadioactive, but otherwisean isotopehas thesamechemical actionas theregular element. Becauseof this, isotopes (such as radioactivecarbon) areused as tracers in biological systemsor processes. 7. An isotopeis oneof several kinds of atoms of thesameelement that havedifferent masses. Theseatoms havethesamenumber of protons in their nuclei, but different numbers of neutrons, and thereforedifferent mass numbers. 8. As noted in the discussion of deuterium, tritiumcan only beseparated from protiumdueto thedifferences in mass. Thechemical properties of isotopes with thesameparent element makethem otherwiseindistinguishable, and hence purely chemical means cannot beused to separatethem. 9. At theturn of thenineteenth century Dalton extended and refined Prout s remarkableconclusion thatcompounds wereof fixed composition by proposing thatatomsof thesameelement had thesameatomic weight. Today, following thediscovery of isotopes in this century, wedefinean atomic weight for a blend of isotopes of an element as the ratio of the average mass per atom of theelement to one-twelfth themass of an atom of 12C 10. Atoms of the same atomic number but different atomic weights are called isotopes. Elements can exist in both stable and unstable (radioactive) forms. 11. Atoms of thesameelement whosenuclei contain a different number of neutrons aresaid to be different isotopes of theelement. A pure element can exist as monatomic units or as diatomic or polyatomic unitscomprising the samekind of atoms. 12. Isotopes areforms of the sameelement having different atomic weights 13. Atoms of thesameelement with differing atomic weights arecalled isotopes. Radioactivedecay is a spontaneous processin which an isotope(theparent) loses particles from its nucleusto form an isotopeof a new element (the daughter) 14. By the way, an isotopemeans theforms of an element having different atomic weights becauseof thedifference in number of neutrons. 15. Carbon has atomic number 6 and atomic weight 12.011, and is represented by thesymbol C. It occurs in two different isotopes. Isotopes sharethesameatomic number, hencethesameidentity as elements and samechemical behavior, but havedifferent atomic weights. Put differently, isotopes havethesamenumber of protonsbut a different number of neutrons. Theisotopes of carbon arecarbon-12 (six protons plussix neutrons) and carbon-14 (six protons plus eight neutrons) 16. Deuterium is an isotopeof Hydrogen. An isotopeis any of two or moreforms of an element having thesameatomic number but with different weights (mass). Hydrogen has 2 isotopes, Deuterium (stable, i.e. non-radioactive) and Tritium (radioactive) 17. Dr. W. D. Harkins, Professor of Physical Chemistryat theUniversity of Chicago, who earlier in theweek had announced his discovery of Zeta rays, described his original work with isotopes, or elements having thesamechemical structurebut different atomic weights. 18. During this time, heworked with American chemist and physicist HaroldUrey at Columbia Universityon gaseous diffusiontechniquesfor theseparation of uranium isotopes (isotopes aredifferent forms of thesameelement having thesameatomic number but different atomic weights) 19. During this time, heworked with American chemist and physicist HaroldUrey at Columbia Universityon gaseous diffusiontechniquesfor theseparation of uranium isotopes (isotopes aredifferent forms of thesameelement having thesameatomic number but different atomic weights). After thewar, he accepted an appointment as a professor of chemistryat theUniversity of Chicago and began to conduct research at theInstituteof Nuclear Studies. 20. Fiveelements haveseven stableisotopes, eight havesix stableisotopes, ninehavefivestableisotopes, ninehavefour stable isotopes, ninehavethreestableisotopes, 16 havetwo stableisotopes (counting m as stable), and 26 elements haveonly a singlestableisotope(of these, 19 are so-called mononuclidic elements, having a singleprimordial stable isotopethat dominates and fixes theatomic weight of thenatural element to high precision; 21. For an element with threenaturally occurring isotopes themethodis thesame: sum themasses of theisotopes weighted by atom-fraction. 22. For an element with threenaturally occurring isotopes themethodis thesame: sum themasses of theisotopes weighted by atom-fraction. This method of calculating theaveragemass takes into account therelativeabundance of all of theisotopes of an element, so that this mass number always gives thesametotal number of atoms, for a natural sampleof any element. 23. He called atoms of thesecond group isotopes, atomsof thesameelement with different atomic weights. In any natural sample of an element, theremay beseveral types of isotopes. As a result, theatomic weight of an element that was calculated by Berzelius was actually an averageof all theisotopeweights for that element. This was thereason that someelements did not fall into theright order on Mendeleev's Periodic Table--theaverageatomic weight depended on how much of each kind of isotopewas present. Soddy suggested placing theelements in thePeriodic Tableby similarity of chemical reactionsand then numbering them in order. 24. He called such chemically identical elements, with slightly differing atomic weights, isotopes (from theGreek words meaning in thesameplace) 25. His assistant, Francis Aston, developed Thomson's instrumentfurther and with theimproved version was ableto discover isotopes-atomsof thesameelement with different atomic weights-in a largenumber of nonradioactive elements. 26. In 1913, Richards proved theexistenceof lead isotopes (two or moreatoms of thesameelement that differ in atomic weight) by investigating theweight of lead from various sources. 27. In 1914 hereceived theNobel Prizein Chemistry for accurately determining theatomic weightsof morethan 25 chemical elementsand ascertaining theexistenceof isotopes, chemical elementsthat havethesameatomic number and position in theperiodic tablebut different atomic masses and physical properties. 28. In a short letter to theeditor of Nature, published on December 4, 1913, hefirst proposed theterm isotopeto designatechemically identicalelements with different atomic weights (in modern terms, elements with thesame atomic numbers but different mass numbers) 29. In a short letter to theeditor of Nature, published on December 4, 1913, hefirst proposed theterm isotopeto designatechemically identicalelements with different atomic weights (in modern terms, elements with thesame
- 6. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Machine way of thinking Crowdsourcing Architecture
- 7. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions High amount of annotations 40 Answer Passages 3000 Questions 10 Workers ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? ?????????????????????????????? 0 - 20 Matching Terms
- 8. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Passage Alignment Template
- 9. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Result of Alignment Answer Passage Terms Question Terms
- 10. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Annotated Relation Types
- 11. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Negation in Passages vs Wiki Answers Is MO polar? o Yes x No Question ID
- 12. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Passage Alignment not perfect β No expertise required β High variety and volume of annotations β Difficult to find the answer β Expensive at $0.06 per task for 10 workers for 120.000 passages = $72.000
- 13. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Question Answering Architecture
- 14. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Question Answering Template
- 15. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Question Answering Template
- 16. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Question Answering Template
- 17. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Answer Justification Question ID JustifyingPassages
- 18. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Reduce dataset 120.000 -10% -75% 27.000
- 19. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions AggregatedWorkerAnswer Crowdsourcing Answers vs Wiki Answers Wiki Answer is Yes Wiki Answer is No Is rex burkhead a senior? Is MO polar? Question IDQuestion ID AggregatedWorkerAnswer
- 20. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Tracking low quality workers No Answer Found Unanswerable Yes No Other Answer Yes No Closed Question Other AnswerOpen Question
- 21. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions AMT vs CF worker efficiency Low Quality Workers
- 22. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Question Answering Task β No expertise required β Judgements on: β Answer Justification β Question Type β Question Answer 120.000 -10% -75% 27.000
- 23. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Future Work β Gamification β Domain specific expert annotators β Improve preprocessing β Relation tasks
- 24. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions Conclusions β Crowd can answer open-domain questions β Aligning passages via Crowdsourcing is expensive β Crowd performs better on AMT than on CF