Learning to read for automated fact checking
•
1 j'aime•1,499 vues
Spreading of mis- and disinformation is growing and is having a big impact on interpersonal communications, politics and even science. Traditional methods, e.g. manual fact-checking by reporters cannot keep up with the growth of information. On the other hand, there has been much progress in natural language processing recently, partly due to the resurgence of neural methods. How can natural language processing methods fill this gap and help to automatically check facts? This talk will explore different ways to frame fact checking and detail our ongoing work on learning to encode documents for automated fact checking, as well as describe future challenges.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (11)
Similaire à Learning to read for automated fact checking
Similaire à Learning to read for automated fact checking (20)
Plus de Isabelle Augenstein
Plus de Isabelle Augenstein (20)
Dernier
Dernier (20)
Learning to read for automated fact checking
- 1. Aarhus University Department of Computer Science Friday Lecture Series 17 November 2017 Isabelle Augenstein augenstein@di.ku.dk @IAugenstein http://isabelleaugenstein.github.io/ Towards Automated Fact Checking of Claims Online
- 2. Types of False Information
- 3. Types of False Information
- 4. Types of False Information http://www.contentrow.com/tools/link-bait-title-generator
- 5. Types of False Information
- 6. Types of False Information https://arxiv.org/abs/1611.04135
- 7. Types of False Information
- 8. Types of False Information • Disinformation: • Intentionally false, spread deliberately • Misinformation: • Unintentionally false information • Clickbait: • Exaggerating information and under-delivering it • Satire: • Intentionally false for humorous purposes • Biased Reporting: • Reporting only some of the facts to serve an agenda
- 9. Types of False Information • Disinformation: • Intentionally false, spread deliberately • Misinformation: • Unintentionally false information • Clickbait: • Exaggerating information and under-delivering it • Satire: • Intentionally false for humorous purposes • Biased Reporting: • Reporting only some of the facts to serve an agenda
- 10. query “Unemployment in the US is 42%” Goal: Fact Checking Machine Reader unemployment(US, 42%) What is the stance of HRC on immigration? stance(HRC, immigration, X)
- 15. Fully Automated Fact Checking 1) Given a claim, retrieve evidence documents for and against it 2) Given evidence documents, find relevant paragraphs/sentences in it 3) For claim and each evidence paragraph/sentence: detect stance of paragraph/sentence towards a claim/target 4) Balance and combine stance judgements (stance judgements + trust of source) 5) Explain stance judgements 17/11/2017 15
- 16. Fully Automated Fact Checking 1) Given a claim, retrieve evidence documents for and against it 2) Given evidence documents, find relevant paragraphs/sentences in it 3) For claim and each evidence paragraph/sentence: detect stance of paragraph/sentence towards a claim/target 4) Balance and combine stance judgements (stance judgements + trust of source) 5) Explain stance judgements 17/11/2017 16
- 17. Stance Detection • Determine attitude expressed in document/paragraph/sentence towards a topic/statement/target • Different classification schemes: • positive, negative, neutral (SemEval 2016 Task 6, RTE, SNLI) • support, deny, query, comment (SemEval 2017 Task 8 RumourEval) • agree, disagree, discuss, unrelated (Fake News Challenge)
- 18. Stance Detection with Conditional Encoding No more #NastyWomen or #BadHombres Task: Is tweet positive, negative or neutral towards a given target (Donald Trump)? Problems: - Interpretation depends on target - Target not always mentioned in tweet - No training data for test target SemEval 2016, EMNLP 2016
- 19. Stance Detection with Conditional Encoding • Challenges • Model: Learn a model that interprets the tweet stance towards a target that might not be mentioned in the tweet itself • Training Data: Learn model without labelled training data for the target with respect to which we are predicting the stance
- 20. Stance Detection Model: Sum of Word Embeddings Legalization of Abortion A foetus has rights too ! Target Tweet s(e)s(a) g(x)Is tweet positive, negative or neutral towards given target?
- 21. Stance Detection Model: Concatenated Sequence Representations Legalization of Abortion A foetus has rights too ! Target Tweet s(e)s(a) g(x)Is tweet positive, negative or neutral towards given target?
- 22. Stance Detection Model: Bidirectional Conditional Encoding x1 c! 1 c1 h! 1 h1 x2 c! 2 c2 h! 2 h2 x3 c! 3 c3 h! 3 h3 x4 c! 4 c4 h! 4 h4 x5 c! 5 c5 h! 5 h5 x6 c! 6 c6 h! 6 h6 x7 c! 7 c7 h! 7 h7 x8 c! 8 c8 h! 8 h8 x9 c! 9 c9 h! 9 h9 Legalization of Abortion A foetus has rights too ! Target Tweet igure 1: Bidirectional encoding of tweet conditioned on bidirectional encoding of target ([c! 3 c1 ]). The stance is predicted using he last forward and reversed output representations ([h! 9 h4 ]).
- 23. Stance Detection with Conditional Encoding • Challenges • Training Data: Learn model without labelled training data for the target with respect to which we are predicting the stance • Solution 1: use training data labelled for other targets (domain adaptation setting) • Solution 2: automatically label training data for target, using a small set of manually defined hashtags (weakly labelled setting)
- 24. Stance Detection with Conditional Encoding • Domain Adaptation Setting • Train on Legalization of Abortion, Atheism, Feminist Movement, Climate Change is a Real Concern and Hillary Clinton, evaluate on Donald Trump tweets Model Stance P R F1 FAVOR 0.3145 0.5270 0.3939 Concat AGAINST 0.4452 0.4348 0.4399 Macro 0.4169 FAVOR 0.3033 0.5470 0.3902 BiCond AGAINST 0.6788 0.5216 0.5899 Macro 0.4901
- 25. Stance Detection with Conditional Encoding • Weakly Supervised Setting • Weakly label Donald Trump tweets using hashtags / expressions, evaluate on Donald Trump tweets positive: make( ?)america( ?)great( ?)again trump( ?)(for|4)( ?)president negative: #dumptrump #notrump
- 26. Stance Detection with Conditional Encoding • Weakly Supervised Setting • Weakly label Donald Trump tweets using hashtags / expressions, evaluate on Donald Trump tweets * state of the art on dataset Model Stance P R F1 FAVOR 0.5506 0.5878 0.5686 Concat AGAINST 0.5794 0.4883 0.5299 Macro 0.5493 FAVOR 0.6268 0.6014 0.6138 BiCond AGAINST 0.6057 0.4983 0.5468 Macro 0.5803 *
- 27. Stance Detection with Conditional Encoding • Conclusions • Modelling sentence pair relationship is important • Weak labelling of in-domain tweets even more important • Partly due to small training data size here • Learning sequence representations also a good approach for small data • State of the art on SemEval 2016 Stance Detection dataset
- 28. Stance Detection for Conversational Structures SemEval 2017 RumourEval Task A (winning system), IP&M 2017 (under review)
- 29. Stance Detection for Conversational Structures SemEval 2017 RumourEval Task A (winning system), IP&M 2017 (under review)
- 30. Stance Detection for Conversational Structures SemEval 2017 RumourEval Task A (winning system), IP&M 2017 (under review)
- 31. Stance Detection for Conversational Structures SemEval 2017 RumourEval Task A (winning system), IP&M 2017 (under review)
- 32. Example Rumours (10 in total, 2 of those only in test) • Putin missing: from March 2015 - Russian president Vladimir Putin did not appear in public for 10 days. Rumours emerged he had been ill or killed. Denied by Putin himself on 11th day. • Gurlitt collection: from November 2014 - Bern Museum of Fine Arts to accept a collection of modernist masterpieces kept by the son of a Nazi-era art dealer. Confirmed. 17/11/2017 32 Stance Detection for Conversational Structures SemEval 2017 RumourEval Task A
- 33. Training data 17/11/2017 33 Stance Detection for Conversational Structures Supporting Denying Querying Commenting Development 69 11 28 173 Testing 94 71 106 778 Training 841 333 330 2734 SemEval 2017 RumourEval Task A
- 34. Confusion matrix 17/11/2017 34 Stance Detection for Conversational Structures Prediction Gold Supporting Denying Querying Commenting Supporting 26 2 1 6 Denying 0 0 0 0 Querying 1 1 36 12 Commenting 6 68 69 760 SemEval 2017 RumourEval Task A (winning system)
- 35. Examples misclassifying denying [As querying] @username Weren’t you the one who abused her? [As supporting] ”Go online & put down ’Hillary Clinton illness,’” Rudy says. Yes – but look up the truth – not health smears https://t.co/EprqiZhAxM [As supporting] @username I demand you retract the lie that people in #Ferguson were shouting ”kill the police”, local reporting has refuted your ugly racism [As commenting] @FoxNews six years ago... real good evidence. Not! 17/11/2017 35 Stance Detection for Conversational Structures SemEval 2017 RumourEval Task A (winning system)
- 36. • Relationship between sequences can be modelled effectively with deep neural models • Even more complicated structures (conversational threads) can be modelled effectively • Many challenges • Hard to collect data, especially with balanced labels • Hard to train deep neural NLP models with little, imbalanced data • Predicted labels do not explain stance judgements 17/11/2017 36 Summary: Stance Detection
- 37. Thanks to my collaborators! USDF: Diana Maynard, Andreas Vlachos, Kalina Bontcheva, Michal Lukasik, Leon Derczynski UCL: Sebastian Riedel, Tim Rocktäschel ATI: Elena Kochkina, Maria Liakata UCPH: Anders Søgaard, Joachim Bingel, Johannes Bjerva, Mareike Hartmann Plus Sebastian Ruder, Arkaitz Zubiaga, Rob Procter, Trevor Cohn, ... 17/11/2017 37