Survey of the current trends, and the future in Natural Language Generation
•
0 j'aime•370 vues
Small talk on NLG/TG. In recent years, NLG has been widely used in many fields. Today, we'll start with a brief introduction to NLG methods including Pipeline, Autoregressive: Seq2seq, Transformer, GAN, and Non-Autoregressive models. Besides, we will roughly talk about NLG tasks/scenarios and some issues we encounter. Nowadays, Neural incorporates Symbolic is a powerful paradigm in future AI, as Bengio mentioned in AAAI 2020 conference. We will introduce how knowledge guides NLG as well.
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Survey of the current trends, and the future in Natural Language Generation
Similaire à Survey of the current trends, and the future in Natural Language Generation (20)
Dernier
Dernier (20)
Survey of the current trends, and the future in Natural Language Generation
- 2. 2 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Current Trends, and the Future
- 3. 3 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Current Trends, and the Future
- 4. 4 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Current Trends, and the Future
- 5. 5 Background and Definition • What is Text Generation? • The difference between NLU, NLP, NLG or (TG) [Survey of the State of the Art in Natural Language Generation- Core tasks, applications and evaluation] NLP Natural Language Processing NLU Natural Language Understanding NLG Natural Language Generation
- 6. 6 Background and Definition • What is Text Generation? • Produce understandable texts in human languages from some underlying non-linguistic representation of information. [Reiter el., 1997] • Both text-to-text generation and data-to-text generation are instances of Text Generation [Reiter at el., 1997] [Gatt at el., 2017, Survey of the State of the Art in Natural Language Generation- Core tasks, applications and evaluation]
- 7. 7 Background and Definition • An overview of the applications that can be categorized under the umbrella of language generation [Gatt at el., 2017, Survey of the State of the Art in Natural Language Generation- Core tasks, applications and evaluation]
- 8. 8 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Current Trends, and the Future
- 10. 10 Rule/Template-based • Rule-based • Template-based • If we have lots of hand-drafted templates, we can … ________ is my friend. Retrieve Check [Sciforce, 2019, A Comprehensive Guide to Natural Language Generation]
- 11. 11 Statistical • Core idea: Build a statistical model from data • Similar to statistical machine translation methodology • Obtain 𝑃 𝑥 𝑦 and 𝑃 𝑦 from data • Compute the argmax with various algorithm • Systems extremely complex • Separately-designed sub-components • Lots of human efforts argmax) 𝑃(𝑦|𝑥; 𝜃) [Kondadadi at el., 2013, A Statistical NLG Framework for Aggregated Planning and Realization] This is my _____. This is __ music. … This is my dog. This is pop music. … 𝑥 (context) 𝑦
- 12. 12 Traditional and Neural Text Generation • Rule/Template-based and statistical models: • Fairly interpretableand well-behaved • Lots of hand-engineering and feature engineering • Neural network models : • Demonstrate better performance in many tasks • Poorly understood and sometimes poorly behaved as well Trade-offs between template/rule-based and neural network models [Xie at el., 2017, Neural Text Generation: A Practical Guide ]
- 13. 13 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Autoregressive • Non-Autoregressive • Text Generation Tasks and Challenges • Current Trends, and the Future
- 14. 14 Language Modeling • Language Modeling: 𝑃 𝑦5 𝑦6, 𝑦8, …, 𝑦5:6 • The task of predicting the next word, given the words so far • A system that produces this probability distribution is called a Language Model • RNNs for language modeling
- 15. 15 Conditional Language Modeling • Conditional Language Modeling: 𝑃 𝑦5 𝑦6, 𝑦8, … , 𝑦5:6, 𝑥) • The task of predicting the next word, given the words so far, and also some other input 𝑥 • 𝑥 input/source • 𝑦 output/target sequence [Xie at el., 2017, Neural Text Generation: A Practical Guide ]
- 16. 16 RNNs for Language Modeling ℎ< ℎ6 ℎ8 ℎ= 𝑥6 𝑥8 𝑥= ℎ> 𝑥> 𝑦> never too late to 𝑤6 𝑤8 𝑤= 𝑤> 𝐸 𝐸 𝐸 𝐸 𝑊B 𝑊B 𝑊B 𝑊B 𝑊C 𝑊C 𝑊C 𝑊C 𝑈 code read one-hot vectors 𝑤E ∈ ℝ|H| word embeddings 𝑥E = 𝐸𝑤E hidden states ℎE = tanh( 𝑊B 𝑥E + 𝑊CℎE:6 + 𝑏6 ) output distribution 𝑦> = softmax 𝑈ℎ> + 𝑏8 ∈ ℝ|H|
- 17. 17 RNNs for Language Modeling • RNNs are good at modeling sequential information • General idea: Use RNN as an encoder for building the semantic representation of the sentence
- 18. 18 Decoding • Recap: decoding algorithms • Question: Once you’ve trained your (conditional) language model, how do you use it to generate text? • Answer: A decoding algorithm is an algorithm you use to generate text from your language model • We’ve learnt about two decoding algorithms: • Greedy decoding • Beam search
- 19. Greedy Decoding • Generate the target sentence by taking argmax on each step of the decoder • argmax)O 𝑃(𝑦E|𝑦6, … , 𝑦E:6, 𝑥) • Due to lack of backtracking, output can be poor (e.g. ungrammatical, unnatural, nonsensical) 19 Decoder (2Seq) ℎ6 ℎ8 ℎ= ℎ> 𝑊B P 𝑊B P 𝑊B P 𝑊B P 𝑊C P 𝑊C P 𝑊C P <START> I can get better performance ℎQ 𝑊B P 𝑊C P ℎQ 𝑊B P 𝑊C P I can get better performance <END> argmax)O argmax)O argmax)O argmax)O argmax)O argmax)O
- 20. 20 Beam Search Decoding • We want to find 𝑦 that maximizes • 𝑃 𝑦 𝑥 = 𝑃 𝑦6 𝑥 𝑃 𝑦8 𝑦6, 𝑥 … 𝑃 𝑦R 𝑦6, … , 𝑦R:6, 𝑥 • Find a high-probability sequence • Beam search • Core idea: • On each step of decoder, keep track of the 𝑘 most probable partial sequences • After you reach some stopping criterion, choose the sequence with the highest probability • Not necessarily the optimal sequence
- 22. 22 What’s the Effect of Changing Beam Size k? • What’s the effect of changing beam size k? • Small 𝑘 has similar problems to greedy decoding (𝑘=1) • Ungrammatical, unnatural, nonsensical, incorrect • Larger 𝑘 means you consider more hypotheses • Increasing 𝑘 reduces some of the problems above • Larger 𝑘 is more computationally expensive • But increasing 𝑘 can introduce other problems • For neural machine translation (NMT): Increasing 𝑘 too much decreases BLEU score (Tu et al., Koehn et al.) • clike chit-chat dialogue: Large 𝑘 can make output more generic (see next slide) [Tu et al., 2017, Neural Machine Translation with Reconstruction] [Koehn et al., 2017, Six Challenges for Neural Machine Translation]
- 26. 26 Decoding • Sampling-based decoding • Pure sampling • On each step t, randomly sample from the probability distribution 𝑃5 to obtain your next word • Top-n sampling • On each step 𝑡, randomly sample from 𝑃5, restricted to just the top-n most probable words • 𝑛 = 1 is greedy search, 𝑛 = 𝑉 is pure sampling • Increase n to get more diverse/riskyoutput • Decrease n to get more generic/safe output • Both of these are more efficient than Beam search
- 27. 27 Decoding • In summary • Greedy decoding • A simple method • Gives low quality output • Beam search • Delivers better quality than greedy • If beam size is too high, it will return unsuitable output (e.g. Generic, short) • Sampling methods • Get more diversity and randomness • Good for open-ended / creative generation (poetry, stories) • Top-n sampling allows you to control diversity
- 28. Decoder (2Seq) 28 Seq2seq ℎ< ℎ6 ℎ8 ℎ= ℎ> 𝑊B 𝑊B 𝑊B 𝑊B 𝑊C 𝑊C 𝑊C 𝑊C Only use neural nets ℎ6 ℎ8 ℎ= ℎ> 𝑊B P 𝑊B P 𝑊B P 𝑊B P 𝑊C P 𝑊C P 𝑊C P <START> I can get better performacen ℎQ 𝑊B P 𝑊C P ℎQ 𝑊B P 𝑊C P I can get better performance <END> Encoder (Seq)
- 29. 29 Seq2seq • Pipeline methods have some problems • It’s hard to propagate the supervision signal to each component • Once the task is changed, all components need to be trained from scratch • Sequence-to-sequence model • 𝑃 𝑦 𝑥 = 𝑃 𝑦6 𝑥 𝑃 𝑦8 𝑦6, 𝑥 … 𝑃(𝑦R|𝑦6, … , 𝑦R:6, 𝑥) • It can be easily modeled by a single neural network and trained in an end-to-end fashion (differentiable) • The seq2seq model is an example of a conditional language model • Encoder RNN produces a representation of the source sentence • Decoder RNN is a language model that generates target sentence conditioned on encoding
- 30. 30 Attention • Seq2seq: the bottleneck problem • The single vector of source sentence encoding needs to capture all information about the source sentence. • The single vector limits the representation capacity of the encoder, which is the information bottleneck.
- 31. 31 Attention • Architecture • Attentionprovides a solution to the bottleneck problem • Core idea: on each step of the decoder, focus on a particular part of the source sequence • Attention helps with vanishing gradient problem Attention
- 32. 32 Transformer • Motivations • Sequential computation in RNNs prevents parallelization • Despite using LSTM or RNNs still need attention mechanism which provides access to any state • Maybe we do not need RNN? • Attention is all you need (2017) [Vaswani at el. 2017, Attention is all you need]
- 35. 35 Transformer • Replace squ2seq+attention with Transformer • The effectiveness of the attention mechanism • The Transformer is powerful and proven to be effective in many text generation tasks
- 36. 36 GAN-based • GAN architecture: 𝑎𝑟𝑔𝑚𝑖𝑛] 𝑚𝑎𝑥^ = 𝑉 𝐺, 𝐷 • Question: Can GANs be applied to text generation tasks ? • Answer: Yes. But …
- 37. 37 GAN-based • Problem: • GANs work better with continuous observed data, but not as well with discrete observed data • The discrete outputs from the generative model make it difficult to pass the gradient to the generative model
- 38. 38 GAN-based • Solution: • Gumbel-softmax • Wasserstein GAN (WGAN) • Replace KL-Divergence with Wasserstein-Divergence • WGAN-GP • Add gradient penalty (GP) to WGAN [Gulrajani at el., 2017, Improved Training of Wasserstein GANs]
- 39. 39 GAN-based • Problem: • Discrete outputs (mentioned) • The discriminative model can only assess a complete sequence • For a partially generated sequence, it is nontrivial to balance its current score and the future one once the entire sequence has been generated. • Solution: • SeqGAN • RL+GAN [Yu at el., 2016, SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient]
- 41. 41 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Autoregressive • Non-Autoregressive • Text Generation Tasks and Challenges • Current Trends, and the Future
- 42. 42 Autoregressive • Given a source 𝑥 = 𝑥6, 𝑥8, … , 𝑥a and target y = (𝑦6, 𝑦8, … , 𝑦c) • 𝑃 𝑦 𝑥 = ∏ 𝑃(𝑦5|𝑦e5, 𝑥, 𝜃fag, 𝜃Pfg)c 5h6 • RNN based, Seq2Seq based, Transformer based ℎ" ℎ# ℎ$ ℎ% ℎ& '( '( '( '( Only use neural nets ℎ# ℎ$ ℎ% ℎ& '( ) '( ) '( ) '( ) '* ) '* ) '* ) <START> ℎ+ '( ) '* ) ℎ+ '( ) '* ) <END>
- 43. 43 Autoregressive • The individual steps of the decoder must be run sequentially rather than in parallel • Time complexity • Lack of global information • Decoding and Transformer based models try to solve this issue
- 44. 44 Non-Autoregressive • Given a source 𝑥 = 𝑥6, 𝑥8, … , 𝑥a and target y = (𝑦6, 𝑦8, … , 𝑦c) • 𝑃 𝑦 𝑥 = 𝑃(𝑚|𝑥) ∏ 𝑃(𝑦5|𝑧, 𝑥)c 5h6
- 45. 45 Non-Autoregressive • 𝑃 𝑦 𝑥 = 𝑃 𝑚 𝑥 ∏ 𝑃 𝑦5 𝑧, 𝑥c 5h6 • Decide the length of the target sequence • The encoder are the same as non-autoregressive encoder; • Input 𝑧 = 𝑓(𝑥; 𝜃fag) • Generate the target sequence in parallel [Gu† at el., 2018, NON-AUTOREGRESSIVE NEURAL MACHINE TRANSLATION]
- 46. 46 Non-Autoregressive • Decode target output at the same time • Fast (20x than Autoregressive) • Can keep context information well during decoding • Similar to BERT masked LM decoding
- 47. 47 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Text Generation Tasks • Control Text Generation • Knowledge-guided Text Generation • Current Trends, and the Future
- 48. 48 Text Generation Tasks • Focus on: Text Summarization, Storytelling, and Poetry Generation • Scenario: Text-to-Text, Data-To-Text (variety of data) • Challenges [Gatt at el., 2017, Survey of the State of the Art in Natural Language Generation- Core tasks, applications and evaluation]
- 50. 50 Summarization
- 51. 51 Summarization: Text-to-text • Pre-neural summarization systems are mostly extractive • They typically had a pipeline: • Content selection: Sentence scoring functions, Graph-based algorithms • Information ordering • Sentence realization
- 52. 52 Summarization: Text-to-text • Neural summarization • Seq2seq+attention • Seq2seq+attention systems are good at writing fluent output, but bad at copying over details • Copy mechanisms • Copy mechanisms use attention to enable a seq2seq system to easily copy words and phrases from the input to the output • There are several papers proposing copy mechanism variants: • Language as a Latent Variable: Discrete Generative Models for Sentence Compression, Miao et al, 2016 • Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond, Nallapatiet al, 2016 • Incorporating Copying Mechanism in Sequence-to-Sequence Learning, Gu et al, 2016
- 53. 53 Summarization: Text-to-text • Neural Summarization: Copy mechanisms • Calculate 𝑝lfa, the probability of generating the next word. • The final distribution: generation distribution, and the copying distribution: See et al, 2017, Get To The Point: Summarization with Pointer-Generator Networks
- 54. 54 Summarization: Challenges • They copy too much • What should be an abstractive system collapses to a mostly extractive system • They’re bad at overall content selection, especially if the input document is long • No overall strategy for selecting content
- 55. 55 Summarization: Better Content Selection • Better content selection • Pre-neural summarization had separate stages for content selection and surface realization (i.e. text generation) • In a standard End2End (seq2seq+attention) summarization system, these two stages are mixed in together • Surface realization: on each step of the decoder: • Word-level content selection: attention • Word-level content selection is bad: no global content selection strategy • One solution: bottom-up summarization
- 56. 56 Summarization: Better Content Selection • Solution: Bottom-up summarization • Content selection stage: Use a neural sequence-tagging model to tag words as includeor don’t-include • Bottom-up attention stage: Use seq2seq+attention system. You can’t attend to words that were tagged don’t-include. (apply a mask)
- 57. 57 Storytelling • Use the given data to generate text • (Image-to-Text) Generate a story-like paragraph, given an image • (Prompt-to-Text) Generate a story, given a brief writing prompt • (Event-to-Text) Generate a story, given events, entities, state, etc. • (LongDocument-to-Text) Generate the next sentence of a story, given the story so far (story continuation)
- 58. 58 Storytelling: Prompt-to-Text • Generating a story from a writing prompt • In 2018, [Fan et al.] released a new story generation dataset collected from Reddit’s WritingPrompts subreddit. • Each story has an associated brief writing prompt. [Fan et al ,2018, Hierarchical Neural Story Generation] Input: Output:
- 59. 59 Storytelling: Prompt-to-Text • [Fan et al,] also proposed a complex seq2seq prompt-to- story model • It’s convolutional-based • Gated multi-head multi-scale self-attention • Model fusion [Fan et al ,2018, Hierarchical Neural Story Generation]
- 60. 60 Storytelling: Prompt-to-Text • The results are impressive! • Related to prompt • Diverse; non-generic • Stylistically dramatic • However: • Mostly atmospheric/descriptive/scene-setting; less events/plot • When generating for longer, mostly stays on the same idea without moving forward to new ideas – coherence issues
- 61. 61 Storytelling: Image-to-Text • Generating a story from an image • Question: How to get around the lack of parallel data? • Answer: Use a common sentence-encoding space, use an image captioning dataset to learn a mapping, train a RNN-LM to decode Input Output
- 65. 65 Storytelling: Challenges • Challenges in storytelling • Stories generated by neural LMs can sound fluent... but are nonsensical, with no coherent plot What’s missing? • LMs model sequences of words. Stories are sequences of events. • To tell a story, we need to understand and model: • Events and the causality structure between them • Characters, their personalities, motivations, histories, and relationships to other characters • State of the world (who and what is where and why) • Narrative structure (e.g. exposition → conflict → resolution) • Good storytelling principles (don’t introduce a story element then never use it)
- 66. 66 Storytelling: Challenges • Challenges in storytelling Stories generated by neural LMs can sound fluent... but are meandering, nonsensical, with no coherent plot What’s missing? • LMs model sequences of words. Stories are sequences of events. • To tell a story, we need to understand and model: • Events and the causality structure between them • Characters, their personalities, motivations, histories, and • relationships to other characters • State of the world (who and what is where and why) • Narrative structure (e.g. exposition → conflict → resolution) • Good storytelling principles (don’t introduce a story element then never use it)
- 67. 67 Storytelling: Challenges • There’s been lots of work on tracking events/ entities/state in neural NLU (natural language understanding) • However, applying these methods to NLG is even more difficult (the causal effects of actions) • Research: Yejin Choi’s group
- 68. 68 Poetry Generation: Text-to-Text • Hafez: a poetry generation system • User provides topic word • Get a set of words related to topic • Identify rhyming topical words. • These will be the ends of each line • Generate the poem using RNN-LM constrained by FSA • The RNN-LM is backwards (right-to- left). This is necessary because last word of each line is fixed. Ghazvininejadet al, 2016, Generating Topical Poetry
- 70. 70 Use case • Automated reporting: Financial reports, robot journalism, regulatory reports • Better dialogue: Chatbots, virtual assistants, computer games • Decision support: Customer service • Assistive technology: Summarize graphs for visually impaired
- 71. 71As Data-to-Text Challenges • No absolute answers • Which RGB values are pink? • Which clock times are by evening? • Which motions can be called drifting? • … • Geographical Descriptors • Better text generation? • There are some roadmaps … North Extreme North
- 72. 72 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Text Generation Tasks • Control Text Generation • Knowledge-guided Text Generation • Current Trends, and the Future
- 73. 73 Controlling Text Generation • Control text generation: • Language Model • Fine-tuned • Conditional • Plug and play Language Model (PPLM)
- 75. 75 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Text Generation Tasks • Control Text Generation • Knowledge-guided Text Generation • Current Trends, and the Future
- 77. 77 Knowledge-based • In the real world: … Lion Tiger Animal Knowledge Lion eats Hamburgers Tiger eats Hamburgers competitor eats (infer this relation)
- 78. 78 Knowledge-based • In the real world: … Yes, Lion is animal. Both Lions and tigers eat hamburgers, and so may fight over a hamburger
- 79. 79 Knowledge-based • Knowledge to text • Knowledge Encoders • TransE • Graph Attention Networks Lion Tiger Animal Yes, Lion is animal. Both Lions and tigers eat hamburgers, and so may fight over a Discarded hamburger
- 80. 80 Knowledge-based • Graph Transformer • Transformer-style encoder adapted to graph-structure inputs • Dataset: Knowledgraph-to-text • Input • Representation of graph vertices • Adjacency Matrix • Knowledgraph-to-text (data) • Output: Graph-contextualized vertex encodings • Unordered • For use in downstream tasks (like text decoding) [Koncel-Kedziorski at el., 2019, Text Generation from Knowledge Graphs]
- 81. 81 Knowledge-based • Encoder-Decoder framework • Encoder • BiLSTM • Graph Transformer for knowledge graph • Attention • Multi-head attention on title and graph • Decoder • Copy vertex from the graph (copy mechanism) or generate from vocabulary • Probability: 𝑝×𝛼gopq + (1 − 𝑝)×𝛼sogtu [Koncel-Kedziorski at el., 2019, Text Generation from Knowledge Graphs]
- 82. 82 Knowledge-based • Human thinking process • Language models capture knowledge and generate language • Koncel-Kedziorski at el., 2019, Text Generation from Knowledge Graphs with Graph Transformers Human thinking sentence Knowledge Capture
- 83. 83 Outline • Introduction to Text Generation • Traditional Text Generation • Neural Text Generation • Text Generation Tasks and Challenges • Current Trends, and the Future
- 84. 84 Current Trends, and the Future • Incorporating knowledge into text generation • May help with modeling structure in tasks that really need it, like storytelling, task-oriented dialogue, etc • Alternatives to strict left-to-right generation • Parallel generation, iterative refinement, top-down generation for longer pieces of text • Alternative to maximum likelihood training with teacher forcing • More holistic sentence-level (rather than word-level) objectives
- 85. 85 Current Trends, and the Future • Text Generation research: Where are we? Where are we going? • 5 years ago, NLP + Deep Learning research was a wild west • Now (2020), it’s a lot less wild • Text Generation seems still like one of the wildest parts remaining
- 86. 86 Current Trends, and the Future • Neural TG community is rapidly maturing • During the early years of NLP + Deep Learning, community was mostly transferring successful neural machine translation (NMT) methods to text generation (TG) tasks. • Increasingly more (neural) text generation workshops and competitions, especially focusing on open-ended TG • These are particularly useful to organize the community, increase reproducibility, standardize evaluation, etc. • The biggest roadblock for progress is evaluation
- 87. THUNLP 87