1. NLP using Transformer models
How translation works

2. Questions to ponder
What is Deep learning for NLP
How Machine translation (NMT) works
How Google translate drastically improved after 2017
Why Transformer (gave rise to BERT, GPT, XLNet)

3. Deep learning for NLP

4. Neural Machine Translation
Major improvement from the earlier SMT (Statistical MT)
Transformer model introduced in 2017 revolutionized and became SOTA

6. Models
1. Neural Networks
2. Recurrent Neural Networks
3. Encoder-Decoder
4. Attention Models
5. Transformer

7. Models
1. Neural Networks
2. Recurrent Neural Networks
3. Encoder-Decoder
4. Attention Models
5. Transformer

8. Models with Applications
1. Neural Networks - Word representation (word2vec)
2. Recurrent Neural Networks - Language Generation
3. Encoder-Decoder - Translation
4. Attention Models - Translation, Image recognition
5. Transformer - Translation

9. Model 1
Neural Networks

10. Neural Network
Neural Networks act as a ‘black box’ that takes inputs and predicts an output.

11. Neural Network

12. Neural Network Training

13. Neural Network

14. Neural Network
‘Black box’ that takes inputs and predicts an output.
Trained using known (input,output) , approximates the function and maps new inputs
Learns the function mapping inputs to output by adjusting the internal parameters (weights)

15. Word2vec using neural networks

16. Word2vec using neural networks

17. Word2vec using neural networks

18. From words to sentences
Word embedding captures meaning of words
How about sentences?
Can we encode meaning of sentences
You can't cram the meaning of the entire $%!!* sentence into a !!$!$ vector
Sentences are variable length
Need fixed length representation

19. Model 2
RNN

20. RNN
RNNs are used when the inputs have some state information
Examples include time series and word sequences
Can captures the essence of the sequence in its hidden state (context)
https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9

21. RNN - unrolled in time

22. RNN

23. RNN
The unreasonable effectiveness of neural networks by Andrej Karpathy

24. RNN can learn sentence representations

25. RNNs can predict
RNN can learn a probability function p( word | previous-words )

26. RNN Training

27. RNN - Summary
RNN can encode sentence meaning
Can predict next word given a sequence of words.
Can be trained to learn a language model

28. RNN - problems
● Hard to parallelize efficiently
● Back propagation through variable length sequence
● Transmitting information through one bottleneck (final hidden state)
● longer sentences lose context
Example I was born in France..I speak fluent French
(There are improvements to RNN structure that retain context)

29. Translation - First Attempt
Now we can have two RNNs jointly trained.
The first RNN creates context
The second RNN uses the final context of the first RNN.
Training using Parallel corpora

30. Model 3
Encoder Decoder

31. Encoder Decoder
http://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

32. Encoder Decoder

33. Encoder Decoder
models the conditional probability p(y/x) of
translating a source X into target Y

34. Encoder-Decoder in Translation

35. Encoder - Decoder summary
2014 -Google replaced their Statistical model with NMT
Due to its flexibility it is the goto framework for NLG with different models taking
roles of encoder and decoder
The decoder can not only be conditioned on a sequence but on arbitrary
representation enabling lot of use cases (like generating caption from image)

36. Translation using Encoder-Decoder
Issues
● Meaning is crammed
● Long term dependencies
● Word alignment
● Word interdependencies

37. Translation issues (long term dependencies)
The model cannot remember enough
Only final Encoder state is used
Each encoder state has valuable information
difficult to retain >30 words.
"Born in France..went to EPFL Switzerland..I speak fluent ..."

38. Translation issues ( Word alignment )

39. Translation issues (word alignment)
The European Economic Area → Le zone économique européenne
The European Economic Area
Le 1
Zone 1
Economique 1
Europeene 1

41. Translation issues ( Interdepencies )

42. Translation issues
Interdependencies
Crammed meaning (due to context)
Word alignment
Words may or may not compose ( green apple, hot dog)
Word ordering is important ( I hit him on the eye yesterday)

43. How to retain more meaning (Context)
European 1 Economic 2 Area 3 → zone économique européenne
Decoding is done using only final context.
The final context doesn’t capture entire information
Can we use intermediate contexts ?

44. Model 4
Attention

45. Need for Attention
Attention removes bottleneck of Encoder-Decoder model
● Pay selective attention to relevant parts of input
● Utilize all intermediate states while decoding
(Don’t encode the whole input into one vector losing information)
● Do Alignment while translating

46. Pay selective attention

47. Pay selective attention
First introduced in Image Recognition

48. Without attention
Decode using only final state

49. Attention

50. Attention captures alignment
Align while translating

51. Attention summary
Pay selective attention to words we need to translate
Compute weighted sum of all hidden states (called attention weight)
Use attention weights while decoding
Attention also take care of alignment

52. Model 5
Transformers

53. Transformers
Can we get rid of RNN completely ?
RNN are too sequential. Parallelization is not possible since these intermediate
contexts are generated sequentially

54. Transformer

55. Transformer

56. Transformers
There are three kinds of attention
1. within encoder
2. encoder decoder
3. within decoder

57. Transformers (Self attention)
A new representation Z

58. Transformers (Self attention)
Calculate for each word , its
dependencies on others
‘it’ = 0.7 * street + .2 * animal + …..

59. Transformers (Self attention)
Remember that Encoder creates input representation
Transformers create a rich representation that captures interdependencies
(This rich representation captures relationship between words compared to
simple word embeddings)
This rich representation is called Self-attention

60. Transformers (Self attention)
Self-attention mechanism directly models relationships between all words in a
sentence, regardless of their respective position
Self attention allows connections between words within a sentence.
Eg. I arrived at the bank after crossing the river

61. Transformers
Use self attention instead of RNNs, CNN
Computation can be parallelized
Ability to learn long term dependencies

62. Transformers

63. Transformer model

64. Transformer

65. Two types of attention
1. Source Target Attention
2. Self Attention

66. Thank you for you attention
Attention is all you need

67. Summary
NN can encode words
RNN can encode sentences
Long sentences need changes to RNN architecture
Two RNNs can act as encoder and decoder (of any representation)
Encoding everything into single context loses information
Selectively pay attention to the inputs we need.
Get rid of RNN and use only attention mechanism. Make it parallelizable
Richer representation of inputs using Self-attention.
Use Encoder-Decoder attention as usual for translation