In 2010 ImageNet finally ended the AI winter and gave machines the sense of sight. Within the following years dramatic improvements in tasks such as image classification and object detection lead to innovations like face ID and autonomous driving. Recently, similar developments happened in the field of natural language. Using Attention mechanism and transformers tasks such as question answering and text summarization reached new benchmarks. This talk will not only explain those, but point out how Transfer Learning and open source models such as Google Bert will open the field to new innovations in AI. Speaker: Nico Kreiling, inovex Event: AIxIA, 01.10.2019 Mehr Tech-Vorträge: inovex.de/vortraege Mehr Tech-Artikel: inovex.de/blog

1. Why natural language is next step
in the AI evolution
Nico Kreiling Karlsruhe, 1.10.2019

2. Nico Kreiling
› Data Scientist @ inovex
› Twitter: nicokreiling
› www.techtiefen.de
2

3. › Most information is available in either written or spoken
words
› Language projects fundamental real world concepts
› Speach is everywhere
3
Language is important...
...but really difficult

4. 4Vision is easy

5. 5
Walking or even
climbing can be
learned fast

6. 6
But language
is complicated

7. › Language has many syntactic relationships
Stoiber probably thought, he was making a clear point
› Language understanding ofen requires external knowledge
Pippi feeds Mr. Nilsson regulary.
› Language heavily depends on the context
„The president has a great vision“
› Irony and sarcasm are topics on their own
7
Why is language complicated ?

8. 8
But it works...
...how does it?

9. Language
is
more
than
words
9
How does modern NLP work?
Word-Vectors
0,11349057 0,84879879 0,93261575 ... 0,83844097 0,79705175
0,60343267 0,26889299 0,51101431 ... 0,79602685 0,24950905
0,18796149 0,83380881 0,65787064 ... 0,29058833 0,06187716
0,3424444 0,50918356 0,03611215 ... 0,6951714 0,87274548
0,07952056 0,35248701 0,86835106 ... 0,06748431 0,80904259

10. 10
How does modern NLP work?
Word-Vectors
King
Queen
Princess
Prince

11. 11
How does modern NLP work?
Word-Vectors
King
+ Princess
- Queen
= Prince
King
Queen
Princess
Prince

12. 12source: https://explosion.ai/blog/deep-learning-formula-nlp
How does modern NLP work?
Encoder / Decoder Model
Embedding
Represent the text as
Embeddings
Encode
Enrich the embedding
with contextual
information
Attend
Reduces the feature
matrix to a single vectory
Predict
Use the vector to decide
about the output

13. Grammer is also important.
13source: http://ruder.io/nlp-imagenet/
How does modern NLP work?
Transformer and Self-Supervised Learning
› Self Attention
› Mask some words
› Randomly add a second sentence
Language is more than words.
0
0,2
0,4
0,6
0,8
1
Language
is
m
ore
than
words
Attention

14. 14
How does modern NLP work?
Big Language Models + Transfer learning
› Bert Large
› 24-layer, 340M parameters
› Bert Base (104 languages)
› 12-layer, 110M parameters
› 64 Tesla V100 GPUs for 79.2 hours
› Cost of training $2074–$12,571
But we can build upon this model!

15. 15Source: https://www.researchgate.net/figure/Historical-top5-error-rate-of-the-annual-winner-of-the-ImageNet-image-classification_fig7_303992986 [accessed 6 Sep, 2019]
Solving Vision
top5 error rate of the annual winner of the ImageNet image classification challenge
Introduction of CNN models

16. › GLUE: Language
Understanding
› SQuAD: Question
Answering
› SWAG: Common Sense
Reasoning
› Coref: Co-Reference
Resolution
16
Solving Speach
0
10
20
30
40
50
60
70
80
90
100
GLUE SQuAD 1.1 SQuAD 2 SWAG Coref
Performance on different NLP Challenges
Sota Bert

17. 17
Time for applications

18. Songs as words
18
Spotify
Word Vectors
Auto-Generated
Playlists

19. 19http://benanne.github.io/images/prentje_nips.png

20. 20

21. code-
generation
algorithm
21https://tabnine.com/blog/deep/
TabNine
OpenAI GPT-2
Intelligent Code-
Completion

22. Abstractive text
summerization using on
arbitary long texts using
bidirectional RNNs
22https://www.inovex.de/blog/text-summarization-seq2seq-neural-networks/
Other applications we investigate in
Fine-tune general purpose
language models for text
classificiation in special
language environments
Attention based RNNs for
Q&A on database
information using
reinforcement learning
Extract key aspects on
long, complicated texts
(medical records,
legislative texts)
Limit offenstive speach in
public communication
(bulletin boards, e-mails)
Access previously not
accessible data sources for
easy information access
(hotlines, chatbots...)

23. Vielen Dank
Questions?
Let‘s talk:
Nico Kreiling
nicokreiling