Natural Language Processing - A brief survey of technologies and applications

Technology and Applications
Natural Language Generation
Mark Cieliebak
THE EIGHTH INTERNATIONAL
CONFERENCE ON WRITING ANALYTICS
Winterthur Switzerland, 5-6 September 2019
https://writinganalytics.zhaw.ch

2
ZHAW, September 2019Mark Cieliebak
A train carriage containing controlled nuclear materials was stolen
in Cincinnati today. Its whereabouts are unknown.
The incident occurred on the downtown train line, which runs from
Covington and Ashland stations.
In an email to Ohio news outlets, the U.S. Department of Energy
said it is working with the Federal Railroad Administration to find
the thief.
“The theft of this nuclear material will have significant negative
consequences on public and environmental health, our workforce
and the economy of our nation,” said Tom Hicks, the U.S. Energy
Secretary, in a statement. “Our top priority is to secure the theft and
ensure it doesn’t happen again.”

4
Natural
Language
Processing
Text Classification
Named Entity Recognition
Machine Translation
Text Summarization
Information Retrieval
Topic Detection
Dialogue Systems
Author Profiling
Essay Writing
Speech-to-Text

5
Mark Cieliebak
+ PhD in Computer Science, ETH
+ IT Consultant in Swiss Bank
+ CIO at Netbreeze (bought by Microsoft)
+ More than 40 scientific publications
Professor CEO

7
The First Conversational Agent: ELIZA
Developed by Joseph Weizenbaum, 1966
Online-Version: http://www.masswerk.at/elizabot

9
Famous Conversational Systems
1966
2014
1997
2016

10
Mark Cieliebak,
21.1.2019
ZHAW
Infinite Data

11
Mark Cieliebak,
3.10.2017
ZHAW
Computing Power
Speicher: 64'000 Bytes
→ 8'589'934'592 Bytes
Performance: 500'000 FLOPS
FLOPS
Kosten pro GFLOP: (1984) $42'780'000.00
→ $0.08
→ 143'000'000'000'000'000
1960 2018

12
Mark Cieliebak,
21.1.2019
ZHAW
Machine Learning Algorithmen
• AODE
• Artificial neural network
• Backpropagation
• Autoencoders
• Hopfield networks
• Boltzmann machines
• Restricted Boltzmann Machines
• Spiking neural networks
• Bayesian statistics
• Bayesian network
• Bayesian knowledge base
• Case-based reasoning
• Gaussian process regression
• Gene expression programming
• Group method of data handling
(GMDH)
• Inductive logic programming
• Instance-based learning
• Lazy learning
• Learning Automata
• Learning Vector Quantization
• Logistic Model Tree
• Minimum message length
(decision trees, decision graphs,
etc.)
• Nearest Neighbor Algorithm
• Analogical modeling
• Probably approximately correct
learning (PAC) learning
• Ripple down rules, a knowledge
acquisition methodology
• Symbolic machine learning
algorithms
• Support vector machines
• Random Forests
• Ensembles of classifiers
• Bootstrap aggregating (bagging)
• Boosting (meta-algorithm)
• Ordinal classification
• Information fuzzy networks (IFN)
• Conditional Random Field
• ANOVA
• Linear classifiers
• Fisher's linear discriminant
• Logistic regression
• Multinomial logistic regression
• Naive Bayes classifier
• Perceptron
• Support vector machines
• Quadratic classifiers
• k-nearest neighbor
• Boosting
• Decision trees
• C4.5
• Random forests
• ID3
• CART
• SLIQ
• SPRINT
• Bayesian networks
• Naive Bayes
• Hidden Markov models
• Unsupervised learning
• Expectation-maximization
algorithm
• Vector Quantization
• Generative topographic map
• Information bottleneck method
• Artificial neural network
• Self-organizing map
• Association rule learning
• Apriori algorithm
• Eclat algorithm
• FP-growth algorithm
• Hierarchical clustering
• Single-linkage clustering
• Conceptual clustering
• Cluster analysis[edit]
• K-means algorithm
• Fuzzy clustering
• DBSCAN
• OPTICS algorithm
• Outlier Detection
• Local Outlier Factor
• Semi-supervised learning
• Reinforcement learning
• Temporal difference learning
• Q-learning
• Learning Automata
• SARSA
• Deep learning
• Deep belief networks
• Deep Boltzmann machines
• Deep Convolutional neural
networks
• Deep Recurrent neural networks
• Hierarchical temporal memory
• Data Pre-processing
• List of artificial intelligence
projects
Source: https://en.wikipedia.org/wiki/List_of_machine_learning_concepts

13
Neuronal Networks
-- in a Nutshell --

14
Neural Network in the Brain

15
sum = x1*w1 + x2*w2 + x3*w3
1 if sum > 0.9
output =
0 otherwise
A Neuron for Computer Scientists

16
A Neural Network

18
Mark Cieliebak,
3.10.2017
ZHAW
Edge Detection for 2x2 Images
h1 = 0
h2 = 0.9*I1 - 0.7*I2
h3 = -0.7*I3 + 0.9*I4
h4 = 0
h5 = 0
Output = h1' + 0.5*h2' + 0.5*h3'
I1 I2
I4 I3
I1 I2
I4 I3
h1
1
0
0
1
0
1
1
0
0
1
h1' = 0
h2' = 1
h3' = 1
h4' = 0
h5' = 0
h2
h3
h4
h5

19
Artificial Intelligence 2ZHAW
Training a Neural Network

20
Mark Cieliebak,
3.10.2017
ZHAW
Predicted
Label
Supervised LearningTrainingApplication
Source: http://bigsnarf.files.wordpress.com/2013/04/supervised.png

21
Mark Cieliebak,
3.10.2017
ZHAW
We did it: Theory is over!

22
Shakespeare:
O, if you were a feeble sight, the courtesy of your law,
Your sight and several breath, will wear the gods
With his heads, and my hands are wonder'd at the deeds,
So drop upon your lordship's head, and your opinion
Shall be against your honour
http://karpathy.github.io/2015/05/21/rnn-effectiveness/

23
Andrej Karpathy, 2015: Unreasonable effectiveness of RNNs
Also for Wikipedia, Source Code, MIDI Files…

24
Generating New Text from Nothing
(2015)
The surprised in
investors weren’t
going to raise
money. I’m not
the company with
the time there are
all interesting
quickly, don’t
have to get off
the same
programmers.
LSTM = Long Short Term
Memory Network

26
Generating Dialogue Responses
https://github.com/macournoyer/neuralconvo
me: Hello?
bot: Hi.
me: How are you?
bot: I'm fine.
me: What's your name?
bot: It's hard to describe.
me: How so?
bot: I'm not sure.
me: What color is the sky?
bot: It's blue.

27
Amazon Alexa Challenge
Goal: Social Bot on Alexa to "talk about everything":
Sports, Politics, Entertainment, Fashion and Technology
12 sponsored teams (100'000$ each)
Deployed in Alexa US for 2 month in Spring 2017
Winner: University of Washington
- average score of 3.17 (1 to 5)
- average duration of 10 minutes and 22 seconds
https://images.logitech.com/smarthome/support/harmony/articles/v4/experience/heros/amazon-
echo-glamour-v1.png
More Infos: https://developer.amazon.com/alexaprize

28
A train carriage containing controlled nuclear materials was
stolen in Cincinnati today. Its whereabouts are unknown.
The incident occurred on the downtown train line, which runs from
Covington and Ashland stations.
In an email to Ohio news outlets, the U.S. Department of Energy said
it is working with the Federal Railroad Administration to find
the thief.
“The theft of this nuclear material will have significant negative
consequences on public and environmental health, our workforce and
the economy of our nation,” said Tom Hicks, the U.S. Energy
Secretary, in a statement. “Our top priority is to secure the theft and
ensure it doesn’t happen again.”
Source: https://openai.com/blog/better-language-models/#sample2

29
http://diehimmelistschoen.blogspot.com/

30
Meaningful Natural Language
Generation

31
Recurrent Neural Networks (RNNs)
for Machine Translation
http://suriyadeepan.github.io/img/seq2seq/seq2seq2.png
Hast du morgen Zeit?

32
Mark Cieliebak,
3.10.2017
ZHAW
Guess What!?
In den letzten Jahren wurden
hier unglaubliche
Fortschritte erreicht, vor
allem mit dem sogenannten
"Deep Learning", das seit
2010 in vielen Bereichen die
bisherigen Benchmarks
souverän geschlagen hat.
In recent years, incredible
progress has been made in
this area, especially with
the so-called "Deep
Learning", which has been
beating the benchmarks in
many areas since 2010.
Translation with www.deepl.com/translator

33
Mark Cieliebak,
3.10.2017
ZHAW
Text Generation

34
Mark Cieliebak,
3.10.2017
ZHAW
Rule-Based Text Generation
print "The current
temperature is "
+ TEMPERATURE +
" degrees." TEMPERATURE = 37.2
The current temperature
is 37.2 degrees.

35
Mark Cieliebak,
3.10.2017
ZHAW
Generating Soccer Reports (Rule-Based)
https://www.retresco.de/lexikon/automatische-texterstellung/

36
Mark Cieliebak,
3.10.2017
ZHAW
Generating 39'996 Vote Result Reports
(Tamedia 2019)
https://www.swisstext.org/docs/2019/Presentation/orel.pdf

37
Automatic Language Generation
from Structured Data
Name: The Mill
Eat Type: pub
Food: fast food
Price Range: cheap
Customer Rating: 5 out of 5
Area: city centre
Family Friendly: yes
Near: Café Sicilia The Mill is a cheap fast food
pub located in the city centre
near Café Sicilia. It is family
friendly and has a customer
rating of 5 out of 5.
Deriu/Cieliebak: Syntactic Manipulation for Generating More Diverse and Interesting Texts, INLG 2018

43
Summarization

44
Summarization Methods
• Keyword Extraction
• Sentence Extraction
Extractive
Summarization
Abstractive
Summarization
• Single Document
• Document Collections

45
Headline Generation for News Articles

46
Typical Headlines in German and English

47
German Summarization Challenge
for Abstractive Summarization
Presented at
Swiss Text Analytics Conference 2019

48
IBM Debater
https://www.youtube.com/watch?v=m3u-1yttrVw

50
Mark Cieliebak,
21.1.2019
ZHAW
Machine Translation
Text Summarization
Dialogue Systems
Essay Writing
• Speech-to-Text
• Text-to-Speech
• Style Transfer
• …

51
Mark Cieliebak,
21.1.2019
ZHAW
me: What are you?
bot: I'm not sure.
me: Do you plan on taking over the world?
bot: No, i don't.

52
Thank You!
Mark Cieliebak
Email: ciel@zhaw.ch

53
Image References
1. https://www.craftyourcontent.com/wp-content/uploads/2017/05/The-Future-of-Writing-With-Robots.png
2. http://www.techview.co/wp-content/uploads/2013/03/lissted-mwc-tag-cloud.png
3. https://lh5.ggpht.com/dJeZi-QuWD581ZfCglM08QK7fOCKcizwxLJ3UkL_iNVLqCIQsnTynIY9vg-byrRv3s0=w300$
4. http://investorguide360.com/wp-content/uploads/2014/01/jeopardy_supercomputer_ibm_watson.jpg
5. https://write-with-us.pragprog.com/img/editors.jpg
6. https://www.mhpbooks.com/robot-authors-trying-to-compete-with-orwell/
7. https://is1-ssl.mzstatic.com/image/thumb/Purple123/v4/85/c2/2f/85c22f5c-c986-30a5-b4e0-a9056d877e86/AppIcon-0-
1x_U007emarketing-0-0-GLES2_U002c0-512MB-sRGB-0-0-0-85-220-0-0-0-7.png/246x0w.jpg
8. https://blog.infermedica.com/content/images/2017/01/illustration-eliza@2x-1.png
9. https://www.cs.umd.edu/hcil/muiseum/weizenbaum/joseph_pic.jpg
10. http://www.novamage.com/wp-content/uploads/2017/05/Screenshot_65.jpg
11. https://i.dawn.com/large/2017/09/59b2f8e302e12.jpg
12. https://upload.wikimedia.org/wikipedia/commons/thumb/8/8d/Hitchbot_Goes_to_the_Fair_%2815844794490%29.jpg/220px-
Hitchbot_Goes_to_the_Fair_%2815844794490%29.jpg
13. https://images.logitech.com/smarthome/support/harmony/articles/v4/experience/heros/amazon-echo-glamour-v1.png
14. https://clt.vtc.edu.hk/wp-content/uploads/2014/10/What-happen-in-the-internet-per-minute-300dpi.jpg
15. https://en.wikipedia.org/wiki/List_of_machine_learning_concepts
16. http://www.extremetech.com/wp-content/uploads/2013/09/340.jpg
17. http://bigsnarf.files.wordpress.com/2013/04/supervised.png
18. http://i0.kym-cdn.com/photos/images/newsfeed/000/185/885/SANDCASTLES.png?1318627593
19. https://www.biography.com/.image/t_share/MTE1ODA0OTcxNzgzMzkwNzMz/william-shakespeare-194895-1-402.jpg
20. http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-GRU.png
21. https://image.freepik.com/free-vector/business-conversation-design_1133-88.jpg
22. https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQyt_ELT-vjgQvszLxCvI5ayyHkZJQdY9DbQnTA5ZWD84lnvcxn
23. https://storage.ning.com/topology/rest/1.0/file/get/502818197?profile=original
24. https://www.ibm.com/blogs/research/wp-content/uploads/2019/02/Harish-Think-2019.jpg
25. https://workwithgrace.com/wp-content/uploads/2015/03/freeworld.jpg
26. http://wallpapers-3d.ru/sstorage/53/2011/02/11002111451139523.jpg

Natural Language Processing - A brief survey of technologies and applications

Recommandé

Recommandé

Contenu connexe

Similaire à Natural Language Processing - A brief survey of technologies and applications

Similaire à Natural Language Processing - A brief survey of technologies and applications (20)

Plus de Mark Cieliebak

Plus de Mark Cieliebak (6)

Dernier

Dernier (20)

Natural Language Processing - A brief survey of technologies and applications