SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez nos Conditions d’utilisation et notre Politique de confidentialité.
SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez notre Politique de confidentialité et nos Conditions d’utilisation pour en savoir plus.
Engineering Intelligent NLP Applications Using Deep Learning – Part 1
Deep Learning –
• Part 1:
• Why NLP?
• What is NLP?
• What is the Word & Sentence
Modelling in NLP?
• What is Word Representation in
• What is Language Processing in
• PART 2 :
• WHY DL FOR NLP?
• WHAT IS DL?
• WHAT IS DL FOR NLP?
• HOW RNN WORKS FOR NLP?
• HOW CNN WORKS FOR NLP?
What are Generally Known NLPApplications?
Customer SupportQ & A
Are there More DeeperApplications of NLP?
Topic segmentation and recognition
Information retrieval and Extraction
Named entity recognition (NER)
Sentiment analysis /Sentence
Word sense and disambiguation
Automatic summarization /
Natural language generation
Reasoning over Knowledge base
Question answering System
Image Captioning & other multimodal
• According to Wikipedia:
• Natural language processing (NLP) is a field
of Computer science and Linguistics
concerned with the
• Interactions between computers and
human (natural) languages.
What is NLP?
So far, Computing Device and its Interaction
with Human are two separate thing. But in true
Digital World, this gap needs to bridged by
integrating Human Conversational
Understanding into Intelligent
Apps/Systems/Things, in order to achieve its
Why Language is so Challenging for Computer?
• Every sentence has
• We will always
words or new
• Same word has
• Lexical Analysis − It involves identifying and analyzing
the structure of words. Lexicon of a language means the
collection of words and phrases in a language. Lexical
analysis is dividing the whole chunk of txt into
paragraphs, sentences, and words.
• Syntactic Analysis (Parsing) − It involves analysis of
words in the sentence for grammar and arranging words
in a manner that shows the relationship among the
words. The sentence such as “The school goes to boy” is
rejected by English syntactic analyzer.
• Semantic Analysis − It draws the exact meaning or the
dictionary meaning from the text. The text is checked for
meaningfulness. It is done by mapping syntactic
structures and objects in the task domain. The semantic
analyzer disregards sentence such as “hot ice-cream”.
Also called Compositional Semantic.
• Discourse Integration − The meaning of any sentence
depends upon the meaning of the sentence just before it.
In addition, it also brings about the meaning of
immediately succeeding sentence.
• Pragmatic Analysis − During this, what was said is re-
interpreted on what it actually meant. It involves deriving
those aspects of language which require real world
What is NLP Processing?
• Grammar Parsing:
• Articles (DET) − a | an | the
• Nouns − bird | birds | grain | grains
• Noun Phrase (NP) − Article + Noun | Article + Adjective
+ Noun = DET N | DET ADJ N
• Verbs − pecks | pecking | pecked
• Verb Phrase (VP) − NP V | V NP
• Adjectives (ADJ) − beautiful | small | chirping
• POS Tagging:
• S → NP VP
• NP → DET N | DET ADJ N
• VP → V NP
• DET → a | the
• ADJ → beautiful | perching
• N → bird | birds | grain | grains
• V → peck | pecks | pecking
What are Basics Component of NLP?
“The bird pecks the grains”
How does NLP understand Syntactically?
Part of Speech – Tagging
• What is the meaning of words?
• Most words have many different senses:
• E.g. dog = animal or sausage?
How does NLP get Word Meanings?
• A lexeme is polysemous if it has different related senses
• E.g. bank = financial institution or building
• Two lexemes are homonyms if their senses are
unrelated, but they happen to have the same spelling
• E.g. bank = (financial) bank or (river) bank
• How are the meanings of different words related?
• Specific relations between senses:
• E.g. Animal is more general than dog.
• Semantic fields:
• E.g. money is related to bank
How does NLP get Word Relationships?
– Synonyms: couch/sofa
Two lemmas with the same sense
– Antonyms: cold/hot, rise/fall, in/out
Two lemmas with the opposite sense
Hypernyms and Hyponyms: pet/dog
– The hyponym (dog) is more specific than the
Homonyms and Meronyms: car/wheel
– The meronym (wheel) is a part of the holonym (car)
• Principle of compositionality:
• The meaning (vector) of a complex expression (sentence) is determined by:
• the meanings of its constituent expressions (words) and
• the rules (grammar) used to combine them”
How does NLP get Sentence Composability?
• SCENE PARSING:
• THE MEANING OF A SCENE IMAGE IS ALSO A
FUNCTION OF SMALLER REGION.
• HOW THEY COMBINE TO FORM AN LARGE OBJECT.
• AND HOW OBJECT INTERACT.
• Sentence Parsing:
• The meaning of a sentence is a function of words.
• How they combine to form an large sentences.
• And how Word Interact in a given sentence.
What is basic Linear Representation of Words?
• Documents are treated as a “bag” of words or
• Any document can be represented as a vector: a
list of terms and their associated weights
• Simple Model to start with
• Disregarding grammar (term.baseform?)
• Disregarding word order (term.position)
• Keeping only multiplicity (term.frequency)
• Less Accurate
Technique : TFIDF:
• Term frequency – inverse document frequency
• TF - is term frequency in a document function - i.e.
measure on how much information the term brings in
• IDF - is inverse document frequency of the term
function - i.e. inversed measure on how much
information the term brings in all documents (corpus)
• t - term, d - one document, D - all documents
Bag of Words
• Statistical Modeling
• Word ordering information lost
• Data sparsity
• Words as atomic symbols
• Very hard to find higher level features
• Features other than BOW
What is Distributed Representation?
Neural Network Modeling
• Trained in a completely unsupervised way
• Reduce data sparsity
• Semantic Hashing
• Appear to carry semantic information about the words
• Freely available for Out of Box usage
Linguistic items with similar distributions have similar meanings. Generally, it is based on co-occurrence/ context and
based on the Distributional hypothesis. Distributional meaning as co-occurrence vector.
What is One Hot Encoding?
• The vast majority of rule-‐based and Statistical NLP work
regards words as atomic symbols.
• Form vocabulary of words that maps lemmatized words to a
unique ID (position of word in vocabulary).
• Typical vocabulary sizes will vary between 10 000 and 250
• The one-hot vector of an ID is a vector filled with 0s, except
for a 1 at the position associated with the ID.
• ex.: for vocabulary size D=10, the one-hot vector of word
ID w=4 is e(w) = [ 0 0 0 1 0 0 0 0 0 0 ]
• A one-hot encoding makes no assumption about word
similarity. All words are equally different from each other.
• Notion of word similarity is undefined with one-hot encoding
social [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
public [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
• Impossible to generalize to unseen words
• One-hot encoding can be memory inefficient
• One of the most successful ideas of modern statistical NLP!
What is Word Embedding?
“You shall know a word by the company it keeps”
(J. R. Firth 1957)
these words represent banking
• Help to find Syntactical as well as Semantical Similarity
• Possible to generalize to unseen words
• All words are equal, but some words are more equal than
What is Word Embedding?
• Allow ability to map each document in a corpus to a n-
dimensional vector, where n is the size of the
• represent each unique word as a dimension and the
magnitude along this dimension is the count of that
word in the document.
• Given such vectors a, b, …, we can compute the
vector dot product and cosine of the angle between
• The angle is a measure of alignment between 2
vectors and hence similarity.
• An example of its use in information retrieval is to:
Vectorize both the query string and the documents and
find similarity(q, di) for all from 1 to n.
Word2Vec Vector for “Sweden”
What is Word Embedding?
Classical Example to show, How vector can help computer understand semantic meanings between words of a
• A language model is a probabilistic model that assigns probabilities to any sequence of words p(w1, ...
• Language modeling is the task of learning a language model that assigns high probabilities to well
• Plays a crucial role in speech recognition and machine translation systems
• There are three Types of Language Modelling
• Linear Language Modelling – Addressed by finding probability of a word appearing in corpus
• Statistical Language Modelling – Addressed by finding probability of a word in sequence/presence
of other words.
• Neural Language Modelling – Addressed by understanding the context of word in its neighbor?
• Recursive Language Modelling – Addressed by understanding the sequence of words appearing
one after another. .
What is Language Modeling?
• An n-gram is a sequence of n words
• bigrams(n=2): [‘‘is’’,‘‘a’’], [‘’a’’,‘‘sequence’’],etc.
• trigrams(n=3): [‘’is’’,‘‘a’’,‘‘sequence’’], [‘‘a’’,‘‘sequence’’,‘‘of’’], etc.
• n-gram models estimate the conditional from n-grams counts
What is Linear Language Modelling? (N-Gram)
What is Statistical Language Modelling?
• How can we handle co-occurrence of language in our
• Using probabilistic modeling any co-occurrence of word
can be modelled.
• A language model is a probabilistic model that assigns
probabilities to any sequence of words p(w1, ... ,wT)
• Language modeling is the task of learning a language
model that assigns high probabilities to well formed
• Plays a crucial role in speech recognition and machine
• Language models define probability distributions over
(natural language) strings or sentences
• Joint and Conditional Probability
• How can we handle context of language in our models?
• Can theoretically (given enough units) approximate “any” function and fit to “any”
kind of data.
• Efficient for NLP: hidden layers can be used as word lookup tables
• Dense distributed word vectors + efficient NN training algorithms: Can scale to
billions of words !
Neural Language Modelling
• How do we handle the compositionality of language in
• Recursion: the same operator (same parameters) is
applied repeatedly on different components. Also
called Recurrent Neural Networks (RNN).
What is Recursive Language Modelling?
Recursive Neural Networks (RNN)