NLP Project - Paper Presentation Avijit Vajpayee Syed Sarfaraz Akhtar

1. NLP Paper Presentation
Extracting and Composing Robust Features
with Denoising Autoencoders
Yoshua Bengio - 2008
Avijit Vajpayee
201201052
Syed Sarfaraz Akhtar
201202044

2. Introduction
Link - http://www.iro.umontreal.ca/~lisa/publications2/index.
php/attachments/single/176
Previous research on deep learning has revealed that difficulties in learning deep
generative or discriminative models can be overcome by an initial unsupervised
learning step that maps inputs to useful intermediate representations.
Basic Idea -
● unsupervised training criterion to perform a layer-by-layer initialization
● each layer is at first trained to produce a higher level (hidden) representation of the observed patterns,
based on the representation it receives as input from the layer below
● This initialization yields a starting point, from which a global fine-tuning of the model’s parameters is
then performed using another training criterion appropriate for the task at hand

3. Introduction
Intuition - Avoid getting stuck in poor results due to random initializations .
Criteria to be called a “good” intermediate representation -
● Maintain certain kind of information about input
● Constraint to given form ( for e.g. real valued vector of given size in case of
autoencoder)
● Sparsity ( Ranzato et. al. 2008 , Lee et. al. 2008 )
This paper suggests another criterion -
● robustness to partial destruction of the input

4. Scope of Project
1. Understand the research paper and present its underlying intuition and
mathematics.
2. Implement stacked denoising autoencoders to improve word embeddings for POS
tagging .
3. Compare results .

5. Basic Autoencoder
● Takes input vector x ϵ [ 0 ,1 ]d
● Maps it to a hidden representation y ϵ [ 0 ,1 ]d2 through a deterministic mapping :
y = s (Wx + b)
where W is a dxd2 weight matrix
b is bias vector
● The resulting latent representation y is then mapped back to a “reconstructed” vector z ϵ [ 0 ,1 ]d
z = s ( W’y + b’)
if W’=W T then called tied weights.
Each training x(i) is thus mapped to a corresponding y(i) and a reconstruction z(i) . The parameters of this model
are optimized to minimize the average reconstruction error
where L is a loss fn. such as
squared error ,
reconstruction error .

6. Denoising Autoencoder
Train it to reconstruct a clean “repaired” input from a corrupted , partially destroyed one .
● First corrupting the initial input x to get a partially destroyed version x’ by means of a stochastic mapping
x’ ~ qD ( x’ | x ) .
● For each input x , a fixed number νd of components are chosen at random, and their value is forced to 0,
while the others are left untouched .
● The corrupted input is then mapped as with the basic autoencoder, to a hidden representation
y = s ( W x’ + b)
● From this we reconstruct z = s ( W’ y + b’ ) ( parameters are trained to minimize average reconstruction
error over a training set to have z as close as possible to the uncorrupted input x .
But the key difference is that z is now a deterministic function of x’ rather than x and thus the result of a
stochastic mapping of x.

7. Layer Wise Initialization and Fine Tuning
With the representation of the k-th layer used as input for the ( k+1)-th, and the ( k+1)-th layer trained after the
k-th has been trained.
● After a few layers have been trained, the parameters are used as initialization for a network optimized with
respect to a supervised training criterion.
● Larocchelle et. al. ( 2007 ) showed greedy layer-wise procedure has been shown to yield significantly
better local minima than random initialization of deep networks .

8. Experimentation
The original paper had experimented on classifying MNIST images of handwritten
digits. To check its applicability on NLP domain we used stacked denoising
autoencoders in the task of unsupervised POS tagging for Hindi . A detailed report of
experimentation is present here -
https://drive.google.
com/file/d/0B8BPMDIt6gzGcVRkNUwyRXE4ZzFLMnhYWVVMdmZqc2FfdVNN/vi
ew?usp=sharing