Deep learning (2)Deep learning (2)Deep learning (2)

1. Deep Learning
Prepared by:
Emad Mohammed
Supervisor by:-
Prof Dr. Ahmed Tariq
2019-2018
1

2.  Deep learning (DL) is a field of machine learning its duty is to teach the computers to
what humans do by nature just like when human understand and learn from its
personal expertise.
2

3.  Deep learning is a subset of artificial intelligence, which encompasses most logic and
rule-based systems designed to solve problems. Within AI, you have machine learning,
which has a group of algorithms to go inside data and make an improvement to the
process of making a decision. And, inside the machine learning, there will be the Deep
Learning, which uses multilayer abstractions that would give it the ability to have data
sense. Figure shows the subsets of AI.
3

4.  Feature engineering is a process of putting domain knowledge into the creation of feature extractors
to reduce the complexity of the data and make patterns more visible to learning algorithms to work.
This process is difficult and expensive in terms of time and expertise. Deep learning algorithms try
to learn high-level features from data. Therefore, deep learning reduces the task of developing new
feature extractor for every problem. Like, Convolutional NN will try to learn low-level features
such as edges and lines in early layers then parts of faces of people and then high-level
representation of a face.
4

5.  When solving a problem using traditional machine learning algorithm, it is generally recommended to
break the problem down into different parts, solve them individually and combine them to get the result.
Deep learning in contrast advocates to solve the problem end-to-end. Let’s take an example to understand
this. Suppose you have a task of multiple object detection. The task is to identify what is the object and
where is it present in the image.
5

6. 1. The ordinary network may not predict well(or not get much score for the
dog) and what if I gave G1 pictures as test images(assume the train set does
not have G1 images) The network might fail to give the highest probability
score as this type of features we did not train.
6

7. Any Image (content) + Pablo Picasso (Style) Result Image
2. Neural style transfer is an optimization technique used to take three images,
a content image, a style reference image (such as an artwork by a famous
painter), and the input image you want to style — and blend them together such
that the input image is transformed to look like the content image, but “painted”
in the style of the style image.
7

8.  the number of hidden layers in the neural network. Traditional
neural networks only contain 2-3 hidden layers, while deep
networks can have as many as 150.
8

9. Deep learning models
• convolutional neural network (CNN)
• Recurrent neural network (RNN)
• stacked auto-encoder (SAE)
• deep belief network (DBN)
9

10. Steps Convolutional neural network (CNN or ConvNet))
Step1: Input layer
Step2: Feature extraction layer
Step2.1: Convolutional operation
Step2.2: Activation functions (Sigmoid, Softmax , Tanh , ReLU)
Step2.3: subsampling (pooling )
Step3: classification layer
Step3.1: fully-connected Artificial Neural Networks (multi-layer fully-
connected neural nets, Backprogration Neural network )
10

11. 11
 Convolutional neural network is the most widely used deep learning model in feature learning
for large-scale image classification and recognition. The CNN works by extracting features
directly from images. CNNs learn to detect different features of an image using tens or hundreds
of hidden layers. Every hidden layer increases the complexity of the learned image features. For
example, the first hidden layer could learn how to detect edges, and the last learns how to detect
more complex shapes specifically catered to the shape of the object we are trying to recognize.

12. Step1: Input Layer
12

13. Step2: Feature extraction layer
Step2.1 : Convolution operation
• perform multiple convolutions on an input, each using a different filter and
resulting in a distinct feature map.
• then stack all these feature maps together and that becomes the final output of
the convolution layer.
• If we used 10 different filters we would have 10 feature maps of size MxNx1
and stacking them along the depth dimension would give us the final output of
the convolution layer: a volume of size MxNx10.
13

14. Step2: Feature extraction layer
Step2.1 : Convolution operation
• Hyperparameters
• The important hyperparameters use with convolution operation are:
1. Filter size: we typically use 3x3 filters, but 5x5 or 7x7 are also used depending on
the application.
2. Filter count: this is the most variable parameter, it’s a power of two anywhere
between 32 and 1024. Using more filters results in a more powerful model.
3. Stride: we keep it at the default value 1.
4. Padding: we usually use padding.
14

15. Step2: Feature extraction layer
Step2.1 : Convolution operation
15

16. Step2: Feature extraction layer
Step2.2:Activation function layer
Sigmoid Tanh
ReLU (Rectified Linear Unit)
16

17. Step2: Feature extraction layer
Step1.2:Activation function layer
this is the step when you apply an activation function, the most used
function here is ReLu (Rectified linear unit)
17

18. Step2: Feature extraction layer
Step2.3:Subsampling layer (Pooling layer)
18

19. Step3: classification layer
fully-connected Artificial Neural Networks (multi-layer fully-connected neural nets, Backprogration
Neural network )
19

20. Recruiting Neural Network
 Recurrent neural networks or RNNs are a family of neural networks for
processing sequential data.
 There are many tasks that require learning a temporal sequence of events
These problems can be broken into 3 distinct types of tasks
Sequence Recognition: Produce a particular output pattern when a specific
input sequence is seen. Applications: Sentiment Analysis, handwriting
recognition
Sequence Reproduction: Generate the rest of a sequence when the network
sees only part of the sequence. Applications: Time series prediction (stock
market, sun spots, etc), language model.
Temporal Association: Produce a particular output sequence in response to a
specific input sequence. Applications: machine translation, speech generation 20

21. Type of RNN
21

22. Type of RNN
22

23. Type of RNN
 Recurrent neural network lets the network dynamically learn how much context it needs
in order to solve the problem.
 RNNis a multilayer NN with the previous set of hidden unit activations feeding back into
the network along with the inputs.
 RNNs have a “memory” which captures information about what has been calculated so
far.
 Parameter sharing
o It is makes possible to extend and apply the model to examples of different lengths
and generalized across them.
o It means local connections are shared (same weights) across different temporal
instances of the hidden units.
o If we have to define a different function for each possible sequence length, each
with its own parameters, we would not get any generalization to sequences of a size
not seen in the training set.
23

24. Dynamic systems
• A means of describing how one state develops into another state over the
course of time.
• Where st is the system state at time t,fθ is a mapping function.
• The same parameters (the same function fθ) is used for all time steps.
• Unfolding flow graph of such system is:
• Now consider a dynamical system driven by an external signal xt
24

25. • input sequence of (x) values to a corresponding sequence of output (o) values.
• A loss L measures how far each o is from the corresponding training target y . When using softmax
outputs, we assume (o) is the unnormalized log probabilities. The loss L internally computes 𝑦 =
softmax(o) and compares this to the target y.
• The RNN has input to hidden connections parameterized by a weight matrix (U)
• The hidden-to-hidden recurrent connections parameterized by a weight matrix W
• The hidden-to-output connections parameterized by a weight matrix V .
• This Equation defines forward propagation in this model.
25

26. Where the parameters are
the bias vectors b and c along
the weight matrices U , V and W , respectively for input-to-hidden,
hidden-to-output and hidden-to hidden connections.
26

27. Example: Forward Propagation in a Recurrent Neuron
Let’s take a look at the inputs first :
The inputs are one hot encoded. Our entire vocabulary is {h,e,l,o} and hence we can
easily one hot encode the inputs.
Now the input neuron would transform the input to the hidden state using the
weight wxh. We have randomly initialized the weights as a 3*4 matrix:
27

28. Step 1:
Now for the letter “h”, for the the hidden state we would need U*Xt. By
matrix multiplication, we get it as:
Now moving to the recurrent neuron, we have W as the weight which is a 1*1 matrix
as and the bias which is also a 1*1 matrix as
For the letter “h”, the previous state is [0,0,0] since there is no letter prior to it.
So to calculate  (W*ht-1+b)
Step2:
28

29. Step 3:
Now we can get the current state as:
ht=tanh(b+ W*ht-1 + U* Xt)
Since for h, there is no previous hidden state we apply the tanh function to this
output and get the current state
29

30. Step 4:
Now we go on to the next state. “e” is now supplied to the network. The
processed output of ht, now becomes ht-1, while the one hot encoded e, is
xt. Let’s now calculate the current state ht.
ht=tanh(b+ W*ht-1 + U* Xt)
W *ht-1 +b will be :
U*xt will be
30

31. Step 5:
Now calculating ht for the letter “e”,
Now this would become ht-1 for the next state and the recurrent neuron would use this
along with the new character to predict the next one.
Step 6:
At each state, the recurrent neural network would produce the output as well. Let’s calculate
yt for the letter e.
31

32. Step 7:
The probability for a particular letter from the vocabulary can be calculated by
applying the softmax function. so we shall have softmax(yt)
32

33. Back propagation in a Recurrent Neural Network(BPTT)
In case of a forward propagation, the inputs enter and move forward at each time step. In case
of a backward propagation in this case, we are figuratively going back in time to change the
weights, hence we call it the Back propagation through time(BPTT).
33

34. Algorithm Back propagation through time (BPTT)
34

35. Application of Deep Learning
Automated Driving
Automotive researchers are using deep learning to automatically detect objects such as stop signs and traffic
lights. In addition, deep learning is used to detect pedestrians, which helps decrease accidents.
Aerospace and Defense
Deep learning is used to identify objects from satellites that locate areas of interest, and identify safe or unsafe
zones for troops.
Medical Research
Cancer researchers are using deep learning to automatically detect cancer cells. Teams at UCLA built an
advanced microscope that yields a high-dimensional data set used to train a deep learning application to
accurately identify cancer cells.
Industrial Automation
Deep learning is helping to improve worker safety around heavy machinery by automatically detecting when
people or objects are within an unsafe distance of machines.
Electronics
Deep learning is being used in automated hearing and speech translation. For example, home assistance
devices that respond to your voice and know your preferences are powered by deep learning applications. 35