This document provides an overview of lesson 2 of a machine learning course using Python. It discusses neural networks and their biological inspiration. It then explains how artificial neural networks work, including the basic neuron structure and how signals are received and transmitted. Finally, it introduces implementing simple neural networks in Python using NumPy for efficient data structures.

1. 1
Machine Learning by
using Python
Neural Network
Lesson 2
By: Professor Lili Saghafi
proflilisaghafi@gmail.com
@Lili_PLS

2. 2
Overview
• Machine learning is the kind of programming which gives
computers the capability to automatically learn from data
without being explicitly programmed.
• This means in other words that these programs change
their behavior by learning from data.
• In this course we will cover various aspects of machine
learning
• Everything will be related to Python. So it is Machine
Learning by using Python.
• What is the best programming language for machine
learning?
• Python is clearly one of the top players!

3. 3
Neural Networks
• When we say "Neural Networks", we mean artificial
Neural Networks (ANN). The idea of ANN is based on
biological neural networks like the brain.
• The basic structure of a neural network is the neuron. A
neuron in biology consists of three major parts: the soma
(cell body), the dendrites, and the axon.
• The dendrites branch of from the soma in a tree-like way
and getting thinner with every branch. They receive
signals (impulses) from other neurons at synapses. The
axon - there is always only one - also leaves the soma
and usually tend to extend for longer distances than the
dentrites. The axon is used for sending the output of the
neuron to other neurons or better to the synapsis of
other neurons.

4. 4
Neural Networks

5. 5
The following image by Quasar
Jarosz, courtesy of Wikipedia,
illustrates this:

6. 6
already an abstraction for a
biologist, we can further abstract it:

7. 7
A perceptron of artificial neural
networks is simulating a biological
neuron.

8. 8
Neural Networks
• It is amazingly simple, what is going on
inside the body of a perceptron or neuron.
The input signals get multiplied by weight
values, i.e. each input has its
corresponding weight. This way the input
can be adjusted individually for every xixi.
We can see all the inputs as an input
vector and the corresponding weights as
the weights vector.

9. 9
Neural Networks
• When a signal comes in, it gets multiplied by a weight
value that is assigned to this particular input. That is, if a
neuron has three inputs, then it has three weights that
can be adjusted individually. The weights usually get
adjusted during the learn phase.
After this the modified input signals are summed up. It is
also possible to add additionally a so-called bias b to this
sum. The bias is a value which can also be adjusted
during the learn phase.
• Finally, the actual output has to be determined. For this
purpose an activation or step function Φ is applied to
weighted sum of the input values.

10. 10
the actual output has to be determined.
For this purpose an activation or step
function Φ is applied to weighted sum
of the input values.

11. 11
The simplest form of an activation
function is a binary function
• The simplest form of
an activation function
is a binary function. If
the result of the
summation is greater
than some threshold
s, the result of Φ will
be 1, otherwise 0.

12. 12
A Simple Neural Network
• The following image shows the general
building principle of a simple artificial
neural network:
•

13. 13
• We will write a very
simple Neural
Network
implementing the
logical "And" and "Or"
functions.
• Let's start with the
"And" function. It is
defined for two inputs:

14. 14
Line Separation
• You could imagine that you have two attributes
describing am eddible object like a fruit for
example: "sweetness" and "sourness"
• We could describe this by points in a two-
dimensional space. The x axis for the sweetness
and the y axis for the sourness. Imagine now that
we have two fruits as points in this space, i.e. an
orange at position (3.5, 1.8) and a lemon at (1.1,
3.9).

15. 15
Line Separation
• We could define dividing lines to define the
points which are more lemon-like and which are
more orange-like. The following program
calculates and renders a bunch of lines. The red
ones are completely unusable for this purpose,
because they are not separating the classes.
Yet, it is obvious that even the green ones are
not all useful.

16. 16
Line Separation

17. 17
Line Separation

18. 18
Line Separation

19. 19
Neural Network Using Python
and Numpy

20. 20
Neural Network Using Python
and Numpy

21. 21
Neural Network Using Python
and Numpy
• We have introduced the basic ideas about
neuronal networks in the previous chapter of our
tutorial.
• We pointed out the similarity between neurons
and neural networks in biology. We also
introduced very small articial neural networks
and introduced decision boundaries and the
XOR problem.
• The focus in our previous chapter had not been
on efficiency.

22. 22
Neural Network Using Python
and Numpy
• We will introduce a Neural Network class in Python in
this chapter, which will use the powerful and efficient
data structures of Numpy. This way, we get a more
efficient network than in our previous chapter. When we
say "more efficient", we do not mean that the artificial
neural networks encountered in this lesson are efficient
and ready for real life usage.
• They are still quite slow compared to implementations
from sklearn for example. The focus is to implement a
very basic neural network and by doing this explaining
the basic ideas.

23. 23
Neural Network Using Python
and Numpy
• Ideas like how the signal flow inside of a
network works, how to implement weights.
how to initialize weight matrices or what
activation functions can be used.
• We will start with a simple neural networks
consisting of three layers, i.e. the input
layer, a hidden layer and an output layer.

24. 24
A Simple Artificial Neural
Network Structure
• You can see a simple neural network structure in
the following diagram. We have an input layer
with three nodes i1,i2,i3i1,i2,i3 These nodes get
the corresponding input values x1,x2,x3x1,x2,x3.
The middle or hidden layer has four
nodes h1,h2,h3,h4h1,h2,h3,h4.
• The input of this layer stems from the input layer.
We will discuss the mechanism soon. Finally,
our output layer consists of the two nodes o1,o2

25. 25
Neural Network Using Python
and Numpy
• We have to note that some would call this
a two layer network, because they don't
count the inputs as a layer.

26. 26
Neural Network Using Python
and Numpy
• The input layer consists of the
nodes i1i1, i2i2 and i3i3. In principle the
input is a one-dimensional vector, like (2,
4, 11). A one-dimensional vector is
represented in numpy like this:

27. 27
A Simple Artificial Neural
Network Structure

28. 28
A Simple Artificial Neural
Network Structure
• In the algorithm, which we will write later,
we will have to transpose it into a column
vector, i.e. a two-dimensional array with
just one column:

29. 29
A Simple Artificial Neural
Network Structure

30. 30
Backpropagation in Neural
Networks

31. 31
Backpropagation in Neural
Networks

32. 32
Backpropagation in Neural
Networks
• We have already written Neural Networks
in Python in the previous chapters of our
tutorial. We could train these networks, but
we didn't explain the mechanism used for
training. We used backpropagation without
saying so. Backpropagation is a
commonly used method for training
artificial neural networks, especially deep
neural networks.

33. 33
Backpropagation in Neural
Networks
• Backpropagation is needed to calculate
the gradient, which we need to adapt the
weights of the weight matrices. The weight
of the neuron (nodes) of our network are
adjusted by calculating the gradient of the
loss function. For this purpose a gradient
descent optimization algorithm is used. It
is also called backward propagation of
errors.

34. 34
Backpropagation in Neural
Networks
• Explaining gradient descent starts in many
articles or tutorials with mountains.
Imagine you are put on a mountain, not
necessarily the top, by a helicopter at
night or heavy fog. Let's further imagine
that this mountain is on an island and you
want to reach sea level. You have to go
down, but you hardly see anything, maybe
just a few metres.

35. 35
Backpropagation in Neural
Networks
• Your task is to find your way down, but you
cannot see the path. You can use the method of
gradient descent. This means that you are
examining the steepness at your current
position. You will proceed in the direction with
the steepest descent. You take only a few steps
and then you stop again to reorientate yourself.
This means you are applying again the
previously described procedure, i.e. you are
looking for the steepest descend.

36. 36
Backpropagation in Neural
Networks
• This procedure is depicted in the following
diagram in a two-dimensional space.

37. 37
Backpropagation in Neural
Networks

38. 38
Backpropagation in Neural
Networks
• Going on like this you will arrive at a position, where
there is no further descend.
• Each directions goes upwards. You may have reached
the deepest level - the global minimum -, but you might
as well be stuck in a basin. If you start at the position on
the right side of our image, everything works out fine, but
from the leftside, you will be stuck in a local minimum. If
you imagine now, - not very realistic - you are dropped
many time at random places on this island, you will find
ways downwards to sea level. This is what we actually
do, when we train a neural network.
•

39. 39
Machine Learning by
using Python
Neural Network
Lesson 2
By: Professor Lili Saghafi