PSO Slide show 2017

1. Iwan Sofana (iwansofana@gmail.com)
Kota Baru Parahyangan
Bandung West Java Indonesia
2017

2. •This slides adapted from many related
slides, papers, & books.
•Special thanks to:
 James Kennedy & Russell Eberhart
 Maurice Clerc
 Riccardo Poli, Tim Blackwell, Andry Pinto, Hugo Alves
 Inês Domingues, Luís Rocha, Susana Cruz
 Jaco F. Schutte
 Matthew Settles
 Satyobroto Talukder
 and many more

3.  Optimization
 Basic PSO
 PSO Algorithm

4. Optimization

5. Optimization determines the best-suited solution to
a problem under given circumstances.

Non Linear Optimization problems are generally
very difficult to solve.

Linear/Non Linear problems are equivalent with
Linear/Non Linear function.
Optimization

6. Function Example

7.  Please find the maxima/minima of the following
functions:
– f(x)=x2
+2
– h(t)=3 + 14t − 5t2
– g(y)=5y3
+ 2x2
− 3x
 How do we know it is a minimum or maximum?
 Clue: use derivatives & math tools.
Case study

8.  Quadratic
 Solution
Case study

9.  Derivative
Case study

10.  Derivative rules
Case study

11.  Base on problem charecteristics:

Unconstrained Optimization.

Constrained Optimization.

Dynamic Optimization.
Optimization

12. Many optimization problems place no restrictions on
the values of that can be assigned to variables of
the problem.
Unconstrained Optimization

13. The process of optimizing an objective function with
respect to some variables in the presence of
constraints on those variables.
Constrained Optimization

14. Many optimization problems have objective
functions that change over time and such
changes in objective function cause changes in
the position of optima.
Dynamic Optimization

15. There are two types of optimization techniques.
 Global optimization technique
 Local optimization technique
Optimization Techniques

16. Global optimization technique seek to find a global
minimum or lowest function value and its
corresponding global minimizer
 Global minimizer (X*)
Global Opt. Technique

17. Local optimization technique try to find a local
minimum and its corresponding local minimizer
 Local minimizer (XL
*)
Local Opt. Technique

18. Case Study
XL
*
X*

19. 
For a known (differentiable) function f , calculus
can fairly easily provide us with the minima and
maxima of f .

However, in real-life optimization tasks, this
objective function f is often not directly known.

The objective function is a “black box”.
Real-Life

20. Basic PSO

21.  Inspired from the nature social behavior and
dynamic movements with communications of
insects, birds and fish

22.  In 1986, Craig Reynolds (a biologist) studied the
flocking behavior of birds.
 He described this process in 3 simple behaviors:
Separation
avoid crowding local
flockmates
(neighbors)
Alignment
move towards the
average heading of
local flockmates
(neighbors)
Cohesion
move toward the
average position of
local flockmates
(neighbors)

23.  In 1990, Heppner and Grenander: research of bird
flocks searching for corn.
 In 1995, James Kennedy (a social psychologist) &
Russell Eberhart (an electrical engineer):
influenced by Heppner and Grenander’s work
developed a powerful optimization method Particle
Swarm Optimization (PSO).
 I believe there are many more researcher... :-)

24. 
Kennedy and Eberhart first introduce The
Particle Swarm Optimization (PSO) algorithm for
a solution to the complex non-linear optimization
problem by imitating the behavior of bird flocks.

The Particle Swarm Optimization (PSO)
algorithm is a multi-agent parallel search
technique which maintains a swarm of particles
and each particle represents a potential solution
in the swarm.
PSO Origins

25. 
All particles fly through a multidimensional
search space where each particle is adjusting its
position according to its own experience and that
of neighbors.
Basic Idea

Each particle adjusts its travelling
speed dynamically corresponding to
the flying experiences of itself and its
neighbors.

26. PSO Algorithm

27. 
Basically, there are two type of PSO algorithm:
– + Local Best (lbest) PSO.
– + Global Best (gbest)

They have been developed which differ in the
size of their neighborhoods.
PSO Algorithm

28.  The local best or personal best PSO (lbest/pbest
PSO) method only allows each particle to be
influenced by the best-fit particle chosen from
its neighborhood.
 The global best PSO (or gbest PSO) is a method
where the position of each particle is influenced
by the best-fit particle in the entire swarm.
Local vs Global

29. 
The larger particle interconnectivity of the
gbest PSO, sometimes it converges faster than
the lbest PSO.

Another is due to the larger diversity of the
lbest PSO, it is less susceptible to being
trapped in local minima.
Local vs Global

30.  Local best or personal best PSO (lbest/pbest
PSO) use ring social topology/structure.
 The global best PSO (or gbest PSO) use star
social network topology/structure.
ring social topology star social topology
Local vs Global

31. local

32. global

33. The PSO algorithm consists of just three steps :
1. Evaluate the fitness of each particle.
2. Update individual and global best fitnesses
and positions.
3. Update velocity and position of each particle.
which are repeated until some stopping condition
is met.
PSO Algorithm

34. PSO algorithm :
PSO Algorithm
………. Gbest PSO
………. Lbest PSO

35. PSO Algorithm

36. 
A uniform distribution (a rectangular
distribution), is a distribution where the
probability of occurrence is the same for all
values. It has constant probability.

For instance, if a die is thrown, then the
probability of obtaining any one of the six
possible outcomes is 1/6.
Uniform Distribution

37. Uniform Distribution

38. Geometrical Illustration

39. Geometrical Illustration

40. 1. Number of particles usually between 20-60 or 10-
50.
2. C1 is the importance of personal best value.
3. C2 is the importance of neighborhood best value
Usually C1 + C2 = 4 or C1 = C2 = 2 (empirically chosen value). Wrong
initialization of C1 and C2 may result in divergent or cyclic behavior
4. If velocity is too low → algorithm too slow.
5. If velocity is too high → algorithm too unstable.
Charecteristic of PSO

41. ●6. When C1,= C2 = 0 then all particles continue flying
at their current speed.
7.When C1 > 0 and C2 = 0 then all particles are
independent.
8. C2 > 0 and C1 = 0 then all particles are attracked
to single point (i.e Gbest).
9. When C1 = C2 then all particles attracked towards
the average Pbest and Gbest.
Charecteristic of PSO

42. ●10. When C1 >> C2 each particle is more strongly
influenced by its Pbest position, resulting in excessive
wandering.
11.When C2 >> C1 and C2 = 0 then all particles are
much more influenced by the global best position,
which causes all particles to run prematurely to the
optima.
Charecteristic of PSO

43. Charecteristic of PSO
Inertia (memory) Personal influence Social influence
Diversification Intensification

44. + Intensification:
explores the previous solutions, finds the best
solution of a given region
+ Diversification:
searches new solutions, finds the regions with
potentially the best solutions
Charecteristic of PSO

45.  Advantages
• Insensitive to scaling of design variables
• Simple implementation
• Easily parallelized for concurrent processing
• Derivative free
• Very few algorithm parameters
• Very efficient global search algorithm
 Disadvantages
• Tendency to a fast and premature convergence in mid
optimum points
• Slow convergence in refined search stage (weak local search
ability)
Charecteristic of PSO

46.  Please find the maxima/minima of the function:
– f(x)= -x2
+5x +20 ; -10 <= x <= 10
– Use 9 particles :
– X1
= -9,6 X2
= -6
– X3
= -2,6 X4
= -1,1
– X5
= 0,6 X6
= 2,3
– X7
= 2,8 X8
= 8,3 X9
= 10
–
Case study

47. Modified PSO

48.  Several approaches
• 2-D Otsu PSO
• Active Target PSO
• Adaptive PSO
• Adaptive Mutation PSO
• Adaptive PSO Guided by Acceleration Information
• Attractive Repulsive Particle Swarm Optimization
• Binary PSO
• Cooperative Multiple PSO
• Dynamic and Adjustable PSO
• Extended Particle Swarms
• …
Davoud Sedighizadeh and Ellips Masehian, “Particle Swarm Optimization Methods, Taxonomy and
Applications”. International Journal of Computer Theory and Engineering, Vol. 1, No. 5, December 2009

49. + Particle Swarm Optimization (PSO) and Ant Colony
Optimization (ACO) are part of Swarm Intelligence
(SI).
+ Swarm intelligence (SI) is the collective behavior
of decentralized, self-organized systems, natural or
artificial.
Key Concepts

50. 1. PSO algorithm basically learned from animal’s
activity or behavior to solve optimization problems.
2. Each member of the population is called a particle
and the population is called a swarm.
3. It does not require any gradient information of the
function to be optimized and uses only primitive
mathematical operators.
Key Concepts

51. 4. PSO is well suited to solve the non-linear, non-convex,
continuous, discrete, integer variable type problems.
5. In PSO, each particle flies through the
multidimensional space and adjusts its position in every
step with its own experience and that of peers toward an
optimum solution by the entire swarm.
6. It doesn’t always work well, still has room for
improvement.
Key Concepts