2. Contents
• What is an optimization?
• Components of an optimization problem
• Classification of optimization problem
• Optimization Taxonomy
• Algorithm by problem type
• Advanced Optimization Technique
• Software Availability
• References
3/22/2018
KaminiSingh,IMEDept.IITKanpur
2
3. What is an optimization ?
• The action of making the best or most effective use of given
situation - (Google dictionary)
• It is a technique of squeezing best performance out of
provided current state of model
3/22/2018
KaminiSingh,IMEDept.IITKanpur
3
4. Components of an optimization
problem
• Objective function
An objective function express performance of a system, need
to be minimized or maximized
• Variable (Design or Decision Variable)
A set of unknowns, define the objective function and
constraints, can be continuous, discrete or boolean
• Constraints
They are conditions, allows the unknowns to take on certain
values but exclude others to render the design to be feasible
3/22/2018
KaminiSingh,IMEDept.IITKanpur
4
6. Types of optimization problem
• Based on existence of constrains
• Based on the nature of the equations involved
• Based on the permissible values of the decision variables
• Based on the number of objective functions
3/22/201
8
KaminiSingh,IMEDept.IITKanpur
6
7. Based on existence of constrains (1/4)
• Unconstrained Optimization Problem (No Constraints)
Objective function
Min/Max ƒ(𝑥𝑖, … . , 𝑥 𝑛)
• Constrained Optimization Problem (One or more than one constrained)
Objective function
Min/Max ƒ(𝑥𝑖, … . , 𝑥 𝑛)
Subject to:
𝑔𝑖 𝑥𝑖, … . , 𝑥 𝑛 = 0 ∀ 𝑖 𝜖 1, … , 𝑝
𝚽𝑗 𝑥𝑖, … . , 𝑥 𝑛 ≤/ ≥ 0 ∀ 𝑗 𝜖 1, … , 𝑚
3/22/2018
KaminiSingh,IMEDept.IITKanpur
7
Equality Constraints
Inequality Constraints
8. Based on the nature of the equations
involved (2/4)
• Linear Programming Problem (LPP)
Linear objective function with linear constraints
• Quadratic Problem 𝑋 =
𝑥
𝑦 and 𝐴 =
𝑎 𝑏 2
𝑏 2 𝑐
; 𝑓 𝑋 = 𝑋 𝑇 𝐴 𝑋
• Non-Linear Programming Problem (NLP)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
8
Maximize 𝐶 𝑇
𝑥
Subject to 𝐴𝑥 ≤ 𝐵
𝑥 ≥ 0
Standard
form
Minimize 𝑓 𝑥, 𝑦 = 𝑎𝑥2
+ 𝑏𝑥𝑦 + 𝑐𝑦2
Min/Max 𝑓 𝑥 = 𝑥1 𝑥2 𝑥3
Subject to 2𝑥1 + 4𝑥2 𝑥3 ≤ 10
𝑥1 + 𝑥2 + 𝑥3 ≤ 7
8𝑥1 + 𝑥2 + 5𝑥3 < 4
𝑥𝑖 ≥ 0 ∀ 𝑖 = 1,2, 3
(with constraints it is become Quadratically
Constrained Quadratic Programming problem )
9. Based on the permissible values of the
decision variables (3/4)
• Integer Programming
Problem (an example of LPP)
• Decision variables (𝑥1, 𝑥2) are
allowed to take only integral
value
3/22/2018
KaminiSingh,IMEDept.IITKanpur
9
• Real-Valued Programming
Problem
• Decision variable allowed to
take any real value in its
domain
• 𝑥𝑖 ≥ 0, ∀ 𝑖 = 1, … , 𝑛
2𝑥1 + 3𝑥2 ≤ 18, 𝐿1
𝑥1 + 3𝑥2 ≤ 9, 𝐿2
𝑥2
𝑥1 The dot represents
the Integer point set
on 2-D coordinate
Optimal Value
𝐿1
𝐿2
2𝑥1 + 3𝑥2 ≤ 18, 𝐿1
𝑥1 + 3𝑥2 ≤ 9, 𝐿2
𝑥2
𝑥1
Optimal Value
𝐿1
𝐿2
10. Based on the number of objective
functions (4/4)
• Single objective
function problem
• Problem in which there is only
single objective function for 𝑥𝑖,
need to be minimized/maximized
Objective function
Min/Max
ƒ(𝑥𝑖, … . , 𝑥 𝑛)
Subject to
𝑔𝑖 𝑥𝑖, … . , 𝑥 𝑛 = 0
∀ 𝑖 𝜖 1, … , 𝑝
3/22/2018 10
KaminiSingh,IMEDept.IITKanpur
• Multi-objective function
problem
• More than one objective function
for 𝑥𝑖 , need to be
minimized/maximized
Objective function
Min/Max
ƒ(𝑥1) 𝑓 𝑥2 … . 𝑓(𝑥 𝑛)
Subject to
𝑔𝑗 𝑥 = 0
∀ 𝑗 𝜖 1, … , 𝑝
15. Genetic Algorithm (1/3)
• Introduced by John Holland in
1970s, which is continuously
evolved by his students and
followers over the years
• A strong tool with adaptive
heuristic search algorithm based on
evolutionary ideas
• Facilitate a more robust Artificial
Intelligence
• Terminology used in GA
Chromosome (String)
Genes (Bits)
Locus
Allele (Value)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
15
1 0 1 1 0
0
0
Chromosome
(String)
Gene (Bit)
Allele
Locus
16. Genetic Algorithm (2/3)
• The Genetic operator
Reproduction
Crossover
Mutation
3/22/2018
KaminiSingh,IMEDept.IITKanpur
16
Crossover
Types of Mutation
Reproduction
17. Genetic Algorithm (3/3)
• Population size (n)
In GA, it is integration of
predecessors with their
reproduction, crossover and
mutation
• Fitness Value
It is the same as the
objective function
(sometime for minimization
problem number of conversion is
done to get optimum way using
exponential function)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
17
Start
Generate Initial Population
Evaluate Fitness function
Stop
Re-generate
process
Selection process
Meets
Optimization
Criteria
Yes
No
18. Simulated Annealing(1/2)
• Simulated annealing (SA)
is a probabilistic technique
for approximating
the global optimum of a
given function
• Technique is inspired from
the annealing process of
metallurgy as name goes
• It search for the global
maxima in the
neighborhood solution
3/22/2018
KaminiSingh,IMEDept.IITKanpur
18
Thermodynamic
Simulation
Combinatorial
Optimization
System State Feasible Solution
Energy Transportation Cost
and Elapsed Time
Change of State Neighboring
Solutions
Temperature Control Parameter
Frozen State Heuristic Solution
Comparison between physical and
simulated annealing
19. Simulated Annealing (2/2)
• It follows the exponential
Boltzmann probability (K
is a Boltzmann constant in
equation [𝑝 = 𝑒(−∆ 𝑇∗𝐾)] )
• ∆ compares the previous
and current objective
function
3/22/2018
KaminiSingh,IMEDept.IITKanpur
19
Calculate 𝑓 𝑥 for trial solution and set
∆ = 𝑓 𝑥 𝑘+1 − 𝑓(𝑥 𝑘)
Accept the Trial
Solution
Start
Select initial Temperature 𝑇0
Generate a trial solution in the neighborhood of present solution
Stop
K= k+1
Decrease the temperature by, 𝑇𝑘+1 = 𝑝 ∗ 𝑇𝑘
∆< 0?
Yes No
Accept the trial solution
with probability, p
(𝑒(−∆ 𝑇∗𝐾)
) > 𝑟
Freezing
point ?
No
No
Yes
20. Hill Climbing Optimization (1/2)
• Hill climbing is a mathematical
optimization technique which
belongs to the family of local
search with iterative algorithm
• Starts with an arbitrary solution to
a problem and attempts to find a
better solution by incrementally
changing a single element of the
solution
• Algorithm Components
Local Maxima (not highest)
Plateau (Random walk)
Ridges (flat like a plateau, but with
drop-offs to the sides)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
20
Local
Maxima Global
Maxima
Plateau
𝑓(𝑥)
State Space (𝑥)
3-D View
Current
state
Shoulder/Plateau
Local
Maxima
𝑓(𝑥)
State Space (𝑥)
Global
Maxima
2-D View
21. Hill Climbing Optimization (2/2)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
21
Local Maxima
Global
Maxima
Plateau
• Not complete since the search will
terminate at local minima, plateaus, and
ridges.
Start
Look one step ahead
Stop
Re-generate
process
k= k+1
𝑓(𝑥 𝑘+1) < 𝑓 𝑥 𝑘
∀ 𝑥𝑖 ≤ 𝑥 𝑘
Yes
No
22. Particle Swarm Intelligence (1/2)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
22
• Particle swarm optimization
(PSO) is a computational
method to optimize a
problem
• It works iteratively trying to
improve a candidate
solution with regard to a
given measure of quality
• It studies the position and
velocities of particle at time
(t)
Random Fly
Personal
best (pbest)
Ordered Fly
Personal best
become global
best (gbest)
Fish following
global best
(gbest)
23. Particle Swarm Intelligence (2/2)
• pBest: personal best solution of
individual bird in flock
• gBest: Global best of the bird
flock
3/22/2018
KaminiSingh,IMEDept.IITKanpur
23
Assign current
fitness as new
pBest Solution
Start
Generate Initial Population
Evaluate Fitness function for each particle, ∆ = 𝑓 𝑥 𝑘+1 −
𝑓(𝑥 𝑘)
Use each particle’s velocity value to update its data
values
Stop
Calculate Velocity for each particle
𝑖𝑓 ∆ is
better than
pBest ?
Yes No
Keep previous
best
Target
reached?
No
No Yes
Assign best particle’s pBest value to gBest
24. Pattern Search (1/2)
• This methodology involves setting up of grids in the decision space
• Evaluates the values of the objective function at each grid point
• To get the optimum point it increment and decrement the grid space
• The best value of the objective function is corresponding to the optimum
point is considered to be an optimum solution
• Stages involved in it are
Pattern (generation as per GPS (Using ‘N’ or ‘N+1’ Law)
Meshes (for each point search for a new algorithm)
Polling (Polls the point obtained after meshing to meet objective function)
Increment and Decrement (as per polling it expands and contracts the pattern to get the
optimal solution)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
24
25. Pattern Search (2/2)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
25
Poll = Minimum objective in the pattern at point (k+1) −Current point value (k)
Start
Generate a pattern of points (both direction of coordinates), specify mesh
size and center this pattern on the current point
Evaluate Fitness function at every point
Stop
k= k+1
Half the Mesh size
Poll < 0
Yes
Nok= k+1, Double
the Mesh Size
Mesh size <
Threshold
Yes
No
• Pattern search finds a
local minimum of an
objective function by
using polling
• Example
26. Ant Colony Optimization (1/2)
• Ant Colony Optimization is
based on the real life example of
ant
• It belongs to the iterative and
sub-group of evolutionary family
as same goes to Particle Swarm
Intelligence
• Ants follows the pheromone
chemical leave behind by their
successor to find the shortest path
3/22/2018
KaminiSingh,IMEDept.IITKanpur
26
A real image of Ant alignment
towards food
27. Ant Colony Optimization (2/2)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
27
Every ant schedule its next
operation
Update probability values each edge
to feasible node
Pheromone Evaporation
Start
Select Initial Parameter
All ants schedule their own first operation
Stop, schedule the
operation
Ant reached
food node?
Yes
Analysis of best
solution found and
update pheromones
according to criteria
Ants
accomplished all
needed trip?
No
No
Yes
Each ant updates its list of feasible operation
• Probability 𝐨𝐟 𝒌 𝒕𝒉ant move from 𝒙 to 𝒚
𝑝 𝑥𝑦
𝑘
= (𝜏 𝑥𝑦
𝛼
)(𝜃 𝑥𝑦
𝛽
) 𝑧𝜖𝑎𝑙𝑙𝑜𝑤𝑒𝑑 𝑥𝑦
(𝜏 𝑥𝑦
𝛼
)(𝜃 𝑥𝑦
𝛽
)
𝜏 𝑥𝑦: amount of pheromone deposited from transition
x to y, 𝛼 ≥ 0 (𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑓𝑜𝑟 𝛼)
𝜃 𝑥𝑦: prior knowledge of state 𝑥𝑦 (1
𝑑 𝑥𝑦
, d is a
distance), 𝛽 ≥ 1 (𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑓𝑜𝑟 𝜃)
• Pheromone Update
Trials updated by 𝜏 𝑥𝑦 ← (1 − 𝜌)𝜏 𝑥𝑦 + 𝑘 ∆𝜏 𝑥𝑦
𝑘
)
𝜌: evaporation coefficient
∆𝜏 𝑥𝑦
𝑘
=
𝑄
𝐿 𝑘
0
𝑖𝑓 𝑎𝑛𝑡 𝑘 𝑢𝑠𝑒𝑠 𝑐𝑢𝑟𝑣𝑒 𝑥𝑦 𝑖𝑛 𝑖𝑡𝑠 𝑡𝑜𝑢𝑟
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝐿 𝑘 is the cost of the kth ant’s tour and Q is a
constant
28. Tabu Search (1/2)
• Tabu (prevent cycling) search introduced by Fred W. Glover in
1986
• It is a metaheuristic based local search methods used
• At each step worsening moves can be accepted, if solution get
stuck to a strict local minim
• A forbidden (Tabu) is introduced to discourage the entrapment to
previously-visited (strict local minima) solutions
3/22/2018
KaminiSingh,IMEDept.IITKanpur
28
29. Tabu Search (2/2)
• X: set of feasible solution
• 𝑥: Current solution, 𝑥 ∈ 𝑋
• 𝑥′′
: Best solution reached
• 𝑥′: Best solution among sample of trialed solution
• 𝑓(𝑥): evaluation function of 𝑥
• N(𝑥): Neighborhood set, 𝑥 ∈ 𝑋 (Trial solution)
• S(𝑥): Neighborhood sample of 𝑥, S(𝑥) ∈ N(𝑥)
• SS(𝑥): Sorted sample in ascending order as per 𝑓(𝑥)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
29
Yes
Every ant schedule its next
operation
Calculate the objective function to each neighbor in the
set, and find the best among them
Find a set of neighbors to the current solution
Start
Select Initial Parameter (TL & AV)
Find randomly an initial solution
Stop
Is the best neighbor
TABU ?
Yes
Is AV
satisfied?
No
No
Yes
Set the current and best solution equal to the initial solution
Is stopping
criteria satisfied
No
• Initialization Parameter
Set TL (Tabu list restriction) and AV
(Aviation value)
Set iteration counter K = 0, and select an
initial solution 𝑥 ∈ 𝑋
Set 𝑥′′
= 𝑥
Start the process as per flow chart by
generating a random set of trial solutions
S(𝑥) ∈ N(𝑥)
30. Fuzzy Optimization (1/4)
3/22/2018
KaminiSingh,IMEDept.IITKanpur
30
(0) (1)(0-1)
• Fuzzy (Vagueness): For cold state
value assigned as zero (𝑥= 0) and for hot
state one (𝑥 =1).
• Vagueness arises for warm state, value lies
of which between zero to one {𝑥 ∈ (0,1)}
• Concepts involve to define fuzziness are
often difficult to model
• fuzzy can be used when:
One or more control variable is constant
System has uncertainties in either the input
or definition
Mathematical model of any process does not
exist or difficult to model or too-complex to
evaluate fast enough for real-time operation
or too costly to design
Fuzziness
• Some Important term:
Singletons: Male-Female, Dead-
Alive (Neither flexibility nor
interval)
Fuzzy number: Around 80℃, bigger
than 25, etc.
Fuzzy set: Continuous set [0 to 1],
[Red to yellow]
Fuzzy linguistic term: Beautiful
lady, good quality etc.
31. Fuzzy Optimization (2/4)
• Characteristic function of Fuzzy term
Control variable: A variable that controls the state of the solution
variables
Defuzzification: Process of converting an output fuzzy set into a
single value for a solution variable
Overlap: The degree to which one domain of fuzzy set overlaps to
another
Solution fuzzy set: A temporary fuzzy set created by fuzzy model as
current solution variable
Solution variable: The variable whose value in the fuzzy logic
system is meant to be find
Fuzzy model: A fuzzy system that coverts input into their fuzzy
representations
3/22/2018
KaminiSingh,IMEDept.IITKanpur
31
32. Fuzzy Optimization (3/4)
There are four steps need to be followed to design a fuzzy model
• First step: Define the model function and operation characteristics
• Second step: Define the control surfaces (fuzzy set) using given r concepts
A label meant to be created associated with variable should be an odd number
Each label should overlap (10-50%) with its neighborhood for smooth stable fuzzy
control surface
Density of fuzzy should be highest around optimal control
• Third step: Define the control behavior [if <fuzzy proposition> then <
fuzzy proposition >]
• Fourth step: Select a method of defuzzification (composite maximum,
calculation of concentration)
• Fifth step: fuzzy sets and membership
• Sixth step: Membership functions
3/22/2018
KaminiSingh,IMEDept.IITKanpur
32
33. Fuzzy Optimization (4/4)
• Membership function:
′𝑥′ a control fuzzy variable such that
𝑥 ∈ 𝑋, X is a classical set of objects,
then membership (𝜇 𝐴) of 𝑥 from X
can be understood as
𝜇 𝐴 (𝑥) =
1
0
𝑖𝑓𝑓 𝑥 ∈ 𝐴
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Where {0,1} is called a valuation set, if
this set is real à is called fuzzy set
𝜇 𝐴 (𝑥) is the degree of membership of
𝑥 in A, the 𝜇 𝐴 (𝑥) more closer to 1 the
more 𝑥 belongs to A
Ã= {(𝑥, 𝜇 𝐴 (𝑥) |𝑥 ∈ 𝑋 )}
3/22/2018
KaminiSingh,IMEDept.IITKanpur
33
Example:
Objective function
Minimize Z= 𝑓(𝑥)
Subjected to: 𝑔𝑗 𝑥 = 0
∀ 𝑗 𝜖 1, … , 𝑝
𝑥 ≥ 0
𝜇 𝑍( 𝑥) = max 𝑥∈𝐴,𝐵{min (𝜇 𝐴
(𝑥), 𝜇 𝐵(𝑥)}
𝑥 is optimal value
{~}defines the fuzziness
34. Software Availability
There are variety of software available out their as per the problem
type some renown are
• MatLab
• SCOPS, NLCOS (for non-linear)
• MCDA package for R (for multi-criteria-decision making)
• SAMPLE (for uncertainty problem)
• MAPLE (with optimization package)
• Lindo (for LP)
• LGO
Note: Other softwares are also available, user can prefer any as per
their compatibility
3/22/2018
KaminiSingh,IMEDept.IITKanpur
34
35. References
• X. Liu, “Network optimization with stochastic traffic flows,” Int. J.
Netw. Manag., vol. 12, no. 4, pp. 225–234, 2002.
• T. Ghose, “Optimization Techniques and an Introduction To Genetic
Algorithms and Simulated Annealing,” thesis GA SA 19, no. X, pp. 1–
19, 2002.
• https://en.wikipedia.org/wiki/Fuzzy_logic
• R. V. Rao and V. J. Savsani, Mechanical Design Optimization Using
Advanced Optimization Techniques. 2012.
• C. R. Dyer, “Informed Search.”
• https://neos-guide.org/content/optimization-taxonomy
• S. A.-H. Soliman and A.-A. H. Mantawy, Modern Optimization
Techniques with Applications in Electric Power Systems. 2012.
• http://www.optimization-online.org/cgi-bin/search.cgi
3/22/2018
KaminiSingh,IMEDept.IITKanpur
35
36. References
• S. E. A. Stephen and J. O. E. A. a, “Review of Ten Non-Traditional
Optimization Techniques,” vol. 3, no. 1, pp. 103–124, 2013.
• S. Mishra, “Chapter 4 Optimization Techniques in Perspective,”
Shodh Ganga a resevoir Indian theses, pp. 26–36, 2013.
• S. H. Z. Edwin K.P Chong, An introduction to optimization, Second
Edition. New Yark/Toronto: Publication, A wiley-Interscience.
• S. Kiranyaz, T. Ince, and M. Gabbouj, Multidimensional Particle
Swarm Optimization for Machine Learning and Pattern Recognition,
vol. 15. 2014.
• https://images.google.com.
• R. Subramani and C. Vijayalakshmi, “A review on advanced
optimization techniques,” ARPN J. Eng. Appl. Sci., vol. 11, no. 19, pp.
11675–11683, 2016.
3/22/2018
KaminiSingh,IMEDept.IITKanpur
36