Derivative Free Optimization and Robust Optimization

SSA KPI
SSA KPISSA KPI
4th International Summer School
Achievements and Applications of Contemporary
Informatics, Mathematics and Physics
National University of Technology of the Ukraine
Kiev, Ukraine, August 5-16, 2009




                        Nonsmooth Optimization
    Derivative Free Optimization and Robust Optimization


             Gerhard-Wilhelm Weber * and Başak Akteke-Öztürk
             Gerhard-                          Akteke-
                               Institute of Applied Mathematics
                       Middle East Technical University, Ankara, Turkey

            * Faculty of Economics, Management and Law,   University of Siegen, Germany
             Center for Research on Optimization and Control, University of Aveiro, Portugal
Introduction

                                      Mathematical Models
 • Experimental Data Analysis
     • Classification problems
      • Identification problems              treated by:
          • Pattern Recognition       SVM, Cluster Analysis,
  • Assignment and Allocation
                                       Neural Systems etc.



• When these methods were born, the most developed and popular
      optimization tools were Linear and Quadratic Programming.

• Optimization parts of these methods are reduced to LP and QP
                                       Linear Discriminant Analysis
Introduction

• progress in Optimization         • new advanced tools,
• Nonsmooth Analysis and           • construct a mathematical model
  Nondifferentiable Opt.             better suited for the problem
                                     under consideration

• Most cases clustering problems are reduced
  to solving nonsmooth optimization problems.

• We are interested in new methods for solving related nonsmooth
   problems
  (e.g.,
  Semidefinite Programming, Semi-Infinite Programming,
   discrete gradient method and cutting angle method).
Nonsmooth Optimization

    Problem:
         minimize
         subject to


•          :           is nonsmooth at many points of interest
        do not have a conventional derivative at these points.

• A less restrictive class of assumptions for       than smoothness:
  convexity and Lipschitzness.
Nonsmooth Functions
Convex Sets


 A set        is called convex
 if
Convex Sets


• The convex hull of a set              :




• The sets         and      coincide if and only if    is convex.

• The set      is called a cone if               for all        ,   ;
  i.e.,
     contains all positive multiples of its elements.
Convex hull
Convex Functions


• A set
  is called an epigraph of function                .

• Let            be a convex set. A function
  is said to be convex if its epigraph is a convex set.
Convex Functions


• are differentiable (smooth) almost everywhere,
• their minimizers are points where the function need not be
  differentiable,
• standard numerical methods do not work


• Examples of convex functions:

    – affinely linear:
    – quadratic:                   (c>0)
    – exponential:
Convex Functions
Convex Optimization

• minimizing a convex function      over a convex feasible set

• Many applications.

• Important, because:

      a strong duality theory
      any local minimum is a global minimum
      includes least-squares problems and linear programs as special cases




      can be solved efficiently and reliably
Lipschitz Continuous


• A function               is called (locally) Lipschitz continuous, if
  for any bounded             there exist a constant         such that




• Lipschitzness is a more restrictive property on functions than
  continuity, i.e., all Lipschitz functions are continuous, but
  they are not guaranteed to be smooth.

• They possess a generalized gradient.
Lipschitz Continuous
Nonsmooth Optimization


• We call the the set ∂f(x) subdifferential of f at x



• Any vector v є ∂f(x) is a subgradient.

• A proper convex function f is subdifferentiable at any point x є   , if
  ∂f(x) is non-empty, convex and compact at x.

• If the convex function f is continuously differentiable, then
Nonsmooth Functions and Subdifferentials
Generalized Derivatives


• The generalized directional derivative of f at x in the direction g is
  defined as




• If the function f is locally Lipschitz continuous, then the generalized
  directional derivative exists.

• The set
  is called the (Clarke) subdifferential of the function f at a point
Nonsmooth Optimization


Nonsmooth optimization

   – more general problem of minimizing functions,

   – lack some, but not all, of the favorable properties of convex functions,

   – minimizers often are again points where the function is nondifferentiable.
Cluster Analysis via Nonsmooth Opt.

Given


Problem:




This is a partitioning clustering problem.
Clustering
Clustering
Cluster Analysis via Nonsmooth Opt.


• k is the number of clusters (given),
• m is the number of available patterns (given),

•          is the j-th cluster’s center (to be found),
•     association weight of pattern       , cluster j (to be found):




• (    ) is an         matrix,

• objective function             has many local minima.
Cluster Analysis via Nonsmooth Opt.

Suggestion (if k is not given a priori):

• Start from a small enough number of clusters k and gradually
  increase the number of clusters for the analysis until a certain
  stopping criteria met.

• This means: If the solution of the corresponding optimization
  problem is not satisfactory, the decision maker needs to consider a
  problem with k + 1 clusters, etc..

• This implies: One needs to solve repeatedly arising optimization
  problems with different values of k - a task even more challenging.

• In order to avoid this difficulty, we suggest a step-by-step calculation
  of clusters.
Cluster Analysis via Nonsmooth Opt.


•   k-means, h-means, j-means
•   dynamic programming
•   branch and bound
•   cutting planes
•   metaheuristics: simulated annealing, tabu search and genetic algorithms
•   an interior point method for minimum sum-of squares clustering
    problem
•   agglomerative and divisive hierarchical clustering incremental approach
Cluster Analsysis via Nonsmooth Opt.

 Reformulated Problem:




• A very complicated objective function: nonsmooth and nonconvex.

• The number of variables in the nonsmooth optimization approach is
  k×n, before it was (m+n)×k.
Robust Optimization


• There is uncertainty or variation in the objective and constraint
  functions, due to parameters or factors that are either
  beyond our control or unknown.

• Refers to the ability of the subject to cope well with uncertainties
  in linear, conic and semidefinite programming .

• Applications in control, engineering design and finance.

• Convex, modelled by SDP or cone quadratic programming.

• Robust solutions are computed in polynomial time, via (convex)
  semidefinite programming problem.
Robust Optimization

• Let us examine Robust Linear Programming



• By a worst case approach the objective is the maximum over all
  possible realizations of the objective


• A robust feasible solution with the smallest possible value of the f(x)
  is sought.

• Robust optimization is no longer a linear programming.
  The problem depends on the geometry of the uncertainty set U;
  i.e.,
  if U is defined as an ellipsoid, the problem becomes a
  conic quadratic program.
Robust Optimization
Robust Optimization

•   Considers that the uncertain parameter c belongs to a bounded, convex,
    uncertainty set


• Stochastic Optimization:                  expected values,
    parameter vector u is modeled as a random variable with known distribution



                                  Robust Counterpart

• Worst Case Optimization: the robust solution is the one that has the best
    worst case, i.e., it solves
Robust Optimization


• A complementary alternative to stochastic programming.

• Seeks a solution that will have a “good” performance under
  many/most/all possible realizations of the uncertain input
  parameters.

• Unlike stochastic programming, no distribution assumptions on
  uncertain parameters –
  each possible value equally important (this can be good or bad)

• Represents a conservative viewpoint when it is worst-case oriented.
Robust Optimization

• Especially useful when

   – some of the problem parameters are estimates and carry estimation
     risk,

   – there are constraints with uncertain parameters that must be satisfied
     regardless of the values of these parameters,

   – the objective functions / optimal solutions are particularly sensitive to
     perturbations,

   – decision-maker can not afford low-probability high-magnitude risks.
Derivative Free Optimization


The problem is to minimize a nonlinear function of several variables

•   the derivatives (sometimes even the values) of this function
    are not available,

•   arise in modern physical, chemical and econometric measurements and in
    engineering applications,

•   computer simulation is employed for the evaluation of the function values.


The methods are known as derivative free methods (DFO).
Derivative Free Optimization


Problem:




•          cannot be computed or just does not exist for every x ,
•     is an arbitrary subset of    ,
•         is called the easy constraint,
• the functions                          represent difficult constraints.
Derivative Free Optimization


Derivative free methods

• build a linear or quadratic model of the objective function,
• apply a trust-region or a line-search to optimize the model;


derivative based methods
                                 use a Taylor polynomial -based model;


DFO methods            use interpolation, regression
                       or other sample-based models.
Derivative Free Optimization




                Six iterations of a trust-region algorithm.
Semidefinite Programming

• Optimization problems where the variable is not a vector but a
  symmetric matrix which is required to be positive semidefinite.

• Linear Programming
                                Semidefinite Programming
  vector of variables
                                a symmetric matrix
  nonnegativity constraint
                                a positive semidefinite constraint

• SDP is convex, has a duality theory and can be solved
  by interior point methods.
SVC via Semidefinite Programming



• I try to reformulate the support vector clustering problem as a
  convex integer program and then relax it to a soft clustering
  formulation which can be feasibly solved by a 0-1 semidefinite
  program.



• In the literature, k-means and clustering methods which use a
  graph cut model are reformulated as a semidefinite program
  and solved by using semidefinite programming relaxations.
Some References


1. Aharon Ben-Tal and Arkadi Nemirovski, Robust optimization
   methodology and applications.
2. Adil Bagirov, Nonsmooth optimization approaches in data
   Classification.
3. Adil Bagirov, Derivative-free nonsmooth optimization and its
  applications.
4. A. M. Bagirov, A. M. Rubinov, N.V. Soukhoroukova and J.
   Yearwood, Unsupervised and supervised data classification via
   nonsmooth and global optimization.
5. Laurent El Ghaoui, Robust Optimization and Applications.
6. Başak A. Öztürk, Derivative Free Optimization methods:
   Application in Stirrer Configuration and Data Clustering.
Thank you very much!

               Questions, please?
1 sur 39

Recommandé

Karmarkar's Algorithm For Linear Programming Problem par
Karmarkar's Algorithm For Linear Programming ProblemKarmarkar's Algorithm For Linear Programming Problem
Karmarkar's Algorithm For Linear Programming ProblemAjay Dhamija
19.6K vues44 diapositives
Linear programming par
Linear programmingLinear programming
Linear programmingGanesh Solanke
4.6K vues43 diapositives
Linear Programming Problem par
Linear Programming ProblemLinear Programming Problem
Linear Programming ProblemA. Dally Maria Evangeline
276 vues15 diapositives
Econometria Jose Nieves par
Econometria Jose NievesEconometria Jose Nieves
Econometria Jose Nievespaquitootd
798 vues23 diapositives
Graphical RepresentationLinear programming par
Graphical RepresentationLinear programming Graphical RepresentationLinear programming
Graphical RepresentationLinear programming abhishekkumar4847
47 vues19 diapositives

Contenu connexe

Tendances

1 resource optimization 2 par
1 resource optimization 21 resource optimization 2
1 resource optimization 2shushay hailu
679 vues22 diapositives
Dynamic Programming par
Dynamic ProgrammingDynamic Programming
Dynamic Programmingparamalways
29.2K vues12 diapositives
Introduction to Dynamic Programming, Principle of Optimality par
Introduction to Dynamic Programming, Principle of OptimalityIntroduction to Dynamic Programming, Principle of Optimality
Introduction to Dynamic Programming, Principle of OptimalityBhavin Darji
14.9K vues11 diapositives
Dynamic programming par
Dynamic programmingDynamic programming
Dynamic programmingMelaku Bayih Demessie
14.8K vues22 diapositives
Global optimization par
Global optimizationGlobal optimization
Global optimizationbpenalver
3.1K vues61 diapositives
ADA complete notes par
ADA complete notesADA complete notes
ADA complete notesVinay Kumar C
59K vues151 diapositives

Tendances(20)

Dynamic Programming par paramalways
Dynamic ProgrammingDynamic Programming
Dynamic Programming
paramalways29.2K vues
Introduction to Dynamic Programming, Principle of Optimality par Bhavin Darji
Introduction to Dynamic Programming, Principle of OptimalityIntroduction to Dynamic Programming, Principle of Optimality
Introduction to Dynamic Programming, Principle of Optimality
Bhavin Darji14.9K vues
Global optimization par bpenalver
Global optimizationGlobal optimization
Global optimization
bpenalver3.1K vues
What Is Dynamic Programming? | Dynamic Programming Explained | Programming Fo... par Simplilearn
What Is Dynamic Programming? | Dynamic Programming Explained | Programming Fo...What Is Dynamic Programming? | Dynamic Programming Explained | Programming Fo...
What Is Dynamic Programming? | Dynamic Programming Explained | Programming Fo...
Simplilearn586 vues
Notion of an algorithm par Nisha Soms
Notion of an algorithmNotion of an algorithm
Notion of an algorithm
Nisha Soms1.5K vues
Elements of dynamic programming par Tafhim Islam
Elements of dynamic programmingElements of dynamic programming
Elements of dynamic programming
Tafhim Islam18.1K vues
Multiobjective optimization and trade offs using pareto optimality par Amogh Mundhekar
Multiobjective optimization and trade offs using pareto optimalityMultiobjective optimization and trade offs using pareto optimality
Multiobjective optimization and trade offs using pareto optimality
Amogh Mundhekar18.1K vues
Design & Analysis of Algorithms Lecture Notes par FellowBuddy.com
Design & Analysis of Algorithms Lecture NotesDesign & Analysis of Algorithms Lecture Notes
Design & Analysis of Algorithms Lecture Notes
FellowBuddy.com2.9K vues
Lecture 4 duality and sensitivity in lp par kongara
Lecture 4 duality and sensitivity in lpLecture 4 duality and sensitivity in lp
Lecture 4 duality and sensitivity in lp
kongara2.6K vues

Similaire à Derivative Free Optimization and Robust Optimization

Introduction to dynamic programming par
Introduction to dynamic programmingIntroduction to dynamic programming
Introduction to dynamic programmingAmisha Narsingani
523 vues22 diapositives
A brief study on linear programming solving methods par
A brief study on linear programming solving methodsA brief study on linear programming solving methods
A brief study on linear programming solving methodsMayurjyotiNeog
33 vues16 diapositives
Paper Study: Melding the data decision pipeline par
Paper Study: Melding the data decision pipelinePaper Study: Melding the data decision pipeline
Paper Study: Melding the data decision pipelineChenYiHuang5
266 vues38 diapositives
linearprogramingproblemlpp-180729145239.pptx par
linearprogramingproblemlpp-180729145239.pptxlinearprogramingproblemlpp-180729145239.pptx
linearprogramingproblemlpp-180729145239.pptxKOUSHIkPIPPLE
11 vues40 diapositives
A brief introduction to Searn Algorithm par
A brief introduction to Searn AlgorithmA brief introduction to Searn Algorithm
A brief introduction to Searn AlgorithmSupun Abeysinghe
225 vues43 diapositives
UNIT IV.pptx par
UNIT IV.pptxUNIT IV.pptx
UNIT IV.pptxSwarndeviKm
4 vues10 diapositives

Similaire à Derivative Free Optimization and Robust Optimization(20)

A brief study on linear programming solving methods par MayurjyotiNeog
A brief study on linear programming solving methodsA brief study on linear programming solving methods
A brief study on linear programming solving methods
MayurjyotiNeog33 vues
Paper Study: Melding the data decision pipeline par ChenYiHuang5
Paper Study: Melding the data decision pipelinePaper Study: Melding the data decision pipeline
Paper Study: Melding the data decision pipeline
ChenYiHuang5266 vues
linearprogramingproblemlpp-180729145239.pptx par KOUSHIkPIPPLE
linearprogramingproblemlpp-180729145239.pptxlinearprogramingproblemlpp-180729145239.pptx
linearprogramingproblemlpp-180729145239.pptx
KOUSHIkPIPPLE11 vues
Artificial Intelligence Course: Linear models par ananth
Artificial Intelligence Course: Linear models Artificial Intelligence Course: Linear models
Artificial Intelligence Course: Linear models
ananth1.4K vues
Linear programming class 12 investigatory project par Divyans890
Linear programming class 12 investigatory projectLinear programming class 12 investigatory project
Linear programming class 12 investigatory project
Divyans89014.7K vues
Automatic Generator Of Minimal Problem Solvers par Deja Lewis
Automatic Generator Of Minimal Problem SolversAutomatic Generator Of Minimal Problem Solvers
Automatic Generator Of Minimal Problem Solvers
Deja Lewis2 vues
4optmizationtechniques-150308051251-conversion-gate01.pdf par BechanYadav4
4optmizationtechniques-150308051251-conversion-gate01.pdf4optmizationtechniques-150308051251-conversion-gate01.pdf
4optmizationtechniques-150308051251-conversion-gate01.pdf
BechanYadav411 vues
Solvers and Applications with CP par iaudesc
Solvers and Applications with CPSolvers and Applications with CP
Solvers and Applications with CP
iaudesc828 vues
Analysis and Design of Algorithms par Bulbul Agrawal
Analysis and Design of AlgorithmsAnalysis and Design of Algorithms
Analysis and Design of Algorithms
Bulbul Agrawal417 vues

Plus de SSA KPI

Germany presentation par
Germany presentationGermany presentation
Germany presentationSSA KPI
2.7K vues24 diapositives
Grand challenges in energy par
Grand challenges in energyGrand challenges in energy
Grand challenges in energySSA KPI
1K vues51 diapositives
Engineering role in sustainability par
Engineering role in sustainabilityEngineering role in sustainability
Engineering role in sustainabilitySSA KPI
3.2K vues22 diapositives
Consensus and interaction on a long term strategy for sustainable development par
Consensus and interaction on a long term strategy for sustainable developmentConsensus and interaction on a long term strategy for sustainable development
Consensus and interaction on a long term strategy for sustainable developmentSSA KPI
869 vues17 diapositives
Competences in sustainability in engineering education par
Competences in sustainability in engineering educationCompetences in sustainability in engineering education
Competences in sustainability in engineering educationSSA KPI
846 vues21 diapositives
Introducatio SD for enginers par
Introducatio SD for enginersIntroducatio SD for enginers
Introducatio SD for enginersSSA KPI
886 vues39 diapositives

Plus de SSA KPI(20)

Germany presentation par SSA KPI
Germany presentationGermany presentation
Germany presentation
SSA KPI2.7K vues
Grand challenges in energy par SSA KPI
Grand challenges in energyGrand challenges in energy
Grand challenges in energy
SSA KPI1K vues
Engineering role in sustainability par SSA KPI
Engineering role in sustainabilityEngineering role in sustainability
Engineering role in sustainability
SSA KPI3.2K vues
Consensus and interaction on a long term strategy for sustainable development par SSA KPI
Consensus and interaction on a long term strategy for sustainable developmentConsensus and interaction on a long term strategy for sustainable development
Consensus and interaction on a long term strategy for sustainable development
SSA KPI869 vues
Competences in sustainability in engineering education par SSA KPI
Competences in sustainability in engineering educationCompetences in sustainability in engineering education
Competences in sustainability in engineering education
SSA KPI846 vues
Introducatio SD for enginers par SSA KPI
Introducatio SD for enginersIntroducatio SD for enginers
Introducatio SD for enginers
SSA KPI886 vues
DAAD-10.11.2011 par SSA KPI
DAAD-10.11.2011DAAD-10.11.2011
DAAD-10.11.2011
SSA KPI835 vues
Talking with money par SSA KPI
Talking with moneyTalking with money
Talking with money
SSA KPI756 vues
'Green' startup investment par SSA KPI
'Green' startup investment'Green' startup investment
'Green' startup investment
SSA KPI676 vues
From Huygens odd sympathy to the energy Huygens' extraction from the sea waves par SSA KPI
From Huygens odd sympathy to the energy Huygens' extraction from the sea wavesFrom Huygens odd sympathy to the energy Huygens' extraction from the sea waves
From Huygens odd sympathy to the energy Huygens' extraction from the sea waves
SSA KPI1.4K vues
Dynamics of dice games par SSA KPI
Dynamics of dice gamesDynamics of dice games
Dynamics of dice games
SSA KPI1.4K vues
Energy Security Costs par SSA KPI
Energy Security CostsEnergy Security Costs
Energy Security Costs
SSA KPI652 vues
Naturally Occurring Radioactivity (NOR) in natural and anthropic environments par SSA KPI
Naturally Occurring Radioactivity (NOR) in natural and anthropic environmentsNaturally Occurring Radioactivity (NOR) in natural and anthropic environments
Naturally Occurring Radioactivity (NOR) in natural and anthropic environments
SSA KPI4.8K vues
Advanced energy technology for sustainable development. Part 5 par SSA KPI
Advanced energy technology for sustainable development. Part 5Advanced energy technology for sustainable development. Part 5
Advanced energy technology for sustainable development. Part 5
SSA KPI581 vues
Advanced energy technology for sustainable development. Part 4 par SSA KPI
Advanced energy technology for sustainable development. Part 4Advanced energy technology for sustainable development. Part 4
Advanced energy technology for sustainable development. Part 4
SSA KPI882 vues
Advanced energy technology for sustainable development. Part 3 par SSA KPI
Advanced energy technology for sustainable development. Part 3Advanced energy technology for sustainable development. Part 3
Advanced energy technology for sustainable development. Part 3
SSA KPI903 vues
Advanced energy technology for sustainable development. Part 2 par SSA KPI
Advanced energy technology for sustainable development. Part 2Advanced energy technology for sustainable development. Part 2
Advanced energy technology for sustainable development. Part 2
SSA KPI542 vues
Advanced energy technology for sustainable development. Part 1 par SSA KPI
Advanced energy technology for sustainable development. Part 1Advanced energy technology for sustainable development. Part 1
Advanced energy technology for sustainable development. Part 1
SSA KPI1.2K vues
Fluorescent proteins in current biology par SSA KPI
Fluorescent proteins in current biologyFluorescent proteins in current biology
Fluorescent proteins in current biology
SSA KPI2K vues
Neurotransmitter systems of the brain and their functions par SSA KPI
Neurotransmitter systems of the brain and their functionsNeurotransmitter systems of the brain and their functions
Neurotransmitter systems of the brain and their functions
SSA KPI12.8K vues

Dernier

discussion post.pdf par
discussion post.pdfdiscussion post.pdf
discussion post.pdfjessemercerail
130 vues1 diapositive
Google solution challenge..pptx par
Google solution challenge..pptxGoogle solution challenge..pptx
Google solution challenge..pptxChitreshGyanani1
117 vues18 diapositives
UWP OA Week Presentation (1).pptx par
UWP OA Week Presentation (1).pptxUWP OA Week Presentation (1).pptx
UWP OA Week Presentation (1).pptxJisc
87 vues11 diapositives
Recap of our Class par
Recap of our ClassRecap of our Class
Recap of our ClassCorinne Weisgerber
74 vues15 diapositives
AI Tools for Business and Startups par
AI Tools for Business and StartupsAI Tools for Business and Startups
AI Tools for Business and StartupsSvetlin Nakov
105 vues39 diapositives
The Open Access Community Framework (OACF) 2023 (1).pptx par
The Open Access Community Framework (OACF) 2023 (1).pptxThe Open Access Community Framework (OACF) 2023 (1).pptx
The Open Access Community Framework (OACF) 2023 (1).pptxJisc
107 vues7 diapositives

Dernier(20)

UWP OA Week Presentation (1).pptx par Jisc
UWP OA Week Presentation (1).pptxUWP OA Week Presentation (1).pptx
UWP OA Week Presentation (1).pptx
Jisc87 vues
AI Tools for Business and Startups par Svetlin Nakov
AI Tools for Business and StartupsAI Tools for Business and Startups
AI Tools for Business and Startups
Svetlin Nakov105 vues
The Open Access Community Framework (OACF) 2023 (1).pptx par Jisc
The Open Access Community Framework (OACF) 2023 (1).pptxThe Open Access Community Framework (OACF) 2023 (1).pptx
The Open Access Community Framework (OACF) 2023 (1).pptx
Jisc107 vues
Structure and Functions of Cell.pdf par Nithya Murugan
Structure and Functions of Cell.pdfStructure and Functions of Cell.pdf
Structure and Functions of Cell.pdf
Nithya Murugan455 vues
Lecture: Open Innovation par Michal Hron
Lecture: Open InnovationLecture: Open Innovation
Lecture: Open Innovation
Michal Hron99 vues
REPRESENTATION - GAUNTLET.pptx par iammrhaywood
REPRESENTATION - GAUNTLET.pptxREPRESENTATION - GAUNTLET.pptx
REPRESENTATION - GAUNTLET.pptx
iammrhaywood91 vues
Narration ppt.pptx par TARIQ KHAN
Narration  ppt.pptxNarration  ppt.pptx
Narration ppt.pptx
TARIQ KHAN131 vues
Use of Probiotics in Aquaculture.pptx par AKSHAY MANDAL
Use of Probiotics in Aquaculture.pptxUse of Probiotics in Aquaculture.pptx
Use of Probiotics in Aquaculture.pptx
AKSHAY MANDAL95 vues
EIT-Digital_Spohrer_AI_Intro 20231128 v1.pptx par ISSIP
EIT-Digital_Spohrer_AI_Intro 20231128 v1.pptxEIT-Digital_Spohrer_AI_Intro 20231128 v1.pptx
EIT-Digital_Spohrer_AI_Intro 20231128 v1.pptx
ISSIP359 vues
Solar System and Galaxies.pptx par DrHafizKosar
Solar System and Galaxies.pptxSolar System and Galaxies.pptx
Solar System and Galaxies.pptx
DrHafizKosar89 vues
Are we onboard yet University of Sussex.pptx par Jisc
Are we onboard yet University of Sussex.pptxAre we onboard yet University of Sussex.pptx
Are we onboard yet University of Sussex.pptx
Jisc93 vues
Narration lesson plan.docx par TARIQ KHAN
Narration lesson plan.docxNarration lesson plan.docx
Narration lesson plan.docx
TARIQ KHAN108 vues

Derivative Free Optimization and Robust Optimization

  • 1. 4th International Summer School Achievements and Applications of Contemporary Informatics, Mathematics and Physics National University of Technology of the Ukraine Kiev, Ukraine, August 5-16, 2009 Nonsmooth Optimization Derivative Free Optimization and Robust Optimization Gerhard-Wilhelm Weber * and Başak Akteke-Öztürk Gerhard- Akteke- Institute of Applied Mathematics Middle East Technical University, Ankara, Turkey * Faculty of Economics, Management and Law, University of Siegen, Germany Center for Research on Optimization and Control, University of Aveiro, Portugal
  • 2. Introduction Mathematical Models • Experimental Data Analysis • Classification problems • Identification problems treated by: • Pattern Recognition SVM, Cluster Analysis, • Assignment and Allocation Neural Systems etc. • When these methods were born, the most developed and popular optimization tools were Linear and Quadratic Programming. • Optimization parts of these methods are reduced to LP and QP Linear Discriminant Analysis
  • 3. Introduction • progress in Optimization • new advanced tools, • Nonsmooth Analysis and • construct a mathematical model Nondifferentiable Opt. better suited for the problem under consideration • Most cases clustering problems are reduced to solving nonsmooth optimization problems. • We are interested in new methods for solving related nonsmooth problems (e.g., Semidefinite Programming, Semi-Infinite Programming, discrete gradient method and cutting angle method).
  • 4. Nonsmooth Optimization Problem: minimize subject to • : is nonsmooth at many points of interest do not have a conventional derivative at these points. • A less restrictive class of assumptions for than smoothness: convexity and Lipschitzness.
  • 6. Convex Sets A set is called convex if
  • 7. Convex Sets • The convex hull of a set : • The sets and coincide if and only if is convex. • The set is called a cone if for all , ; i.e., contains all positive multiples of its elements.
  • 9. Convex Functions • A set is called an epigraph of function . • Let be a convex set. A function is said to be convex if its epigraph is a convex set.
  • 10. Convex Functions • are differentiable (smooth) almost everywhere, • their minimizers are points where the function need not be differentiable, • standard numerical methods do not work • Examples of convex functions: – affinely linear: – quadratic: (c>0) – exponential:
  • 12. Convex Optimization • minimizing a convex function over a convex feasible set • Many applications. • Important, because: a strong duality theory any local minimum is a global minimum includes least-squares problems and linear programs as special cases can be solved efficiently and reliably
  • 13. Lipschitz Continuous • A function is called (locally) Lipschitz continuous, if for any bounded there exist a constant such that • Lipschitzness is a more restrictive property on functions than continuity, i.e., all Lipschitz functions are continuous, but they are not guaranteed to be smooth. • They possess a generalized gradient.
  • 15. Nonsmooth Optimization • We call the the set ∂f(x) subdifferential of f at x • Any vector v є ∂f(x) is a subgradient. • A proper convex function f is subdifferentiable at any point x є , if ∂f(x) is non-empty, convex and compact at x. • If the convex function f is continuously differentiable, then
  • 16. Nonsmooth Functions and Subdifferentials
  • 17. Generalized Derivatives • The generalized directional derivative of f at x in the direction g is defined as • If the function f is locally Lipschitz continuous, then the generalized directional derivative exists. • The set is called the (Clarke) subdifferential of the function f at a point
  • 18. Nonsmooth Optimization Nonsmooth optimization – more general problem of minimizing functions, – lack some, but not all, of the favorable properties of convex functions, – minimizers often are again points where the function is nondifferentiable.
  • 19. Cluster Analysis via Nonsmooth Opt. Given Problem: This is a partitioning clustering problem.
  • 22. Cluster Analysis via Nonsmooth Opt. • k is the number of clusters (given), • m is the number of available patterns (given), • is the j-th cluster’s center (to be found), • association weight of pattern , cluster j (to be found): • ( ) is an matrix, • objective function has many local minima.
  • 23. Cluster Analysis via Nonsmooth Opt. Suggestion (if k is not given a priori): • Start from a small enough number of clusters k and gradually increase the number of clusters for the analysis until a certain stopping criteria met. • This means: If the solution of the corresponding optimization problem is not satisfactory, the decision maker needs to consider a problem with k + 1 clusters, etc.. • This implies: One needs to solve repeatedly arising optimization problems with different values of k - a task even more challenging. • In order to avoid this difficulty, we suggest a step-by-step calculation of clusters.
  • 24. Cluster Analysis via Nonsmooth Opt. • k-means, h-means, j-means • dynamic programming • branch and bound • cutting planes • metaheuristics: simulated annealing, tabu search and genetic algorithms • an interior point method for minimum sum-of squares clustering problem • agglomerative and divisive hierarchical clustering incremental approach
  • 25. Cluster Analsysis via Nonsmooth Opt. Reformulated Problem: • A very complicated objective function: nonsmooth and nonconvex. • The number of variables in the nonsmooth optimization approach is k×n, before it was (m+n)×k.
  • 26. Robust Optimization • There is uncertainty or variation in the objective and constraint functions, due to parameters or factors that are either beyond our control or unknown. • Refers to the ability of the subject to cope well with uncertainties in linear, conic and semidefinite programming . • Applications in control, engineering design and finance. • Convex, modelled by SDP or cone quadratic programming. • Robust solutions are computed in polynomial time, via (convex) semidefinite programming problem.
  • 27. Robust Optimization • Let us examine Robust Linear Programming • By a worst case approach the objective is the maximum over all possible realizations of the objective • A robust feasible solution with the smallest possible value of the f(x) is sought. • Robust optimization is no longer a linear programming. The problem depends on the geometry of the uncertainty set U; i.e., if U is defined as an ellipsoid, the problem becomes a conic quadratic program.
  • 29. Robust Optimization • Considers that the uncertain parameter c belongs to a bounded, convex, uncertainty set • Stochastic Optimization: expected values, parameter vector u is modeled as a random variable with known distribution Robust Counterpart • Worst Case Optimization: the robust solution is the one that has the best worst case, i.e., it solves
  • 30. Robust Optimization • A complementary alternative to stochastic programming. • Seeks a solution that will have a “good” performance under many/most/all possible realizations of the uncertain input parameters. • Unlike stochastic programming, no distribution assumptions on uncertain parameters – each possible value equally important (this can be good or bad) • Represents a conservative viewpoint when it is worst-case oriented.
  • 31. Robust Optimization • Especially useful when – some of the problem parameters are estimates and carry estimation risk, – there are constraints with uncertain parameters that must be satisfied regardless of the values of these parameters, – the objective functions / optimal solutions are particularly sensitive to perturbations, – decision-maker can not afford low-probability high-magnitude risks.
  • 32. Derivative Free Optimization The problem is to minimize a nonlinear function of several variables • the derivatives (sometimes even the values) of this function are not available, • arise in modern physical, chemical and econometric measurements and in engineering applications, • computer simulation is employed for the evaluation of the function values. The methods are known as derivative free methods (DFO).
  • 33. Derivative Free Optimization Problem: • cannot be computed or just does not exist for every x , • is an arbitrary subset of , • is called the easy constraint, • the functions represent difficult constraints.
  • 34. Derivative Free Optimization Derivative free methods • build a linear or quadratic model of the objective function, • apply a trust-region or a line-search to optimize the model; derivative based methods use a Taylor polynomial -based model; DFO methods use interpolation, regression or other sample-based models.
  • 35. Derivative Free Optimization Six iterations of a trust-region algorithm.
  • 36. Semidefinite Programming • Optimization problems where the variable is not a vector but a symmetric matrix which is required to be positive semidefinite. • Linear Programming Semidefinite Programming vector of variables a symmetric matrix nonnegativity constraint a positive semidefinite constraint • SDP is convex, has a duality theory and can be solved by interior point methods.
  • 37. SVC via Semidefinite Programming • I try to reformulate the support vector clustering problem as a convex integer program and then relax it to a soft clustering formulation which can be feasibly solved by a 0-1 semidefinite program. • In the literature, k-means and clustering methods which use a graph cut model are reformulated as a semidefinite program and solved by using semidefinite programming relaxations.
  • 38. Some References 1. Aharon Ben-Tal and Arkadi Nemirovski, Robust optimization methodology and applications. 2. Adil Bagirov, Nonsmooth optimization approaches in data Classification. 3. Adil Bagirov, Derivative-free nonsmooth optimization and its applications. 4. A. M. Bagirov, A. M. Rubinov, N.V. Soukhoroukova and J. Yearwood, Unsupervised and supervised data classification via nonsmooth and global optimization. 5. Laurent El Ghaoui, Robust Optimization and Applications. 6. Başak A. Öztürk, Derivative Free Optimization methods: Application in Stirrer Configuration and Data Clustering.
  • 39. Thank you very much! Questions, please?