1. From the math to an F# interface
Implementing the D.E. algorithm
A numerical optimization library in F#
N. El Bachir1
1 FinBoost Limited
29 April 2010
N. El Bachir F# Solvers

2. From the math to an F# interface
Implementing the D.E. algorithm
Contents
1 From the math to an F# interface
The mathematical problem
Mapping the problem into an F# interface
2 Implementing the D.E. algorithm
Differential Evolution algorithm
F# implementation
N. El Bachir F# Solvers

3. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Optimization under constraints
Find x ∗ ∈ A sucht that
x ∗ = arg min f (x), (1)
x
s.t. g (x) (2)
Objective function f defined on domain A (A ⊆ Rn ) :
f : A→R (3)
Constraints g (x) an arbitrary Boolean combination of :
equations g (x) = 0,
weak inequalities g (x) ≥ 0, and
weak inequalities g (x) > 0.
N. El Bachir F# Solvers

4. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Some examples
Very simple : f (x) = x 2 , A = R and no constraint. Solution :
x∗ = 0
Another example : f (x, y ) = x − y ,A = R2 s.t.
−3x 2 + 2xy − y 2 ≥ −1. Solution : x ∗ = 0, y ∗ = 1 and
f (0, 1) = −1.
Third example : f (x, y ) = x 2 + (y − 1 )2 , A = R2 s.t. y ≥ 0 and
2
y ≥ x + 1.Solution : x ∗ = −0.25, y ∗ = 0.75 and
f (−0.25, 0.75) = 0.125.
We are interested in numerical algorithms to solve more
complex problems.
N. El Bachir F# Solvers

5. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Numerical optimization
Iterative procedure :
start with initial guess(es) x 0 ,
keep updating x j → x j+1 = F (x j ) according to some
transformation rule F ,
stop when some criteria are met
# of iterations,
convergence in x,
convergence in objective function.
To handle constraints :
Transform into unconstrained optimization
drop non-active inequalities.
search value of the variable that dominates the constraint.
Lagrange multipliers.
use of merit/penalty functions.
Feasible directions method.
Repairing/rejecting unfeasible solutions.
N. El Bachir F# Solvers

6. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Solve
let Solve (problem) (strategy) (stopWhen): Solution =
let rec helper (pblm) (stp): Solution =
let solution = strategy.oneStepSolve problem st
let isFinished,stopper =
solution.stop.Finished solution
match isFinished with
| true -> solution
| _ -> let solution =
{
best=solution.best
bestValue=
solution.bestValue
stop = stopper
}
helper pblm solution.stop
helper problem stop
N. El Bachir F# Solvers

7. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Main abstractions
type Problem =
abstract Domain : Domain with get
abstract Objective : RnOnDomain -> real
abstract Constraint : Constraint with get
and Constraint =
abstract isSatisfied : RnOnDomain -> bool
and Strategy =
abstract oneStepSolve :
Problem -> StopWhen -> Solution
and StopWhen =
abstract Finished : Solution -> bool*StopWhen
and Solution =
{
best : RnOnDomain
bestValue : real
stop : StopWhen
}
N. El Bachir F# Solvers

8. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Domains on Rn
type RnOnDomain =
val rn : Rn
val dim: int
new(x:Rn,domain:Domain) =
let rn =
if domain.contains(x) then x
else failwith ("Not in the domain")
let dim = match x with
| Real1 x -> 1
| RealN x -> Array.length x
{
rn = rn
dim = dim
}
N. El Bachir F# Solvers

9. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
More on domains
type Domain =
val lowBound : BoundVal[]
val upBound : BoundVal[]
new(bound:Bounds)=
let low,up =
let low_,up_= match bound with
| Real theBound ->
[|theBound.lower|],[|theBound.upper|]
| Realn theBound ->
theBound.lower,theBound.upper
if low_.Length = up_.Length then low_,up_
else failwith
("Invalid domain: " +
"# of upper bounds != # of lower bounds.")
{
lowBound = low
upBound = up
}
N. El Bachir F# Solvers

10. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
More on domains
member this.contains(x) : bool =
let X = match x with
| Real1 x -> [| x |]
| RealN x -> x
if not( X.Length = this.lowBound.Length )
then failwith
("Input has different dimensions than domain")
else
let above_lower =
[| for k in 0..(this.lowBound.Length - 1) ->
let x = X.[k]
match z with
| Inclusive y -> x >= y
| Exclusive y -> x > y |] //let below_upper = ...
Array.forall2
(fun x y -> x && y) above_lower below_upper
member this.newRn(x:Rn) = new RnOnDomain(x,this)
N. El Bachir F# Solvers

11. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Reals and intervals
type BoundVal =
| Inclusive of float
| Exclusive of float
type Range = {
lower : BoundVal
upper : BoundVal
}
type Range_nd = {
lower : BoundVal[]
upper : BoundVal[]
}
type Bounds =
| Real of Range
| Realn of Range_nd
type Rn =
| Real1 of float
| RealN of float[]
N. El Bachir F# Solvers

12. From the math to an F# interface The mathematical problem
Implementing the D.E. algorithm Mapping the problem into an F# interface
Simple example
let bisectPbm =
{ new Problem with
member this.Domain = domain1D
member this.Objective (x:RnOnDomain)
: float= objFunc
member this.Constraint = (
new PosConst domain1D
):> Constraint
}
let solution = (
Solve (bisectProblem) (
new BisectionMethod()) oneStep
).best.Rn
N. El Bachir F# Solvers

13. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
Quick intro to D.E.
Stochastic global optimization approach which mimicks
evolutionary concepts.
Initialization consists in picking an initial population randomly
Updating procedure (generation evolution) given by
Mutation,
Recombination,
Selection.
N. El Bachir F# Solvers

14. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
Initial population
D parameters. Initial population of size N, will evolve in each
generation G with selection
x i,G = [x1,i,G , . . . , xD,i,G ] (4)
Bounds for each parameter :
xjL ≤ xj,i,1 ≤ xjU (5)
Initial population randomly selected on the intervals
xjL , xjU (6)
N. El Bachir F# Solvers

15. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
A mutation strategy
For each member i of the population, a mutant is generated by
randomly picking distinct indices k1 , k2 , and k3 and combining
them. For example, using a mutation factor F ∈ [0, 2] :
v i,G +1 = x k1 ,G + F × x k2 ,G − x k3 ,G (7)
Recombination step will then create a trial member from a
mutant and a member from previous generation.
N. El Bachir F# Solvers

16. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
Recombination/Crossover
Given a crossover probability CR,
randj,i U[0, 1] for i = 1, . . . , N and j = 1, . . . , D
V a random integer from 1, . . . , D
Trial members for the next generation given by
v j,i,G +1 if randj,i ≤ CR or j = V
u j,i,G +1 = (8)
x j,i,G +1 otherwise
N. El Bachir F# Solvers

17. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
Selection of the fittest
Each trial member is compared to its previous generation and
the fittest is selected for each pair (x i,G , u i,G +1 ) :
u i,G +1 if f (u i,G +1 ) ≤ f (x i,G )
x i,G +1 = (9)
x i,G otherwise
The process of mutation, recombination and selection
continues until either a maximum number of iterations is
reached or some convergence criteria is attained.
N. El Bachir F# Solvers

18. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
Some snippets
Initialization of the population :
initpop = [| for j in 0 .. N-1 -> randomize domain |]
// randomize checks the bounds for each dim
// returns an array of rands, 1 per dim within the bounds
Mutation and crossover :
for n = 0 to N-1 do
for d = 0 to D-1 do
let ui=
if unitRand() < CR then
bm.[n].[d] + F*(pm1.[n].[d] - pm2.[n].[d])
else pop.[n].[d]
trial.[n].[d] <- ui
N. El Bachir F# Solvers

19. From the math to an F# interface Differential Evolution algorithm
Implementing the D.E. algorithm F# implementation
More snippets
Convergence criteria :
type MaxItOrConv (iter:int, tol:float) =
interface StopWhen with
member this.Finished solution =
let run_iter =
if iter = 0 then true
else false
let distance = solution.bestValue < tol
run_iter || distance, solution.stop
N. El Bachir F# Solvers