This document presents a new 3-level surrogate model selection approach that simultaneously selects the best model type, kernel function, and hyperparameters. It uses Regional Error Estimation of Surrogates (REES) to quantify the median and maximum errors of different surrogates. The approach is tested on benchmark problems using radial basis function, Kriging, and support vector regression models. Results show at least 60% reduction in error compared to traditional methods, demonstrating the effectiveness of the new approach. Future work will apply the method to more complex problems and develop an online platform for collaborative surrogate model selection.

1. A Novel Approach to Simultaneous Selection of
Surrogate Models, Constitutive Kernels, and
Hyper-parameter Values
Ali Mehmani*, Souma Chowdhury#, and Achille Messac#
* Syracuse University, Department of Mechanical and Aerospace Engineering
# Mississippi State University, Bagley College of Engineering
10th Multi-Disciplinary Design Optimization Conference
AIAA Science and Technology Forum and Exposition
January 13 – 17, 2014 National Harbor, Maryland

2. Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual system behavior in many
routine engineering analysis and design activities:
2

3. Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual system behavior in many
routine engineering analysis and design activities:
풘풊 흍( 풙 − 풙풊 )
3
Model Type Kriging RBF SVR . . .
Linear Exponential Gaussian Cubic Multiquadric . . .
Kernel / basis function
Correlation parameter Shape parameter . . .
Hyper-parameter
풇 풙 =
풏
풊=ퟏ
흍 풓 = (풓ퟐ + 풄ퟐ) ퟏ/ퟐ
풓= 풙 − 풙풊
풄풍풐풘풆풓 < 풄 < 풄풖풑풑풆풓

4. Research Objective
 Develop a new model selection approach, which
simultaneously select the best model type, kernel function, and
hyper-parameter.
Types of model Types of basis/kernel Hyper-parameter(s)
4
• RBF,
•Kriging,
• E-RBF,
• SVR,
•QRS,
• …
• Linear
•Gaussian
• Multiquadric
• Inverse multiquadric
•Kriging
• …
• Shape parameter in RBF,
• Smoothness and width
parameters in Kriging,
•Kernel parameter in SVM,
• …

5. Presentation Outline
5
• Surrogate model selection
• REES-based Model Selection
• 3-Level model selection
• Regional Error Estimation of Surrogate (REES)
• Numerical Examples
• Concluding Remarks

6. Surrogate model selection
6
Experienced-based model selection
• Dimension and nature of sample points,
• Level of a noise,
• Application domain,
• …
Suitable Surrogate
Automated model selection
Error measures are used to
select the best surrogate
• RMSE,
• Cross-validation,
• REES,
• …
 Hyper-parameter selection (Kriging-Guassian) using cross validation and
maximum likelihood estimation (Martin and Simpson)
 Model type and basis function selection using cross validation (Viana and Haftka)
 Model type selection using leave-one-out cross validation (Drik Gorisson et al.)

7. 3-Level model selection
 In 3-level model selection, the selection criteria could depend
7
on the user preference.
Standard surrogate-based analysis
Structural optimization applications
lower median error
lower maximum error

8. 3-Level model selection
8
Median error
Maximum error
Two model selection criteria
evaluated using advanced surrogate error
estimation method presented in REES
 Depending on the problem and the available data set, the
median and maximum errors might be
mutually conflicting
mutually promoting
Pareto models
A single optimum model

9. 3-Level model selection
 To implement a 3-level model selection, two approaches are
9
proposed:
(i) Cascaded technique, and
(ii) One-Step technique.

10. 3-Level model selection
10
Cascaded technique
 For each candidate kernel function, hyper-parameter optimization is
performed to minimize the median and maximum error.
 Hyper-parameter optimization is
the process of quantitative search to
find optimum hyper-parameter
value(s).
 Post hyper-parameter optimizations,
Pareto filter is used to reach the final
Pareto models.

11. 3-Level model selection
11
Cascaded technique
Solutions of the hyper-parameter optimization in the cascaded
technique for multiquadric basis function of RBF surrogate for
Baranin-hoo function

12. 3-Level model selection
The three-level model selection could also be performed by solving
a single uniquely formulated mixed integer nonlinear
programming (MINLP) problem.
12
One-Step technique
 To escape the potentially high computational cost of the cascaded
technique
Subjected to
model type
basis function
hyper-parameter(s)

13. Regional Error Estimation of Surrogate
(REES)
The REES method is derived from the hypothesis that the accuracy of
approximation models is related to the amount of data resources
leveraged to train the model.
• Mehmani, A., Chowdhury, S., Zhang, Jie, and Messac, A., “Quantifying Regional Error in
Surrogates by Modeling its Relationship with Sample Density,” 54th
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference,
Paper No. AIAA 2013-1751, Boston, Massachusetts, April 8-11, 2013.
• Mehmani, et. al., “Model Selection based on Generalized-Regional Error Estimation for
Surrogate,” 10th World Congress on Structural and Multidisciplinary Optimization, Paper No.
5447, Orlando, Florida, May 19-24, 2013.
• Mehmani, et. al.., “Regional Error Estimation of Surrogates (REES),” 14th AIAA/ISSMO
Multidisciplinary Analysis and Optimization Conference, Paper No. AIAA 2012-5707,
Indianapolis, Indiana, September 17-19, 2012.
13

14. REES
Surrogate Model
Training Sample Point
Point

15. 1st Iter.
2nd Iter.
3rd Iter.
4th Iter.
Error
Surrogate
ε1
ε2
ε3
ε4
?
Training Point
Test Point
8 16
12 12
16 8
20 4
24 0
1st
2nd
3rd
4th
REES

16. Training Point
Test Point
1st
Median of RAEs
Intermediate Actual model
surrogate model
ε = 풎풆풅 |
풇풊 − 풇풊
1st
풇풊
| ,
퐢 = ퟏ, … , ퟏퟔ
...
REES
16

17. 17
Median of RAEs
Momed
It. 1
REES
t1 t2 t3 t4
Number of Training Points

18. 18
Median of RAEs
t1 t2 t3 t4
Number of Training Points
Momed
It. 1 It. 2
REES

19. 19
Median of RAEs
REES
t1 t2 t3 t4
Number of Training Points
Momed
It. 1 It. 2 It. 3

20. 20
It. 1 It. 3
Median of RAEs
It. 2
t1 t2 t3 t4
Number of Training Points
Momed
It. 4
Model of Median
REES

21. 21
It. 1 It. 3
Median of RAEs
It. 2
t1 t2 t3 t4
Number of Training Points
Momed
It. 4
Model of Median
Predicted Median Error
REES

22. Predicted Maximum Error
22
It. 1 It. 3
Median of RAEs
It. 2
t1 t2 t3 t4
Number of Training Points
Momed
It. 4
Model of Median
Momax Mode of maximum
error distribution at
each iteration
Predicted Median Error
REES

23. The effectiveness of the new 3-level model selection method is
investigated by considering the following three candidate surrogates:
Model type Kernel function Hyper-parameter
RBF
Kriging
The methods are implemented on three benchmark problems and an
engineering design problem are tested.
23
Gaussian basis function
Multiquadric basis function
Gaussian correlation function
Exponential correlation function
Radial basis kernel function
Sigmoid kernel function
SVR
Numerical Examples

24. 24
Numerical Examples
Numerical Setting
 The numerical settings for the implementation of REES-based model selection
for the benchmark problems
 The numerical settings for the hyper-parameter optimization
24

25. Hyper-parameter optimization of Cascaded technique in different surrogate type and
Kernel functions for Branin-Hoo function with 2 design variables
25
Numerical Examples

26. 26
Numerical Examples
Numerical Setting
 The numerical settings for One-Step technique
Integer design variables

27. 27
Results; Branin-hoo 2 design variables
Computational cost
One-Step Technique
RBF-Multiquadric
Cascaded technique

28. 28
Results; Branin-hoo 2 design variables
One-Step Technique
Cascaded technique
Actual error
RBF-Multiquadric
RBF-Multiquadric

29. Results; Hartmann function with 6 design variables
29
Computational cost
One-Step Technique
SVR-Radial basis
Cascaded technique

30. Results; Hartmann function with 6 design variables
30
One-Step Technique
SVR-Radial basis
Cascaded technique
Actual error
SVR-Radial basis

31. Results; Dixon & Price function with 18 design variables
31
Computational cost
One-Step Technique
RBF-Multiquadric
Cascaded technique

32. Results; Dixon & Price function with 18 design variables
32
One-Step Technique
RBF-Multiquadric
Cascaded technique
Actual error
RBF-Multiquadric

33. 33
Numerical Examples
Emed Emed Emed
Emax Emax Emax
Initial Value Final Solution
RBF-Multiquadric (C=0.9)

34. 34
Concluding Remarks
 A new 3-level model selection approach is developed to select the best
surrogate among available surrogate candidates based on the level of
accuracy.
(i) model type selection,
(ii) kernel function selection, and
(iii) hyper-parameter selection.
 This approach is based on the model independent error measure given by the
Regional Error Estimation of Surrogates (REES) method.
 The preliminary results on problems indicate at least 60% reduction in
maximum and median error values.

35. 35
Future Work
 Implementation of One-Step technique with
 larger pool of surrogates with different number of kernels
 higher dimensional and more computationally intensive problems
 Develop an open online platform called Collaborative Surrogate
Model Selection (COSMOS) to allow users to submit
- training data for identifying an ideal model from existing pool
of surrogate models, and
- their own new surrogate into the pool of surrogate candidates.
If interested in COSMOS please contact me at amehmani@syr.edu

36. Acknowledgement
 I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Dr.
Souma Chowdhury for their immense help and
support in this research.
 I would also like to thank my friend and colleague
Weiyang Tong for his valuable contributions to this
paper.
 Support from the NSF Awards is also
acknowledged.
36

37. Questions
and
Comments
37
Thank you

38. Quantifying the mode of median and maximum errors
38

39. 39
A chi-square (χ2) goodness-of-fit criterion is used to select the
type of distribution from a list of candidates such as lognormal,
Gamma, Weibull, logistic, log logistic, inverse Gaussian, and
generalized extreme value distribution.