Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Model Selection based on
Regional Error Estimation of Surrogates (REES)
Ali Mehmani, Souma Chowdhury, Jie Zhang, Weiyang T...
Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual...
Surrogate model: Model selection
3
Statistical model selection approaches provide support
information for users
• to selec...
Surrogate model: Model selection
4
Types of model Types of basis/kernel Parameter estimation
•RBF,
•Kriging,
•E-RBF,
•SVR,...
Research Objective
• Investigate the effectiveness of a Regional Error Estimation of
Surrogate (REES) to select the best s...
Presentation Outline
6
• Surrogate model selection
• Regional Error Estimation of Surrogate
• Numerical examples: benchmar...
Presentation Outline
7
• Surrogate model selection
• Regional Error Estimation of Surrogate
• Numerical examples: benchmar...
Surrogate model selection
8
• Size and location of sample points,
• Dimension and level of a noise,
• Application domain,
...
Surrogate model: Model selection
Types of model Types of basis/kernel Parameter estimation
•RBF,
•Kriging,
•E-RBF,
•SVR,
•...
Presentation Outline
10
• Review of surrogate model error measurement methods
• Regional Error Estimation of Surrogate (RE...
REES: Concept
11
Model accuracy Available resources
In general, this concept can be applied to different types of approxim...
REES: Methodology
12
 The REES method formulates the variation of error as a
function of the number of training points us...
Step 1 : Generation of sample data
The entire set of sample points is represented by 𝑿 .
13
Sample Point
Step 2 : Estimati...
 A position of sample points which are selected as training
points, at each iteration, is critical to the surrogate accur...
MedianofRAEs
Number of Training Points
It. 1
t1 t2 t3 t4
Momed
Choose number of iterations, Nit
Choose number of combinati...
MedianofRAEs
t1 t2 t3 t4
It. 2It. 1
Momed
variation of the error with sample density
Number of Training Points
MedianofRAEs
t1 t2 t3 t4
It. 3It. 2It. 1
Momed
variation of the error with sample density
Number of Training Points
MedianofRAEs
t1 t2 t3 t4
It. 3It. 2It. 1 It. 4
Momed
variation of the error with sample density
Number of Training Points
Step 3 : Prediction of error in the final surrogate
 The final surrogate model is constructed using the full set of train...
ModeofMedianofRAEs
Number of Training Points
t1 t2 t3 t4
It. 3It. 2It. 1 It. 4
Predicted Overall Error
Momed
Prediction of...
t1 t2 t3 t4
Predicted Overall Error
Momed
Momax
Number of Training Points
Prediction of error in the final surrogate
It. 3...
t1 t2 t3 t4
Predicted Overall Error
Momed
Momax
Predicted Maximum Error
Number of Training Points
Prediction of error in t...
Presentation Outline
23
• Review of surrogate model error measurement methods
• Regional Error Estimation of Surrogate
• N...
Numerical Examples
24
 The effectiveness of the model selection based on REES is explored
to select the best surrogate be...
Numerical Examples
Branin-Hoo (2 variables) Hartmann (6 variables)
Dixon & Price (18 variables)Dixon & Price (12 variables...
Numerical Examples
VESD regression models used to predict the maximum error
Dixon & Price (18 variables)Dixon & Price (12 ...
Numerical Examples
Wind Farm Power Generation
27
Surrogates are developed using Kriging, RBF, E-RBF, and QRS to
represent ...
Numerical Examples
28
Model Selection based the actual error on additional test points
Set Number of additional test point...
Numerical Examples
29
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
Model selection based on the overall ...
Numerical Examples
30
Model Selection based the Normalized Prediction Sum of Square (PRESS)
Set Number of additional train...
Numerical Examples
31
Model selection based on the n-PRESS
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
...
Numerical Examples
32
Model selection based on the maximum error estimated using REES
Function
Surrogate model selection
R...
Numerical Examples
33
% Success of Model Selection (overall error)
REES PRESS
W considering QRS 100% 0%
W/OUT considering ...
Concluding Remarks
34
 We developed a new model selection approach to select the best
surrogate among available surrogate...
Acknowledgement
35
 I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Prof.
Souma C...
36
Thank you
Questions
and
Comments
Numerical Examples
37
Numerical setup for test problems
Prochain SlideShare
Chargement dans…5
×

ModelSelection1_WCSMO_2013_Ali

380 vues

Publié le

The analysis of complex system behavior often demands expensive experiments or computational simulations. Surrogates modeling techniques are often used to provide a tractable and inexpensive approximation of such complex system behavior. Owing to the lack of knowledge regarding the suitability of particular surrogate modeling techniques, model selection approach can be helpful to choose the best surrogate technique. Popular model selection approaches include: (i) split sample, (ii) cross-validation, (iii) bootstrapping, and (iv) Akaike's information criterion (AIC) (Queipo et al. 2005; Bozdogan et al. 2000). However, the effectiveness of these model selection methods is limited by the lack of accurate measures of local and global errors in surrogates.

This paper develops a novel and model-independent concept to quantify the local/global reliability of surrogates, to assist in model selection (in surrogate applications). This method is called the Generalized-Regional Error Estimation of Surrogate (G-REES). In this method, intermediate surrogates are iteratively constructed over heuristic subsets of the available sample points (i.e., intermediate training points), and tested over the remaining available sample points (i.e., intermediate test points). The fraction of sample points used as intermediate training points is fixed at each iteration, with the total number of iterations being pre-specified. The estimated median and maximum relative errors for the heuristic subsets at each iteration are used to fit a distribution of the median and maximum error, respectively. The statistical mode of the median and the maximum error distributions are then determined. These mode values are then represented as functions of the density of training points (at the corresponding iteration). Regression methods, called Variation of Error with Sample Density (VESD), are used for this purpose. The VESD models are then used to predict the expected median and maximum errors, when all the sample points are used as training points.

The effectiveness of the proposed model selection criterion is explored to find the best surrogate between candidates including: (i) Kriging, (ii) Radial Basis Functions (RBF), (iii) Extended Radial Basis Functions (ERBF), and (iv) Quadratic Response Surface (QRS), for standard test functions and a wind farm capacity factor function. The results will be compared with the relative accuracy of the surrogates evaluated on additional test points, and also with the prediction sum of square (PRESS) error given by leave-one-out cross-validation.

The application of G-REES to a standard test problem with two design variables (Branin-hoo function) show that the proposed method predicts the median and the maximum value of the global error with a higher level of confidence compared to PRESS. It also shows that model selection based on G-REES method is significantly more reliable than that currently performed using error measures such as PRESS. The

Publié dans : Formation
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

ModelSelection1_WCSMO_2013_Ali

  1. 1. Model Selection based on Regional Error Estimation of Surrogates (REES) Ali Mehmani, Souma Chowdhury, Jie Zhang, Weiyang Tong, and Achille Messac Syracuse University, Department of Mechanical and Aerospace Engineering 10th World Congress on Structural and Multidisciplinary Optimization May 19 - 24, 2013, Orlando, FL
  2. 2. Surrogate model • Surrogate models are commonly used for providing a tractable and inexpensive approximation of the actual system behavior in many routine engineering analysis and design activities: 2
  3. 3. Surrogate model: Model selection 3 Statistical model selection approaches provide support information for users • to select a best model, • to select a best kernel function, and • to determine an optimum model’s parameter.
  4. 4. Surrogate model: Model selection 4 Types of model Types of basis/kernel Parameter estimation •RBF, •Kriging, •E-RBF, •SVR, •QRS, • … •Linear •Gaussian •Multiquadric •Inverse multiquadric •Kriging •… •Shape parameter in RBF, •Smoothness and width parameters in Kriging, •Kernel parameter in SVM, • … 𝑓 𝑥 = 𝑖=1 𝑛 𝑤𝑖 ψ( 𝑥 − 𝑥 𝑖 ) RBF ψ 𝑟 = (𝑟2 + 𝑐2 ) 1/2 𝑟= 𝑥 − 𝑥 𝑖 Multiquadric Shape parameter ψ 𝑟 = (𝑟2 + 𝒄2 ) 1/2
  5. 5. Research Objective • Investigate the effectiveness of a Regional Error Estimation of Surrogate (REES) to select the best surrogate model based on the level of accuracy. 5 Overall fidelity information Minimum fidelity information REES e.g., Hybrid surrogate e.g., Conservative surrogate
  6. 6. Presentation Outline 6 • Surrogate model selection • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  7. 7. Presentation Outline 7 • Surrogate model selection • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  8. 8. Surrogate model selection 8 • Size and location of sample points, • Dimension and level of a noise, • Application domain, • … Suitable Surrogate Error measures are used to select the best surrogate  lack of any general guidelines regarding the suitability of different surrogate models for different applications Application-based model selection (Manual selection) Error-based model selection (Automatic Selection)
  9. 9. Surrogate model: Model selection Types of model Types of basis/kernel Parameter estimation •RBF, •Kriging, •E-RBF, •SVR, •QRS, • … •Linear •Gaussian •Multiquadric •Inverse multiquadric •Kriging •… •Shape parameter in RBF, •Smoothness and width parameters in Kriging, •Kernel parameter in SVM, • …  Split samples (or holdout samples)  Bootstrapping  Cross-Validation (Predictive Sum of Square based on k-fold or leave-one-out cv)  Akaike information criterion (AIC), and  Schwarz's Bayesian information criterion (BIC) There exist many parameter/model/kernel selection approaches 9
  10. 10. Presentation Outline 10 • Review of surrogate model error measurement methods • Regional Error Estimation of Surrogate (REES) • Numerical examples: benchmark and an engineering design problems
  11. 11. REES: Concept 11 Model accuracy Available resources In general, this concept can be applied to different types of approximation models; - Surrogate modeling, - Finite Element Analysis, and - ...
  12. 12. REES: Methodology 12  The REES method formulates the variation of error as a function of the number of training points using intermediate surrogates.  This formulation is used to predict the level of error in the final surrogate.
  13. 13. Step 1 : Generation of sample data The entire set of sample points is represented by 𝑿 . 13 Sample Point Step 2 : Estimation of the variation of error with sample density Test Point Training Point First Iteration: Test Point Training Point Second Iteration: Test Point Training Point Third Iteration: Test Point Training Point Forth Iteration: Training Point Final Surrogate: REES: Methodology
  14. 14.  A position of sample points which are selected as training points, at each iteration, is critical to the surrogate accuracy. 14  Intermediate surrogates are iteratively constructed (at each iteration) over a heuristic subsets of sample points. REES: Methodology
  15. 15. MedianofRAEs Number of Training Points It. 1 t1 t2 t3 t4 Momed Choose number of iterations, Nit Choose number of combinations, Kt Define intermediate training and test points Construct an intermediate surrogate Estimate Median and Maximum errors Fit a distribution over all combinations Determine Momedian and Momaximum errors FOR t=1,..,Nit FOR k=1,…, Kt variation of the error with sample density
  16. 16. MedianofRAEs t1 t2 t3 t4 It. 2It. 1 Momed variation of the error with sample density Number of Training Points
  17. 17. MedianofRAEs t1 t2 t3 t4 It. 3It. 2It. 1 Momed variation of the error with sample density Number of Training Points
  18. 18. MedianofRAEs t1 t2 t3 t4 It. 3It. 2It. 1 It. 4 Momed variation of the error with sample density Number of Training Points
  19. 19. Step 3 : Prediction of error in the final surrogate  The final surrogate model is constructed using the full set of training data.  Regression models are applied to relate the evaluated error at each iteration to the size of training points,  These regression models are called the variation of error with sample density (VESD). The regression models are used to predict the level of the error in the final surrogate. 19 Exponential regression model Multiplicative regression model Linear regression model  In the choice of these functions we assume a smooth monotonic decrease of the error with the training point density. REES: Methodology
  20. 20. ModeofMedianofRAEs Number of Training Points t1 t2 t3 t4 It. 3It. 2It. 1 It. 4 Predicted Overall Error Momed Prediction of error in the final surrogate
  21. 21. t1 t2 t3 t4 Predicted Overall Error Momed Momax Number of Training Points Prediction of error in the final surrogate It. 3It. 2It. 1 It. 4 Mode of maximum error distribution at each iteration
  22. 22. t1 t2 t3 t4 Predicted Overall Error Momed Momax Predicted Maximum Error Number of Training Points Prediction of error in the final surrogate It. 3It. 2It. 1 It. 4
  23. 23. Presentation Outline 23 • Review of surrogate model error measurement methods • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  24. 24. Numerical Examples 24  The effectiveness of the model selection based on REES is explored to select the best surrogate between all candidates including (i) Kriging, (ii) RBF, (iii) E-RBF, and (iv) Quadratic Response Surface (QRS) on four benchmark problems and an engineering design problem.  The results of the REES method in selecting the best surrogate are compared with the model selection based on actual errors evaluated using additional test points, and model selection based on a normalized PRESS.
  25. 25. Numerical Examples Branin-Hoo (2 variables) Hartmann (6 variables) Dixon & Price (18 variables)Dixon & Price (12 variables) VESD regression models used to predict the overall error
  26. 26. Numerical Examples VESD regression models used to predict the maximum error Dixon & Price (18 variables)Dixon & Price (12 variables) Branin-Hoo (2 variables) Hartmann (6 variables)
  27. 27. Numerical Examples Wind Farm Power Generation 27 Surrogates are developed using Kriging, RBF, E-RBF, and QRS to represent the power generation of an array-like wind farm. VESD used to predict the overall error VESD used to predict the maximum error
  28. 28. Numerical Examples 28 Model Selection based the actual error on additional test points Set Number of additional test points 𝑵 𝒕𝒆𝒔𝒕 for all surrogate candidates, 𝒋 = 𝟏, … , 𝑱 for 𝒊 = 𝟏, … , 𝑵𝒕𝒆𝒔𝒕 𝑦𝑖 = System (𝑥𝑖) 𝑦𝑖 = Surrogate(𝑥𝑖) 𝑅𝐴𝐸𝑖 = 𝑦𝑖 − 𝑦𝑖 𝑦𝑖 Fit a distribution over all RAEs Determine the mode of the error distribution as an actual error Select the best surrogate with the smallest error
  29. 29. Numerical Examples 29 Function Surrogate model selection Rank 1 Rank 2 Rank 3 Rank 4 Model selection based on the overall error estimated using REES Branin-Hoo ERBF Kriging RBF QRS Hartmann - 6 QRS Kriging RBF ERBF Dixon and Price - 12 ERBF QRS Kriging RBF Dixon and Price - 18 ERBF QRS Kriging RBF Wind Farm Kriging QRS RBF ERBF Model selection based on the actual error Function Surrogate model selection Rank 1 Rank 2 Rank 3 Rank 4 Branin-Hoo ERBF RBF Kriging QRS Hartmann - 6 QRS Kriging RBF ERBF Dixon and Price - 12 ERBF Kriging RBF QRS Dixon and Price - 18 ERBF Kriging RBF QRS Wind Farm Kriging RBF ERBF QRS
  30. 30. Numerical Examples 30 Model Selection based the Normalized Prediction Sum of Square (PRESS) Set Number of additional training points 𝑵 for all surrogate candidates, 𝒋 = 𝟏, … , 𝑱 for 𝒊 = 𝟏, … , 𝑵 𝑦𝑖 = Surrogate(𝑥𝑖) 𝑅𝐴𝐸 𝐶𝑉 𝑖 = 𝑦−𝑖 𝑖 − 𝑦𝑖 𝑦𝑖 Fit a distribution over all 𝑅𝐴𝐸 𝐶𝑉 Determine the root mean square of errors (RAEs), n-PRESS error Select the best surrogate with the smallest n-PRESS error 𝑦−𝑖 𝑖 = Intermediate Surrogate(𝑥𝑖)
  31. 31. Numerical Examples 31 Model selection based on the n-PRESS Function Surrogate model selection Rank 1 Rank 2 Rank 3 Rank 4 Model selection based on the RMSE on additional test points Function Surrogate model selection Rank 1 Rank 2 Rank 3 Rank 4 Wind Farm QRS ERBF Kriging RBF Dixon and Price - 18 QRS ERBF RBF Kriging Dixon and Price - 12 QRS ERBF Kriging RBF Hartmann - 6 Kriging QRS RBF ERBF Branin-Hoo ERBF Kriging RBF QRS Wind Farm Kriging ERBF RBF QRS Dixon and Price - 18 ERBF RBF Kriging QRS Dixon and Price - 12 ERBF RBF Kriging QRS Hartmann - 6 RBF Kriging ERBF QRS Branin-Hoo RBF ERBF Kriging QRS
  32. 32. Numerical Examples 32 Model selection based on the maximum error estimated using REES Function Surrogate model selection Rank 1 Rank 2 Rank 3 Rank 4 Function 75th percentile of the RMSE Rank 1 Rank 2 Rank 3 Rank 4 75th percentile of the 𝐑𝐀𝐄 𝐂𝐕 Rank 1 Rank 2 Rank 3 Rank 4 Model selection based on the 75th percentile of the RMSE and 𝐑𝐀𝐄 𝐂𝐕 Wind Farm Kriging QRS ERBF RBF Dixon and Price - 18 ERBF QRS Kriging RBF Dixon and Price - 12 ERBF QRS Kriging RBF Hartmann - 6 RBF Kriging QRS ERBF Branin-Hoo ERBF RBF Kriging QRS Wind Farm Kriging QRS ERBF RBF QRS ERBF Kriging RBF Dixon and Price - 18 ERBF RBF Kriging QRS QRS ERBF RBF Kriging QRS ERBF Kriging RBF QRS RBF Kriging ERBF QRS Kriging RBF ERBF Dixon and Price - 12 ERBF RBF Kriging QRS Hartmann - 6 Kriging RBF QRS ERBF Branin-Hoo ERBF RBF Kriging QRS
  33. 33. Numerical Examples 33 % Success of Model Selection (overall error) REES PRESS W considering QRS 100% 0% W/OUT considering QRS 100% 40% % Success of Model Selection (maximum error) REES 𝐑𝐀𝐄 𝐂𝐕 W considering QRS 80% 0% W/OUT considering QRS 80% 40% A Summary and Comparison
  34. 34. Concluding Remarks 34  We developed a new model selection approach to select the best surrogate among available surrogate models based on the level of accuracy.  The REES method is defined based on the hypothesis that: “The accuracy of the approximation model is related to the amount of available resources”  The REES method proposes two error measures; Overall error measure, Maximum error measure to assess the confidence level of the surrogate model.  The preliminary results on benchmark and wind farm power generation problems indicate that in all of cases the REES method selects the best surrogate with a higher level of confidence in comparison to the n- PRESS.
  35. 35. Acknowledgement 35  I would like to acknowledge my research adviser Prof. Achille Messac, and my co-adviser Prof. Souma Chowdhury for their immense help and support in this research.  Support from the NSF Awards is also acknowledged.
  36. 36. 36 Thank you Questions and Comments
  37. 37. Numerical Examples 37 Numerical setup for test problems

×