This paper investigates the characterization of the uncertainty in the prediction of surrogate models. In the practice of engineering, where predictive models are pervasively used, the knowledge of the level of modeling error in any region of the design space is uniquely helpful for design exploration and model improvement. The lack of methods that can explore the spatial variation of surrogate error levels in a wide variety of surrogates (i.e., model-independent methods) leaves an important gap in our ability to perform design domain exploration. We develop a novel framework, called domain segmentation based on uncertainty in the surrogate (DSUS) to segregate the design domain based on the level of local errors. The errors in the surrogate estimation are classified into physically meaningful classes based on the user's understanding of the system and/or the accuracy requirements for the concerned system analysis. The leave-one-out cross-validation technique is used to quantity the local errors. Support vector machine (SVM) is implemented to determine the boundaries between error classes, and to classify any new design point into the pertinent error class. We also investigate the effectiveness of the leave-one-out cross-validation technique in providing a local error measure, through comparison with actual local errors. The utility of the DSUS framework is illustrated using two different surrogate modeling methods: (i) the Kriging method and (ii) the adaptive hybrid functions (AHF). The DSUS framework is applied to a series of standard test problems and engineering problems. In these case studies, the DSUS framework is observed to provide reasonable accuracy in classifying the design-space based on error levels. More than 90% of the test points are accurately classified into the appropriate error classes.

References

References
1.
Haldar
,
A.
, and
Mahadevan
,
S.
,
2000
,
Probability, Reliability, and Statistical Methods in Engineering Design
,
Wiley
,
New York
.
2.
Picheny
,
V.
,
2009
, “
Improving Accuracy and Compensating for Uncertainty in Surrogate Modeling
,” Ph.D. thesis, Aerospace Engineering, University of Florida, Gainesville, FL.
3.
Keane
,
A. J.
, and
Nair
,
P. B.
,
2005
,
Computational Approaches for Aerospace Design: The Pursuit of Excellence
,
Wiley
,
New York
.
4.
Myers
,
R. H.
, and
Montgomery
,
D. C.
,
2002
,
Response Surface Methodology: Process and Product Optimization Using Designed Experiments
,
2nd ed.
,
Wiley
,
New York
.
5.
Giunta
,
A. A.
, and
Watson
,
L.
,
1998
, “
A Comparison of Approximation Modeling Techniques: Polynomial Versus Interpolating Models
,” NASA, Technical Report AIAA-98-4758.
6.
Sakata
,
S.
,
Ashida
,
F.
, and
Zako
,
M.
,
2003
, “
Structural Optimization Using Kriging Approximation
,”
Comput. Methods Appl. Mech. Eng.
,
192
(
7–8
), pp.
923
939
.10.1016/S0045-7825(02)00617-5
7.
Cressie
,
N.
,
1993
,
Statistics for Spatial Data
,
Wiley
,
New York
.
8.
Hardy
,
R. L.
,
1971
, “
Multiquadric Equations of Topography and Other Irregular Surfaces
,”
J. Geophys. Res.
,
76
, pp.
1905
1915
.10.1029/JB076i008p01905
9.
Jin
,
R.
,
Chen
,
W.
, and
Simpson
,
T.
,
2001
, “
Comparative Studies of Metamodelling Techniques Under Multiple Modelling Criteria
,”
Struct. Multidiscip. Optim.
,
23
(
1
), pp.
1
13
.10.1007/s00158-001-0160-4
10.
Mullur
,
A. A.
, and
Messac
,
A.
,
2005
, “
Extended Radial Basis Functions: More Flexible and Effective Metamodeling
,”
AIAA J.
,
43
(
6
), pp.
1306
1315
.10.2514/1.11292
11.
Duda
,
R. O.
,
Hart
,
P. E.
, and
Stork
,
D. G.
,
2000
,
Pattern Classification
,
2nd ed.,
Wiley
,
New York
.
12.
Yegnanarayana
,
B.
,
2004
,
Artificial Neural Networks
,
PHI Learning Pvt. Ltd.
,
New Delhi, India
.
13.
Clarke
,
S. M.
,
Griebsch
,
J. H.
, and
Simpson
,
T. W.
,
2005
, “
Analysis of Support Vector Regression for Approximation of Complex Engineering Analyses
,”
ASME J. Mech. Des.
,
127
(
6
), pp.
1077
1087
.10.1115/1.1897403
14.
Vapnik
,
V.
,
1995
,
The Nature of Statistical Learning Theory
,
Springer
,
New York
.
15.
Basudhar
,
A.
, and
Missoum
,
S.
,
2008
, “
Adaptive Explicit Decision Functions for Probabilistic Design and Optimization Using Support Vector Machines
,”
Comput. Struct.
,
86
(
19–20
), pp.
1904
1917
.10.1016/j.compstruc.2008.02.008
16.
Forrester
,
A. I. J.
, and
Keane
,
A. J.
,
2009
, “
Recent Advances in Surrogate-Based Optimization
,”
Prog. Aerosp. Sci.
,
45
(
1–3
), pp.
50
79
.10.1016/j.paerosci.2008.11.001
17.
Queipo
,
N.
,
Haftka
,
R.
,
Shyy
,
W.
,
Goel
,
T.
,
Vaidyanathan
,
R.
, and
Tucker
,
P.
,
2005
, “
Surrogate-Based Analysis and Optimization
,”
Prog. Aerosp. Sci.
,
41
(
1
), pp.
1
28
.10.1016/j.paerosci.2005.02.001
18.
Wang
,
G.
, and
Shan
,
S.
,
2007
, “
Review of Metamodeling Techniques in Support of Engineering Design Optimization
,”
ASME J. Mech. Des.
,
129
(
4
), pp.
370
380
.10.1115/1.2429697
19.
Simpson
,
T. W.
,
Toropov
,
V.
,
Balabanov
,
V.
, and
Viana
,
F. A. C.
,
2008
, “
Design and Analysis of Computer Experiments in Multidisciplinary Design Optimization: A Review of How Far we Have Come or Not
,”
12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA
.
20.
Zerpa
,
L. E.
,
Queipo
,
N. V.
,
Pintos
,
S.
, and
Salager
,
J.
,
2005
, “
An Optimization Methodology of Alkaline-Urfactant-Polymer Flooding Processes Using Field Scale Numerical Simulation and Multiple Surrogates
,”
J. Pet. Sci. Eng.
,
47
(
3–4
), pp.
197
208
.10.1016/j.petrol.2005.03.002
21.
Goel
,
T.
,
Haftka
,
R. T.
,
Shyy
,
W.
, and
Queipo
,
N. V.
,
2007
, “
Ensemble of Surrogates
,”
Struct. Multidiscip. Optim.
,
33
(
3
), pp.
199
216
.10.1007/s00158-006-0051-9
22.
Sanchez
,
E.
,
Pintos
,
S.
, and
Queipo
,
N. V.
,
2008
, “
Toward an Optimal Ensemble of Kernel-Based Approximations With Engineering Applications
,”
Struct. Multidiscip. Optim.
,
36
(
3
), pp.
247
261
.10.1007/s00158-007-0159-6
23.
Acar
,
E.
, and
Rais-Rohani
,
M.
,
2009
, “
Ensemble of Metamodels With Optimized Weight Factors
,”
Struct. Multidiscip. Optim.
,
37
(
3
), pp.
279
294
.10.1007/s00158-008-0230-y
24.
Viana
,
F. A. C.
,
Haftka
,
R. T.
, and
Steffen
,
V.
,
2009
, “
Multiple Surrogates: How Cross-Validation Errors Can Help us to Obtain the Best Predictor
,”
Struct. Multidiscip. Optim.
,
39
(
4
), pp.
439
457
.10.1007/s00158-008-0338-0
25.
Apley
,
D. W.
,
Liu
,
J.
, and
Chen
,
W.
,
2006
, “
Understanding the Effects of Model Uncertainty in Robust Design With Computer Experiments
,”
ASME J. Mech. Des.
,
128
(
4
), pp.
945
958
.10.1115/1.2204974
26.
Kennedy
,
M. C.
, and
O'Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc.: Ser. B
,
63
(
3
), pp.
425
464
.10.1111/1467-9868.00294
27.
Neufeld
,
D.
,
Behdinan
,
K.
, and
Chung
,
J.
,
2010
, “
Aircraft Wing Box Optimization Considering Uncertainty in Surrogate Models
,”
Struct. Multidiscip. Optim.
,
42
(
5
), pp.
745
753
.10.1007/s00158-010-0532-8
28.
Eldred
,
M.
,
Giunta
,
A.
,
Wojtkiewicz
,
S. F.
, and
Trucano
,
T.
,
2002
, “
Formulations for Surrogate-Based Optimization Under Uncertainty
,”
9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, AIAA
.
29.
Jones
,
D.
,
Schonlau
,
M.
, and
Welch
,
W.
,
1998
, “
Efficient Global Optimization of Expensive Black-Box Functions
,”
J. Global Optim.
,
13
(
4
), pp.
455
492
.10.1023/A:1008306431147
30.
Viana
,
F. A. C.
, and
Haftka
,
R. T.
,
2009
, “
Importing Uncertainty Estimates From one Surrogate to Another
,”
50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA
.
31.
Xiong
,
Y.
,
Chen
,
W.
, and
Tsui
,
K.
,
2008
, “
A New Variable-Fidelity Optimization Framework Based on Model Fusion and Objective-Oriented Sequential Sampling
,”
ASME J. Mech. Des.
,
130
(
11
), p.
111401
.10.1115/1.2976449
32.
Chen
,
S.
,
Xiong
,
Y.
, and
Chen
,
W.
,
2009
, “
Multiresponse and Multistage Metamodeling Approach for Design Optimization
,”
AIAA J.
,
47
(
1
), pp.
206
218
.10.2514/1.38187
33.
Huang
,
D.
,
Allen
,
T. T.
,
Notz
,
W. I.
, and
Zeng
,
N.
,
2006
, “
Global Optimization of Stochastic Black-Box Systems Via Sequential Kriging Meta-Models
,”
J. Global Optim.
,
34
(
3
), pp.
441
466
.10.1007/s10898-005-2454-3
34.
Zhang
,
J.
,
Chowdhury
,
S.
,
Zhang
,
J.
,
Messac
,
A.
, and
Castillo
,
L.
,
2013
, “
Adaptive Hybrid Surrogate Modeling for Complex Systems
,”
AIAA J.
,
51
(
3
), pp.
643
656
.10.2514/1.J052008
35.
Zhang
,
J.
,
Chowdhury
,
S.
,
Messac
,
A.
, and
Castillo
,
L.
,
2012
, “
A Response Surface-Based Cost Model for Wind Farm Design
,”
Energy Policy
,
42
, pp.
538
550
.10.1016/j.enpol.2011.12.021
36.
Chowdhury
,
S.
,
Zhang
,
J.
,
Messac
,
A.
, and
Castillo
,
L.
,
2012
, “
Unrestricted Wind Farm Layout Optimization (UWFLO): Investigating Key Factors Influencing the Maximum Power Generation
,”
Renewable Energy
,
38
(
1
), pp.
16
30
.10.1016/j.renene.2011.06.033
37.
Chowdhury
,
S.
,
Zhang
,
J.
,
Messac
,
A.
, and
Castillo
,
L.
,
2013
, “
Optimizing the Arrangement and the Selection of Turbines for Wind Farms Subject to Varying Wind Conditions
,”
Renewable Energy
,
52
, pp.
273
282
.10.1016/j.renene.2012.10.017
38.
Forrester
,
A. I. J.
,
Sóbester
,
A.
, and
Keane
,
A. J.
,
2008
,
Engineering Design via Surrogate Modelling: A Practical Guide
,
Wiley
,
New York
.
39.
Joseph
,
V. R.
,
Hung
,
Y.
, and
Sudjianto
,
A.
,
2008
, “
Blind Kriging: A New Method for Developing Metamodels
,”
ASME J. Mech. Des.
,
130
(
3
), p.
031102
.10.1115/1.2829873
40.
Viana
,
F. A. C.
, and
Haftka
,
R. T.
,
2009
, “
Cross Validation Can Estimate How Well Prediction Variance Correlates With Error
,”
AIAA J.
,
47
(
9
), pp.
2266
2270
.10.2514/1.42162
41.
Viana
,
F. A. C.
,
Picheny
,
V.
, and
Haftka
,
R. T.
,
2010
, “
Using Cross Validation to Design Conservative Surrogates
,”
AIAA J.
,
48
(
10
), pp.
2286
2298
.10.2514/1.J050327
42.
Duan
,
K.
, and
Keerthi
,
S. S.
,
2005
, “
Which is the Best Multiclass SVM Method? An Empirical Study
,”
Multiple Classifier Syst.
,
3541
, pp.
732
760
.
43.
Hsu
,
C. W.
, and
Lin
,
C. J.
,
2002
, “
A Comparison of Methods for Multiclass Support Vector Machines
,”
IEEE Trans. Neural Networks
,
13
(
2
), pp.
415
425
.10.1109/72.991427
44.
Chang
,
C. C.
, and
Lin
,
C. J.
,
2011
, “
LIBSVM: A Library for Support Vector Machines
,”
ACM Trans. Intell. Syst. Technol.
,
2
(
3
), pp.
27:1
27:27
.10.1145/1961189.1961199
45.
Zhang
,
J.
,
Chowdhury
,
S.
, and
Messac
,
A.
,
2012
, “
An Adaptive Hybrid Surrogate Model
,”
Struct. Multidiscip. Optim.
,
46
(
2
), pp.
223
238
.10.1007/s00158-012-0764-x
46.
Audze
,
P.
, and
Eglais
,
V.
,
1997
, “
New Approach for Planning Out of Experiments
,”
Prob. Dyn. Strengths
,
35
, pp.
104
107
.
47.
Lophaven
,
S. N.
,
Nielsen
,
H. B.
, and
Sondergaard
,
J.
,
2002
,
Dace—A Matlab Kriging Toolbox, Version 2.0
,” Technical University of Denmark, Technical Report, Informatics and Mathematical Modelling Report IMM-REP-2002-12.
48.
Zhang
,
J.
,
Chowdhury
,
S.
,
Messac
,
A.
, and
Castillo
,
L.
,
2011
, “
A Comprehensive Measure of the Energy Resource Potential of a Wind Farm Site
,”
ASME 2011 5th International Conference on Energy Sustainability, ASME
.
49.
Chowdhury
,
S.
,
Messac
,
A.
, and
Khire
,
R. A.
,
2011
, “
Comprehensive Product Platform Planning (cp3) Framework
,”
ASME J. Mech. Des.
,
133
(
10
), p.
101004
.10.1115/1.4004969
50.
Zhang
,
J.
,
Chowdhury
,
S.
,
Messac
,
A.
,
Castillo
,
L.
, and
Lebron
,
J.
,
2010
, “
Response Surface Based Cost Model for Onshore Wind Farms Using Extended Radial Basis Functions
,”
ASME 2010 International Design Engineering Technical Conferences (IDETC), ASME
.
51.
GE
,
2010
, GE Energy 1.5MW Wind Turbine Brochure,
General Electric
, http://www.gepower.com/
52.
Chowdhury
,
S.
,
Messac
,
A.
, and
Khire
,
R. A.
,
2013
, “
Investigating the Commonality Attributes for Scaling Product Families Using Comprehensive Product Platform Planning (cp3)
,”
Struct. Multidiscip. Optim.
,
48
, pp.
1089
1107
.
53.
Goldberg
,
M.
,
2009
,
Jobs and Economic Development Impact (JEDI) Model
,
National Renewable Energy Laboratory
,
Golden, CO
.
54.
NDSU
,
2012
, “
The North Dakota Agricultural Weather Network
,” http://ndawn.ndsu.nodak.edu
You do not currently have access to this content.