Abstract

A surrogate model is a common tool to approximate system response at untested points for design space exploration. Adaptive sampling has been studied for improving the accuracy of surrogates iteratively by introducing additional samples (simulations and experiments). New samples are often selected based on the estimated uncertainty in the design space. While some surrogates such as kriging have readily available uncertainty models for their predictions, other surrogates do not. Consequently, there have been studies of using the process of leaving-samples-out (LSO) used in cross-validation tools to estimate prediction uncertainty, such as the universal prediction distribution (UPD). In this paper, an adaptive sampling scheme for general surrogates is proposed based on LSO, similar to cross-validation and interquartile range (IQR). Multiple submodels are first developed from LSO. The uncertainty is then estimated from the IQR of these surrogates at a given point. New samples are added iteratively at the point with maximum IQR for design space exploration. The proposed scheme is illustrated using kriging, radial basis function, and neural network surrogates. The proposed scheme is evaluated using four algebraic test functions. Multiple sets of initial samples were produced to account for randomness. For these test functions, the proposed scheme was found to be more accurate and robust than kriging with its own uncertainty model. The proposed scheme was more accurate than the UPD for three out of the four test functions. For a fixed number of samples, the IQR-based adaptive sampling also proved to be more accurate than all-at-once sampling in most cases even when the estimated uncertainty was only mildly correlated with prediction errors.

References

1.
Forrester
,
A. I.
, and
Keane
,
A. J.
,
2009
, “
Recent Advances in Surrogate-Based Optimization
,”
Prog. Aerosp. Sci.
,
45
(
1
), pp.
50
79
. 10.1016/j.paerosci.2008.11.001
2.
Queipo
,
N. V.
,
Haftka
,
R. T.
,
Shyy
,
W.
,
Goel
,
T.
,
Vaidyanathan
,
R.
, and
Tucker
,
P. K.
,
2005
, “
Surrogate-Based Analysis and Optimization
,”
Prog. Aerosp. Sci.
,
41
(
1
), pp.
1
28
. 10.1016/j.paerosci.2005.02.001
3.
Frangos
,
M.
,
Marzouk
,
Y.
,
Willcox
,
K.
, and
van Bloemen Waanders
,
B.
,
2010
, “Surrogate and Reduced-Order Modeling: A Comparison of Approaches for Large-Scale Statistical Inverse Problems,”
Large-Scale Inverse Problems and Quantification of Uncertainty
,
L.
Biegler
,
G.
Biros
,
O.
Ghattas
,
M.
Heinkenschloss
,
D.
Keyes
,
B.
Mallick
,
Y.
Marzouk
,
L.
Tenorio
,
B.
van Bloemen Waanders
, and
K.
Willcox
, eds.,
Wiley
,
New York
, pp.
123
149
.
4.
Wang
,
G. G.
, and
Shan
,
S.
,
2007
, “
Review of Metamodeling Techniques in Support of Engineering Design Optimization
,”
ASME J. Mech. Des.
,
129
(
4
), pp.
370
380
. 10.1115/1.2429697
5.
Zhang
,
Y.
,
Meeker
,
J.
,
Schutte
,
J.
,
Kim
,
N.
, and
Haftka
,
R.
,
2016
, “
On Approaches to Combine Experimental Strength and Simulation With Application to Open-Hole-Tension Configuration
,”
Proceedings of the American Society for Composites: Thirty-First Technical Conference
,
Williamsburg, VA
,
Sept. 19–22
,
DEStech Publications
,
Lancaster, PA
, Paper No. 67.
6.
Bui-Thanh
,
T.
,
Willcox
,
K.
, and
Ghattas
,
O.
,
2008
, “
Model Reduction for Large-Scale Systems With High-Dimensional Parametric Input Space
,”
SIAM J. Sci. Comput.
,
30
(
6
), pp.
3270
3288
. 10.1137/070694855
7.
Guénot
,
M.
,
Lepot
,
I.
,
Sainvitu
,
C.
,
Goblet
,
J.
, and
Filomeno Coelho
,
R.
,
2013
, “
Adaptive Sampling Strategies for non-Intrusive POD-Based Surrogates
,”
Eng. Comput.
,
30
(
4
), pp.
521
547
. 10.1108/02644401311329352
8.
Zhang
,
Y.
,
Neelakantan
,
A.
,
Kumar
,
N.
,
Park
,
C.
,
Haftka
,
R. T.
,
Kim
,
N. H.
, and
Lam
,
H.
,
2017
, “
Multi-Fidelity Surrogate Modeling for Application/Architecture Co-Design
,”
International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems
,
Denver, CO
,
Nov. 13
,
Springer
,
Cham
, pp.
179
196
.
9.
Hao
,
P.
,
Wang
,
B.
, and
Li
,
G.
,
2012
, “
Surrogate-Based Optimum Design for Stiffened Shells With Adaptive Sampling
,”
AIAA J.
,
50
(
11
), pp.
2389
2407
. 10.2514/1.J051522
10.
Chen
,
Z.
,
Qiu
,
H.
,
Gao
,
L.
,
Li
,
X.
, and
Li
,
P.
,
2014
, “
A Local Adaptive Sampling Method for Reliability-Based Design Optimization Using Kriging Model
,”
Struct. Multidiscip. Optim.
,
49
(
3
), pp.
401
416
. 10.1007/s00158-013-0988-4
11.
Jiang
,
Z.
,
Li
,
W.
,
Apley
,
D. W.
, and
Chen
,
W.
,
2015
, “
A Spatial-Random-Process Based Multidisciplinary System Uncertainty Propagation Approach With Model Uncertainty
,”
ASME J. Mech. Des.
,
137
(
10
), p.
101402
. 10.1115/1.4031096
12.
Giselle Fernández-Godino
,
M.
,
Park
,
C.
,
Kim
,
N. H.
, and
Haftka
,
R. T.
,
2019
, “
Issues in Deciding Whether to Use Multifidelity Surrogates
,”
AIAA [Q7]J.
,
57
(
5
), pp.
1
16
. 10.2514/1.J057750
13.
Zhang
,
Y.
,
Kim
,
N. H.
,
Park
,
C.
, and
Haftka
,
R. T.
,
2018
, “
Multifidelity Surrogate Based on Single Linear Regression
,”
AIAA J.
,
56
(
12
), pp.
4944
4952
. 10.2514/1.J057299
14.
Picheny
,
V.
,
Ginsbourger
,
D.
,
Roustant
,
O.
,
Haftka
,
R. T.
, and
Kim
,
N.-H.
,
2010
, “
Adaptive Designs of Experiments for Accurate Approximation of a Target Region
,”
ASME J. Mech. Des.
,
132
(
7
), p.
071008
. 10.1115/1.4001873
15.
Haftka
,
R. T.
,
Villanueva
,
D.
, and
Chaudhuri
,
A.
,
2016
, “
Parallel Surrogate-Assisted Global Optimization With Expensive Functions—A Survey
,”
Struct. Multidiscip. Optim.
,
54
(
1
), pp.
3
13
. 10.1007/s00158-016-1432-3
16.
Mackman
,
T.
,
Allen
,
C.
,
Ghoreyshi
,
M.
, and
Badcock
,
K.
,
2013
, “
Comparison of Adaptive Sampling Methods for Generation of Surrogate Aerodynamic Models
,”
AIAA J.
,
51
(
4
), pp.
797
808
. 10.2514/1.J051607
17.
Ginsbourger
,
D.
,
Le Riche
,
R.
, and
Carraro
,
L.
,
2010
, “Kriging is Well-Suited to Parallelize Optimization,”
Computational Intelligence in Expensive Optimization Problems. Adaptation Learning and Optimization
, Vol.
2
,
Y.
Tenne
, and
C. K.
Goh
, eds.,
Springer
,
Berlin, Heidelberg
.
18.
Zhu
,
P.
,
Zhang
,
S.
, and
Chen
,
W.
,
2015
, “
Multi-Point Objective-Oriented Sequential Sampling Strategy for Constrained Robust Design
,”
Eng. Optim.
,
47
(
3
), pp.
287
307
. 10.1080/0305215X.2014.887705
19.
Ling
,
Y.
,
Ghosh
,
S.
,
Asher
,
I.
,
Kristensen
,
J.
,
Ryan
,
K.
, and
Wang
,
L.
,
2018
, “
An Intelligent Sampling Framework for Multi-Objective Optimization in High Dimensional Design Space
,”
Proceedings of the 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference
,
Kissimmee, FL
,
Jan. 8–12
, p.
0912
.
20.
Romero
,
D. A.
,
Amon
,
C. H.
, and
Finger
,
S.
,
2006
, “
On Adaptive Sampling for Single and Multi-Response Bayesian Surrogate Models
,”
Proceedings of the ASME 2006 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Philadelphia, PA
,
Sept. 10–13
, pp.
393
404
.
21.
Viana
,
F. A.
,
Haftka
,
R. T.
, and
Steffen
,
V.
, Jr.
,
2009
, “
Multiple Surrogates: How Cross-Validation Errors Can Help us to Obtain the Best Predictor
,”
Struct. Multidiscip. Optim.
,
39
(
4
), pp.
439
457
. 10.1007/s00158-008-0338-0
22.
Claeskens
,
G.
, and
Hjort
,
N.
,
2008
,
Model Selection and Model Averaging (Cambridge Series in Statistical and Probabilistic Mathematics)
,
Cambridge University Press
,
Cambridge, UK
.
23.
Shan
,
S.
, and
Wang
,
G. G.
,
2010
, “
Survey of Modeling and Optimization Strategies to Solve High-Dimensional Design Problems With Computationally-Expensive Black-Box Functions
,”
Struct. Multidiscip. Optim.
,
41
(
2
), pp.
219
241
. 10.1007/s00158-009-0420-2
24.
Matsumura
,
T.
,
Haftka
,
R. T.
, and
Kim
,
N. H.
,
2015
, “
Accurate Predictions From Noisy Data: Replication Versus Exploration With Applications to Structural Failure
,”
Struct. Multidiscip. Optim.
,
51
(
1
), pp.
23
40
. 10.1007/s00158-014-1115-x
25.
Viana
,
F. A.
,
Haftka
,
R. T.
, and
Watson
,
L. T.
,
2013
, “
Efficient Global Optimization Algorithm Assisted by Multiple Surrogate Techniques
,”
J. Glob. Optim.
,
56
(
2
), pp.
669
689
. 10.1007/s10898-012-9892-5
26.
Jin
,
R.
,
Chen
,
W.
, and
Sudjianto
,
A.
,
2002
, “
On Sequential Sampling for Global Metamodeling in Engineering Design
,”
Proceedings of the ASME 2002 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Sept. 29–Oct. 2
,
Montreal, Quebec, Canada
, pp.
539
548
.
27.
Ben Salem
,
M.
,
Roustant
,
O.
,
Gamboa
,
F.
, and
Tomaso
,
L.
,
2017
, “
Universal Prediction Distribution for Surrogate Models
,”
SIAM/ASA J. Uncertain. Quantif.
,
5
(
1
), pp.
1086
1109
. 10.1137/15M1053529
28.
Xu
,
S.
,
Liu
,
H.
,
Wang
,
X.
, and
Jiang
,
X.
,
2014
, “
A Robust Error-Pursuing Sequential Sampling Approach for Global Metamodeling Based on Voronoi Diagram and Cross Validation
,”
ASME J. Mech. Des.
,
136
(
7
), p.
071009
. 10.1115/1.4027161
29.
Garbo
,
A.
, and
German
,
B. J.
,
2019
, “
Performance Assessment of a Cross-Validation Sampling Strategy With Active Surrogate Model Selection
,”
Struct. Multidiscip. Optim.
,
59
(
6
), pp.
2257
2272
. 10.1007/s00158-018-02190-7
30.
Mehmani
,
A.
,
Chowdhury
,
S.
, and
Messac
,
A.
,
2015
, “
Predictive Quantification of Surrogate Model Fidelity Based on Modal Variations With Sample Density
,”
Struct. Multidiscip. Optim.
,
52
(
2
), pp.
353
373
. 10.1007/s00158-015-1234-z
31.
Rousseeuw
,
P. J.
, and
Croux
,
C.
,
1993
, “
Alternatives to the Median Absolute Deviation
,”
J. Am. Stat. Assoc.
,
88
(
424
), pp.
1273
1283
. 10.1080/01621459.1993.10476408
32.
Neal
,
R. M.
,
2012
, “
Bayesian Learning for Neural Networks
,” Dissertation,
University of Toronto Toronto
,
Ont., Canada
.
33.
Khuri
,
A. I.
, and
Mukhopadhyay
,
S.
,
2010
, “
Response Surface Methodology
,”
Wiley Interdiscip. Rev.: Comput. Stat.
,
2
(
2
), pp.
128
149
. 10.1002/wics.73
34.
Jones
,
D. R.
,
Schonlau
,
M.
, and
Welch
,
W. J.
,
1998
, “
Efficient Global Optimization of Expensive Black-Box Functions
,”
J. Glob. Optim.
,
13
(
4
), pp.
455
492
. 10.1023/A:1008306431147
35.
Chaudhuri
,
A.
,
2014
, “
Implications of Optimization Cost for Balancing Exploration and Exploitation in Global Search and for Experimental Optimization
,” Dissertation,
University of Florida
,
Gainesville, FL
.
36.
Rasmussen
,
C. E.
,
2004
, “Gaussian Processes in Machine Learning,”
Advanced Lectures on Machine Learning
.
ML 2003. Lecture Notes in Computer Science
, Vol.
3176
,
O.
Bousquet
,
U.
von Luxburg
, and
G.
Rätsch
, eds.,
Springer
,
Berlin, Heidelberg
.
37.
Viana
,
F. A.
,
2011
, “
Multiple Surrogates for Prediction and Optimization
,” Dissertation,
University of Florida
,
Gainesville, FL
.
38.
Kohavi
,
R.
,
1995
, “
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
,”
IJCAI’95 Proceedings of the 14th International Joint Conference on Artificial Intelligence—Volume 2
,
Aug. 20–25
,
Montreal, Quebec, Canada
, pp.
1137
1114
.
39.
Park
,
J.-S.
,
1994
, “
Optimal Latin-Hypercube Designs for Computer Experiments
,”
J. Stat. Plann. Inference
,
39
(
1
), pp.
95
111
. 10.1016/0378-3758(94)90115-5
40.
Surjanovic
,
S.
, and
Bingham
,
D.
,
2013
,
Virtual Library of Simulation Experiments: Test Functions and Datasets
. https://www.sfu.ca/∼ssurjano/. Accessed July 3, 2019.
41.
Wang
,
Z.
, and
Wang
,
P.
,
2015
, “
An Integrated Performance Measure Approach for System Reliability Analysis
,”
ASME J. Mech. Des.
,
137
(
2
), p.
021406
. 10.1115/1.4029222
42.
Hu
,
Z.
, and
Mahadevan
,
S.
,
2016
, “
A Single-Loop Kriging Surrogate Modeling for Time-Dependent Reliability Analysis
,”
ASME J. Mech. Des.
,
138
(
6
), p.
061406
. 10.1115/1.4033428
43.
Roustant
,
O.
,
Ginsbourger
,
D.
, and
Deville
,
Y.
,
2013
, “
DiceKriging, DiceOptim: Two R Packages for the Analysis of Computer Experiments by Kriging-Based Metamodeling and Optimization
,”
J. Stat. Softw.
,
51
(
1
), pp.
1
55
. 10.18637/jss.v051.i01
44.
Lee Rodgers
,
J.
, and
Nicewander
,
W. A.
,
1988
, “
Thirteen Ways to Look at the Correlation Coefficient
,”
Am. Stat.
,
42
(
1
), pp.
59
66
. 10.1080/00031305.1988.10475524
45.
Jones
,
E.
,
Oliphant
,
E.
, and
Peterson
,
P.
,
SciPy: Open Source Scientific Tools for Python
,
2001
, http://www.scipy.org/. Accessed July 3, 2019.
46.
Pedregosa
,
F.
,
Varoquaux
,
G.
,
Gramfort
,
A.
,
Michel
,
V.
,
Thirion
,
B.
,
Grisel
,
O.
,
Blondel
,
M.
,
Prettenhofer
,
P.
,
Weiss
,
R.
, and
Dubourg
,
V.
,
2011
, “
Scikit-Learn: Machine Learning in Python
,”
J. Mach. Learn. Res.
,
12
, pp.
2825
2830
.
You do not currently have access to this content.