The design optimization process relies often on computational models for analysis or simulation. These models must be validated to quantify the expected accuracy of the obtained design solutions. It can be argued that validation of computational models in the entire design space is neither affordable nor required. In previous work, motivated by the fact that most numerical optimization algorithms generate a sequence of candidate designs, we proposed a new paradigm where design optimization and calibration-based model validation are performed concurrently in a sequence of variable-size local domains that are relatively small compared to the entire design space. A key element of this approach is how to account for variability in test data and model predictions in order to determine the size of the local domains at each stage of the sequential design optimization process. In this article, we discuss two alternative techniques for accomplishing this: parametric and nonparametric bootstrapping. The parametric bootstrapping assumes a Gaussian distribution for the error between test and model data and uses maximum likelihood estimation to calibrate the prediction model. The nonparametric bootstrapping does not rely on the Gaussian assumption providing; therefore, a more general way to size the local domains for applications where distributional assumptions are difficult to verify, or not met at all. If distribution assumptions are met, parametric methods are preferable over nonparametric methods. We use a validation literature benchmark problem to demonstrate the application of the two techniques. Which technique to use depends on whether the Gaussian distribution assumption is appropriate based on available information.

References

References
1.
Schlesinger
,
S.
,
Crosby
,
R. E.
,
Cagne
,
R. E.
,
Innis
,
G. S.
,
Lalwani
,
C. S.
,
Loch
,
J.
,
Sylvester
,
R. J.
,
Wright
,
R. D.
,
Kheir
,
N.
, and
Bartos
,
D.
, 1979, “
Terminology for Model Credibility
,”
Simulation
,
32
(
3
), pp.
103
104
.
2.
Sargent
,
R. G.
, 1998, “
Verification and Validation of Simulation Models
,” Proceedings of the 1998 Winter Simulation Conference, Dec. 13–16,
Washington, DC
, pp.
121
130
.
3.
Kennedy
,
M. C.
, and
O’Hagan
,
A.
, 2001, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc. Ser. B (Stat. Methodol.)
,
63
, pp.
425
450
.
4.
Oberkampf
,
W. L.
,
Trucano
,
T. G.
, and
Hirsch
,
C.
, 2004, “
Verification, Validation, and Predictive Capability in Computational Engineering and Physics
,”
Appl. Mech. Rev.
,
57
(
5
), pp.
345
384
.
5.
Bayarri
,
M. J.
,
Walsh
,
D.
,
Berger
,
J. O.
,
Cafeo
,
J.
,
Garcia-Donato
,
G.
,
Liu
,
F.
,
Palomo
,
J.
,
Parthasarathy
,
R. J.
,
Paulo
,
R.
, and
Sacks
,
J.
, 2007, “
Computer Model Validation With Functional Output
,”
Ann. Stat.
,
35
,
1874
1906
.
6.
Oberkampf
,
W. L.
, and
Roy
,
C. J.
, 2010,
Verification and Validation in Scientific Computing
,
Cambridge University Press
,
Cambridge
.
7.
Zhang
,
R.
, and
Mahadevan
,
S.
, 2003, “
Bayesian Methodology for Reliability Model Acceptance
,”
Reliab. Eng. Syst. Saf.
,
80
, pp.
95
103
.
8.
Mahadevan
,
S.
, and
Rebba
,
R.
, 2005, “
Validation of Reliability Computational Models Using Bayes Networks
,”
Reliab. Eng. Syst. Saf.
,
87
, pp.
223
232
.
9.
Rebba
,
R.
, and
Mahadevan
,
S.
, 2006, “
Model Predictive Capability Assessment Under Uncertainty
,”
AIAA J.
,
44
(
10
), pp.
2376
2384
.
10.
Jiang
,
X.
, and
Mahadevan
,
S.
, 2008, “
Bayesian Validation Assessment of Multivariate Computational Models
,”
J. Appl. Stat.
,
35
(
1
), pp.
49
65
.
11.
Jiang
,
X.
, and
Mahadevan
,
S.
, 2008, “
Bayesian Wavelet Method for Multivariate Model Assessment of Dynamic Systems
,”
J. Sound Vib.
,
312
(
4–5
), pp.
694
712
.
12.
Jiang
,
X.
, and
Mahadevan
,
S.
, 2009, “
Bayesian Structural Equation Modeling for Hierarchical Model Validation
,”
Reliab. Eng. Syst. Saf.
,
94
(
4
), pp.
796
809
.
13.
Chen
,
W.
,
Xiong
,
Y.
,
Tsui
,
K.-L.
, and
Wang
,
S.
, 2008, “
A Design-Driven Validation Approach Using Bayesian Prediction Models
,”
J. Mech. Des.
,
130
, p.
021101
.
14.
Li
,
J.
,
Mourelatos
,
Z. P.
,
Kokkolaras
,
M.
,
Papalambros
,
P.
, and
Gorsich
,
D.
, 2010, “
Validating Designs Through Sequential Simulation-Based Optimization
,” Proceedings of the ASME 2010 IDETC/CIE, Aug. 15–18,
Montreal, Quebec, Canada
, Paper No. DETC2010-28431.
15.
Drignei
,
D.
,
Mourelatos
,
Z. P.
,
Kokkolaras
,
M.
,
Pandey
,
V.
, and
Koscik
,
G.
, 2011, “
A Variable-Size Local Domain Approach for Increased Design Confidence in Simulation-Based Optimization
,”
Struct. Multidiscip. Optim.
,
46
, pp.
83
92
.
16.
Schervish
,
M. J.
, 1995,
Theory of Statistics
,
Springer
,
New York
.
17.
Rizzo
,
M. L.
, 2007,
Statistical Computing With R
,
Chapman and Hall
,
London
.
18.
Drignei
,
D.
,
Forest
,
C.
, and
Nychka
,
D.
, 2008, “
Parameter Estimation for Computationally Intensive Nonlinear Regression With an Application to Climate Modeling
,”
Ann. Appl. Stat.
,
2
, pp.
1217
1230
.
19.
Higgins
,
J. J.
, 2003,
Introduction to Modern Nonparametric Statistics
,
Duxbury Press
, Pacific Grove, CA.
20.
Efron
,
B.
, and
Tibshirani
,
R. J.
, 1993,
An Introduction to the Bootstrap
,
Chapman and Hall
,
New York
.
21.
Silverman
,
B. W.
, and
Young
,
G. A.
, 1987, “
The Bootstrap: To Smooth or not to Smooth?
,”
Biometrika
,
74
, pp.
469
479
.
22.
Dowding
,
K. J.
,
Pilch
,
M.
, and
Hills
,
R. G.
, 2008, “
Formulation of the Thermal Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
, pp.
2385
2389
.
You do not currently have access to this content.