The optimal solution of a design optimization problem is dependent on the predictive models used to evaluate the objective and constraints. Since different models give different predictions and can yield different design decisions, when more than one model is available, the choice of model used to represent the objectives/constraints of the design becomes important. This paper addresses the problem of model selection among physics-based models during the prediction stage, which is in contrast to model selection during the calibration and validation stages, and therefore affects design under uncertainty. Model selection during calibration addresses the problem of selecting the model that is likely to provide the best generalization of the calibration data over the entire domain. Model selection during the validation stage examines the validity of a calibrated model by comparing its predictions against the validation data. This paper presents an approach that allows for model selection during the prediction stage, which selects the “best” model for each prediction point. The proposed approach is based on the estimation of the model prediction error under stationary/nonstationary uncertainty. By selecting the best model at each prediction point, the proposed approach partitions the input domain of the models into nonoverlapping regions. The effects of measurement noise, sparseness of validation data, and model prediction uncertainty are included in deriving a probabilistic selection criterion for model selection. The effects of these uncertainties on the classification errors are analyzed. The proposed approach is demonstrated for the problem of selecting between two parametric models for energy dissipation in a mechanical lap joint under dynamic loading, and for the problem of selecting fatigue crack growth models.

References

References
1.
Kennedy
,
M. C.
, and
O'Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc. Ser. B (Stat. Methodol.)
,
63
(
3
), pp.
425
464
.10.1111/1467-9868.00294
2.
Rebba
,
R.
,
Mahadevan
,
S.
, and
Huang
,
S.
,
2006
, “
Validation and Error Estimation of Computational Models
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1390
1397
.10.1016/j.ress.2005.11.035
3.
American Society of Mechanical Engineers, V&V 20
,
2009
, Available at ASME Codes & Standards website.
4.
Mahadevan
,
S.
, and
Liang
,
B.
,
2011
, “
Error and Uncertainty Quantification and Sensitivity Analysis in Mechanics Computational Models
,”
Int. J. Uncertainty Quantification
,
1
(
2
), pp.
147
161
.10.1615/IntJUncertaintyQuantification.v1.i2.30
5.
Roache
,
P. J.
,
1998
, “
Verification and Validation in Computational Science and Engineering
,”
Comput. Sci. Eng.
, Albuquerque, NM: Hermosa Publishers, 1998, pp.
8
9
.
6.
Rangavajhala
,
S.
,
Sura
,
V. S.
,
Hombal
,
V. K.
, and
Mahadevan
,
S.
,
2011
, “
Discretization Error Estimation in Multidisciplinary Simulations
,”
AIAA J.
,
49
(
12
), pp.
2673
2683
.10.2514/1.J051085
7.
Akaike
,
H.
,
1978
, “
A Bayesian Analysis of the Minimum AIC Procedure
,”
Ann. Inst. Stat. Math.
,
30
(
1
), pp.
9
14
.10.1007/BF02480194
8.
Burnham
,
K. P.
, and
Anderson
,
D. R.
,
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach
,
Springer-Verlag
,
Berlin
.
9.
Rissanen
,
J.
,
1978
, “
Modeling by Shortest Data Description
,”
Automatica
,
14
(
5
), pp.
465
471
.10.1016/0005-1098(78)90005-5
10.
Schwarz
,
G.
,
1978
, “
Estimating the Dimension of a Model
,”
Ann. Stat.
,
6
(
2
), pp.
461
464
.10.1214/aos/1176344136
11.
Spiegelhalter
,
D. J.
,
Best
,
N. G.
,
Carlin
,
B. P.
, and
Van Der Linde
,
A.
,
2002
, “
Bayesian Measures of Model Complexity and Fit
,”
J. R. Stat. Soc. Ser. B (Stat. Methodol.)
,
64
, pp.
583
639
.10.1111/1467-9868.00353
12.
Liu
,
Y.
,
Chen
,
W.
,
Arendt
,
P.
, and
Huang
,
H.-Z.
,
2011
, “
Toward a Better Understanding of Model Validation Metrics
,”
ASME J. Mech. Des.
,
133
(
7
), p.
071005
.10.1115/1.4004223
13.
Oberkampf
,
W. L.
, and
Roy
,
C. J.
,
2010
,
Verification and Validation in Scientific Computing
,
Cambridge University Press
,
Cambridge
.
14.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2008
, “
Computational Methods for Model Reliability Assessment
,”
Reliab. Eng. Syst. Saf.
,
93
(
8
), pp.
1197
1207
.10.1016/j.ress.2007.08.001
15.
Romero
,
V. J.
,
2011
, “
Elements of a Pragmatic Approach for Dealing With Bias and Uncertainty in Experiments Through Predictions
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2011-7342.
16.
Hills
,
R. G.
, and
Leslie
,
I.
,
2003
, “
Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2003-0706.
17.
Oberkampf
,
W. L.
, and
Barone
,
M. F.
,
2006
, “
Measures of Agreement Between Computation and Experiment: Validation Metrics
,”
J. Comput. Phys.
,
217
(
1
), pp.
5
36
.10.1016/j.jcp.2006.03.037
18.
Urbina
,
A.
, and
Paez
,
T. L.
,
2003
, “
Statistical Validation of Structural Dynamics Models
,”
J. IEST
,
46
(
1
), pp.
141
148
. Available at http://iest.metapress.com/content/p3k33743858u56hx/?p=2987fd7493ee4efea050d89b4f290d98&pi=9
19.
Zhang
,
R.
, and
Mahadevan
,
S.
,
2003
, “
Bayesian Methodology for Reliability Model Acceptance
,”
Reliab. Eng. Syst. Saf.
,
80
(
1
), pp.
95
103
.10.1016/S0951-8320(02)00269-7
20.
Jiang
,
X.
, and
Mahadevan
,
S.
,
2007
, “
Bayesian Risk-Based Decision Method for Model Validation Under Uncertainty
,”
Reliab. Eng. Syst. Saf.
,
92
(
6
), pp.
707
718
.10.1016/j.ress.2006.03.006
21.
Sankararaman
,
S.
, and
Mahadevan
,
S.
,
2011
, “
Model Validation Under Epistemic Uncertainty
,”
Reliab. Eng. Syst. Saf.
,
96
, pp.
1232
1241
.10.1016/j.ress.2010.07.014
22.
Bernardo
,
J. M.
,
Smith
,
A. F. M.
, and
Berliner
,
M.
,
2000
,
Bayesian Theory
,
Wiley
,
New York, New York, USA
.
23.
Messer
,
M.
,
Panchal
,
J. H.
,
Krishnamurthy
,
V.
,
Klein
,
B.
,
Douglas Yoder
,
P.
,
Allen
,
J. K.
, and
Mistree
,
F.
,
2010
, “
Model Selection Under Limited Information Using a Value-of-Information-Based Indicator
,”
ASME J. Mech. Des.
,
132
(
12
), p.
121008
.10.1115/1.4002751
24.
Radhakrishnan
,
R.
, and
McAdams
,
D. A.
,
2005
, “
A Methodology for Model Selection in Engineering Design
,”
ASME J. Mech. Des.
,
127
(
3
),
378
387
.10.1115/1.1830048
25.
Robinson
,
T. D.
,
Eldred
,
M. S.
,
Willcox
,
K. E.
, and
Haimes
,
R.
,
2008
, “
Surrogate-Based Optimization Using Multifidelity Models With Variable Parameterization and Corrected Space Mapping
,”
AIAA J.
,
46
(
11
), pp.
2814
2822
.10.2514/1.36043
26.
Rangavajhala
,
S.
,
Liang
,
C.
,
Mahadevan
,
S.
, and
Hombal
,
V. K.
,
2011
, “
Concurrent Optimization of Mesh Refinement and Design Parameters in Multidisciplinary Design
,”
J. Aircr.
,
49
, pp.
1786
1795
.10.2514/1.C031564
27.
Sugiyama
,
M.
, and
Ogawa
,
H.
,
2001
, “
Subspace Information Criterion for Model Selection
,”
Neural Comput.
,
13
(
8
), pp.
1863
1889
.10.1162/08997660152469387
28.
Segalman
,
D. J.
,
2005
, “
A Four-Parameter Iwan Model for Lap-Type Joints
,”
ASME J. Appl. Mech.
,
72
, pp.
752
760
.10.1115/1.1989354
29.
Smallwood
,
D. O.
,
Gregory
,
D. L.
, and
Coleman
,
R. G.
,
2001
, “
A Three Parameter Constitutive Model for a Joint Which Exhibits a Power Law Relationship Between Energy Loss and Relative Displacement
,”
Proceedings of the 72nd Shock and Vibration Symposium
.
30.
Sarkar
,
S.
,
2009
, “
Probabilistic Durability Analysis of Cementitious Materials Under External Sulfate Attack
,” Ph.D. thesis, Vanderbilt University, Nashville, TN.
31.
Viana
,
F. A. C.
, and
Haftka
,
R.T.
,
2008
, “
Using Multiple Surrogates for Metamodeling
,”
Proceedings of the 7th ASMO-UK/ISSMO International Conference on Engineering Design Optimization
.
32.
Zhao
,
D.
, and
Xue
,
D.
,
2011
, “
A Multi-Surrogate Approximation Method for Metamodeling
,”
Eng. Comput.
,
27
(
2
), pp.
139
153
.10.1007/s00366-009-0173-y
33.
Queipo
,
N. V.
,
Haftka
,
R. T.
,
Shyy
,
W.
,
Goel
,
T.
,
Vaidyanathan
,
R.
, and
Kevin Tucker
,
P.
,
2005
, “
Surrogate-Based Analysis and Optimization
,”
Prog. Aerosp. Sci.
,
41
(
1
), pp.
1
28
.10.1016/j.paerosci.2005.02.001
34.
Villanueva
,
D.
,
Le Riche
,
R.
,
Picard
,
G.
, and
Haftka
,
R. T.
,
2012
, “
Surrogate-Based Agents for Constrained Optimization
,”
14th AIAA Non-Deterministic Approaches Conference
,
Honolulu
, AIAA Paper No. 2012-1935.
35.
Rasmussen
,
C. E.
, and
Williams
,
C. K. I.
,
2006
,
Gaussian Processes for Machine Learning
,
Springer
,
New York
.
36.
Beck
,
J. L.
, and
Yuen
,
K. V.
,
2004
, “
Model Selection Using Response Measurements: Bayesian Probabilistic Approach
,”
J. Eng. Mech.
,
130
, pp.
192
203
.10.1061/(ASCE)0733-9399(2004)130:2(192)
37.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2006
, “
Validation of Models With Multivariate Output
,”
Reliab. Eng. Syst. Saf.
,
91
(
8
), pp.
861
871
.10.1016/j.ress.2005.09.004
38.
Shao
,
J.
,
1993
, “
Linear Model Selection by Cross-Validation
,”
J. Am. Stat. Assoc.
,
88
, pp.
486
494
.10.1080/01621459.1993.10476299
39.
McFarland
,
J. M.
,
2008
, “
Uncertainty Analysis for Computer Simulations Through Validation and Calibration
,” Ph.D. thesis, Vanderbilt University, Nashville, TN.
40.
Sacks
,
J.
,
Welch
,
W. J.
,
Mitchell
,
T. J.
, and
Wynn
,
H. P.
,
1989
, “
Design and Analysis of Computer Experiments
,”
Stat. Sci.
,
4
(
4
), pp.
409
435
.10.1214/ss/1177012413
41.
MacKay
,
D. J. C.
,
1998
, “
Introduction to Gaussian Processes
,”
NATO ASI Series, Ser. F
,
168
, pp.
133
166
.
42.
Paciorek
,
C. J.
,
2003
, “
Nonstationary Gaussian Processes for Regression and Spatial Modelling
,” Ph.D. thesis,
Carnegie Mellon University, Pittsburg, PA
.
43.
Goldberg
,
P. W.
,
Williams
,
C. K. I.
, and
Bishop
,
C. M.
,
1998
, “
Regression With Input-Dependent Noise: A Gaussian Process Treatment
,”
Adv. Neural Inf. Process. Syst.
,
10
, pp.
493
499
.
44.
Kersting
,
K.
,
Plagemann
,
C.
,
Pfaff
,
P.
, and
Burgard
,
W.
,
2007
, “
Most Likely Heteroscedastic Gaussian Process Regression
,”
Proceedings of the 24th International Conference on Machine Learning
,
ACM
, pp.
393
400
.
45.
Iwan
,
W. D.
,
1966
, “
A Distributed-Element Model for Hysteresis and Its Steady-State Dynamic Response
,”
ASME J. Appl. Mech.
,
33
, pp.
893
900
.10.1115/1.3625199
46.
Segalman
,
D. J.
,
2002
, “
A Four-Parameter Iwan Model for Lap-Type Joints
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2002-3828.
47.
Lilliefors
,
H. W.
,
1967
, “
On the Kolmogorov-Smirnov Test for Normality With Mean and Variance Unknown
,”
J. Am. Stat. Assoc.
,
62
(
318
), pp.
399
402
.10.1080/01621459.1967.10482916
48.
Anderson
,
T. L.
,
2005
,
Fracture Mechanics: Fundamentals and Applications
,
CRC Press
,
Boca Raton
.
49.
Paris
,
P. C.
, and
Erdogan
,
F.
,
1963
, “
A Critical Analysis of Crack Propagation Laws
,”
J. Basic Eng.
,
85
(
4
), pp.
528
534
.10.1115/1.3656900
50.
Forman
,
R. G.
,
Kearney
,
V. E.
, and
Engle
,
R. M.
,
1967
, “
Numerical Analysis of Crack Propagation in Cyclic-Loaded Structures
,”
J. Basic Eng.
,
89
(
3
), pp.
459
463
.10.1115/1.3609637
51.
Klesnil
,
M.
, and
Lukáš
,
P.
,
1972
, “
Influence of Strength and Stress History on Growth and Stabilisation of Fatigue Cracks
,”
Eng. Fract. Mech.
,
4
(
1
), pp.
77
92
.10.1016/0013-7944(72)90078-1
52.
Shantz
,
C.
,
2010
, “
Uncertainty Quantification in Fatigue Crack Propagation Under Multiaxial Variable Amplitude Loading
,” Ph.D. thesis, Vanderbilt University, Nashville, TN.
53.
Sankararaman
,
S.
,
Ling
,
Y.
,
Shantz
,
C.
, and
Mahadevan
,
S.
,
2011
, “
Inference of Equivalent Initial Flaw Size Under Multiple Sources of Uncertainty
,”
Int. J. Fatigue
,
33
(
2
), pp.
75
89
.10.1016/j.ijfatigue.2010.06.008
54.
Fatemi
,
A.
, and
Shamsaei
,
N.
,
2011
, “
Multiaxial Fatigue: An overview and Some Approximation Models for Life Estimation
,”
Elsv., Int. J. Fatigue
,
33
, pp.
948
958
.
55.
Forth
,
S. C.
,
Favrow
,
L. H.
,
Keat
,
W. D.
, and
Newman
,
J. A.
,
2003
, “
Three-Dimensional Mixed-Mode Fatigue Crack Growth in a Functionally Graded Titanium Alloy
,”
Eng. Fract. Mech.
,
70
(
15
), pp.
2175
2185
.10.1016/S0013-7944(02)00237-0
56.
Reis
,
L.
,
Li
,
B.
, and
de Freitas
,
M.
,
2009
, “
Crack Initiation and Growth Path Under Multiaxial Fatigue Loading in Structural Steels
,”
Int. J. Fatigue
,
31
(
11–12
), pp.
1660
1668
.10.1016/j.ijfatigue.2009.01.013
57.
Carter
,
B. J.
,
Wawrzynek
,
P. A.
, and
Ingraffea
,
A. R.
,
2000
, “
Automated 3-D Crack Growth Simulation
,”
Int. J. Numer. Methods Eng.
,
47
, pp.
229
253
.10.1002/(SICI)1097-0207(20000110/30)47:1/3<229::AID-NME769>3.0.CO;2-2
58.
Maligno
,
A. R.
,
Rajaratnam
,
S.
,
Leen
,
S. B.
, and
Williams
,
E. J.
,
2010
, “
A Three-Dimensional (3d) Numerical Study Of Fatigue Crack Growth Using Remeshing Techniques
,”
Eng. Fract. Mech.
,
77
(
1
), pp.
94
111
.10.1016/j.engfracmech.2009.09.017
59.
Ingrafea
,
T.
, et al. .,
1996
,
FRANC3D, 3D Fracture Analysis Code
,
The Cornell University Fracture Group
,
Cornell University
,
Ithaca, NY
.
60.
Jiang
,
X.
, and
Mahadevan
,
S.
,
2006
, “
Bayesian Cross-Entropy Methodology for Optimal Design of Validation Experiments
,”
Meas. Sci. Technol.
,
17
, pp.
1895
1908
.10.1088/0957-0233/17/7/031
You do not currently have access to this content.