To use predictive models in engineering design of physical systems, one should first quantify the model uncertainty via model updating techniques employing both simulation and experimental data. While calibration is often used to tune unknown calibration parameters of a computer model, the addition of a discrepancy function has been used to capture model discrepancy due to underlying missing physics, numerical approximations, and other inaccuracies of the computer model that would exist even if all calibration parameters are known. One of the main challenges in model updating is the difficulty in distinguishing between the effects of calibration parameters versus model discrepancy. We illustrate this identifiability problem with several examples, explain the mechanisms behind it, and attempt to shed light on when a system may or may not be identifiable. In some instances, identifiability is achievable under mild assumptions, whereas in other instances, it is virtually impossible. In a companion paper, we demonstrate that using multiple responses, each of which depends on a common set of calibration parameters, can substantially enhance identifiability.

References

References
1.
Chen
,
W.
,
Yin
,
X.
,
Lee
,
S.
, and
Liu
,
W.
, 2010, “
A Multiscale Design Methodology for Hierarchical Systems With Random Field Uncertainty
,”
ASME J. Mech. Des.
,
132
(
4
), p.
041006
.
2.
Chen
,
W.
,
Xiong
,
Y.
,
Tsui
,
K. L.
, and
Wang
,
S.
, 2008, “
A Design-Driven Validation Approach Using Bayesian Prediction Models
,”
ASME J. Mech. Des.
,
130
(
2
), p.
021101
.
3.
Wang
,
Z.
,
Huang
,
H.-Z.
, and
Liu
,
Y.
, 2010, “
A Unified Framework for Integrated Optimization Under Uncertainty
,”
ASME J. Mech. Des.
,
132
(
5
), p.
051008
.
4.
Singh
,
A.
,
Mourelatos
,
Z. P.
, and
Li
,
J.
, 2010, “
Design for Lifecycle Cost Using Time-Dependent Reliability
,”
ASME J. Mech. Des.
,
132
(
9
), p.
091008
.
5.
Apley
,
D. W.
, and
Kim
,
J. B.
, 2011, “
A Cautious Approach to Robust Parameter Design With Model Uncertainty
,”
IIE Trans.
,
43
(
7
), pp.
471
782
.
6.
Lancaster
,
T.
, 2004,
An Introduction to Modern Bayesian Econometrics
,
Blackwell Publishing
,
Malden, MA
.
7.
Arendt
,
P.
,
Apley
,
D.
, and
Chen
,
W.
, 2012, “
Improving Identifiability in Model Calibration Using Multiple Responses
,”
ASME J. Mech. Des.
,
134
(
10
), p.
100909
.
8.
Kennedy
,
M. C.
, and
O’Hagan
,
A.
, 2001, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc. Ser. B (Stat. Methodol.)
,
63
(
3
), pp.
425
464
.
9.
Box
,
G.
,
Hunter
,
W.
, and
Hunter
,
J.
, 1978,
Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building
,
Wiley
,
New York
.
10.
McKay
,
M. D.
,
Beckman
,
R. J.
, and
Conover
,
W. J.
, 1979, “
A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code
,”
Technometrics
,
21
(
2
), pp.
239
245
.
11.
Sacks
,
J.
,
Welch
,
W. J.
,
Mitchell
,
T. J.
, and
Wynn
,
H. P.
, 1989, “
Design and Analysis of Computer Experiments
,”
Stat. Sci.
,
4
(
4
), pp.
409
423
.
12.
Liu
,
Y.
,
Chen
,
W.
,
Arendt
,
P.
, and
Huang
,
H.-Z.
, 2011, “
Toward a Better Understanding of Model Validation Metrics
,”
ASME J. Mech. Des.
,
133
(
7
), p.
071005
.
13.
Xiong
,
Y.
,
Chen
,
W.
,
Tsui
,
K. L.
, and
Apley
,
D. W.
, 2009, “
A Better Understanding of Model Updating Strategies in Validating Engineering Models
,”
Comput. Methods Appl. Mech. Eng.
,
198
(
15–16
), pp.
1327
1337
.
14.
Lindgren
,
L.-E.
,
Alberg
,
H.
, and
Domkin
,
K.
, “
Constitutive Modelling and Parameter Optimization
,” Proc. 7th International Conference on Computational Plasticity, Barcelona.
15.
McFarland
,
J.
,
Mahadevan
,
S.
,
Romero
,
V.
, and
Swiler
,
L.
, 2008, “
Calibration and Uncertainty Analysis for Computer Simulations With Multivariate Output
,”
AIAA J.
,
46
(
5
), pp.
1253
1265
.
16.
Wang
,
S.
,
Tsui
,
K. L.
, and
Chen
,
W.
, 2009, “
Bayesian Validation of Computer Models
,”
Technometrics
,
51
(
4
), pp.
439
451
.
17.
Higdon
,
D.
,
Kennedy
,
M. C.
,
Cavendish
,
J.
,
Cafeo
,
J.
, and
Ryne
,
R.
, 2004, “
Combining Field Data and Computer Simulations for Calibration and Prediction
,”
SIAM J. Sci. Comput. (USA)
,
26
(
2
), pp.
448
466
.
18.
Bayarri
,
M. J.
,
Berger
,
J. O.
,
Paulo
,
R.
,
Sacks
,
J.
,
Cafeo
,
J. A.
,
Cavendish
,
J.
,
Lin
,
C. H.
, and
Tu
,
J.
, 2007, “
A Framework for Validation of Computer Models
,”
Technometrics
,
49
(
2
), pp.
138
154
.
19.
Loeppky
,
J.
,
Bingham
,
D.
, and
Welch
,
W.
, 2006, “
Computer Model Calibration or Tuning in Practice
,” Technical Report University of British Columbia, Vancouver, BC, CA.
20.
Apley
,
D.
,
Liu
,
J.
, and
Chen
,
W.
, 2006, “
Understanding the Effects of Model Uncertainty in Robust Design With Computer Experiments
,”
ASME J. Mech. Des.
,
128
(
4
), pp.
945
958
.
21.
Hasselman
,
T. K.
,
Yap
,
K.
,
Lin
,
C.-H.
, and
Cafeo
,
J.
, 2005, “
A Case Study in Model Improvement for Vehicle Crashworthiness Simulation
,” Proceedings of the 23rd International Modal Analysis Conference, Orlando, FL.
22.
Easterling
,
R. G.
, and
Berger
,
J. O.
, 2002, “
Statistical Foundations for the Validation of Computer Models
,” Presented at Computer Model Verification and Validation in the 21st Century Workshop, Johns Hopkins University.
23.
Qian
,
P. Z. G.
, and
Wu
,
C. F. J.
, 2008, “
Bayesian Hierarchical Modeling for Integrating Low-Accuracy and High-Accuracy Experiments
,”
Technometrics
,
50
(
2
), pp.
192
204
.
24.
Drignei
,
D.
, 2009, “
A Kriging Approach to the Analysis of Climate Model Experiments
,”
J. Agri. Biol. Environ. Stat.
,
14
(
1
), pp.
99
114
.
25.
Liu
,
F.
,
Bayarri
,
M. J.
,
Berger
,
J. O.
,
Paulo
,
R.
, and
Sacks
,
J.
, 2008, “
A Bayesian Analysis of the Thermal Challenge Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2457
2466
.
26.
Rasmussen
,
C. E.
, and
Williams
,
C. K. I.
, 2006,
Gaussian Processes for Machine Learning
,
MIT, Cambridge
,
MA
.
27.
Jin
,
R.
,
Chen
,
W.
, and
Simpson
,
T. W.
, 2001, “
Comparative Studies of Metamodelling Techniques Under Multiple Modelling Criteria
,”
Struct. Multidiscip. Optim.
,
23
(
1
), pp.
1
13
.
28.
Santner
,
T. J.
,
Williams
,
B. J.
, and
Notz
,
W. I.
, 2003,
The Design and Analysis of Computer Experiments
,
Springer-Verlag
,
New York
.
29.
Cressie
,
N.
, 1993,
Statistics for Spatial Data
,
Wiley
,
New York.
30.
Schabenberger
,
O.
, and
Gotway
,
C.
, 2005,
Statistical Methods for Spatial Data Analysis
,
Chapman & Hall/CRC
,
Boca Raton, FL
.
31.
Liu
,
F.
,
Bayarri
,
M. J.
, and
Berger
,
J. O.
, 2009, “
Modularization in Bayesian Analysis, With Emphasis on Analysis of Computer Models
,”
Bayesian Anal.
,
4
(
1
), pp.
119
150
.
32.
Kennedy
,
M. C.
,
Anderson
,
C. W.
,
Conti
,
S.
, and
O’Hagan
,
A.
, 2006, “
Case Studies in Gaussian Process Modelling of Computer Codes
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1301
1309
.
33.
Jin
,
R.
, 2004, “
Enhancements of Metamodeling Techniques in Engineering Design
,” Ph.D. thesis, University of Illinois at Chicago, Chicago, IL.
34.
Rasmussen
,
C. E.
, 1996, “
Evaluation of Gaussian Processes and Other Methods for Non-Linear Regression
,” Ph.D. thesis, University of Toronto, Toronto, Ontario, Canada.
35.
Kennedy
,
M. C.
, and
O’Hagan
,
A.
, 2000, “Supplementary details on Bayesian Calibration of ComputerModels,” available at http://www.isds.duke.edu/~fei/samsi/Oct_09/01Sup-KenOHa.pdf (last accessed on August 26, 2012).
36.
Arendt
,
P.
,
Apley
,
D. W.
, and
Chen
,
W.
, 2010, “
Updating Predictive Models: Calibration, Bias Correction, and Identifiability
,” ASME 2010 International Design Engineering Technical Conferences, Montreal, Quebec, Canada.
37.
Chakrabarty
,
J.
, 2006,
Theory of Plasticity
,
Elsevier Butterworth-Heinemann
,
Burlington, MA
.
38.
Joseph
,
V.
, and
Melkote
,
S.
, 2009, “
Statistical Adjustments to Engineering Models
,”
J. Qual. Technol.
,
41
(
4
), pp.
362
375
. Available at http://asq.org/qic/display-item/index.html?item=30615http://asq.org/qic/display-item/index.html?item=30615 (last accessed on August 26, 2012).
You do not currently have access to this content.