Abstract

The modern scientific process often involves the development of a predictive computational model. To improve its accuracy, a computational model can be calibrated to a set of experimental data. A variety of validation metrics can be used to quantify this process. Some of these metrics have direct physical interpretations and a history of use, while others, especially those for probabilistic data, are more difficult to interpret. In this work, a variety of validation metrics are used to quantify the accuracy of different calibration methods. Frequentist and Bayesian perspectives are used with both fixed effects and mixed-effects statistical models. Through a quantitative comparison of the resulting distributions, the most accurate calibration method can be selected. Two examples are included which compare the results of various validation metrics for different calibration methods. It is quantitatively shown that, in the presence of significant laboratory biases, a fixed effects calibration is significantly less accurate than a mixed-effects calibration. This is because the mixed-effects statistical model better characterizes the underlying parameter distributions than the fixed effects model. The results suggest that validation metrics can be used to select the most accurate calibration model for a particular empirical model with corresponding experimental data.

References

References
1.
Oberkampf
,
W. L.
, and
Roy
,
C. J.
,
2010
,
Verification and Validation in Scientific Computing
, 1st ed.,
Cambridge University Press
,
New York
.
2.
Roache
,
P. J.
,
1998
,
Verification and Validation in Computational Science and Engineering
,
Hermosa Publishing
,
Albuquerque, NM
.
3.
Katsma
,
K. R.
, et al.,
1989
, “
Quantifying Reactor Safety Margins: Application of CSAU Evaluation Methodology to a Large-Break LOCA
,” Nuclear Regulatory Commission, Washington, DC, Report No. NUREG/CR-5249.
4.
Oberkampf
,
W. L.
,
Pilch
,
M.
, and
Trucano
,
T. G.
,
2007
, “
Predictive Capability Maturity Model for Computational Modeling and Simulation
,” Sandia National Laboratories, Albuquerque NM, Report No. SAND2007-5948.
5.
Babuška
,
I.
,
Nobile
,
F.
, and
Tempone
,
R.
,
2008
, “
A Systematic Approach to Model Validation Based on Bayesian Updates and Prediction Related Rejection Criteria
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2517
2539
.10.1016/j.cma.2007.08.031
6.
Trucano
,
T. G.
,
Swiler
,
L. P.
,
Igusa
,
T.
,
Oberkampf
,
W. L.
, and
Pilch
,
M.
,
2006
, “
Calibration, Validation, and Sensitivity Analysis: What's What
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1331
1357
.10.1016/j.ress.2005.11.031
7.
Maupin
,
K.
,
Swiler
,
L.
, and
Porter
,
N. W.
,
2018
, “
Validation Metrics for Deterministic and Probabilistic Data
,”
ASME J. Verif. Valid. Uncertainty Quantif.
,
3
(
3
), p. 031002.10.1115/1.4042443
8.
Jaynes
,
E. T.
,
2003
,
Probability Theory: The Logic of Science
,
Cambridge University Press
,
New York
.
9.
Gilks
,
W. R.
,
Richardson
,
S.
, and
Spiegelhalter
,
D. J.
,
1996
,
Markov Chain Monte Carlo in Practice
,
Chapman and Hall
,
London
.
10.
Draper
,
N. R.
, and
Smith
,
H.
,
1981
,
Applied Regression Analysis
,
2
nd ed.,
Wiley
,
New York
.
11.
Smith
,
R. C.
,
2014
,
Uncertainty Quantification: Theory, Implementation, and Applications
,
Siam
,
Philadelphia, PA
.
12.
Gelman
,
A.
,
2014
,
Bayesian Data Analysis
,
CRC Press
,
Boca Raton, FL
.
13.
Kennedy
,
M. C.
, and
O'Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc. Ser. B (Stat. Methodol.)
,
63
(
3
), pp.
425
464
.10.1111/1467-9868.00294
14.
Kruschke
,
J. K.
,
2015
,
Doing Bayesian Data Analysis: A Tutorial With R, JAGS, and Stan
,
Academic Press
,
Boston, IL
.
15.
Porter
,
N. W.
,
2018
, “
Development of a Novel Residual Formulation of CTF and Application of Parameter Estimation Techniques
,”
Ph.D. thesis
,
North Carolina State University
, Raleigh, NC.https://repository.lib.ncsu.edu/bitstream/handle/1840.20/35650/etd.pdf?sequence=1
16.
Hastings
,
W. K.
,
1970
, “
Monte Carlo Sampling Methods Using Markov Chains and Their Applications
,”
Biometrika
,
57
(
1
), pp.
97
109
.10.1093/biomet/57.1.97
17.
Haario
,
H.
,
Laine
,
M.
,
Mira
,
A.
, and
Saksman
,
E.
,
2006
, “
DRAM: Efficient Adaptive MCMC
,”
Stat. Comput.
,
16
(
4
), pp.
339
354
.10.1007/s11222-006-9438-0
18.
Haario
,
H.
,
Saksman
,
E.
, and
Tamminen
,
J.
,
2001
, “
An Adaptive Metropolis Algorithm
,”
Bernoulli
,
7
(
2
), pp.
223
242
.10.2307/3318737
19.
Mira
,
A.
,
2001
, “
On Metropolis-Hastings Algorithm With Delayed Rejection
,”
Metron
,
59
(
3
), pp.
231
241
.ftp://metron.sta.uniroma1.it/RePEc/articoli/2001-LIX-3_4-16.pdf
20.
Schmidt
,
K. L.
,
2016
, “
Uncertainty Quantification for Mixed-Effects Models With Applications in Nuclear Engineering
,” Ph.D. thesis,
North Carolina State University
, Raleigh, NC.
21.
Brooks
,
S. P.
, and
Gelman
,
A.
,
1998
, “
General Methods for Monitoring Convergence of Iterative Simulations
,”
J. Comput. Graph. Stat.
,
7
(
4
), pp.
434
455
.10.2307/1390675
22.
Gelman
,
A.
, and
Rubin
,
D. B.
,
1992
, “
Inference From Iterative Simulation Using Multiple Sequences
,”
Stat. Sci.
,
7
(
4
), pp.
457
511
.10.1214/ss/1177011136
23.
Laine
,
M.
,
2008
, “
Adaptive MCMC Methods With Applications in Environmental and Geophysical Models
,” Ph.D. thesis,
Lappeenranta University of Technology
, Lappeenranta, Finland.
24.
Laine
,
M.
,
2017
, “MCMC Toolbox for Matlab,” accessed Sept. 13, 2019, mjlaine.github.io/mcmcstat/
25.
Mathworks
,
2008
, “nlmefit,” Mathworks, Natick, MA, accessed Sept. 13, 2019, mathworks.com/help/stats/nlmefit.html
26.
Pinheiro
,
J.
, and
Bates
,
D. M.
,
2000
,
Mixed-Effects Models in S and S-Plus
,
Springer
,
New York
.
27.
Nelder
,
J. A.
, and
Mead
,
R.
,
1965
, “
A Simplex Method for Function Minimization
,”
Comput. J.
,
7
(
4
), pp.
308
313
.10.1093/comjnl/7.4.308
28.
Gamerman
,
D.
,
1997
,
Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference
,
Chapman and Hall
,
Boca Raton, FL
.
29.
Wakefield
,
J. C.
,
Smith
,
A. F. M.
,
Racine-Poon
,
A.
, and
Gelfand
,
A. E.
,
1994
, “
Bayesian Analysis of Linear and Non-Linear Population Models Using the Gibbs Sampler
,”
J. R. Stat. Soc. Ser. C (Appl. Stat.)
,
43
(
1
), pp.
201
221
.10.2307/2986121
30.
Susiluoto
,
J.
,
Raivonen
,
M.
,
Backman
,
L.
,
Laine
,
M.
,
Makela
,
J.
,
Peltola
,
O.
,
Vesala
,
T.
, and
Aalto
,
T.
,
2018
, “
Calibrating the sqHIMMELI v1.0 Wetland Methane Emission Model With Hierarchical Modeling and Adaptive MCMC
,”
Geosci. Model Develop.
,
11
(
3
), pp.
1199
1228
.10.5194/gmd-11-1199-2018
31.
Maupin
,
K. A.
, and
Swiler
,
L. P.
,
2017
, “
Validation Metrics for Deterministic and Probabilistic Data
,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2016-1421.
32.
Debnath
,
L.
, and
Mikusiński
,
P.
,
2005
,
Hibert Spaces With Applications
, 3rd ed.,
Academic Press
,
Cambridge, MA
.
33.
Mahalanobis
,
P. C.
,
1936
, “
On the Generalized Distance in Statistics
,”
Proc. Natl. Inst. Sci.
,
2
(
1
), pp.
49
55
.
34.
Liu
,
Y.
,
Chen
,
W.
,
Arendt
,
P.
, and
Huang
,
H.
,
2011
, “
Toward a Better Understanding of Model Validation Metrics
,”
ASME J. Mech. Des.
,
133
(
7
), p. 071005.10.1115/1.4004223
35.
Kullback
,
S.
, and
Leibler
,
R. A.
,
1951
, “
On Information and Sufficiency
,”
Ann. Math. Stat.
,
22
(
1
), pp.
79
86
.10.1214/aoms/1177729694
36.
Weisbach
,
J. L.
,
1845
,
Lehrbuch Der Ingenieur- Und Maschinen-Mechanik (Textbook of Engineering and Machine Mechanics)
,
Vieweg
,
Brunswick, Germany
.
37.
Poiseuille
,
J. L. M.
,
1841
, “
Recherches Expérimentales Sur le Mouvement Des Liquides Dans Les Tubes de Tres-Petits Diamtres (Experimental Research on the Movement of Liquids in Tubes of Very Small Diameters)
,”
Comptes Rendus, Acadmie Des Sciences
, Paris, France.
38.
McAdams
,
W. H.
,
1954
,
Heat Transmission
,
3
rd ed.,
McGraw-Hill
,
New York
.
39.
Koo
,
E. C.
,
1932
, “
Mechanisms of Isothermal and Non-Isothermal Flow of Fluids in Pipes
,” Ph.D. thesis,
Massachusetts Institute of Technology
, Cambridge, MA.
40.
Swanson
,
C. J.
,
Julian
,
B.
,
Ihas
,
G. G.
, and
Donnelly
,
R. J.
,
2002
, “
Pipe Flow Measurements Over a Wide Range of Reynolds Numbers Using Liquid Helium and Various Gases
,”
J. Fluid Mech.
,
461
, pp.
51
60
.10.1017/S0022112002008595
41.
Muriel
,
A.
,
1998
, “
Quantum Kinetic Model of Turbulence
,”
Phys. D Nonlinear Phenom.
,
124
(
1–3
), pp.
225
247
.10.1016/S0167-2789(98)00197-3
42.
Patel
,
V. C.
, and
Head
,
M. R.
,
1969
, “
Some Observations on Skin Friction and Velocity Profiles in Fully Developed Pipe and Channel Flows
,”
J. Fluid Mech.
,
38
(
1
), pp.
181
201
.10.1017/S0022112069000115
43.
Schlichting
,
H.
, and
Gersten
,
K.
,
1982
,
Boundary Layer Theory
,
Springer
,
Berlin, Germany
.
44.
Everts
,
M.
, and
Meyer
,
J. P.
,
2018
, “
Relationship Between Pressure Drop and Heat Transfer of Developing and Fully Developed Flow in Smooth Horizontal Circular Tubes in the Laminar, Transitional, Quasi-Turbulent and Turbulent Flow Regimes
,”
Int. J. Heat Mass Transfer
,
117
, pp.
1231
1250
.10.1016/j.ijheatmasstransfer.2017.10.072
45.
Nikuradse
,
J.
,
1966
, “
Laws of Turbulent Flow in Smooth Pipes,” Tech. Rep. TT 359, National Advisory Committee for Aeronautics. Translation of "Gesetzmässingkeiten Der Turbulenten Stromung in Glatten Rohren
,”
Verein Deutscher Ingenieure-Forschungshelft
,
356
(
3
), pp.
1
74
.
46.
Zagarola
,
M. V.
, and
Smits
,
A. J.
,
1998
, “
Mean-Flow Scaling of Turbulent Pipe Flow
,”
J. Fluid Mech.
,
373
, pp.
33
79
.10.1017/S0022112098002419
47.
Furuichi
,
N.
,
Terao
,
Y.
,
Wada
,
Y.
, and
Tsuji
,
Y.
,
2015
, “
Friction Factor and Mean Velocity Profile for Pipe Flow at High Reynolds Numbers
,”
Phys. Fluids
,
27
(
9
), p.
095108
.10.1063/1.4930987
48.
Mullins
,
J.
, and
Mahadevan
,
S.
,
2016
, “
Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
,”
ASME J. Verif., Valid. Uncertainty Quantif.
,
1
(
1
), p. 011006.10.1115/1.4032371
49.
Sargsyan
,
K.
,
Najm
,
H. N.
, and
Ghanem
,
R.
,
2015
, “
On the Statistical Calibration of Physical Models
,”
Int. J. Chem. Kinetics
,
47
(
4
), pp.
246
276
.10.1002/kin.20906
50.
Arendt
,
P. D.
,
Apley
,
D. W.
, and
Chen
,
W.
,
2012
, “
Quantification of Model Uncertainty: Calibration, Model Discrepancy, and Identifiability
,”
ASME J. Mech. Des.
,
134
(
10
), pp.
1
12
.10.1115/1.4007390
You do not currently have access to this content.