Computational modeling and simulation are paramount to modern science. Computational models often replace physical experiments that are prohibitively expensive, dangerous, or occur at extreme scales. Thus, it is critical that these models accurately represent and can be used as replacements for reality. This paper provides an analysis of metrics that may be used to determine the validity of a computational model. While some metrics have a direct physical meaning and a long history of use, others, especially those that compare probabilistic data, are more difficult to interpret. Furthermore, the process of model validation is often application-specific, making the procedure itself challenging and the results difficult to defend. We therefore provide guidance and recommendations as to which validation metric to use, as well as how to use and decipher the results. An example is included that compares interpretations of various metrics and demonstrates the impact of model and experimental uncertainty on validation processes.

References

References
1.
Roach
,
P. J.
,
1998
,
Verification and Validation in Computational Science and Engineering
,
Hermosa Publishing
, Socorro, NM.
2.
Oberkampf
,
W. J.
, and
Roy
,
C. J.
,
2010
,
Verification and Validation in Scientific Computing
,
Cambridge University Press
, Cambridge, UK.
3.
Babuska
,
I.
,
Nobile
,
F.
, and
Tempone
,
R.
,
2008
, “
A Systematic Approach to Model Validation Based on Bayesian Updates and Prediction Related Rejection Criteria
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29
), pp.
2517
2539
.
4.
Tinsley Oden
,
J.
,
Moser
,
R.
, and
Ghattas
,
O.
,
2010
, “
Computer Predictions With Quantified Uncertainty—Part I
,”
SIAM News
,
43
(
9
), pp. 1–4.https://www.ices.utexas.edu/sites/oden/wp-content/uploads/2013/06/2011-1842-SIAM-NEWS-2010_computerpredictionsPt1.pdf
5.
Tinsley Oden
,
J.
,
Moser
,
R.
, and
Ghattas
,
O.
,
2010
, “
Computer Predictions With Quantified Uncertainty—Part II
,”
SIAM News
,
43
(
10
), pp. 1–3.https://pdfs.semanticscholar.org/6a43/63d913fddcf5bb889e93f131010ef86e195e.pdf
6.
Hawkins-Daarud
,
A.
,
Prudhomme
,
S.
,
van der Zee
,
K. G.
, and
Tinsley Oden
,
J.
,
2013
, “
Bayesian Calibration, Validation, and Uncertainty Quantification of Diffuse Interface Models of Tumor Growth
,”
J. Math. Biol.
,
67
(
6–7
), pp.
1457
1485
.
7.
Oden
,
J. T.
,
Prudencio
,
E. E.
, and
Bauman
,
P. T.
,
2013
, “
Virtual Model Validation of Complex Multiscale Systems: Applications to Nonlinear Elastostatics
,”
Comput. Methods Appl. Mech. Eng.
,
266
, pp.
162
184
.
8.
Farrell
,
K.
,
Oden
,
J. T.
, and
Faghihi
,
D.
,
2015
, “
A Bayesian Framework for Adaptive Selection, Calibration, and Validation of Coarse-Grained Models of Atomistic Systems
,”
J. Comput. Phys.
,
295
, pp.
189
208
.
9.
Liu
,
Y.
,
Chen
,
W.
,
Arendt
,
P.
, and
Huang
,
H. Z.
,
2011
, “
Toward a Better Understanding of Model Validation Metrics
,”
ASME J. Mech. Des.
,
133
(
7
), p.
071005
.
10.
Bayarri
,
M. J.
,
Berger
,
J. O.
,
Paulo
,
R.
,
Sacks
,
J.
,
Cafeo
,
J. A.
,
Cavendish
,
J.
,
Lin
,
C.-H.
, and
Tu
,
J.
,
2007
, “
A Framework for Validation of Computer Models
,”
Technometrics
,
49
(
2
), pp.
138
154
.
11.
Debnath
,
L.
, and
Mikusiński
,
P.
,
2005
,
Hilbert Spaces With Applications
,
Elsevier Academic Press
, Cambridge, UK.
12.
Mahalanobis
,
P. C.
,
1936
, “
On the Generalised Distance in Statistics
,”
Proc. Natl. Inst. Sci. India
,
2
(
1
), pp.
49
55
.
13.
Kullback
,
S.
,
1959
,
Information Theory and Statistics
(Probability and Mathematical Statistics),
Wiley
, Hoboken, NJ.
14.
Kullback
,
S.
, and
Leibler
,
R. A.
,
1951
, “
On Information and Sufficiency
,”
Ann. Math. Stat.
,
22
(
1
), pp.
79
86
.
15.
Pérez-Cruz
,
F.
,
2008
, “
Kullback-Leibler Divergence Estimation of Continuous Distributions
,”
IEEE
International Symposium on Information Theory
, Toronto, ON, Canada, July 6–11, pp.
1666
1670
.
16.
Yamano
,
T.
,
2014
, “
A Note on Bound for Jensen-Shannon Divergence by Jeffreys
,”
First International Electronic Conference on Entropy and Its Applications
(
ECEA-1
), Nov. 3–21, pp.
3
21
.https://sciforum.net/manuscripts/2630/manuscript.pdf
17.
Ghosh
,
J. K.
, and
Ramamoorthi
,
R. V.
,
2006
,
Bayesian Nonparametrics
(Springer Series in Statistics),
Springer
,
New York
.
18.
Ferson
,
S.
,
Oberkampf
,
W. L.
, and
Ginzburg
,
L.
,
2008
, “
Model Validation and Predictive Capability for the Thermal Challenge Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2408
2430
.
19.
Moody
,
L. F.
,
1944
, “
Friction Factors for Pipe Flow
,”
Trans. ASME
,
66
(
8
), pp.
671
684
.
20.
Poiseuille
,
J. L. M.
,
1841
, “
Recherches Expérimentales Sur le Mouvement Des Liquides Dans Les Tubes de Très-Petits Diamètres
,”
Comptes Rendus
,
Académie Des Sciences
, Orlando, FL.
21.
McAdams
,
W. H.
,
1933
,
Heat Transmission
,
1st ed.
,
McGraw-Hill
,
New York
.
22.
Churchill
,
S. W.
,
1973
, “
Empirical Expressions for the Shear Stress in Turbulent Flow in Commercial Pipe
,”
AIChE J.
,
19
(
2
), pp.
375
376
.
23.
Zigrang
,
D. J.
, and
Sylvester
,
N. D.
,
1982
, “
Explicit Approximations to the Solution of Colebrook's Friction Factor Equation
,”
AIChE J.
,
28
(
3
), pp.
514
515
.
24.
Swanson
,
C. J.
,
Julian
,
B.
,
Ihas
,
G. G.
, and
Donnelly
,
R. J.
,
2002
, “
Pipe Flow Measurements Over a Wide Range of Reynolds Numbers Using Liquid Helium and Various Gases
,”
J. Fluid Mech.
,
461
, pp.
51
60
.
25.
Furuichi
,
N.
,
Terao
,
Y.
,
Wada
,
Y.
, and
Tsuji
,
Y.
,
2015
, “
Friction Factor and Mean Velocity Profile for Pipe Flow at High Reynolds Numbers
,”
Phys. Fluids
,
27
(
9
), p.
095108
.
26.
Porter
,
N. W.
,
Mousseau
,
V. A.
, and
Avramova
,
M. N.
,
2018
, “
Quantified Validation With Uncertainty Analysis for Turbulent Single Phase Friction Models
,”
ANS Best Estimate Plus Uncertainty International Conference
, Lucca, Italy, May 13–19, pp. 1–11.
27.
Adams
,
B. M.
,
Bohnhoff
,
W. J.
,
Dalbey
,
K. R.
,
Eddy
,
J. P.
,
Ebeida
,
M. S.
,
Eldred
,
M. S.
,
Frye
,
J. R.
,
Gerarci
,
G.
,
Hooper
,
R. W.
,
Hough
,
P. D.
,
Hu
,
K. T.
,
Jakeman
,
J. D.
,
Khalil
,
M.
,
Maupin
,
K. A.
,
Monschke
,
J. A.
,
Ridgway
,
E. M.
,
Rushdi
,
A.
,
Stephens
,
J. A.
,
Swiler
,
L. P.
,
Vigil
,
D. M.
, and
Wildey
,
T. M.
,
2017
, “
Dakota, a Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.7 User's Manual
,” Sandia National Laboratories, Albuquerque, NM, Report No.
SAND2014-4633
.https://www.osti.gov/biblio/1177077-dakota-multilevel-parallel-object-oriented-framework-design-optimization-parameter-estimation-uncertainty-quantification-sensitivity-analysis
28.
Saltelli
,
A.
,
Ratto
,
M.
,
Andres
,
T.
,
Campolongo
,
F.
,
Cariboni
,
J.
,
Gatelli
,
D.
,
Saisana
,
M.
, and
Tarantola
,
S.
,
2008
,
Global Sensitivity Analysis: The Primer
,
Wiley
, Hoboken, NJ.
29.
Saltelli
,
A.
,
Tarantola
,
S.
,
Campolongo
,
F.
, and
Ratto
,
M.
,
2004
,
Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models
,
Wiley
, Hoboken, NJ.
30.
Saltelli
,
A.
,
Annoni
,
P.
,
Azzini
,
I.
,
Campolongo
,
F.
,
Ratto
,
M.
, and
Tarantola
,
S.
,
2010
, “
Variance Based Sensitivity Analysis of Model Output: Design and Estimator for the Total Sensitivity Index
,”
Comput. Phys. Commun.
,
181
(
2
), pp.
259
270
.
31.
Jaynes
,
E. T.
, and
Bretthorst
,
G. L.
,
2003
,
Probability Theory: The Logic of Science
,
Cambridge University Press
, Cambridge, UK.
32.
Prudencio
,
E. E.
, and
Schulz
,
K. W.
,
2012
, “
The Parallel C++ Statistical Library ‘QUESO’: Quantification of Uncertainty for Estimation, Simulation and Optimization
,”
Euro-Par 2011: Parallel Processing Workshops
, Bordeaux, France, Aug. 29–Sept. 2, pp.
398
407
.https://link.springer.com/chapter/10.1007/978-3-642-29737-3_44
33.
Schlichting
,
H.
, and
Gersten
,
K.
,
2016
,
Boundary-Layer Theory
,
Springer
,
Berlin
.
34.
Zagarola
,
M. V.
, and
Smits
,
A. J.
,
1998
, “
Mean-Flow Scaling of Turbulent Pipe Flow
,”
J. Fluid Mech.
,
373
, pp.
33
79
.
You do not currently have access to this content.