Abstract

The purpose of this paper is not to present new results; rather, it is to show that the current approach to model validation is not consistent with the accepted mathematics of probability theory. Specifically, we argue that the Sandia V&V Challenge Problem is ill-posed in that the answers sought do not, mathematically, exist. We apply our arguments to show the types of mistakes present in the papers presented in the Journal of Verification, Validation and Uncertainty Quantification, Volume 1,1 along with the challenge problem. Further, we argue that, when the problem is properly posed, both the applicable methodology and the solution techniques are easily drawn from the well-developed mathematics of probability and decision theory. The unfortunate aspect of the challenge problem as currently stated is that it leads to incorrect and inappropriate mathematical approaches that should be avoided and corrected in the current literature.

References

References
1.
Hu
,
K.
,
Carnes
,
B.
, and
Romero
,
V.
,
2016
, “
Introduction: The 2014 Sandia Verification and Validation Challenge Workshop
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
010301
.10.1115/1.4032569
2.
Hu
,
K.
, and
Orient
,
G. E.
,
2016
, “
The 2014 Sandia Verification and Validation Challenge: Problem Statement
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011001
. 10.1115/1.4032498
3.
Hazelrigg
,
G. A.
,
2012
,
Fundamentals of Decision Making for Engineering Design and Systems Engineering
.
4.
Caves
,
C. M.
,
Fuchs
,
C. A.
, and
Schack
,
R.
,
2001
, “
Quantum Probabilities as Bayesian Probabilities
,” arXiv:quant-ph/0106133v2.
5.
Howard
,
R. A.
, and
Abbas
,
A. E.
,
2016
,
Foundations of Decision Analysis
,
Pearson
,
Essex, UK
.
6.
Lindley
,
D.
,
2013
,
Understanding Uncertainty
,
Wiley
,
Hoboken, NJ
.
7.
Ash
,
R. B.
,
2008
,
Basic Probability Theory
,
Dover Books on Mathematics
,
Mineola, New York
.
8.
Kolmogorov
,
A.
,
1956
,
Grundbegriffe der Wahrscheinlich-Keitsrechnung
,
Springer
,
New York
.
9.
Bernardo
,
J. M.
, and
Smith
,
A. F. M.
,
2000
,
Bayesian Theory
,
Wiley
,
New York
.
10.
Hájek
,
A.
,
2005
, “
Scotching Dutch Books
,”
Philos. Perspect.
,
19
(
1
), pp. 139–151.10.1111/j.1520-8583.2005.00057.x
11.
Hajek
,
A.
,
2009
, “
Dutch Book Arguments
,”
The Oxford Handbook of Rational and Social Choice
,
P.
Anand
,
P.
Pattanaik
, and
C.
Puppe
, eds.,
Oxford University Press
,
New York
, Chap.
7
.
12.
Ramsey
,
F. P.
,
1926
, “
Truth and Probability
,”
The Foundations of Mathematics and Other Logical Essays
,
R. B.
Braithwaite
, ed.,
Kegan, Paul, Trench, Trubner & Co
,
London
, Chap.
7
.
13.
de Finetti
,
B.
,
1990
,
Theory of Probability
,
Wiley
,
New York
.
14.
de Finetti
,
B.
,
1931
, “
Sul Significato Soggettivo Della Probabilità
,”
Fundam. Math.
,
17
, pp.
298
329
.10.4064/fm-17-1-298-329
15.
Shakespeare
,
W.
,
1606
, “
The Tragedy of Macbeth
,”.
16.
Oreskes
,
N.
,
Shrader-Frechette
,
K.
, and
Belitz
,
K.
,
1994
, “
Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences
,”
Science
,
263
(
5147
), pp.
641
646
.10.1126/science.263.5147.641
17.
Houghton Mifflin Harcourt, 2016, The American Heritage Dictionary of the English Language, 5th ed., Houghton Mifflin Harcourt Publisher, Boston, MA, accessed Mar. 5, 2020, www.yourdictionary.com/valid
18.
ASME
, 2007, “
Guide for Verification and Validation in Computational Solid Mechanics
,” ASME, New York, accessed Mar. 2, 2020, https://www.asme.org/codes-standards/find-codes-standards/v-v-10-guide-verification-validation-computational-solid-mechanics
19.
Beghini
,
L. L.
, and
Hough
,
P. D.
,
2016
, “
Sandia Verification and Validation Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011002
.10.1115/1.4032369
20.
Choudhary
,
A.
,
Voyles
,
I. T.
,
Roy
,
C. J.
,
Oberkampf
,
W. L.
, and
Patil
,
M.
,
2016
, “
Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011003
.10.1115/1.4031285
21.
Li
,
W.
,
Chen
,
S.
,
Jiang
,
Z.
,
Apley
,
D. W.
,
Lu
,
Z.
, and
Chen
,
W.
,
2016
, “
Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011004
. 10.1115/1.4031983
22.
Xi
,
Z.
, and
Yang
,
R.-J.
,
2016
, “
Reliability Analysis With Model Uncertainty Coupling With Parameter and Experiment Uncertainties: A Case Study of 2014 Verification and Validation Challenge Problem
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011005
.10.1115/1.4031984
23.
Mullins
,
J.
, and
Mahadevan
,
S.
,
2016
, “
Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011006
.10.1115/1.4032371
24.
Paez
,
P. J.
,
Paez
,
T. L.
, and
Hasselman
,
T. K.
,
2016
, “
Economic Analysis of Model Validation for a Challenge Problem
,”
ASME J. Verif., Validation Uncertainty Quantif.
,
1
(
1
), p.
011007
.10.1115/1.4032370
25.
Zadeh
,
L.
,
1992
,
Fuzzy Logic for the Management of Uncertainty
,
Wiley
,
New York
.
26.
Haack
,
S.
, and
Logic
,
D.
,
1996
,
Fuzzy Logic: Beyond the Formalism
,
The University of Chicago Press
,
Chicago, IL
.
27.
Haack
,
S.
,
1978
,
Philosophy of Logics
,
Cambridge University Press
,
Cambridge, UK
.
28.
Hofstadter
,
D. R.
,
1979
,
Gödel, Escher, Bach
,
Basic Book
,
New York
.
29.
Howard
,
R. A.
,
1966
, “
Information Value Theory
,”
IEEE Trans. Syst. Sci. Cybern.
, 2(1), pp.
22
26
.10.1109/TSSC.1966.300074
30.
de Neufville
,
R.
,
1990
,
Applied Systems Analysis: Engineering Planning and Technology Management
,
McGraw-Hill
,
New York
.
31.
Hazelrigg
,
G. A.
,
1977
, “
Economic Viability of Pursuing a Space Power Concept
,”
J. Energy
,
1
(
2
), pp.
93
99
.10.2514/3.62319
32.
von Neumann
,
J.
, and
Morgenstern
,
O.
,
1953
,
The Theory of Games and Economic Behavior
, 3rd ed.,
Princeton University Press
,
Princeton, NJ
.
33.
Luce
,
D.
, and
Raiffa
,
1957
,
Games and Decisions
,
Wiley
,
New York
.
34.
Arrow
,
K. J.
,
1963
,
Social Choice and Individual Values
, 2nd ed.,
Wiley
,
New York
.
35.
Tribus
,
M.
,
1969
,
Rational Descriptions, Decisions and Designs
,
Pergamon Press
,
New York
.
36.
Bernoulli
,
D.
,
1968
, “
Exposition of a New Theory on the Measurement of Risk” (1738)
,”
Utility Theory: A Book of Readings
,
Wiley
,
New York
, pp.
199
214
.
37.
Dodgson
,
C. L.
,
1865
,
Alice in Wonderland
,
Macmillan
,
London
.
38.
Hazelrigg
,
G. A.
,
2009
, “
The Cheshire Cat on Engineering Design
,”
Qual. Reliab. Eng. Int.
,
25
(
6
), pp.
759
769
.10.1002/qre.1028
You do not currently have access to this content.