In this discussion paper, we explore different ways to assess the value of verification and validation (V&V) of engineering models. We first present a literature review on the value of V&V and then use value chains and decision trees to show how value can be assessed from a decision maker's perspective. In this context, the value is what the decision maker is willing to pay for V&V analysis with the understanding that the V&V results are uncertain. The 2014 Sandia V&V Challenge Workshop is used to illustrate these ideas.

References

References
1.
ASME V&V 20
,
2009
,
Standard for Verification and Validation in Computational Fluids and Heat Transfer
,
The American Society of Mechanical Engineers
,
New York
.
2.
ASME V&V 10-2006
,
2006
,
Guide for Verification and Validation in Computational Solid Mechanics
,
The American Society of Mechanical Engineers
,
New York
.
3.
AIAA,
1998
, “
Guide for the Verification and Validation of Computational Fluid Dynamics Simulations
,”
AIAA
Paper No. G-077-1998.
4.
Hu
,
K. T.
,
Carnes
,
B.
, and
Romero
,
V.
,
2016
, “
The 2014 Sandia Verification and Validation Challenge Workshop
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
), p.
010202
.
5.
Schroeder
,
B. B.
,
Hu
,
K. T.
,
Winokur
,
J. G.
, and
Mullins
,
J. G.
,
2016
, “
Summary of the 2014 Sandia V&V Challenge Workshop
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
),
015501
.
6.
Choudhary
,
A.
,
Voyles
,
I. T.
,
Roy
,
C. J.
,
Oberkampf
,
W. L.
, and
Patil
,
M.
,
2016
, “
Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
), p.
011003
7.
Li
,
W.
,
Chen
,
S.
,
Jiang
,
Z.
,
Apley
,
D. W.
,
Lu
,
Z.
, and
Chen
,
W.
,
2016
, “
Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
),
011004
.
8.
Mullins
,
J.
, and
Mahadevan
,
S.
,
2016
, “
Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
), p.
011006
.
9.
Beghini
,
L. L.
, and
Hough
,
P. D.
,
2016
, “
Sandia V&V Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
),
011002
.
10.
Xi
,
Z.
, and
Yang
,
R. J.
,
2016
, “
Reliability Analysis With Model Uncertainty Coupling With Parameter and Experimental Uncertainties: A Case Study of 2014 V&V Challenge Problem
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
), p.
011005
.
11.
Paez
,
P. J.
,
Paez
,
T.
, and
Hasselman
,
T. K.
,
2016
, “
Economics Analysis of Model Validation for a Challenge Problem
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
), p.
011007
.
12.
Youngblood
,
S. M.
, “
Roadmap for VV&A Technology Advancement
,”
2004
, Foundations'04: A Workshop for V&V in the 21st Century, Defense Modeling and Simulation Office, Arizona State University.
13.
Pace
,
D.
,
2002
, “
Foundations'02 Overview
,”
Foundations'02 a Workshop on Model and Simulation Verification and Validation for the 21st Century
,
D.
Pace
, ed., JHU/APL, Laurel, MD.
14.
Oberkampf
,
W. L.
,
1998
, “
Bibliography for Verification and Validation in Computational Simulation
,” Sandia National Laboratories, Report No. SAND98-2041.
15.
Nitta
,
C. K.
, and
Logan
,
R. W.
,
2004
, “
ASCI V&V at LLNL: An Unclassified Bibliography
,” Lawrence Livermore National Laboratory, Report No. UCRL-AR-203864.
16.
Jahangirian
,
M.
,
Taylor
,
S. J. E.
, and
Young
,
T.
,
2010
, “
Economics of Modeling and Simulation: Reflections and Implications for Healthcare
,”
2010 Winter Simulation Conference (WSC)
.
17.
Kilikauskas
,
M. L.
, and
Hall
,
D. H.
,
2002
, “
Estimating V&V Resource Requirements and Schedule Impact
,”
Foundations for V&V in the 21st Century Workshop
,
S.
Youngblood
, ed.,
Johns Hopkins University Applied Physics Laboratory
,
Laurel, MD
.
18.
Back
,
G.
,
Love
,
G.
, and
Falk
,
J.
,
2000
, “
The Doing of Model Verification and Validation: Balancing Cost and Theory
,”
18th International Conference of the System Dynamics Society
,
System Dynamics Society
,
Bergen, Norway
.
19.
Gray
,
P.
,
1976
, “
The Economics of Simulation
,”
76 Bicentennial Winter Conference on Simulation, Winter Simulation Conference
,
Gaithersburg, MD
, pp.
17
25
.
20.
Pace
,
D.
,
2004
, “
Modeling and Simulation Verification and Validation Challenges
,”
Johns Hopkins APL Tech. Dig.
,
25
(
2
), pp.
163
172
.
21.
Oberkampf
,
W. L.
,
Pilch
,
M.
, and
Trucano
,
T. G.
,
2007
, “
Predictive Capability Maturity Model for Computational Modeling and Simulation
,” Sandia National Laboratories, Report No. SAND2007-5948.
22.
Easterling
,
R. G.
,
2001
, “
Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations
,” Sandia National Laboratories, Report No. SAND2001-0243.
23.
Rizzi
,
A.
, and
Vos
,
J.
,
1998
, “
Toward Establishing Credibility in Computational Fluid Dynamics Simulations
,”
AIAA J.
,
36
(
5
), pp.
668
675
.
24.
Blattnig
,
S. R.
,
Green
,
L.
,
Luckring
,
J.
,
Morrison
,
J.
,
Tripathi
,
R.
, and
Zang
,
T.
,
2008
, “
Towards a Credibility Assessment of Models and Simulations
,”
AIAA
Paper No. 2008-2156.
25.
Hemez
,
F.
,
Atamturktur
,
H. S.
, and
Unal
,
C.
,
2010
, “
Defining Predictive Maturity for Validated Numerical Simulations
,”
Comput. Struct.
,
88
(
7–8
), pp.
497
505
.
26.
Balci
,
O.
, and
Sargent
,
R. G.
,
1981
, “
A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models
,”
Commun. ACM
,
24
(
4
), pp.
190
197
.
27.
Muessig
,
P. R.
,
Laack
,
D. R.
, and
Wrobleski
,
J. W.
, Jr.
,
1997
, “
Optimizing the Selection of VV&A Activities: A Risk/Benefit Approach
,”
29th Winter Conference on Simulation
,
IEEE Computer Society
,
Atlanta, GA
, pp.
60
66
.
28.
Youngblood
,
S. M.
,
Stutzman
,
M.
,
Pace
,
D. K.
, and
Pandolfini
,
P. P.
,
2011
, “
Risk Based Methodology for Verification, Validation, and Accreditation (VV&A), M&S Use Risk Methodology (MURM)
,” The Johns Hopkins University Applied Physics Laboratory, Technical Report No. NSAD-R-2011-011.
29.
Elele
,
J. N.
, and
Smith
,
J.
,
2010
, “
Risk-Based Verification, Validation, and Accreditation Process
,”
Proc. SPIE
7705
.
30.
Logan
,
R. W.
,
Nitta
,
C. K.
, and
Chidester
,
S. K.
,
2005
, “
Risk Reduction as the Product of Model Assessed Reliability, Confidence, and Consequence
,”
J. Def. Model. Simul.: Appl., Methodol., Technol.
,
2
(
4
), pp.
191
207
.
31.
Nitta
,
C.
,
Logan
,
R.
,
Chidester
,
S.
, and
Foltz
,
M. F.
,
2004
, “
Benefit/Cost Ratio in Systems Engineering: Integrated Models, Tests, Design, and Production
,” Lawrence Livermore National Laboratory, Report No. UCRL-TR-207610.
32.
Paez
,
P. J.
,
Paez
,
T. L.
,
Hasselman
,
T. K.
, and
Hu
,
K.
,
2015
, “
The Economics of Model Validation and Solution of the 2014 Sandia V&V Challenge Problem
,” Sandia National Laboratories, Report No. SAND2015-10560.
33.
Waite
,
W.
,
Lightner
,
G.
,
Gravitz
,
R.
,
Severinghaus
,
R.
,
Waite
,
E.
,
Swenson
,
S.
,
Feinberg
,
J.
,
Cooley
,
T.
,
Gordon
,
S.
,
Oswalt
,
I.
,
2008
, “
Metrics for Modeling and Simulation (M&S) Investments
,” NAVAIR, Report No. TJ-042608-RP013.
34.
Oswalt
,
I.
,
Cooley
,
T.
,
Waite
,
W.
,
Waite
,
E.
,
Gordon
,
S.
,
Severinghaus
,
R.
,
Feinberg
,
J.
,
Lightner
,
G.
,
2015
,
Calculating Return on Investment for U.S. Department of Defense Modeling and Simulation
,
Defense ARJ and Defense AT&L Publications
.
35.
Gibson
,
R.
,
Medeiros
,
D. J.
,
Sudar
,
A.
,
Waite
,
B.
, and
Rohrer
,
M. W.
,
2003
, “
Increasing Return on Investment From Simulation
,”
2003 Winter Simulation Conference
,
S.
Chick
,
P. J.
Sánchez
,
D.
Ferrin
, and
D. J.
Morrice
, eds., pp.
2027
2032
.
36.
Carter
,
J. R.
, III
,
2001
, “
A Business Case for Modeling and Simulation
,” Aviation and Missile Research, Development, and Engineering Center, Special Report No. RD-AS-01-02.
37.
Brown
,
C. D.
,
Grant
,
G.
,
Kotchman
,
D.
,
Reyenga
,
R.
, and
Szanto
,
T.
,
2000
, “
Building a Business Case for Modeling and Simulation
,”
Acquis. Rev. Q.
, pp.
311
328
.
38.
Dabney
,
J. B.
,
Barber
,
G.
, and
Ohi
,
D.
,
2005
, “
Computing Return on Investment of Risk-Reducing Systems Engineering Disciplines
,”
Space Systems Engineering and Risk Management Conference
,
Los Angeles, CA
.
39.
Dabney
,
J. B.
,
Barber
,
G.
, and
Ohi
,
D.
,
2004
, “
Estimating Direct Return on Investment of Independent Verification and Validation
,”
Eighth IASTED International Conference
,
Cambridge, MA
.
40.
Lederer
,
P. J.
, and
Rhee
,
S.-K.
,
1995
, “
Economics of Total Quality Management
,”
J. Oper. Manage.
,
12
(
3–4
), pp.
353
367
.
41.
Abdel-Hamid
,
T. K.
,
1988
, “
The Economics of Software Quality Assurance: A Simulation-Based Case Study
,”
MIS Q.
,
12
(
3
), pp.
395
411
.
42.
Shreve
,
C. M.
, and
Kelman
,
I.
,
2014
, “
Does Mitigation Save? Reviewing Cost-Benefit Analyses of Disaster Risk Reduction
,”
Int. J. Disaster Risk Reduct.
,
10
, pp.
213
235
.
43.
Wethli
,
K.
,
2014
,
2016
, “
Benefit-Cost Analysis for Risk Management: Summary of Selected Examples
,”
World Development Report 2014
,
The World Bank
,
Washington, DC
.
44.
Hu
,
K. T.
, and
Orient
,
G. E.
,
2016
, “
The 2014 Sandia V&V Challenge: Problem Statement
,”
ASME J. Verif., Validation, Uncertainty Quantif.
,
1
(
1
), p.
011001
.
45.
Hu
,
K. T.
,
2013
, “
2014 V&V Challenge: Problem Statement
,” Sandia National Laboratories, Report No. SAND2013-10486P.
46.
Porter
,
M. E.
,
1985
,
Competitive Advantage: Creating and Sustaining Superior Performance
,
Simon and Schuster
,
New York
.
47.
Paté-Cornell
,
M. E.
, and
Dillon
,
R. L.
,
2006
, “
The Respective Roles of Risk and Decision Analyses in Decision Support
,”
Decis. Anal.
,
3
(
4
), pp.
220
232
.
48.
Clemen
,
R. T.
,
1996
,
Making Hard Decisions: An Introduction to Decision Analysis
,
Duxbury Press
,
Boston, MA
.
49.
Berger
,
J. O.
,
1985
,
Statistical Decision Theory and Bayesian Analysis
,
2nd ed.
,
Springer-Verlag
,
Berlin
.
50.
Artzner
,
P.
,
Delbaen
,
F.
,
Eber
,
J.-M.
, and
Heath
,
D.
,
1999
, “
Coherent Measures of Risk
,”
Math. Finance
,
9
(
3
), pp.
203
228
.
51.
Adeyemo
,
A. M.
,
2013
, “
Stochastic Dominance for Project Screening and Selection Under Uncertainty
,” Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA.
52.
Bertsekas
,
D. P.
, and
Tsitsiklis
,
J. N.
,
2002
,
Introduction to Probability
,
Athena Scientific
,
Belmont, MA
.
53.
Lee
,
J. R.
,
1998
, “
Certainty in Stockpile Computing: Recommending a Verification and Validation Program for Scientific Software
,” Sandia National Laboratories, Report No. SAND98-2420.
54.
Sandia,
1998
, “
Strategic Computing & Simulation Validation & Verification Program: Program Plan
,” Sandia National Laboratories, http://www.sandia.gov/asc/pubs_pres/pubs/vnvprogplan_FY98.html (Last accessed Nov. 26, 2011).
55.
Klein
,
R.
,
Doebling
,
S.
,
Graziani
,
F.
,
Pilch
,
M.
, and
Trucano
,
T. G.
,
2006
, “
ASC Predictive Science Academic Alliance Program Verification and Validation Whitepaper
,” Lawrence Livermore National Laboratories, Los Alamos National Laboratories, Sandia National Laboratories, Report No. UCRL-TR-220711.
56.
Schwitters
,
R.
,
2003
, “
Requirements for ASCI
,” MITRE, FSR-03-330.
57.
Hodges
,
A.
,
Froehlich
,
G.
,
Peercy
,
D.
,
Pilch
,
M.
,
Meza
,
J.
,
Peterson
,
M.
,
LaGrange
,
J.
,
Cox
,
L.
,
Koch
,
K.
,
Storch
,
N.
,
Nitta
,
C.
, and
Dube
,
E.
,
2001
, “
ASCI Software Quality Engineering, Goals, Principles, and Guidelines
,” DOE/DP/ASC-SQE-2000-FDRFR-VERS2.
You do not currently have access to this content.