In this discussion paper, we explore different ways to assess the value of verification and validation (V&V) of engineering models. We first present a literature review on the value of V&V and then use value chains and decision trees to show how value can be assessed from a decision maker's perspective. In this context, the value is what the decision maker is willing to pay for V&V analysis with the understanding that the V&V results are uncertain. The 2014 Sandia V&V Challenge Workshop is used to illustrate these ideas.
Issue Section:
Research Papers
References
1.
ASME V&V 20
, 2009
, Standard for Verification and Validation in Computational Fluids and Heat Transfer
, The American Society of Mechanical Engineers
, New York
.2.
ASME V&V 10-2006
, 2006
, Guide for Verification and Validation in Computational Solid Mechanics
, The American Society of Mechanical Engineers
, New York
.3.
AIAA,
1998
, “Guide for the Verification and Validation of Computational Fluid Dynamics Simulations
,” AIAA
Paper No. G-077-1998.4.
Hu
, K. T.
, Carnes
, B.
, and Romero
, V.
, 2016
, “The 2014 Sandia Verification and Validation Challenge Workshop
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), p. 010202
.5.
Schroeder
, B. B.
, Hu
, K. T.
, Winokur
, J. G.
, and Mullins
, J. G.
, 2016
, “Summary of the 2014 Sandia V&V Challenge Workshop
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), 015501
.6.
Choudhary
, A.
, Voyles
, I. T.
, Roy
, C. J.
, Oberkampf
, W. L.
, and Patil
, M.
, 2016
, “Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), p. 011003
7.
Li
, W.
, Chen
, S.
, Jiang
, Z.
, Apley
, D. W.
, Lu
, Z.
, and Chen
, W.
, 2016
, “Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), 011004
.8.
Mullins
, J.
, and Mahadevan
, S.
, 2016
, “Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), p. 011006
.9.
Beghini
, L. L.
, and Hough
, P. D.
, 2016
, “Sandia V&V Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), 011002
.10.
Xi
, Z.
, and Yang
, R. J.
, 2016
, “Reliability Analysis With Model Uncertainty Coupling With Parameter and Experimental Uncertainties: A Case Study of 2014 V&V Challenge Problem
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), p. 011005
.11.
Paez
, P. J.
, Paez
, T.
, and Hasselman
, T. K.
, 2016
, “Economics Analysis of Model Validation for a Challenge Problem
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), p. 011007
.12.
Youngblood
, S. M.
, “Roadmap for VV&A Technology Advancement
,” 2004
, Foundations'04: A Workshop for V&V in the 21st Century, Defense Modeling and Simulation Office, Arizona State University.13.
Pace
, D.
, 2002
, “Foundations'02 Overview
,” Foundations'02 a Workshop on Model and Simulation Verification and Validation for the 21st Century
, D.
Pace
, ed., JHU/APL, Laurel, MD.14.
Oberkampf
, W. L.
, 1998
, “Bibliography for Verification and Validation in Computational Simulation
,” Sandia National Laboratories, Report No. SAND98-2041.15.
Nitta
, C. K.
, and Logan
, R. W.
, 2004
, “ASCI V&V at LLNL: An Unclassified Bibliography
,” Lawrence Livermore National Laboratory, Report No. UCRL-AR-203864.16.
Jahangirian
, M.
, Taylor
, S. J. E.
, and Young
, T.
, 2010
, “Economics of Modeling and Simulation: Reflections and Implications for Healthcare
,” 2010 Winter Simulation Conference (WSC)
.17.
Kilikauskas
, M. L.
, and Hall
, D. H.
, 2002
, “Estimating V&V Resource Requirements and Schedule Impact
,” Foundations for V&V in the 21st Century Workshop
, S.
Youngblood
, ed., Johns Hopkins University Applied Physics Laboratory
, Laurel, MD
.18.
Back
, G.
, Love
, G.
, and Falk
, J.
, 2000
, “The Doing of Model Verification and Validation: Balancing Cost and Theory
,” 18th International Conference of the System Dynamics Society
, System Dynamics Society
, Bergen, Norway
.19.
Gray
, P.
, 1976
, “The Economics of Simulation
,” 76 Bicentennial Winter Conference on Simulation, Winter Simulation Conference
, Gaithersburg, MD
, pp. 17
–25
.20.
Pace
, D.
, 2004
, “Modeling and Simulation Verification and Validation Challenges
,” Johns Hopkins APL Tech. Dig.
, 25
(2
), pp. 163
–172
.21.
Oberkampf
, W. L.
, Pilch
, M.
, and Trucano
, T. G.
, 2007
, “Predictive Capability Maturity Model for Computational Modeling and Simulation
,” Sandia National Laboratories, Report No. SAND2007-5948.22.
Easterling
, R. G.
, 2001
, “Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations
,” Sandia National Laboratories, Report No. SAND2001-0243.23.
Rizzi
, A.
, and Vos
, J.
, 1998
, “Toward Establishing Credibility in Computational Fluid Dynamics Simulations
,” AIAA J.
, 36
(5
), pp. 668
–675
.24.
Blattnig
, S. R.
, Green
, L.
, Luckring
, J.
, Morrison
, J.
, Tripathi
, R.
, and Zang
, T.
, 2008
, “Towards a Credibility Assessment of Models and Simulations
,” AIAA
Paper No. 2008-2156. 25.
Hemez
, F.
, Atamturktur
, H. S.
, and Unal
, C.
, 2010
, “Defining Predictive Maturity for Validated Numerical Simulations
,” Comput. Struct.
, 88
(7–8
), pp. 497
–505
.26.
Balci
, O.
, and Sargent
, R. G.
, 1981
, “A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models
,” Commun. ACM
, 24
(4
), pp. 190
–197
.27.
Muessig
, P. R.
, Laack
, D. R.
, and Wrobleski
, J. W.
, Jr., 1997
, “Optimizing the Selection of VV&A Activities: A Risk/Benefit Approach
,” 29th Winter Conference on Simulation
, IEEE Computer Society
, Atlanta, GA
, pp. 60
–66
.28.
Youngblood
, S. M.
, Stutzman
, M.
, Pace
, D. K.
, and Pandolfini
, P. P.
, 2011
, “Risk Based Methodology for Verification, Validation, and Accreditation (VV&A), M&S Use Risk Methodology (MURM)
,” The Johns Hopkins University Applied Physics Laboratory, Technical Report No. NSAD-R-2011-011.29.
Elele
, J. N.
, and Smith
, J.
, 2010
, “Risk-Based Verification, Validation, and Accreditation Process
,” Proc. SPIE
7705
.30.
Logan
, R. W.
, Nitta
, C. K.
, and Chidester
, S. K.
, 2005
, “Risk Reduction as the Product of Model Assessed Reliability, Confidence, and Consequence
,” J. Def. Model. Simul.: Appl., Methodol., Technol.
, 2
(4
), pp. 191
–207
.31.
Nitta
, C.
, Logan
, R.
, Chidester
, S.
, and Foltz
, M. F.
, 2004
, “Benefit/Cost Ratio in Systems Engineering: Integrated Models, Tests, Design, and Production
,” Lawrence Livermore National Laboratory, Report No. UCRL-TR-207610.32.
Paez
, P. J.
, Paez
, T. L.
, Hasselman
, T. K.
, and Hu
, K.
, 2015
, “The Economics of Model Validation and Solution of the 2014 Sandia V&V Challenge Problem
,” Sandia National Laboratories, Report No. SAND2015-10560.33.
Waite
, W.
, Lightner
, G.
, Gravitz
, R.
, Severinghaus
, R.
, Waite
, E.
, Swenson
, S.
, Feinberg
, J.
, Cooley
, T.
, Gordon
, S.
, Oswalt
, I.
, 2008
, “Metrics for Modeling and Simulation (M&S) Investments
,” NAVAIR, Report No. TJ-042608-RP013.34.
Oswalt
, I.
, Cooley
, T.
, Waite
, W.
, Waite
, E.
, Gordon
, S.
, Severinghaus
, R.
, Feinberg
, J.
, Lightner
, G.
, 2015
, Calculating Return on Investment for U.S. Department of Defense Modeling and Simulation
, Defense ARJ and Defense AT&L Publications
.35.
Gibson
, R.
, Medeiros
, D. J.
, Sudar
, A.
, Waite
, B.
, and Rohrer
, M. W.
, 2003
, “Increasing Return on Investment From Simulation
,” 2003 Winter Simulation Conference
, S.
Chick
, P. J.
Sánchez
, D.
Ferrin
, and D. J.
Morrice
, eds., pp. 2027
–2032
.36.
Carter
, J. R.
, III, 2001
, “A Business Case for Modeling and Simulation
,” Aviation and Missile Research, Development, and Engineering Center, Special Report No. RD-AS-01-02.37.
Brown
, C. D.
, Grant
, G.
, Kotchman
, D.
, Reyenga
, R.
, and Szanto
, T.
, 2000
, “Building a Business Case for Modeling and Simulation
,” Acquis. Rev. Q.
, pp. 311
–328
.38.
Dabney
, J. B.
, Barber
, G.
, and Ohi
, D.
, 2005
, “Computing Return on Investment of Risk-Reducing Systems Engineering Disciplines
,” Space Systems Engineering and Risk Management Conference
, Los Angeles, CA
.39.
Dabney
, J. B.
, Barber
, G.
, and Ohi
, D.
, 2004
, “Estimating Direct Return on Investment of Independent Verification and Validation
,” Eighth IASTED International Conference
, Cambridge, MA
.40.
Lederer
, P. J.
, and Rhee
, S.-K.
, 1995
, “Economics of Total Quality Management
,” J. Oper. Manage.
, 12
(3–4
), pp. 353
–367
.41.
Abdel-Hamid
, T. K.
, 1988
, “The Economics of Software Quality Assurance: A Simulation-Based Case Study
,” MIS Q.
, 12
(3
), pp. 395
–411
.42.
Shreve
, C. M.
, and Kelman
, I.
, 2014
, “Does Mitigation Save? Reviewing Cost-Benefit Analyses of Disaster Risk Reduction
,” Int. J. Disaster Risk Reduct.
, 10
, pp. 213
–235
.43.
Wethli
, K.
, 2014
, 2016
, “Benefit-Cost Analysis for Risk Management: Summary of Selected Examples
,” World Development Report 2014
, The World Bank
, Washington, DC
.44.
Hu
, K. T.
, and Orient
, G. E.
, 2016
, “The 2014 Sandia V&V Challenge: Problem Statement
,” ASME J. Verif., Validation, Uncertainty Quantif.
, 1
(1
), p. 011001
.45.
Hu
, K. T.
, 2013
, “2014 V&V Challenge: Problem Statement
,” Sandia National Laboratories, Report No. SAND2013-10486P.46.
Porter
, M. E.
, 1985
, Competitive Advantage: Creating and Sustaining Superior Performance
, Simon and Schuster
, New York
.47.
Paté-Cornell
, M. E.
, and Dillon
, R. L.
, 2006
, “The Respective Roles of Risk and Decision Analyses in Decision Support
,” Decis. Anal.
, 3
(4
), pp. 220
–232
.48.
Clemen
, R. T.
, 1996
, Making Hard Decisions: An Introduction to Decision Analysis
, Duxbury Press
, Boston, MA
.49.
Berger
, J. O.
, 1985
, Statistical Decision Theory and Bayesian Analysis
, 2nd ed., Springer-Verlag
, Berlin
.50.
Artzner
, P.
, Delbaen
, F.
, Eber
, J.-M.
, and Heath
, D.
, 1999
, “Coherent Measures of Risk
,” Math. Finance
, 9
(3
), pp. 203
–228
.51.
Adeyemo
, A. M.
, 2013
, “Stochastic Dominance for Project Screening and Selection Under Uncertainty
,” Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA.52.
Bertsekas
, D. P.
, and Tsitsiklis
, J. N.
, 2002
, Introduction to Probability
, Athena Scientific
, Belmont, MA
.53.
Lee
, J. R.
, 1998
, “Certainty in Stockpile Computing: Recommending a Verification and Validation Program for Scientific Software
,” Sandia National Laboratories, Report No. SAND98-2420.54.
Sandia,
1998
, “Strategic Computing & Simulation Validation & Verification Program: Program Plan
,” Sandia National Laboratories, http://www.sandia.gov/asc/pubs_pres/pubs/vnvprogplan_FY98.html (Last accessed Nov. 26, 2011).55.
Klein
, R.
, Doebling
, S.
, Graziani
, F.
, Pilch
, M.
, and Trucano
, T. G.
, 2006
, “ASC Predictive Science Academic Alliance Program Verification and Validation Whitepaper
,” Lawrence Livermore National Laboratories, Los Alamos National Laboratories, Sandia National Laboratories, Report No. UCRL-TR-220711.56.
Schwitters
, R.
, 2003
, “Requirements for ASCI
,” MITRE, FSR-03-330.57.
Hodges
, A.
, Froehlich
, G.
, Peercy
, D.
, Pilch
, M.
, Meza
, J.
, Peterson
, M.
, LaGrange
, J.
, Cox
, L.
, Koch
, K.
, Storch
, N.
, Nitta
, C.
, and Dube
, E.
, 2001
, “ASCI Software Quality Engineering, Goals, Principles, and Guidelines
,” DOE/DP/ASC-SQE-2000-FDRFR-VERS2.Copyright © 2016 by ASME
You do not currently have access to this content.
