A discussion of the five responses to the 2014 Sandia Verification and Validation (V&V) Challenge Problem, presented within this special issue, is provided hereafter. Overviews of the challenge problem workshop, workshop participants, and the problem statement are also included. Brief summations of teams' responses to the challenge problem are provided. Issues that arose throughout the responses that are deemed applicable to the general verification, validation, and uncertainty quantification (VVUQ) community are the main focal point of this paper. The discussion is oriented and organized into big picture comparison of data and model usage, VVUQ activities, and differentiating conceptual themes behind the teams' VVUQ strategies. Significant differences are noted in the teams' approaches toward all VVUQ activities, and those deemed most relevant are discussed. Beyond the specific details of VVUQ implementations, thematic concepts are found to create differences among the approaches; some of the major themes are discussed. Finally, an encapsulation of the key contributions, the lessons learned, and advice for the future are presented.

References

References
1.
Hu
,
K. T.
,
Carnes
,
B.
, and
Romero
,
V.
, “
The 2014 Sandia V&V Challenge Problem Workshop
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
2.
Hu
,
K. T.
,
2013
, “
2014 V&V Challenge: Problem Statement
,” Sandia National Laboratories, Albuquerque, NM, Technical Report No. SAND2013-10486P.
3.
Hu
,
K. T.
, and
Orient
,
G. E.
, “
The 2014 Sandia V&V Challenge Problem Statement
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
4.
Helton
,
J. C.
, and
Oberkampf
,
W. L.
,
2004
, “
Alternative Representations of Epistemic Uncertainty
,”
Reliab. Eng. Syst. Saf.
,
85
, pp.
1
10
.
5.
Hills
,
R. G.
,
Pilch
,
M.
,
Dowding
,
K. J.
,
Red-Horse
,
J.
,
Paez
,
T. L.
,
Babuška
,
I.
, and
Tempone
,
R.
,
2008
, “
Validation Challenge Workshop
,”
Comput. Methods Appl. Mech.
,
197
, pp.
2375
2380
.
6.
Beghini
,
L. L.
, and
Hough
,
P. D.
, “
Sandia V&V Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
7.
Choudhary
,
A.
,
Voyles
, I
. T.
,
Roy
,
C. J.
,
Oberkampf
,
W. L.
, and
Patil
,
M.
, “
Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
8.
Li
,
W.
,
Chen
,
S.
,
Jiang
,
Z.
,
Apley
,
D. W.
,
Lu
,
Z.
, and
Chen
,
W.
, “
Integrating Calibration, Bias Correction, and Machine Learning for the Challenge Problem
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
9.
Xi
,
Z.
, and
Yang
,
R.-J.
, “
Reliability Analysis With Model Uncertainty Coupling With Parameter and Experimental Uncertainties: A Case Study of 2014 V&V Challenge Problem
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
10.
Mullins
,
J.
, and
Mahadevan
,
S.
, “
Bayesian Information Fusion for Model Calibration, Validation, and Prediction
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
11.
Hu
,
K. T.
, and
Paez
,
T. L.
, “
Why Do Verification and Validation?
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
12.
Paez
,
P. J.
,
Paez
,
T. L.
, and
Hasselman
,
T. J.
, “
The Economics of V&V
,”
J. Verif., Validation Uncertainty Quantif.
,
1
(
1
).
13.
Shields
,
M. D.
,
Teferra
,
K.
, and
Kim
,
H.
,
2014
. “
V&V Challenge Problem: An Efficient Monte Carlo Method Incorporating the Effects of Model Error
,”
ASME
Paper No. V&V2014-7214.
14.
Adams
,
B. M.
,
Bauman
,
L. E.
,
Bohnhoff
,
W. J.
,
Dalbey
,
K. R.
,
Eddy
,
J. P.
,
Ebieda
,
M. S.
,
Eldred
,
M. S.
,
Hough
,
P. D.
,
Hu
,
K. T.
,
Jakeman
,
J. D.
,
Swiler
,
L. P.
,
Stephens
,
J. A.
,
Vigil
,
D. M.
, and
Wildey
,
T. M.
,
2014
, “
Dakota—A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.1 User's Manual
,” Sandia National Laboratories, Albuquerque, NM, Technical Report No. SAND2014-4633.
15.
Bichon
,
B. J.
,
Eldred
,
M. S.
,
Swiler
,
L. P.
,
Mahadevan
,
S.
, and
McFarland
,
J. M.
,
2008
, “
Efficient Global Reliability Analysis for Nonlinear Implicit Performance Functions
,”
AIAA J.
,
46
(
10
), pp.
2459
2468
.
16.
Oberkampf
,
W.
,
Pilch
,
M.
, and
Trucano
,
T.
,
2007
, “
Predictive Capability Maturity Model for Computational Modeling and Simulation
,” Sandia National Laboratories, Albuquerque, NM, Technical Report No. SAND2007-5948.
17.
Ferson
,
S.
,
Oberkampf
,
W. L.
, and
Ginzburg
,
L.
,
2008
, “
Model Validation and Predictive Capability for the Thermal Challenge Problem
,”
Comput. Methods Appl. Mech.
,
197
, pp.
2408
2430
.
18.
Voyles
,
I. T.
, and
Roy
,
C. J.
,
2014
, “
Evaluation of Model Validation Techniques in the Presence of Uncertainty
,” AIAA Paper No. 2014-0120.
19.
Voyles
, I
. T.
, and
Roy
,
C. J.
,
2015
, “
Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties
,” AIAA Paper No. 2015-1374.
20.
Kennedy
,
M. C.
, and
O'Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc. B
,
63
(
3
), pp.
425
464
.
21.
Oberkampf
,
W. L.
, and
Trucano
,
T. G.
,
2002
, “
Verification and Validation in Computational Fluid Dynamics
,”
Prog. Aerosp. Sci.
,
38
(
3
), pp.
209
272
.
22.
Ferson
,
S.
,
Joslyn
,
C. A.
,
Helton
,
J. C.
,
Oberkampf
,
W. L.
, and
Sentz
,
K.
,
2004
, “
Summary From the Epistemic Uncertainty Workshop: Consensus Amid Diversity
,”
Reliab. Eng. Syst. Saf.
,
85
, pp.
355
369
.
23.
Knupp
,
P.
, and
Salari
,
K.
,
2002
,
Verification of Computer Codes in Computational Science and Engineering
,
CRC Press
,
Boca Raton, FL
.
24.
Roy
,
C. J.
,
2005
, “
Review of Code and Solution Verification Procedures for Computational Simulation
,”
J. Comput. Phys.
,
205
(
1
), pp.
131
156
.
25.
Saltelli
,
A.
,
Tarantola
,
S.
,
Campolongo
,
F.
, and
Ratto
,
M.
,
2004
,
Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models
,
Wiley
,
Chichester, UK
.
26.
Trucano
,
T. G.
,
Swiler
,
L. P.
,
Igusa
,
T.
,
Oberkampf
,
W. L.
, and
Pilch
,
M.
,
2006
, “
Calibration, Validation, and Sensitivity Analysis: What's What
,”
Reliab. Eng. Syst. Saf.
,
91
, pp.
1331
1357
.
27.
ASME V&V 10 Committee
,
2006
, “
Guide for Verification and Validation in Computational Solid Mechanics
,” The American Society of Mechanical Engineers, New York, Technical Report No. V&V 10-2006.
28.
ASME V&V 20 Committee
,
2009
, “
Standard for Verification and Validation in Computational Fluids and Heat Transfer
,” The American Society of Mechanical Engineers, New York, Technical Report No V&V 20-2009.
29.
Ling
,
Y.
,
Mullins
,
J.
, and
Mahadevan
,
S.
,
2014
, “
Selection of Model Discrepancy Priors in Bayesian Calibration
,”
J. Comput. Phys.
,
276
, pp.
665
680
.
30.
AIAA Standards
,
2002
, “
Guide for the Verification and Validation of Computational Fluid Dynamics Simulations
,”
AIAA
Paper No. G-077-1998.
31.
Hu
,
K. T.
,
2014
, “
The Sandia National Laboratories 2014 Verification & Validation Challenge Workshop
,”
ASME
Paper No. V&V2014-7211.
32.
Sargent
,
R. G.
,
2011
, “
Verification and Validation of Simulation Models
,” Proceedings of the Winter Simulation Conference, Phoenix, AZ, pp.183–198.
You do not currently have access to this content.