Developers of computer codes, analysts who use the codes, and decision makers who rely on the results of the analyses face a critical question: How should confidence in modeling and simulation be critically assessed? Verification and validation (V&V) of computational simulations are the primary methods for building and quantifying this confidence. Briefly, verification is the assessment of the accuracy of the solution to a computational model. Validation is the assessment of the accuracy of a computational simulation by comparison with experimental data. In verification, the relationship of the simulation to the real world is not an issue. In validation, the relationship between computation and the real world, ie, experimental data, is the issue. This paper presents our viewpoint of the state of the art in V&V in computational physics. (In this paper we refer to all fields of computational engineering and physics, eg, computational fluid dynamics, computational solid mechanics, structural dynamics, shock wave physics, computational chemistry, etc, as computational physics.) We describe our view of the framework in which predictive capability relies on V&V, as well as other factors that affect predictive capability. Our opinions about the research needs and management issues in V&V are very practical: What methods and techniques need to be developed and what changes in the views of management need to occur to increase the usefulness, reliability, and impact of computational physics for decision making about engineering systems? We review the state of the art in V&V over a wide range of topics, for example, prioritization of V&V activities using the Phenomena Identification and Ranking Table (PIRT), code verification, software quality assurance (SQA), numerical error estimation, hierarchical experiments for validation, characteristics of validation experiments, the need to perform nondeterministic computational simulations in comparisons with experimental data, and validation metrics. We then provide an extensive discussion of V&V research and implementation issues that we believe must be addressed for V&V to be more effective in improving confidence in computational predictive capability. Some of the research topics addressed are development of improved procedures for the use of the PIRT for prioritizing V&V activities, the method of manufactured solutions for code verification, development and use of hierarchical validation diagrams, and the construction and use of validation metrics incorporating statistical measures. Some of the implementation topics addressed are the needed management initiatives to better align and team computationalists and experimentalists in conducting validation activities, the perspective of commercial software companies, the key role of analysts and decision makers as code customers, obstacles to the improved effectiveness of V&V, effects of cost and schedule constraints on practical applications in industrial settings, and the role of engineering standards committees in documenting best practices for V&V. There are 207 references cited in this review article.
Issue Section:
Review Articles
1.
DoD (1994), DoD directive No 5000.59: Modeling and Simulation (M&S) Management, Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering.
2.
DoD (1996), Verification, Validation, and Accreditation (VV&A) Recommended Practices Guide, Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering.
3.
Cohen ML, Rolph JE, and Steffey DL (eds) (1998), Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements, National Academy Press, Washington DC.
4.
IEEE (1984), IEEE Standard Dictionary of Electrical and Electronics Terms, ANSI/IEEE Std 100-1984, New York.
5.
ANS (1987), Guidelines for the Verification and Validation of Scientific and Engineering Computer Programs for the Nuclear Industry, Am Nuc Soc, ANSI/ANS-10.4-1987, La Grange Park IL.
6.
IEEE (1991), IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990, New York.
7.
ISO (1991), ISO 9000-3: Quality Management and Quality Assurance Standards-Part 3: Guidelines for the Application of ISO 9001 to the Development, Supply, and Maintenance of Software, Int Standards Organization, Geneva, Switzerland.
8.
DoD (1996), DoD Instruction 5000.61: Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering.
9.
AIAA (1998), Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, American Institute of Aeronautics and Astronautics, AIAA, AIAA-G-077-1998, Reston VA.
10.
Oberkampf WL (1994), A proposed framework for computational fluid dynamics code calibration/validation, American Institute of Aeronautics and Astronautics, AIAA Paper No. 94-2540, 18th AIAA Aerospace Ground Testing Conf, Colorado Springs CO.
11.
Oberkampf WL and Trucano TG (2000), Validation methodology in computational fluid dynamics, Am Inst of Aeronaut and Astronaut, AIAA 2000-2549, Fluids 2000 Conf, Denver CO.
12.
Oberkampf
WL
and Trucano
TG
(2002
), Verification and validation in computational fluid dynamics, Prog. Aerosp. Sci.
38
(3
), 209
–272
.13.
Schlesinger
S
(1979
), Terminology for model credibility
, Simulation
32
(3
), 103
–104
.14.
Roache PJ (1998), Verification and Validation in Computational Science and Engineering, Hermosa Publ, Albuquerque NM.
15.
Knepell PL and Arangno DC (1993), Simulation Validation: A Confidence Assessment Methodology, 1st Edition, IEEE Comput Soc Press, Washington DC.
16.
Roache
PJ
(1998
), Verification of codes and calculations
, AIAA J.
36
(5
), 696
–702
.17.
Lin SJ, Barson SL, and Sindir MM (1992), Development of evaluation criteria and a procedure for assessing predictive capability and code performance, Advanced Earth-to-Orbit Propulsion Technology Conf, Marshall Space Flight Center, Huntsville AL.
18.
Marvin
JG
(1995
), Perspective on computational fluid dynamics validation
, AIAA J.
33
(10
), 1778
–1787
.19.
Cosner RR (1995), CFD validation requirements for technology transition, Am Inst of Aeronaut and Astronaut, AIAA Paper No. 95-2227, 26th AIAA Fluid Dynamics Conf, San Diego CA.
20.
Sindir MM, Barson SL, Chan DC, and Lin WH (1996), On the development and demonstration of a code validation process for industrial applications, Am Inst of Aeronaut and Astronaut, AIAA Paper No. 96-2032, 27th AIAA Fluid Dynamics Conf, New Orleans LA.
21.
Sindir MM and Lynch ED (1997), Overview of the state-of-practice of computational fluid dynamics in advanced propulsion system design, Am. Inst. Aeronaut. Astronaut, AIAA Paper No. 97-2124, 28th AIAA Fluid Dynamics Conf, Snowmass CO.
22.
Kleijner
JPC
(1995
), Verification and validation of simulation models
, Eur. J. Oper. Res.
82
, 145
–162
.23.
Balci
O
(1997
), Principles of simulation of model validation, verification, and testing
, Trans. Soc. Comput. Simul. Int.
14
, 3
–12
.24.
Kleindorfer
GB
, O’Neill
L
, and Ganeshan
R
(1998
), Validation in simulation: various positions in the philosophy of science
, Manage. Sci.
44
(8
), 1087
–1099
.25.
Murray-Smith
DJ
(1998
), Methods for the external validation of continuous systems simulation models: A review
, Mathematical and Computer Modelling of Dynamics Systems
4
, 5
–31
.26.
Robinson
S
(1999
), Simulation verification, validation, and confidence: A tutorial
, Trans. Soc. Comput. Simul. Int.
16
, 63
–69
.27.
Zeigler BP, Praehofer H, and Kim TG (2000), Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems, 2nd Edition, Academic Press, San Diego CA.
28.
Casey M and Wintergerste T (eds) (2000), ERCOFTAC Special Interest Group on Quality and Trust in Industrial CFD: Best Practices Guidelines, European Research Community on Flow, Turbulence, and Combustion.
29.
Pilch M, Trucano TG, Moya JL, Froehlich GK, Hodges AL, and Peercy DE (2001), Guidelines for Sandia ASCI Verification and Validation Plans-Content and Format: Version 2, Sandia National Laboratories, SAND2000-3101, Albuquerque NM.
30.
Trucano TG, Pilch M, and Oberkampf WL (2002), General concepts for experimental validation of ASCI code applications, Sandia National Laboratories, SAND2002-0341, Albuquerque NM.
31.
Easterling RG (2001), Measuring the predictive capability of computational models: principles and methods, issues and illustrations, Sandia National Laboratories, SAND2001-0243, Albuquerque NM.
32.
Chiles J-P and Delfiner P (1999), Geostatistics: Modeling Spatial Uncertainty, John Wiley, New York.
33.
Oberkampf WL, Diegert KV, Alvin KF, and Rutherford BM (1998), Variability, uncertainty, and error in computational simulations, ASME, ASME-HTD-Vol 357-2, AIAA/ASME Joint Thermophysics and Heat Transfer Conf, Albuquerque NM.
34.
Oberkampf WL, DeLand SM, Rutherford BM, Diegert KV, and Alvin KF (2000), Estimation of total uncertainty in computational simulation, Sandia National Laboratories, SAND2000-0824, Albuquerque NM.
35.
Oberkampf
WL
, DeLand
SM
, Rutherford
BM
, Diegert
KV
, and Alvin
KF
(2002
), Error and uncertainty in modeling and simulation
, Reliability Eng. Sys. Safety
75
(3
), 333
–357
.36.
Coleman HW and Steele Jr WG (1999), Experimentation and Uncertainty Analysis for Engineers, 2nd Edition, John Wiley, New York.
37.
Hora
SC
(1996
), Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management
, Reliability Eng. Sys. Safety
54
, 217
–223
.38.
Ferson
S
, and Ginzburg
LR
(1996
), Different methods are needed to propagate ignorance and variability
, Reliability Eng. Sys. Safety
54
, 133
–144
.39.
Helton
JC
(1997
), Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty
, J. Stat. Comput. Simul.
57
, 3
–76
.40.
Pate´-Cornell
ME
(1996
), Uncertainties in risk analysis: Six levels of treatment
, Reliability Eng. Sys. Safety
54
, 95
–111
.41.
Cullen AC and Frey HC (1999), Probabilistic Techniques in Exposure Assessment: A Handbook for Dealing with Variability and Uncertainty in Models and Inputs, Plenum Press, New York.
42.
Frank
MV
(1999
), Treatment of uncertainties in space nuclear risk assessment with examples from Cassini mission applications
, Reliability Eng. Sys. Safety
66
, 203
–221
.43.
Smithson M (1989), Ignorance and Uncertainty: Emerging Paradigms, Springer-Verlag, New York.
44.
Almond RG (1995), Graphical Belief Modeling, 1st Edition, Chapman & Hall, London.
45.
Kohlas J and Monney P-A (1995), A Mathematical Theory of Hints-An Approach to the Dempster-Shafer Theory of Evidence, Springer, Berlin.
46.
Klir GJ and Wierman MJ (1998), Uncertainty-based information: elements of generalized information theory, Physica-Verlag, Heidelberg.
47.
Dubois D and Prade H (eds) (2000), Fundamentals of Fuzzy Sets, Kluwer Academic Publ, Boston MA.
48.
Kramosil I (2001), Probabilistic Analysis of Belief Functions, Kluwer, New York.
49.
NRC (1990), Severe accident risks: An assessment for five US nuclear power plants, US Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Div of Systems Research, NUREG-1150, Washington DC.
50.
Modarres M (1993), What Every Engineer Should Know about Reliability and Risk Analysis, Marcel Dekker, New York.
51.
Kafka
P
(1994
), Important issues using PSA technology for design of new systems and plants
, Reliability Eng. Sys. Safety
45
(1–2), 205
–213
.52.
Kumamoto H and Henley EJ (1996), Probabilistic Risk Assessment and Management for Engineers and Scientists, 2nd Edition, IEEE Press, New York.
53.
Helton
JC
, Anderson
DR
, Baker
BL
, Bean
JE
, Berglund
JW
, Beyeler
W
, Economy
K
, Garner
JW
, Hora
SC
, Iuzzolino
HJ
, Knupp
P
, Marietta
MG
, Rath
J
, Rechard
RP
, Roache
PJ
, et al. (1996
), Uncertainty and sensitivity analysis results obtained in the 1992 performance assessment for the Waste Isolation Pilot Plant
, Reliability Eng. Sys. Safety
51
(1
), 53
–100
.54.
Pate´-Cornell
ME
(1999
), Conditional uncertainty analysis and implications for decision making: The case of WIPP
, Risk Anal
19
(5
), 1003
–1016
.55.
Helton
JC
, Anderson
DR
, Basabilvazo
G
, Jow
H-N
, and Marietta
MG
(2000
), Conceptual structure of the 1996 performance assessment for the Waste Isolation Pilot Plant
, Reliability Eng. Sys. Safety
69
(1–3), 151
–165
.56.
Mohanty
S
, Codell
RB
, Ahn
TM
, and Cragnolino
GA
(2000
), An approach to the assessment of high-level radioactive waste containment II: Radionuclide releases from an engineered barrier system
, Nucl. Eng. Des.
201
, 307
–325
.57.
Boyack
BE
, Catton
I
, Duffey
RB
, Griffith
P
, Katsma
KR
, Lellouche
GS
, Levy
S
, Rohatgi
US
, Wilson
GE
, Wulff
W
, and Zuber
N
(1990
), Quantifying reactor safety margins, Part 1: An overview of the code scaling, applicability, and uncertainty evaluation methodology
, Nucl. Eng. Des.
119
, 1
–15
.58.
Wilson
GE
, Boyack
BE
, Catton
I
, Duffey
RB
, Griffith
P
, Katsma
KR
, Lellouche
GS
, Levy
S
, Rohatgi
US
, Wulff
W
, and Zuber
N
(1990
), Quantifying reactor safety margins, Part 2: Characterization of important contributors to uncertainty
, Nucl. Eng. Des.
119
, 17
–31
.59.
Wulff
W
, Boyack
BE
, Catton
I
, Duffey
RB
, Griffith
P
, Katsma
KR
, Lellouche
GS
, Levy
S
, Rohatgi
US
, Wilson
GE
, and Zuber
N
(1990
), Quantifying reactor safety margins, Part 3: Assessment and ranging of parameters
, Nucl. Eng. Des.
119
, 33
–65
.60.
Wilson
GE
, and Boyack
BE
(1998
), The role of the PIRT in experiments, code development, and code applications associated with reactor safety assessment
, Nucl. Eng. Des.
186
, 23
–37
.61.
Zuber
N
, Wilson
GE
, Ishii
M
, Wulff
W
, Boyack
BE
, Dukler
AE
, Griffith
P
, Healzer
JM
, Henry
RE
, Lehner
JR
, Levy
S
, and Moody
FJ
(1998
), An integrated structure and scaling methodology for severe accident technical issue resolution: Development of methodology
, Nucl. Eng. Des.
186
(1–2), 1
–21
.62.
Roache
PJ
(2002
), Code verification by the method of manufactured solutions
, ASME J. Fluids Eng.
114
(1
), 4
–10
.63.
Peercy DE (2000), personal communication.
64.
Hatton
L
(1997
), The T experiments: Errors in scientific software
, IEEE Comput. Sci. Eng.
4
(2
), 27
–38
.65.
Stevenson
DE
(1999
), A critical look at quality in large-scale simulations
, Comput. Sci. Eng.
1
(3
), 53
–63
.66.
Gustafson
J
(1998
), Computational verifiability and feasibility of the ASCI program
, IEEE Comput. Sci. Eng.
5
(1
), 36
–45
.67.
Johnson
DM
(1996
), A review of fault management techniques used in safety-critical avionic systems
, Prog. Aerosp. Sci.
32
(5
), 415
–431
.68.
FAA (1988), System design analysis, Federal Aviation Administration, Advisory Circular 25.1309-1A, Washington DC.
69.
NIST (2002), The economic impacts of inadequate infrastructure for software testing, National Institute of Standards & Technology, Planning Report 02-3, Washington DC.
70.
Shih
TM
(1985
), A procedure to debug computer programs
, Int. J. Numer. Methods Eng.
21
(6
), 1027
–1037
.71.
Salari K and Knupp P (2000), Code verification by the method of manufactured solutions, Sandia National Laboratories, SAND2000-1444, Albuquerque NM.
72.
Steinberg
S
, and Roache
PJ
(1985
), Symbolic manipulation and computational fluid dynamics
, J. Comput. Phys.
57
(2
), 251
–284
.73.
Knupp P and Salari K (2002), Verification of Computer Codes in Computational Science and Engineering, Chapman & Hall/CRC, Boca Raton FL.
74.
Keller
HB
(1969
), Accurate difference methods for linear ordinary differential systems subject to linear constraints
, SIAM (Soc. Ind. Appl. Math.) J. Numer. Anal.
6
, 8
–30
.75.
Srivastava
BN
, Werle
MJ
, and Davis
RT
(1979
), A finite difference technique involving discontinuous derivatives
, Comput. Fluids
7
(1
), 69
–74
.76.
Blottner
FG
(1982
), Influence of boundary approximations and conditions on finite difference solutions
, J. Comput. Phys.
48
(2
), 246
–269
.77.
Turkel
E
(1986
), Accuracy of schemes with nonuniform meshes for compressible fluid-flows
, Appl. Numer. Math.
2
(6
), 529
–550
.78.
Ferziger JH and Peric M (1996), Computational Methods for Fluid Dynamics, Springer-Verlag, New York.
79.
Axelsson O (1996), Iterative Solution Methods, Cambridge Univ Press, Cambridge, UK.
80.
Carpenter
MH
and Casper
JH
(1999
), Accuracy of shock capturing in two spatial dimensions
, AIAA J.
37
(9
), 1072
–1079
.81.
Roy CJ, McWherter-Payne MA, and Oberkampf WL (2000), Verification and validation for laminar hypersonic flowfields, Am Inst of Aeronaut and Astronaut, AIAA2000-2550, Fluids 2000 Conf, Denver, CO.
82.
Botella
O
and Peyret
R
(2001
), Computing singular solutions of the Navier-Stokes equations with the Chebyshev-Collocation method
, Int. J. Numer. Methods Fluids
36
(2
), 125
–163
.83.
Roy
CJ
and Blottner
FB
(2001
), Assessment of one- and two-equation turbulence models for hypersonic flows
, J. Spacecr. Rockets
38
(5
), 699
–710
.84.
Diskin B and Thomas JL (2002), Analysis of boundary conditions for factorizable discretizations of the euler equations, NASA/ICASE, NASA/CR-2002-211648, Hampton VA.
85.
Haworth
DC
, Tahry
SHE
, and Huebler
MS
(1993
), A global approach to error estimation and physical diagnostics in multidimensional computational fluid dynamics
, Int. J. Numer. Methods Fluids
17
(1
), 75
–97
.86.
Singhal AK (1998), Key elements of verification and validation of CFD software, American Institute of Aeronautics and Astronautics, AIAA 98-2639, 29th AIAA Fluid Dynamics Conf, Albuquerque NM.
87.
Wallace DR, Ippolito LM, and Cuthill BB (1996), Reference information for the software verification and validation process, Rept 500-234.
88.
Beizer B (1990), Software Testing Techniques, Van Nostrand Reinhold, New York.
89.
Kaner C, Falk J, and Nguyen HQ (1999), Testing Computer Software, 2nd Edition, John Wiley, New York.
90.
PSI (1994), Purecoverage: User’s guide, Pure Software, Inc., Sunnyvale CA.
91.
NRC (1996), Statistical Software Engineering, Natl Res Council, Natl Academy Press, Washington DC.
92.
Bowen JP and Hinchey MG (1995), Applications of Formal Methods, Prentice-Hall, Englewood Cliffs NJ.
93.
Partsch HA (1990), Specification and Transformation of Programs, Springer-Verlag, New York.
94.
Rushby J (1993), Formal methods and the certification of critical systems, Computer Science Lab, SRI Int, SRI-CSL-93-7, Menlo Park CA.
95.
Bowen
JP
and Hinchey
MG
(1995
), 10-commandments of formal methods
, Computer
28
(4
), 56
–63
.96.
Clark Jr EM, Grumberg O, and Peled D (1999), Model Checking, MIT Press, Cambridge MA.
97.
Kurshan
RP
(2000
), Program verification
, Not. Am. Math. Soc.
, 47
(5
), 534
–545
.98.
Fairley RE (1985), Software Engineering Concepts, McGraw-Hill, New York.
99.
Baber R (1987), The Spine of Software: Designing Provably Correct Software: Theory and Practice, John Wiley, New York.
100.
DeMillo RA, McCracken WM, Martin RJ, and Passafiume JF (1987), Software Testing and Evaluation, Benjamin/Cummings, Menlo Park CA.
101.
Dahl O (1992), Verifiable Programming, Prentice-Hall, Englewood Cliffs NJ.
102.
Dyer M (1992), The Cleanroom Approach to Quality Software Development, John Wiley, New York.
103.
Lewis RO (1992), Independent Verification and Validation, 1st Edition, John Wiley, New York.
104.
Rook P (1990), Software Reliability Handbook, Elsevier Science Publ, New York.
105.
Paulk MC, Weber CV, Curtis B, and Chrissis MB (eds) (1995), The Capability Maturity, Model: Guidelines for Improving the Software Process, Addison-Wesley, Reading, MA.
106.
Jones C (1997), Applied Software Measurement, 2nd Edition, McGraw-Hill, New York.
107.
Persse JR (2001), Implementing the Capability Maturity Model, John Wiley, New York.
108.
Hirsch C (1988), Numerical Computation of Internal and External Flows: Vol 1: Fundamentals of Numerical Discretization, John Wiley, New York.
109.
Hirsch C (1990), Numerical Computation of Internal and External Flows: Vol 2: Computational Methods for Inviscid and Viscous Flows, John Wiley, New York.
110.
Oden JT (1993), Error estimation and control in computational fluid dynamics, The Mathematics of Finite Elements and Applications, JR Whiteman (ed), John Wiley, New York, 1–23.
111.
Morton KW (1996), Numerical Solution of Convection-Diffusion Problems, CRC Press, Boca Raton FL.
112.
Laney CB (1998), Computational Gasdynamics, Cambridge Univ Press, Cambridge, UK.
113.
Ainsworth M and Oden JT (2000), A Posteriori Error Estimation in Finite Element Analysis, John Wiley, New York.
114.
Babuska I and Strouboulis T (2001), The Finite Element Method and its Reliability, Oxford Univ Press, Oxford, UK.
115.
Roache
PJ
(1990
), Need for control of numerical accuracy
, J. Spacecr. Rockets
27
(2
), 98
–102
.116.
Roache
PJ
(1994
), Perspective: A method for uniform reporting of grid refinement studies
, ASME J. Fluids Eng.
116
, 405
–413
.117.
Roache PJ (1997), Quantification of uncertainty in computational fluid dynamics, Annu. Rev. Fluid Mech., 126–160.
118.
Roy CJ (2001), Grid convergence error analysis for mixed-order numerical schemes, Am Inst of Aeronaut and Astronaut, AIAA -2001-2606, AIAA Fluid Dynamics Conf, Anaheim CA.
119.
Eca L and Hoekstra M (2002), An evaluation of verification procedures for CFD applications, Proc of 24th Symp on Naval Hydrodynamics, Fukuoka, Japan.
120.
Cadafalch
J
, Perez-Segarra
CC
, Consul
R
, and Oliva
A
(2002
), Verification of finite volume computations on steady state fluid flow and heat transfer
, ASME J. Fluids Eng.
, 124
(1
), 11
–21
.121.
Chen
C-FA
, Lotz
RD
, and Thompson
BE
(2002
), Assessment of numerical uncertainty around shocks and corners on blunt trailing-edge supercritical airfoils
, Comput. Fluids
31
(1
), 25
–40
.122.
Venditti
DA
, and Darmofal
DL
(2000
), Adjoint error estimation and grid adaptation for functional outputs: Application to quasi-one-dimensional flow
, J. Comput. Phys.
164
(1
), 204
–227
.123.
Jameson
A
, and Martinelli
L
(1998
), Mesh refinement and modeling errors in flow simulation
, AIAA J.
36
(5
), 676
–686
.124.
Barth TJ and Larson MG (2002), A posteriori error estimates for higher order Godunov finite volume methods on unstructured meshes, Natl Aeronaut and Space Administration, NASA-TR-NAS-02-001.
125.
Estep
DJ
, Larson
MG
, and Williams
RD
(2000
), Estimating the error of numerical solutions of systems of reaction-diffusion equations
, Mem. Am. Math. Soc.
146
(696
), 1
–109
.126.
Babuska
I
, Ihlenburg
F
, Strouboulis
T
, and Gangaraj
SK
(1997
), A posteriori error estimation for finite element solutions of Helmholtz’ equation, Part II: Estimation of the pollution error
, Int. J. Numer. Methods Eng.
40
, 3883
–3900
.127.
Babuska
I
and Oh
H-S
(1987
), Pollution problem of the p- and h-p versions of the finite element method
, Commun. Appl. Numer. Methods
3
, 553
–561
.128.
Babuska
I
, Strouboulis
T
, Upadhyay
CS
, and Gangaraj
SK
(1995
), A posteriori estimation and adaptive control of the pollution error in the h-version of the finite element method
, Int. J. Numer. Methods Eng.
38
, 4207
–4235
.129.
Oden
JT
, Feng
Y
, and Prudhomme
S
(1998
), Local and pollution error estimation for Stokesian flow
, Int. J. Numer. Methods Fluids
27
, 33
–39
.130.
Zhang XD, Pelletier D, Trepanier JY, and Camarero R (2000), Verification of error estimators for the euler equations, Am Inst of Aeronaut and Astronau, AIAA-2000-1001, 38th AIAA Aerospace Sciences Meeting, Reno NV.
131.
Chorin AJ, Kast AP, and Kupferman R (1998), On the prediction of large-scale dynamics using unresolved computations, Lawrence Berkeley National Lab, LBNL-42283, Berkeley CA.
132.
Chorin
AJ
, Kast
AP
, and Kupferman
R
(1998
), Optimal prediction of underresolved dynamics
, Proc. Natl. Acad. Sci. U.S.A.
95
, 4094
–4098
.133.
Chorin
AJ
, Kast
AP
, and Kupferman
R
(1999
), Unresolved computation and optimal prediction
, Commun. Pure Appl. Math.
52
, 1231
–1254
.134.
Glimm J, Hou S, Kim H, Sharp DH, and Ye K (1999), A probability model for errors in the numerical solutions of a partial differential equation, Los Alamos Natl Lab, LAUR-99-5352, Los Alamos NM.
135.
Glimm J and Sharp DH (1997), Stochastic methods for the prediction of complex multiscale phenomena, Los Alamos Natl Lab, LAUR-97-3748, Los Alamos NM.
136.
McMillan C (1996), personal communication.
137.
Younger SM (1997), personal communication.
138.
Cox DR (1958), Planning of Experiments, John Wiley, New York.
139.
Dean A and Voss D (1999), Design and Analysis of Experiments, Springer-Verlag, New York.
140.
Gunter
BH
(1993
), How statistical design concepts can improve experimentation in the physical sciences
, Comput. Phys.
7
(3
), 262
–272
.141.
Law AM and Kelton WD (1991), Simulation Modeling and Analysis, 2nd Edition, McGraw-Hill, New York.
142.
Bossel H (1994), Modeling and Simulation, 1st Edition, AK Peters, Wellesley MA.
143.
Haimes YY (1998), Risk Modeling, Assessment, and Management, John Wiley, New York.
144.
Oberkampf
WL
and Aeschliman
DP
(1992
), Joint computational/experimental aerodynamics research on a hypersonic vehicle: Part 1, Experimental results
, AIAA J.
30
(8
), 2000
–2009
.145.
Walker
MA
, and Oberkampf
WL
(1992
), Joint computational/experimental aerodynamics research on a hypersonic vehicle: Part 2, Computational results
, AIAA J.
30
(8
), 2010
–2016
.146.
Oberkampf WL, Aeschliman DP, Tate RE, and Henfling JF (1993), Experimental aerodynamics research on a hypersonic vehicle, Sandia Natl Labs, SAND92-1411, Albuquerque NM.
147.
Aeschliman DP, Oberkampf WL, and Blottner FG (1995), A proposed methodology for CFD code verification, calibration, and validation, ICIASF, Paper 95-CH3482-7, 16th Int Congress on Instrumentation for Aerospace Simulation Facilities, Dayton OH.
148.
Oberkampf WL, Aeschliman DP, Henfling JF, and Larson DE (1995), Surface pressure measurements for CFD code validation in hypersonic flow, Am Inst of Aeronaut and Astronaut, AIAA Paper No 95-2273, 26th AIAA Fluid Dynamics Conf, San Diego CA.
149.
Aeschliman
DP
and Oberkampf
WL
(1998
), Experimental methodology for computational fluid dynamics code validation
, AIAA J.
36
(5
), 733
–741
.150.
Oberkampf
WL
and Blottner
FG
(1998
), Issues in computational fluid dynamics code verification and validation
, AIAA J.
36
(5
), 687
–695
.151.
Gamerman D (1997), Markov Chain Monte Carlo, Chapman & Hall, London.
152.
Kleijnen JPC (1987), Statistical Tools for Simulation Practitioners, 1st Edition, Marcel Dekker, New York.
153.
LeGore T (1990), Predictive software validation methodology for use with experiments having limited replicability, Benchmark Test Cases for Computational Fluid Dynamics, I Celik and CJ Freitas (eds), ASME, New York, 21–27.
154.
Helton
JC
(1993
), Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal
, Reliability Eng. Sys. Safety
42
(2–3), 327
–367
.155.
Helton
JC
(1999
), Uncertainty and sensitivity analysis in performance assessment for the Waste Isolation Pilot Plant
, Comput. Phys. Commun.
117
(1–2), 156
–180
.156.
Stockman
CT
, Garner
JW
, Helton
JC
, Johnson
JD
, Shinta
A
, and Smith
LN
(2000
), Radionuclide transport in the vicinity of the repository and associated complementary cumulative distribution functions in the 1996 performance assessment for the Waste Isolation Pilot Plant
, Reliability Eng. Sys. Safety
69
(1–3), 369
–396
.157.
Melchers RE (1999), Structural Reliability Analysis and Prediction, John Wiley, New York.
158.
Ditlevsen O and Madsen HO (1996), Structural Reliability Methods, John Wiley.
159.
Haldar A and Mahadevan S (2000), Probability, Reliability, and Statistical Methods in Engineering Design, John Wiley, New York.
160.
Hamill TM and Wilks DS (1995), A probabilistic forecast contest and the difficulty in assessing short-range forecast uncertainty, Weather and Forecasting 10(3), 620–631.
161.
Du
J
, Mullen
SL
, and Sanders
F
(1997
), Short-range ensemble forecasting of quantitative precipitation
, Mon. Weather Rev.
125
(10
), 2427
–2459
.162.
Alapaty
K
, Raman
S
, and Niyogi
DS
(1997
), Uncertainty in the specification of surface characteristics: A study of prediction errors in the boundary layer
, Boundary-Layer Meteorol.
82
(3
), 473
–500
.163.
Chlond
A
and Wolkau
A
(2000
), Large-eddy simulation of a nocturnal stratocumulus-topped marine atmospheric boundary layer: An uncertainty analysis
, Boundary-Layer Meteorol.
95
(1
), 31
–55
.164.
Palmer
TN
(2000
), Predicting uncertainty in forecasts of weather and climate
, Rep. Prog. Phys.
63
, 71
–116
.165.
Sanders
F
, Mullen
SL
, and Baumhefner
DP
(2000
), Ensemble simulations of explosive cyclogenesis at ranges of 2–5 days
, Mon. Weather Rev.
128
(8/Pt 2), 2920
–2934
.166.
Glimm
J
and Sharp
D
(1998
), Stochastic methods for the prediction of complex multiscale phenomena
, Q. Appl. Math.
56
(4
), 741
–765
.167.
Earman J (1992), Bayes or Bust?, MIT Press, Cambridge MA.
168.
Gelman AB, Carlin JS, Stern HS, and Rubin DB (1995), Bayesian Data Analysis, Chapman & Hall, London.
169.
French S and Smith JQ (eds) (1997), The Practice of Bayesian Analysis, Hodder Arnold, London.
170.
De Volder
B
, Glimm
J
, Grove
JW
, Kang
Y
, Lee
Y
, Pao
K
, Sharp
DH
, and Ye
K
(2002
), Uncertainty quantification for multiscale simulations
, ASME J. Fluids Eng.
124
(1
), 29
–41
.171.
Gass
SI
(1993
), Model accreditation: A rationale and process for determining a numerical rating
, Eur. J. Oper. Res.
66
, 250
–258
.172.
Lee LH and Poolla K (1994), Statistical Validation for Uncertainty Models, Springer-Verlag, Lecture Notes in Control and Information Sciences, Vol 202, Feedback Control, Complexity, and Identification: A festschrift for Professor George Zames, Montreal, Canada.
173.
Lee
LH
and Poolla
K
(1996
), On statistical model validation
, ASME J. Dyn. Syst., Meas., Control
118
, 226
–236
.174.
Draper
D
(1995
), Assessment and propagation of model uncertainty
, J. R. Stat. Soc. Ser. B. Methodol.
57
(1
), 45
–97
.175.
Kleijnen
JPC
(1995
), Case-study: Statistical validation of simulation models
, Eur. J. Oper. Res.
87
, 21
–34
.176.
Laskey
KB
(1996
), Model uncertainty: Theory and practical implications
, IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans
26
(3
), 340
–348
.177.
Coleman
HW
and Stern
F
(1997
), Uncertainties and CFD code validation
, ASME J. Fluids Eng.
119
, 795
–803
.178.
Hills RG and Trucano TG (1999), Statistical validation of engineering and scientific models: Background, Sandia Natl Labs, SAND99-1256, Albuquerque NM.
179.
Hanson
KM
(1999
), A framework for assessing uncertainties in simulation predictions
, Physica D
133
, 179
–188
.180.
Hills RG and Trucano TG (2002), Statistical validation of engineering and scientific models: A maximum likelihood based metric, Sandia Natl Labs, SAND2001-1783, Albuquerque NM.
181.
Hills RG and Trucano TG (2001), Statistical validation of engineering and scientific models with application to CTH, Sandia Natl Labs, SAND2001-0312, Albuquerque NM.
182.
Trucano TG, Easterling RG, Dowding KJ, Paez TL, Urbina A, Romero VJ, Rutherford RM, and Hills RG (2001), Description of the Sandia validation metrics project, Sandia Natl Labs, SAND2001-1339, Albuquerque NM.
183.
Dowding K (2001), Quantitative validation of mathematical models, ASME Int Mech Eng Congress, New York.
184.
Easterling RG (2001), Quantifying the uncertainty of computational predictions, Sandia Natl Labs, SAND2001-0919C, Albuquerque NM.
185.
Paez T and Urbina A (2001), Validation of structural dynamics models via hypothesis testing, Society of Experimental Mechanics Annual Conf, Portland, OR.
186.
Urbina A and Paez TL (2001), Statistical validation of structural dynamics models, Institute of Environmental Sciences and Technology, Annual Technical Meeting and Exposition of the Inst of Environmental Sciences and Technology, Phoenix AZ.
187.
Paez TL and Urbina A (2002), Validation of mathematical models of complex structural dynamics systems, International Institute of Acoustics and Vibration, Proc of 9th Int Congress on Sound and Vibration, Orlando FL.
188.
Wilson
RV
, Stern
F
, Coleman
HW
, and Paterson
EG
(2001
), Comprehensive approach to verification and validation of CFD simulations, Part 2: Application for RANS simulation of a cargo/container ship
, ASME J. Fluids Eng.
123
(4
), 803
–810
.189.
Stern
F
, Wilson
RV
, Coleman
HW
, and Paterson
EG
(2001
), Comprehensive approach to verification and validation of CFD simulations, Part 1: Methodology and procedures
, ASME J. Fluids Eng.
123
(4
), 793
–802
.190.
Denker M and Woyczynski WA (1998), Introductory Statistics and Random Phenomena, Birkhauser, Boston.
191.
Brandt S (1999), Data Analysis: Statistical and Computational Methods for Scientists and Engineers, 3rd Edition, Springer, New York.
192.
Singpurwalla ND and Wilson SP (1999), Statistical Methods in Software Engineering: Reliability and Risk, Springer-Verlag, Berlin.
193.
Yee
HC
, and Sweby
PK
(1998
), Aspects of numerical uncertainties in time marching to steady-state numerical solutions
, AIAA J.
36
(5
), 712
–724
.194.
Glimm
J
, Hou
SL
, Kim
HJ
, Lee
Y
, Sharp
DH
, Ye
K
, and Zou
WS
(2001
), Risk management for petroleum reservoir production: A simulation-based study of prediction
, Comput. Geosci.
5
(3
), 173
–197
.195.
Hills RG and Leslie IH (2003), Statistical validation of engineering and scientific models: Validation experiments to application, Sandia Natl Labs, SAND2003-0706, Albuquerque NM.
196.
Zhang
R
, and Mahadevan
S
(2003
), Bayesian methodology for reliability model acceptance
, Reliability Eng. Sys. Safety
80
(1
), 95
–103
.197.
Campbell K (2002), A brief survey of statistical model calibration ideas, Los Alamos Natl Lab, LA-UR-02-3157, Los Alamos NM.
198.
Hasselman
TK
(2001
), Quantification of uncertainty in structural dynamic models
, ASCE, J. Aerosp. Eng.
14
(4
), 158
–165
.199.
Hasselman TK, Wathugala GW, and Crawford J (2002), A hierarchical approach for model validation and uncertainty quantification, Vienna Univ of Technology, Austria, http://wccm.tuwien.ac.at, Fifth World Congress on Computational Mechanics, Vienna, Austria.
200.
Logan RW and Nitta CK (2002), Verification & validation (V&V) methodology and quantitative reliability at confidence (QRC): Basis for an investment strategy, Lawrence Livermore Natl Lab, UCRL-ID-150874, Livermore CA.
201.
Denton JD (1996), Lessons from Rotor 37, Proc of 3rd Int Symp on Experimental and Computational Aerothermodynamics of Internal Flows, Beijing, China.
202.
Hutton AG and Casey MV (2001), Quality and trust in industrial CFD—A European perspective, Am Inst of Aeronau and Astronaut, AIAA2001-0656, 39th AIAA Aerospace Sciences Meeting, Reno NV.
203.
Levy DW, Zickuhr T, Vassberg J, Agrawal S, Wahls RW, Pirzadeh S, and Hemsch MJ (2002), Summary of data from the 1st AIAA CFD Drag Prediction Workshop, Am Inst of Aeronaut and Astronaut, AIAA-2002-0841, 40th AIAA Aerospace Sciences Meeting and Exhibit, Reno NV.
204.
Hemsch M (2002), Statistical analysis of CFD solutions from the Drag Prediction Workshop, Am Inst of Aeronaut and Astronaut, AIAA-2002-0842 40th AIAA Aerospace Sciences Meeting and Exhibit, Reno NV.
205.
ERCOFTAC (2000), Portal to Fluid Dynamics Database Resources, European Research Community on Flow, Turbulence, and Combustion, http://ercoftac.mech.surrey.ac.uk.
206.
NAFEMS (2000), CFD Working Group, International Association for the Engineering Analysis Community, www.NAFEMS.org.
207.
QNET-CFD (2001), Thematic Network on Quality and Trust for the Industrial Applications of CFD, www.qnet-cfd.net.
Copyright © 2004
by ASME
You do not currently have access to this content.
