This paper proposes a comprehensive approach to prediction under uncertainty by application to the Sandia National Laboratories verification and validation challenge problem. In this problem, legacy data and experimental measurements of different levels of fidelity and complexity (e.g., coupon tests, material and fluid characterizations, and full system tests/measurements) compose a hierarchy of information where fewer observations are available at higher levels of system complexity. This paper applies a Bayesian methodology in order to incorporate information at different levels of the hierarchy and include the impact of sparse data in the prediction uncertainty for the system of interest. Since separation of aleatory and epistemic uncertainty sources is a pervasive issue in calibration and validation, maintaining this separation in order to perform these activities correctly is the primary focus of this paper. Toward this goal, a Johnson distribution family approach to calibration is proposed in order to enable epistemic and aleatory uncertainty to be separated in the posterior parameter distributions. The model reliability metric approach to validation is then applied, and a novel method of handling combined aleatory and epistemic uncertainty is introduced. The quality of the validation assessment is used to modify the parameter uncertainty and add conservatism to the prediction of interest. Finally, this prediction with its associated uncertainty is used to assess system-level reliability (a prediction goal for the challenge problem).

References

References
1.
Roy
,
C.
, and
Oberkampf
,
W.
,
2011
, “
A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing
,”
Comput. Methods Appl. Mech. Eng.
,
200
(
25–28
), pp.
2131
2144
.
2.
Romero
,
V.
,
Luketa
,
A.
, and
Sherman
,
M.
,
2010
, “
Application of a Versatile ‘Real-Space’ Validation Methodology to a Fire Model
,”
J. Thermophys. Heat Transfer
,
24
(
4
), pp.
730
744
.
3.
Hills
,
R. G.
, and
Leslie
,
I. H.
,
2003
, “
Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application
,”
Sandia Technical Report No. SAND2003-0706
.
4.
Trucano
,
T.
,
Swiler
,
L.
,
Igusa
,
T.
,
Oberkampf
,
W.
, and
Pilch
,
M.
,
2006
, “
Calibration, Validation, and Sensitivity Analysis: What's What
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1331
1357
.
5.
Higdon
,
D.
,
Kennedy
,
M.
,
Cavendish
,
J.
,
Cafeo
,
J.
, and
Ryne
,
R.
,
2004
, “
Combining Field Data and Computer Simulations for Calibration and Prediction
,”
SIAM J. Sci. Comput.
,
26
(
2
), pp.
448
466
.
6.
Sankararaman
,
S.
, and
Mahadevan
,
S.
,
2012
, “
Comprehensive Framework for Integration of Calibration, Verification and Validation
,”
AIAA
Paper No. 2012-1366.
7.
Arendt
,
P. D.
,
Apley
,
D. W.
, and
Chen
,
W.
,
2012
, “
Quantification of Model Uncertainty: Calibration, Model Discrepancy, and Identifiability
,”
ASME J. Mech. Des.
,
134
(
10
), p.
100908
.
8.
Ling
,
Y.
,
Mullins
,
J.
, and
Mahadevan
,
S.
,
2014
, “
Selection of Model Discrepancy Priors in Bayesian Calibration
,”
J. Comput. Phys.
,
276
, pp.
665
680
.
9.
Hartmann
,
C.
,
Smeyers-Verbeke
,
J.
,
Penninckx
,
W.
,
Heyden
,
Y. V.
,
Vankeerberghen
,
P.
, and
Massart
,
D.
,
1995
, “
Reappraisal of Hypothesis Testing for Method Validation: Detection of Systematic Error by Comparing the Means of Two Methods or of Two Laboratories
,”
Anal. Chem.
,
67
(
24
), pp.
4491
4499
.
10.
Hills
,
R. G.
, and
Trucano
,
T. G.
,
1999
, “
Statistical Validation of Engineering and Scientific Models: Background
,”
Sandia Technical Report No. SAND99-1256
.
11.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2006
, “
Validation and Error Estimation of Computational Models
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1390
1397
.
12.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2006
, “
Validation of Models With Multivariate Output
,”
Reliab. Eng. Syst. Saf.
,
91
(
8
), pp.
861
871
.
13.
O'Hagan
,
A.
,
1995
, “
Fractional Bayes Factors for Model Comparison
,”
J. R. Stat. Soc., Ser. B (Methodological)
,
57
(
1
), pp.
99
138
.
14.
Wang
,
S.
,
Chen
,
W.
, and
Tsui
,
K.-L.
,
2009
, “
Bayesian Validation of Computer Models
,”
Technometrics
,
51
(
4
), pp.
439
451
.
15.
Ferson
,
S.
,
Oberkampf
,
W.
, and
Ginzburg
,
L.
,
2008
, “
Model Validation and Predictive Capability for the Thermal Challenge Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2408
2430
.
16.
Ferson
,
S.
, and
Oberkampf
,
W.
,
2009
, “
Validation of Imprecise Probability Models
,”
Int. J. Reliab. Saf.
,
3
(
1
), pp.
3
22
.
17.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2008
, “
Computational Methods for Model Reliability Assessment
,”
Reliab. Eng. Syst. Saf.
,
93
(
8
), pp.
1197
1207
.
18.
Sankararaman
,
S.
, and
Mahadevan
,
S.
,
2013
, “
Assessing the Reliability of Computational Models Under Uncertainty
,”
AIAA
Paper No. 2013-1873.
19.
O'Hagan
,
A.
, and
Oakley
,
J. E.
,
2004
, “
Probability is Perfect, but We Can't Elicit It Perfectly
,”
Reliab. Eng. Syst. Saf.
,
85
(
1–3
), pp.
239
248
.
20.
Jaulin
,
L.
,
Kieffer
,
M.
,
Didrit
,
O.
, and
Walter
,
E.
,
2001
,
Applied Interval Analysis
,
Springer-Verlag
,
New York
.
21.
Shafer
,
G.
,
1976
,
A Mathematical Theory of Evidence
,
Princeton University Press
,
Princeton, NJ
.
22.
Dubois
,
D.
, and
Prade
,
H.
,
1986
,
Possibility Theory: An Approach to Computerized Processing of Uncertainty
,
Plenum Press
,
New York
.
23.
Ross
,
T. J.
,
1995
,
Fuzzy Logic With Engineering Applications
,
McGraw-Hill
,
New York
.
24.
Klir
,
G. J.
, and
Wierman
,
M. J.
,
1998
,
Uncertainty-Based Information: Elements of Generalized Information Theory
,
2nd ed.
, Vol.
15
,
Physica-Verlag
,
Heidelberg, DE
.
25.
Helton
,
J.
, and
Sallaberry
,
C.
,
2012
, “
Uncertainty and Sensitivity Analysis: From Regulatory Requirements to Conceptual Structure and Computational Implementation
,”
Uncertainty Quantification in Scientific Computing, IFIP Advances in Information and Communication Technology
, Vol.
377
,
Springer
,
Berlin
, pp.
60
77
.
26.
Oberkampf
,
W. L.
,
Helton
,
J. C.
,
Joslyn
,
C. A.
,
Wojtkiewicz
,
S. F.
, and
Ferson
,
S.
,
2004
, “
Challenge Problems: Uncertainty in System Response Given Uncertain Parameters
,”
Reliab. Eng. Syst. Saf.
,
85
(
1–3
), pp.
11
19
.
27.
Kiureghian
,
A.
,
2009
, “
Aleatory or Epistemic? Does It Matter?
Struct. Saf.
,
31
(
2
), pp.
105
112
.
28.
Hu
,
K.
,
2014
, “
2014 V&V Challenge: Problem Statement
,” Sandia Technical Report No. SAND2013-10486P.
29.
Kennedy
,
M. C.
, and
O'Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc., Ser. B (Stat. Methodol.)
,
63
(
5
), pp.
425
464
.
30.
Metropolis
,
N.
,
Rosenbluth
,
A.
,
Rosenbluth
,
M.
,
Teller
,
A.
, and
Teller
,
E.
,
1953
, “
Equation of State Calculations by Fast Computing Machines
,”
J. Chem. Phys.
,
21
(
6
), p.
1087
.
31.
Hastings
,
W.
,
1970
, “
Monte Carlo Sampling Methods Using Markov Chains and Their Applications
,”
Biometrika
,
57
(
1
), pp.
97
109
.
32.
Gilks
,
W.
, and
Wild
,
P.
,
1992
, “
Adaptive Rejection Sampling for Gibbs Sampling
,”
J. R. Stat. Soc., Ser. C (Appl. Stat.)
,
41
(
2
), pp.
337
348
.
33.
Neal
,
R.
,
2003
, “
Slice Sampling
,”
Ann. Stat.
,
31
(
3
), pp.
705
741
.
34.
Cressie
,
N. A. C.
,
1993
,
Statistics for Spatial Data
,
Revised edition
,
Wiley
,
New York
.
35.
Sacks
,
J.
,
Schiller
,
S. B.
, and
Welch
,
W.
,
1989
, “
Design of Computer Experiments
,”
Technometrics
,
31
(
1
), pp.
41
47
.
36.
Xiu
,
D.
,
2010
,
Numerical Methods for Stochastic Computations: A Spectral Method Approach
,
Princeton University Press
,
Princeton, NJ
.
37.
Press
,
W. H.
,
Teukolsky
,
S. A.
,
Vetterling
,
W. T.
, and
Flanner
,
B. P.
,
2007
,
Numerical Recipes: The Art of Scientific Computing
,
3rd ed.
,
Cambridge University Press
,
New York
.
38.
McCulloch
,
W.
, and
Pitts
,
W.
,
1943
, “
A Logical Calculus of Ideas Immanent in Nervous Activity
,”
Bull. Math. Biophys.
,
5
(
4
), pp.
115
133
.
39.
Rasmussen
,
C. E.
, and
Williams
,
C. K. I.
,
2006
,
Gaussian Processes for Machine Learning
,
The MIT Press
,
Cambridge, MA
.
40.
Johnson
,
N. L.
,
1949
, “
Systems of Frequency Curves Generated by Methods of Translation
,”
Biometrika
,
36
(
1/2
), pp.
149
176
.
41.
DeBrota
,
D. J.
,
Roberts
,
S. D.
,
Dittus
,
R. S.
,
Wilson
,
J. R.
,
Swain
,
J. J.
, and
Venkatraman
,
S.
,
1988
, “
Input Modeling With the Johnson System of Distributions
,”
Winter Simulations Conference
(WSC '88)
,
M.
Abrams
,
P.
Haigh
, and
J.
Comfort
, eds., pp.
165
179
.
42.
Marhadi
,
K.
,
Venkataraman
,
S.
, and
Pai
,
S. S.
,
2012
, “
Quantifying Uncertainty in Statistical Distribution of Small Sample Data Using Bayesian Inference of Unbounded Johnson Distribution
,”
Int. J. Reliab. Saf.
,
6
(
4
), pp.
311
337
.
43.
Sankararaman
,
S.
, and
Mahadevan
,
S.
,
2013
, “
Separating the Contributions of Variability and Parameter Uncertainty in Probability Distributions
,”
Reliab. Eng. Syst. Saf.
,
112
, pp.
187
199
.
44.
Ellingwood
,
B.
,
Galambos
,
T.
,
MacGregor
,
J.
, and
Cornell
,
C. A.
,
1980
,
Development of a Probability Based Load Criterion for American National Standard A58: Building Code Requirements for Minimum Design Loads in Buildings and Other Structures
, Vol.
577
,
National Bureau of Standards Publication
, Gaithersburg, MD.
You do not currently have access to this content.