Abstract

Computer model calibration typically operates by fine-tuning parameter values in a computer model so that the model output faithfully predicts reality. By using performance targets in place of observed data, we show that calibration techniques can be repurposed for solving multi-objective design problems. Our approach allows us to consider all relevant sources of uncertainty as an integral part of the design process. We demonstrate our proposed approach through both simulation and fine-tuning material design settings to meet performance targets for a wind turbine blade.

References

References
1.
Kennedy
,
M. C.
, and
O’Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc.: Ser. B
,
63
(
3
), pp.
425
464
. 10.1111/1467-9868.00294
2.
Higdon
,
D.
,
Kennedy
,
M.
,
Cavendish
,
J. C.
,
Cafeo
,
J. A.
, and
Ryne
,
R. D.
,
2004
, “
Combining Field Data and Computer Simulations for Calibration and Prediction
,”
SIAM J. Sci. Comput.
,
26
(
2
), pp.
448
466
. 10.1137/S1064827503426693
3.
Williams
,
B.
,
Higdon
,
D.
,
Gattiker
,
J.
,
Moore
,
L.
,
McKay
,
M.
, and
Keller-McNulty
,
S.
,
2006
, “
Combining Experimental Data and Computer Simulations, With An Application to Flyer Plate Experiments
,”
Bayesian Anal.
,
1
(
4
), pp.
765
792
. 10.1214/06-BA125
4.
Loeppky
,
J. L.
,
Bingham
,
D.
, and
Welch
,
W. J.
,
2006
,
Computer Model Calibration or Tuning in Practice
,
Department of Statistics, University of British Columbia, Vancouver
.
5.
Bayarri
,
M. J.
,
Berger
,
J. O.
,
Paulo
,
R.
,
Sacks
,
J.
,
Cafeo
,
J. A.
,
Cavendish
,
J.
,
Lin
,
C.-H.
, and
Tu
,
J.
,
2007
, “
A Framework for Validation of Computer Models
,”
Technometrics
,
49
(
2
), pp.
138
154
. 10.1198/004017007000000092
6.
Bayarri
,
M. J.
,
Berger
,
J. O.
,
Cafeo
,
J.
,
Garcia-Donato
,
G.
,
Liu
,
F.
,
Palomo
,
J.
,
Parthasarathy
,
R. J.
,
Paulo
,
R.
,
Sacks
,
J.
, and
Walsh
,
D.
,
2007
, “
Computer Model Validation With Functional Output
,”
Ann. Stat.
,
35
(
5
), pp.
1874
1906
. 10.1214/009053607000000163
7.
Paulo
,
R.
,
García-Donato
,
G.
, and
Palomo
,
J.
,
2012
, “
Calibration of Computer Models With Multivariate Output
,”
Comput. Stat. Data Anal.
,
56
(
12
), pp.
3959
3974
. 10.1016/j.csda.2012.05.023
8.
Brynjarsdóttir
,
J.
, and
O’Hagan
,
A.
,
2014
, “
Learning About Physical Parameters: The Importance of Model Discrepancy
,”
Inverse Prob.
,
30
(
11
), p.
114007
. 10.1088/0266-5611/30/11/114007
9.
Peitz
,
S.
, and
Dellnitz
,
M.
,
2018
,
Gradient-Based Multiobjective Optimization With Uncertainties
,
Springer International Publishing
,
Cham
, pp.
159
182
.
10.
Vasilopoulos
,
I.
,
Asouti
,
V. G.
,
Giannakoglou
,
K. C.
, and
Meyer
,
M.
,
2019
, “
Gradient-Based Pareto Front Approximation Applied to Turbomachinery Shape Optimization
,”
Eng. Comput.
, pp.
1
11
. http://dx.doi.org/10.1007/s00366-019-00832-y
11.
Jin
,
Y.
, and
Branke
,
J.
,
2005
, “
Evolutionary Optimization in Uncertain Environments – A Survey
,”
IEEE Trans. Evol. Comput.
,
9
(
3
), pp.
303
317
. 10.1109/TEVC.2005.846356
12.
Deb
,
K.
, and
Gupta
,
H.
,
2006
, “
Introducing Robustness in Multi-Objective Optimization
,”
Evol. Comput.
,
14
(
4
), pp.
463
494
. 10.1162/evco.2006.14.4.463
13.
Zhou
,
A.
,
Qu
,
B.-Y.
,
Li
,
H.
,
Zhao
,
S.-Z.
,
Suganthan
,
P. N.
, and
Zhang
,
Q.
,
2011
, “
Multiobjective Evolutionary Algorithms: A Survey of the State of the Art
,”
Swarm Evol. Comput.
,
1
(
1
), pp.
32
49
. 10.1016/j.swevo.2011.03.001
14.
Picheny
,
V.
,
Binois
,
M.
, and
Habbal
,
A.
,
2019
, “
A Bayesian Optimization Approach to Find Nash Equilibria
,”
J. Global Optim.
,
73
(
1
), pp.
171
192
. 10.1007/s10898-018-0688-0
15.
Jones
,
D. R.
,
Schonlau
,
M.
, and
Welch
,
W. J.
,
1998
, “
Efficient Global Optimization of Expensive Black-Box Functions
,”
J. Global Optim.
,
13
(
4
), pp.
455
492
. 10.1023/A:1008306431147
16.
Chevalier
,
C.
,
Bect
,
J.
,
Ginsbourger
,
D.
,
Vazquez
,
E.
,
Picheny
,
V.
, and
Richet
,
Y.
,
2014
, “
Fast Parallel Kriging-Based Stepwise Uncertainty Reduction With Application to the Identification of An Excursion Set
,”
Technometrics
,
56
(
4
), pp.
455
465
. 10.1080/00401706.2013.860918
17.
Picheny
,
V.
,
2015
, “
Multiobjective Optimization Using Gaussian Process Emulators Via Stepwise Uncertainty Reduction
,”
Stat. Comput.
,
25
(
6
), pp.
1265
1280
. 10.1007/s11222-014-9477-x
18.
Tuo
,
R.
, and
Wang
,
W.
,
2020
,
Uncertainty quantification for Bayesian Optimization
.
preprint arXiv:2002.01569
.
19.
Pandita
,
P.
,
Bilionis
,
I.
,
Panchal
,
J.
,
Gautham
,
B. P.
,
Joshi
,
A.
, and
Zagade
,
P.
,
2018
, “
Stochastic Multiobjective Optimization on a Budget: Application to Multipass Wire Drawing With Quantified Uncertainties
,”
Int. J. Uncertainty Quantif.
,
8
(
3
), pp.
233
249
. 10.1615/Int.J.UncertaintyQuantification.2018021315
20.
Olalotiti-Lawal
,
F.
, and
Datta-Gupta
,
A.
,
2015
, “
A Multi-Objective Markov Chain Monte Carlo Approach for History Matching and Uncertainty Quantification
,”
J. Petr. Sci. Eng.
,
166
, pp.
759
777
. https://doi.org/10.1016/j.petrol.2018.03.062
21.
Gelfand
,
A. E.
, and
Smith
,
A. F. M.
,
1990
, “
Sampling-Based Approaches to Calculating Marginal Densities
,”
J. Am. Stat. Assoc.
,
85
(
410
), pp.
398
409
. 10.1080/01621459.1990.10476213
22.
Miettinen
,
K.
,
2008
,
Introduction to Multiobjective Optimization: Noninteractive Approaches
,
Springer
,
Berlin, Heidelberg
, pp.
1
23
.
23.
Chen
,
W.
,
Xiong
,
Y.
,
Tsui
,
K. L.
, and
Wang
,
S.
,
2008
, “
A Design-Driven Validation Approach Using Bayesian Prediction Models
,”
Trans. ASME: J. Mech. Des.
,
130
(
2
), p.
021101
. 10.1115/1.2809439
24.
Drignei
,
D.
,
Mourelatos
,
Z. P.
,
Pandey
,
V.
, and
Kokkolaras
,
M.
,
2012
, “
Concurrent Design Optimization and Calibration-Based Validation Using Local Domains Sized by Bootstrapping
,”
ASME J. Mech. Des.
,
134
(
10
), p.
100910
. 10.1115/1.4007572
25.
Xi
,
Z.
,
2019
, “
Model-Based Reliability Analysis With Both Model Uncertainty and Parameter Uncertainty
,”
ASME J. Mech. Des.
,
141
(
5
), p.
051404
. 10.1115/1.4041946
26.
Rubin
,
D. B.
,
1974
, “
Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies
,”
J. Educ. Psychol.
,
66
(
5
), pp.
688
701
. 10.1037/h0037350
27.
Adams
,
R. M.
,
1974
, “
Theories of Actuality
,”
Noûs
,
8
(
3
), p.
211
.
28.
Lewis
,
D. K.
,
1986
, “On the Plurality of Worlds,”
Central Works of Philosophy
, Vol.
5
,
J.
Shand
, ed.,
McGill-Queen's Press
,
Montreal, Canada
.
29.
Jiang
,
W.
, and
Tanner
,
M. A.
,
2008
, “
Gibbs Posterior for Variable Selection in High-Dimensional Classification and Data Mining
,”
Ann. Stat.
,
36
(
5
), pp.
2207
2231
. 10.1214/07-AOS547
30.
Hemez
,
F.
, and
Atamturktur
,
S.
,
2011
, “
The Dangers of Sparse Sampling for the Quantification of Margin and Uncertainty
,”
Reliab. Eng. Syst. Saf.
,
96
(
9
), pp.
1220
1231
. 10.1016/j.ress.2011.02.015
31.
Van Buren
,
K. L.
,
Mollineaux
,
M. G.
,
Hemez
,
F. M.
, and
Atamturktur
,
S.
,
2013
, “
Simulating the Dynamics of Wind Turbine Blades: Part II, Model Validation and Uncertainty Quantification
,”
Wind Energy
,
16
(
5
), pp.
741
758
. 10.1002/we.1522
32.
Van Buren
,
K. L.
,
Atamturktur
,
S.
, and
Hemez
,
F. M.
,
2014
, “
Model Selection Through Robustness and Fidelity Criteria: Modeling the Dynamics of the CX-100 Wind Turbine Blade
,”
Mech. Syst. Sig. Process.
,
43
(
1–2
), pp.
246
259
. 10.1016/j.ymssp.2013.10.010
33.
Sacks
,
J.
,
Welch
,
W. J.
,
Mitchell
,
T. J.
, and
Wynn
,
H. P.
,
1989
, “
Design and Analysis of Computer Experiments
,”
Stat. Sci.
,
4
(
4
), pp.
409
423
. 10.1214/ss/1177012413
34.
Santner
,
T. J.
,
Williams
,
B. J.
, and
Notz
,
W. I.
,
2003
,
The Design and Analysis of Computer Experiments
,
Springer
,
New York
.
35.
Pratola
,
M.
, and
Chkrebtii
,
O.
,
2018
, “
Bayesian Calibration of Multistate Stochastic Simulators
,”
Stat. Sin.
,
28
, pp.
693
719
.
36.
O’Hagan
,
A.
,
1978
, “
Curve Fitting and Optimal Design for Prediction
,”
J. R. Stat. Soc.: Ser. B
,
40
(
1
), pp.
1
42
.
37.
Kennedy
,
M. C.
,
Anderson
,
C. W.
,
Conti
,
S.
, and
O’Hagan
,
A.
,
2006
, “
Case Studies in Gaussian Process Modelling of Computer Codes
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1301
1309
. 10.1016/j.ress.2005.11.028
38.
Bastos
,
L. S.
, and
O’Hagan
,
A.
,
2009
, “
Diagnostics for Gaussian Process Emulators
,”
Technometrics
,
51
(
4
), pp.
425
438
. 10.1198/TECH.2009.08019
39.
Gramacy
,
R. B.
, and
Lee
,
H. K. H.
,
2008
, “
Bayesian Treed Gaussian Process Models With An Application to Computer Modeling
,”
J. Am. Stat. Assoc.
,
103
(
483
), pp.
1119
1130
. 10.1198/016214508000000689
40.
Qian
,
P. Z. G.
,
Wu
,
H.
, and
Wu
,
C. F. J.
,
2008
, “
Gaussian Process Models for Computer Experiments With Qualitative and Quantitative Factors
,”
Technometrics
,
50
(
3
), pp.
383
396
. 10.1198/004017008000000262
41.
Tuo
,
R.
, and
Wu
,
C. F. J.
,
2016
, “
A Theoretical Framework for Calibration in Computer Models: Parametrization, Estimation and Convergence Properties
,”
SIAM/ASA J. Uncertainty Quantif.
,
4
(
1
), pp.
767
795
. 10.1137/151005841
42.
Liu
,
F.
,
Bayarri
,
M. J.
, and
Berger
,
J. O.
,
2009
, “
Modularization in Bayesian Analysis, With Emphasis on Analysis of Computer Models
,”
Bayesian Anal.
,
4
(
1
), pp.
119
150
. 10.1214/09-BA404
43.
Deb
,
K.
, and
Sundar
,
J.
,
2006
, “
Reference Point Based Multi-Objective Optimization Using Evolutionary Algorithms
,”
Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation – GECCO ’06
,
Seattle, WA
,
July
, p.
635
.
44.
ANSYS, Inc.
,
2017
,
Ansys® Academic Research Mechanical, Release 18.1
.
45.
Matlab
,
2017
,
Version 9.2.0 (R2017a)
,
The MathWorks, Inc.
,
Natick, Massachusetts
.
46.
Berg
,
J. C.
, and
Resor
,
B. R.
,
2012
, “
Numerical Manufacturing and Design Tool (NuMAD v2.0) for Wind Turbine Blades: User’s Guide
,”
Sandia National Laboratories Report SAND2012-7028
.
47.
Mori
,
T.
, and
Tanaka
,
K.
,
1973
, “
Average Stress in Matrix and Average Elastic Energy of Materials With Misfitting Inclusions
,”
Acta Metall.
,
21
(
5
), pp.
571
574
. 10.1016/0001-6160(73)90064-3
48.
McKay
,
M. D.
,
Beckman
,
R. J.
, and
Conover
,
W. J.
,
1979
, “
Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code
,”
Technometrics
,
21
(
2
), pp.
239
245
.
49.
Metropolis
,
N.
,
Rosenbluth
,
A. W.
,
Rosenbluth
,
M. N.
,
Teller
,
A. H.
, and
Teller
,
E.
,
1953
, “
Equation of State Calculations by Fast Computing Machines
,”
J. Chem. Phys.
,
21
(
6
), pp.
1087
1092
. 10.1063/1.1699114
50.
Hastings
,
W.
,
1970
, “
Monte Carlo Sampling Methods Using Markov Chains and Their Applications
,”
Biometrika
,
57
(
1
), pp.
97
109
. 10.1093/biomet/57.1.97
51.
Geman
,
S.
, and
Geman
,
D.
,
1984
, “
Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
6
(
6
), pp.
721
741
. 10.1109/TPAMI.1984.4767596
52.
Roberts
,
G.
,
Gelman
,
A.
, and
Gilks
,
W.
,
1997
, “
Weak Convergence and Optimal Scaling of Random Walk Metropolis Algorithms
,”
Ann. Appl. Prob.
,
7
(
1
), p.
120
.
53.
Gelman
,
A.
,
Carlin
,
J. B.
,
Stern
,
H. S.
,
Dunson
,
D. B.
,
Vehtari
,
A.
, and
Rubin
,
D. B.
,
2013
,
Bayesian Data Analysis
, 3rd ed.,
CRC Press
,
London
.
54.
Gelman
,
A.
, and
Rubin
,
D. B.
,
1992
, “
Inference From Iterative Simulation Using Multiple Sequences
,”
Stat. Sci.
,
7
(
4
), pp.
457
472
. 10.1214/ss/1177011136
55.
Deb
,
K.
,
Pratap
,
A.
,
Agarwal
,
S.
, and
Meyarivan
,
T.
,
2002
, “
A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II
,”
IEEE Trans. Evol. Comput.
,
6
(
2
), pp.
182
197
. 10.1109/4235.996017
56.
Shahriari
,
B.
,
Swersky
,
K.
,
Wang
,
Z.
,
Adams
,
R. P.
, and
de Freitas
,
N.
,
2016
, “
Taking the Human out of the Loop: A Review of Bayesian Optimization
,”
Proc. IEEE
,
104
(
1
), pp.
148
175
. 10.1109/JPROC.2015.2494218
57.
Conti
,
S.
, and
O’Hagan
,
A.
,
2010
, “
Bayesian Emulation of Complex Multi-Output and Dynamic Computer Models
,”
J. Stat. Plan. Inference
,
140
(
3
), pp.
640
651
. 10.1016/j.jspi.2009.08.006
58.
Saibaba
,
A. K.
,
Bardsley
,
J.
,
Brown
,
D. A.
, and
Alexanderian
,
A.
,
2019
, “
Efficient Marginalization-Based MCMC Methods for Hierarchical Bayesian Inverse Problems
,”
SIAM/ASA J. Uncertainty Quantif.
,
7
(
3
), pp.
1105
1131
. 10.1137/18M1220625
59.
Constantine
,
P. G.
,
2015
,
Active Subspaces: Emerging Ideas for Dimension Reduction in Parameter Studies
,
SIAM
,
Philadelphia
.
60.
Calvetti
,
D.
,
Kaipio
,
J. P.
, and
Somersalo
,
E.
,
2014
, “
Inverse Problems in the Bayesian Framework
,”
Inverse Prob.
,
30
(
11
), p.
110301
. 10.1088/0266-5611/30/11/110301
61.
Golchi
,
S.
,
Bingham
,
D. R.
,
Chipman
,
H.
, and
Campbell
,
D. A.
,
2015
, “
Monotone Emulation of Computer Experiments
,”
SIAM/ASA J. Uncertainty Quantif.
,
3
(
1
), pp.
370
392
. 10.1137/140976741
62.
Wang
,
X.
, and
Berger
,
J. O.
,
2016
, “
Estimating Shape Constrained Functions Using Gaussian Processes
,”
SIAM/ASA J. Uncertainty Quantif.
,
4
(
1
), pp.
1
25
. 10.1137/140955033
63.
Maatouk
,
H.
, and
Bay
,
X.
,
2017
, “
Gaussian Process Emulators for Computer Experiments With Inequality Constraints
,”
Math. Geosci.
,
49
(
5
), pp.
557
582
. 10.1007/s11004-017-9673-2
64.
Ding
,
L.
,
Mak
,
S.
, and
Wu
,
C. F. J.
,
2019
,
arxiv preprint, 1908.08868
.
65.
Atamturktur
,
S.
, and
Brown
,
D. A.
,
2015
, “
State-Aware Calibration for Inferring Systematic Bias in Computer Models of Complex Systems
,”
NAFEMS World Congress Proceedings
,
Anaheim, CA
,
June 21–24
.
66.
Stevens
,
G. N.
,
Atamturktur
,
S.
,
Brown
,
D. A.
,
Williams
,
B. J.
, and
Unal
,
C.
,
2018
, “
Statistical Inference of Empirical Constituents in Partitioned Analysis From Integral-Effect Experiments
,”
Eng. Comput.
,
35
(
2
), pp.
672
691
. 10.1108/EC-07-2016-0264
67.
Brown
,
D. A.
, and
Atamturktur
,
S.
,
2018
, “
Nonparametric Functional Calibration of Computer Models
,”
Stat. Sin.
,
28
, pp.
721
742
.
68.
Berry
,
D. S.
,
2008
, “
Blade System Design Studies Phase II: Final Project Report
,”
Sandia National Laboratories Report, SAND2008-4648
.
69.
Berry
,
D. S.
, and
Ashwill
,
T.
,
2007
, “
Design of 9-Meter Carbon-Fiberglass Prototype Blades: CX-100 and TX-100
,”
Sandia National Laboratories Report SAND2007-0201
.
70.
Resor
,
B. R.
, and
Paquette
,
J.
,
2012
, “
A NuMAD model of the Sandia CX-100 Blade
,”
Sandia National Laboratories Report, SAND2012-9273
.
You do not currently have access to this content.