Abstract

Emulation plays an important role in engineering design. However, most emulators such as Gaussian processes (GPs) are exclusively developed for interpolation/regression and their performance significantly deteriorates in extrapolation. To address this shortcoming, we introduce evolutionary Gaussian processes (EGPs) that aim to increase the extrapolation capabilities of GPs. An EGP differs from a GP in that its training involves automatic discovery of some free-form symbolic bases that explain the data reasonably well. In our approach, this automatic discovery is achieved via evolutionary programming (EP) which is integrated with GP modeling via maximum likelihood estimation, bootstrap sampling, and singular value decomposition. As we demonstrate via examples that include a host of analytical functions as well as an engineering problem on materials modeling, EGP can improve the performance of ordinary GPs in terms of not only extrapolation, but also interpolation/regression and numerical stability.

References

1.
Goodfellow
,
I.
,
Bengio
,
Y.
, and
Courville
,
A.
,
2016
,
Deep Learning
,
MIT Press
,
London, UK
.
2.
Bostanabad
,
R.
,
2020
, “
Reconstruction of 3d Microstructures From 2d Images Via Transfer Learning
,”
Comput.-Aided Design
,
128
(
2
), p.
102906
.
3.
Hassaninia
,
I.
,
Bostanabad
,
R.
,
Chen
,
W.
, and
Mohseni
,
H.
,
2017
, “
Characterization of the Optical Properties of Turbid Media by Supervised Learning of Scattering Patterns
,”
Sci. Rep.
,
7
(
1
), p.
15259
.
4.
Bostanabad
,
R.
,
Liang
,
B.
,
Gao
,
J. Y.
,
Liu
,
W. K.
,
Cao
,
J.
,
Zeng
,
D.
,
Su
,
X. M.
,
Xu
,
H. Y.
,
Li
,
Y.
, and
Chen
,
W.
,
2018
, “
Uncertainty Quantification in Multiscale Simulation of Woven Fiber Composites
,”
Comput. Methods. Appl. Mech. Eng.
,
338
(
21
), pp.
506
532
.
5.
Bostanabad
,
R.
,
Chan
,
Y. C.
,
Wang
,
L. W.
,
Zhu
,
P.
, and
Chen
,
W.
,
2019
, “
Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111402 (11 pages)
.
6.
Planas
,
R.
,
Oune
,
N.
, and
Bostanabad
,
R.
,
2020
, “
Extrapolation With Gaussian Random Processes and Evolutionary Programming
,”
ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Virtual, Online
.
7.
Cressie
,
N.
,
1990
, “
The Origins of Kriging
,”
Math. Geology
,
22
(
3
), pp.
239
252
.
8.
Martin
,
J.
, and
Simpson
,
T.
,
2005
, “
Use of Kriging Models to Approximate Deterministic Computer Models
,”
AIAA. J.
,
43
(
4
), pp.
853
863
.
9.
Stein
,
M. L.
,
2012
,
Interpolation of Spatial Data: Some Theory for Kriging
,
Springer Science & Business Media
,
New York
.
10.
Bostanabad
,
R.
,
Liang
,
B.
,
van Beek
,
A.
,
Gao
,
J.
,
Liu
,
W. K.
,
Cao
,
J.
,
Zeng
,
D.
,
Su
,
X.
,
Xu
,
H.
, and
Li
,
Y.
,
2020
,
Multiscale Simulation of Fiber Composites With Spatially Varying Uncertainties
,
Y.
Wang
and
D. L.
McDowell
, eds.,
Elsevier
,
Atlanta, GA
, pp.
355
384
.
11.
Rasmussen
,
C. E.
,
2006
,
Gaussian Processes for Machine Learning
,
The MIT Press
,
Cambridge, MA
.
12.
Awad
,
M.
, and
Khanna
,
R.
,
2015
, “Support Vector Regression,”
Efficient Learning Machines
,
Apress
,
Berkeley, CA
, pp.
67
80
.
13.
Chen
,
T.
, and
Guestrin
,
C.
,
2016
, “
Xgboost: A Scalable Tree Boosting System
,”
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
,
San Francisco, CA
,
ACM
, pp.
785
794
.
14.
Hastie
,
T.
,
Tibshirani
,
R.
,
Friedman
,
J.
,
Hastie
,
T.
,
Friedman
,
J.
, and
Tibshirani
,
R.
,
2009
,
The Elements of Statistical Learning
, Vol.
2
,
Springer
,
New York, NY
.
15.
Bongard
,
J.
, and
Lipson
,
H.
,
2007
, “
Automated Reverse Engineering of Nonlinear Dynamical Systems
,”
Proc. Natl. Acad. Sci. USA
,
104
(
24
), pp.
9943
8
.
16.
Schmidt
,
M.
, and
Lipson
,
H.
,
2009
, “
Distilling Free-Form Natural Laws From Experimental Data
,”
Science
,
324
(
5923
), pp.
81
5
.
17.
Tsoulos
,
I. G.
, and
Lagaris
,
I. E.
,
2006
, “
Solving Differential Equations With Genetic Programming
,”
Genetic Program. Evolvable Mach.
,
7
(
1
), pp.
33
54
.
18.
Wang
,
Z.
, and
Xu
,
H.
,
2021
, “
Quantitative Representation of Aleatoric Uncertainties in Network-Like Topological Structural Systems
,”
ASME J. Mech. Des.
,
143
(
3
), p.
031713
.
19.
Bae
,
S.
,
Park
,
C.
, and
Kim
,
N. H.
,
2020
, “
Estimating Effect of Additional Sample on Uncertainty Reduction in Reliability Analysis Using Gaussian Process
,”
ASME J. Mech. Des.
,
142
(
11
), p.
111706
.
20.
Oune
,
N.
, and
Bostanabad
,
R.
,
2021
, Latent Map Gaussian Processes for Mixed Variable Metamodeling.
21.
Wilson
,
A. G.
, and
Adams
,
R. P.
,
2013
, Gaussian Process Kernels for Pattern Discovery and Extrapolation.
22.
Ba
,
S.
, and
Joseph
,
V. R.
,
2012
, “
Composite Gaussian Process Models for Emulating Expensive Functions
,”
Ann. Appl. Stat.
,
6
(
4
), pp.
1838
1860
.
23.
Zhang
,
N.
, and
Apley
,
D. W.
,
2014
, “
Fractional Brownian Fields for Response Surface Metamodeling
,”
J. Q. Tech.
,
46
(
4
), pp.
285
301
.
24.
Plumlee
,
M.
, and
Apley
,
D. W.
,
2017
, “
Lifted Brownian Kriging Models
,”
Technometrics
,
59
(
2
), pp.
165
177
.
25.
Paulo
,
R.
,
2005
, “
Default Priors for Gaussian Processes
,”
Ann. Stat.
,
33
(
2
), pp.
556
582
.
26.
Brunton
,
S. L.
,
Proctor
,
J. L.
, and
Kutz
,
J. N.
,
2016
, “
Discovering Governing Equations From Data by Sparse Identification of Nonlinear Dynamical Systems
,”
Proc. Natl. Acad. Sci. USA
,
113
(
15
), pp.
3932
7
.
27.
Long
,
Z.
,
Lu
,
Y.
,
Ma
,
X.
, and
Dong
,
B.
,
2018
, “
PDE-NET: Learning PDEs from Data
,”
Proceedings of the 35th International Conference on Machine Learning
,
Stockholmsmässan, Stockholm, Sweden
,
July 10–15
.
28.
Somacal
,
A.
,
Boechi
,
L.
,
Jonckheere
,
M.
,
Lefieux
,
V.
,
Picard
,
D.
, and
Smucler
,
E.
,
2020
, Uncovering Differential Equations from Data With Hidden Variables.
29.
Schaeffer
,
H.
,
2017
, “
Learning Partial Differential Equations Via Data Discovery and Sparse Optimization
,”
Proc. Math. Phys. Eng. Sci.
,
473
(
2197
), p.
20160446
.
30.
Rudy
,
S. H.
,
Brunton
,
S. L.
,
Proctor
,
J. L.
, and
Kutz
,
J. N.
,
2017
, “
Data-Driven Discovery of Partial Differential Equations
,”
Sci. Adv.
,
3
(
4
), p.
e1602614
.
31.
Martius
,
G.
, and
Lampert
,
C. H.
,
2016
, Extrapolation and Learning Equations.
32.
Kim
,
S.
,
Lu
,
P. Y.
,
Mukherjee
,
S.
,
Gilbert
,
M.
,
Jing
,
L.
,
Čeperić
,
V.
, and
Soljačić
,
M.
,
2020
, “
Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
,”
IEEE Trans. Neural Netw. Learn. Syst.
,
31
(
8
), pp.
1
12
.
33.
Schmidt
,
M.
, and
Lipson
,
H.
,
2010
,
Symbolic Regression of Implicit Equations
,
Springer
,
Ithaca, NY
, pp.
73
85
.
34.
Chollet
,
F.
,
2017
,
Deep Learning With Python
,
Manning Publications Co
,
New York, NY
.
35.
Bostanabad
,
R.
,
Kearney
,
T.
,
Tao
,
S. Y.
,
Apley
,
D. W.
, and
Chen
,
W.
,
2018
, “
Leveraging the Nugget Parameter for Efficient Gaussian Process Modeling
,”
Int. J. Numer. Methods Eng.
,
114
(
5
), pp.
501
516
.
36.
Zhang
,
W. Z.
,
Bostanabad
,
R.
,
Liang
,
B.
,
Su
,
X. M.
,
Zeng
,
D.
,
Bessa
,
M. A.
,
Wang
,
Y. C.
,
Chen
,
W.
, and
Cao
,
J.
,
2019
, “
A Numerical Bayesian-Calibrated Characterization Method for Multiscale Prepreg Preforming Simulations With Tension-Shear Coupling
,”
Compos. Sci. Technol.
,
170
(
3
), pp.
15
24
.
37.
Xu
,
H.
,
2020
, “
Constructing Oscillating Function-Based Covariance Matrix to Allow Negative Correlations in Gaussian Random Field Models for Uncertainty Quantification
,”
ASME J. Mech. Des.
,
142
(
7
), p.
074501
.
38.
Gramacy
,
R. B.
, and
Apley
,
D. W.
,
2015
, “
Local Gaussian Process Approximation for Large Computer Experiments
,”
J. Computat. Graphical Stat.
,
24
(
2
), pp.
561
578
.
39.
MacDonald
,
B.
,
Ranjan
,
P.
, and
Chipman
,
H.
,
2015
, “
GPfit: An R Package for Fitting a Gaussian Process Model to Deterministic Simulator Outputs
,”
J. Stat. Soft.
,
64
(
12
), pp.
1
23
.
40.
Ranjan
,
P.
,
Haynes
,
R.
, and
Karsten
,
R.
,
2011
, “
A Computationally Stable Approach to Gaussian Process Interpolation of Deterministic Computer Simulation Data
,”
Technometrics
,
53
(
4
), pp.
366
378
.
41.
Sacks
,
J.
,
Schiller
,
S. B.
, and
Welch
,
W. J.
,
1989
, “
Designs for Computer Experiments
,”
Technometrics
,
31
(
1
), pp.
41
47
.
42.
Toal
,
D.
,
Bressloff
,
N.
, and
Keane
,
A.
,
2008
, “
Kriging Hyperparameter Tuning Strategies
,”
AIAA. J.
,
46
(
5
), pp.
1240
1252
.
43.
Audet
,
C.
, and
Dennis Jr
,
J. E.
,
2002
, “
Analysis of Generalized Pattern Searches
,”
SIAM J. Optim.
,
13
(
3
), pp.
889
903
.
44.
Zhao
,
L.
,
Choi
,
K. K.
, and
Lee
,
I.
,
2011
, “
Metamodeling Method Using Dynamic Kriging for Design Optimization
,”
AIAA. J.
,
49
(
9
), pp.
2034
2046
.
45.
Toal
,
D.
,
Bressloff
,
N. W.
,
Keane
,
A. J.
, and
Holden
,
C. M. E.
,
2011
, “
The Development of a Hybridized Particle Swarm for Kriging Hyperparameter Tuning. Engineering Optimization
,”
Eng. Optim.
,
43
(
6
), pp.
675
699
.
46.
Tao
,
S.
,
Shintani
,
K.
,
Bostanabad
,
R.
,
Chan
,
Y.-C.
,
Yang
,
G.
,
Meingast
,
H.
, and
Chen
,
W.
,
2017
, “
Enhanced Gaussian Process Metamodeling and Collaborative Optimization for Vehicle Suspension Design Optimization
,”
ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Cleveland, OH
.
47.
Koza
,
J. R.
,
1994
, “
Genetic Programming As a Means for Programming Computers by Natural-selection
,”
Stat. Comput.
,
4
(
2
), pp.
87
112
.
48.
Fortin
,
F. A.
,
De Rainville
,
F. M.
,
Gardner
,
M. A.
,
Parizeau
,
M.
, and
Gagne
,
C.
,
2012
, “
Deap: Evolutionary Algorithms Made Easy
,”
J. Mach. Learn. Res.
,
13
(
Jul.
), pp.
2171
2175
.
49.
Koller
,
D.
, and
Friedman
,
N.
,
2009
,
Probabilistic Graphical Models: Principles and Techniques
,
MIT Press
,
London, UK
.
50.
Izzo
,
D.
,
Biscani
,
F.
, and
Mereta
,
A.
,
2017
,
Differentiable Genetic Programming
,
J.
McDermott
,
M.
Castelli
,
L.
Sekanina
,
I.
Wolf
, and
P.
García-Sánchez
, eds.,
Springer
,
Amsterdam, The Netherlands
, pp.
35
51
.
51.
Xiong
,
J.
,
Shi
,
S.-Q.
, and
Zhang
,
T.-Y.
,
2020
, “
A Machine-Learning Approach to Predicting and Understanding the Properties of Amorphous Metallic Alloys
,”
Mater. Des.
,
187
(
6
), p.
108378
.
52.
Guyon
,
I.
, and
Elisseeff
,
A.
,
2003
, “
An Introduction to Variable and Feature Selection
,”
J. Mach. Learning Res.
,
3
(
Mar.
), pp.
1157
1182
.
53.
Sobol’
,
I. M.
,
1967
, “
On the Distribution of Points in a Cube and the Approximate Evaluation of Integrals
,”
Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki
,
7
(
4
), pp.
784
802
.
54.
Sobol
,
I. M.
,
1998
, “
On Quasi-monte Carlo Integrations
,”
Math. Comput. Simul.
,
47
(
2–5
), pp.
103
112
.
55.
Arruda
,
E. M.
, and
Boyce
,
M. C.
,
1993
, “
A Three-Dimensional Constitutive Model for the Large Stretch Behavior of Rubber Elastic Materials
,”
J. Mech. Phys. Solids.
,
41
(
2
), pp.
389
412
.
56.
Bostanabad
,
R.
,
Chen
,
W.
, and
Apley
,
D. W.
,
2016
, “
Characterization and Reconstruction of 3d Stochastic Microstructures Via Supervised Learning
,”
J. Microsc.
,
264
(
3
), pp.
282
297
.
57.
Bessa
,
M. A.
,
Bostanabad
,
R.
,
Liu
,
Z.
,
Hu
,
A.
,
Apley
,
D. W.
,
Brinson
,
C.
,
Chen
,
W.
, and
Liu
,
W. K.
,
2017
, “
A Framework for Data-Driven Analysis of Materials Under Uncertainty: Countering the Curse of Dimensionality
,”
Comput. Methods. Appl. Mech. Eng.
,
320
, pp.
633
667
.
58.
Belytschko
,
T.
,
Liu
,
W. K.
,
Moran
,
B.
, and
Elkhodary
,
K.
,
2013
,
Nonlinear Finite Elements for Continua and Structures
,
John Wiley & sons
.
59.
Friedman
,
J. H.
, and
Stuetzle
,
W.
,
1981
, “
Projection Pursuit Regression
,”
J. Am. Stat. Assoc.
,
76
(
376
), pp.
817
823
.
60.
Friedman
,
J. H.
,
1991
, “
Multivariate Adaptive Regression Splines
,”
Ann. Stat.
,
19
(
1
), pp.
1
67
.
61.
Shan
,
S.
, and
Wang
,
G. G.
,
2010
, “
Metamodeling for High Dimensional Simulation-Based Design Problems
,”
ASME J. Mech. Des.
,
132
(
5
), p.
051009
.
You do not currently have access to this content.