Abstract
Emulation plays an important role in engineering design. However, most emulators such as Gaussian processes (GPs) are exclusively developed for interpolation/regression and their performance significantly deteriorates in extrapolation. To address this shortcoming, we introduce evolutionary Gaussian processes (EGPs) that aim to increase the extrapolation capabilities of GPs. An EGP differs from a GP in that its training involves automatic discovery of some free-form symbolic bases that explain the data reasonably well. In our approach, this automatic discovery is achieved via evolutionary programming (EP) which is integrated with GP modeling via maximum likelihood estimation, bootstrap sampling, and singular value decomposition. As we demonstrate via examples that include a host of analytical functions as well as an engineering problem on materials modeling, EGP can improve the performance of ordinary GPs in terms of not only extrapolation, but also interpolation/regression and numerical stability.
Issue Section:
Design Automation
References
1.
Goodfellow
, I.
, Bengio
, Y.
, and Courville
, A.
, 2016
, Deep Learning
, MIT Press
, London, UK
.2.
Bostanabad
, R.
, 2020
, “Reconstruction of 3d Microstructures From 2d Images Via Transfer Learning
,” Comput.-Aided Design
, 128
(2
), p. 102906
. 3.
Hassaninia
, I.
, Bostanabad
, R.
, Chen
, W.
, and Mohseni
, H.
, 2017
, “Characterization of the Optical Properties of Turbid Media by Supervised Learning of Scattering Patterns
,” Sci. Rep.
, 7
(1
), p. 15259
. 4.
Bostanabad
, R.
, Liang
, B.
, Gao
, J. Y.
, Liu
, W. K.
, Cao
, J.
, Zeng
, D.
, Su
, X. M.
, Xu
, H. Y.
, Li
, Y.
, and Chen
, W.
, 2018
, “Uncertainty Quantification in Multiscale Simulation of Woven Fiber Composites
,” Comput. Methods. Appl. Mech. Eng.
, 338
(21
), pp. 506
–532
. 5.
Bostanabad
, R.
, Chan
, Y. C.
, Wang
, L. W.
, Zhu
, P.
, and Chen
, W.
, 2019
, “Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design
,” ASME J. Mech. Des.
, 141
(11
), p. 111402 (11 pages)
. 6.
Planas
, R.
, Oune
, N.
, and Bostanabad
, R.
, 2020
, “Extrapolation With Gaussian Random Processes and Evolutionary Programming
,” ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Virtual, Online
.7.
Cressie
, N.
, 1990
, “The Origins of Kriging
,” Math. Geology
, 22
(3
), pp. 239
–252
. 8.
Martin
, J.
, and Simpson
, T.
, 2005
, “Use of Kriging Models to Approximate Deterministic Computer Models
,” AIAA. J.
, 43
(4
), pp. 853
–863
. 9.
Stein
, M. L.
, 2012
, Interpolation of Spatial Data: Some Theory for Kriging
, Springer Science & Business Media
, New York
.10.
Bostanabad
, R.
, Liang
, B.
, van Beek
, A.
, Gao
, J.
, Liu
, W. K.
, Cao
, J.
, Zeng
, D.
, Su
, X.
, Xu
, H.
, and Li
, Y.
, 2020
, Multiscale Simulation of Fiber Composites With Spatially Varying Uncertainties
, Y.
Wang
and D. L.
McDowell
, eds., Elsevier
, Atlanta, GA
, pp. 355
–384
.11.
Rasmussen
, C. E.
, 2006
, Gaussian Processes for Machine Learning
, The MIT Press
, Cambridge, MA
.12.
Awad
, M.
, and Khanna
, R.
, 2015
, “Support Vector Regression,” Efficient Learning Machines
, Apress
, Berkeley, CA
, pp. 67
–80
.13.
Chen
, T.
, and Guestrin
, C.
, 2016
, “Xgboost: A Scalable Tree Boosting System
,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
, San Francisco, CA
, ACM
, pp. 785
–794
.14.
Hastie
, T.
, Tibshirani
, R.
, Friedman
, J.
, Hastie
, T.
, Friedman
, J.
, and Tibshirani
, R.
, 2009
, The Elements of Statistical Learning
, Vol. 2
, Springer
, New York, NY
.15.
Bongard
, J.
, and Lipson
, H.
, 2007
, “Automated Reverse Engineering of Nonlinear Dynamical Systems
,” Proc. Natl. Acad. Sci. USA
, 104
(24
), pp. 9943
–8
. 16.
Schmidt
, M.
, and Lipson
, H.
, 2009
, “Distilling Free-Form Natural Laws From Experimental Data
,” Science
, 324
(5923
), pp. 81
–5
. 17.
Tsoulos
, I. G.
, and Lagaris
, I. E.
, 2006
, “Solving Differential Equations With Genetic Programming
,” Genetic Program. Evolvable Mach.
, 7
(1
), pp. 33
–54
. 18.
Wang
, Z.
, and Xu
, H.
, 2021
, “Quantitative Representation of Aleatoric Uncertainties in Network-Like Topological Structural Systems
,” ASME J. Mech. Des.
, 143
(3
), p. 031713
. 19.
Bae
, S.
, Park
, C.
, and Kim
, N. H.
, 2020
, “Estimating Effect of Additional Sample on Uncertainty Reduction in Reliability Analysis Using Gaussian Process
,” ASME J. Mech. Des.
, 142
(11
), p. 111706
. 20.
Oune
, N.
, and Bostanabad
, R.
, 2021
, Latent Map Gaussian Processes for Mixed Variable Metamodeling.21.
Wilson
, A. G.
, and Adams
, R. P.
, 2013
, Gaussian Process Kernels for Pattern Discovery and Extrapolation.22.
Ba
, S.
, and Joseph
, V. R.
, 2012
, “Composite Gaussian Process Models for Emulating Expensive Functions
,” Ann. Appl. Stat.
, 6
(4
), pp. 1838
–1860
.23.
Zhang
, N.
, and Apley
, D. W.
, 2014
, “Fractional Brownian Fields for Response Surface Metamodeling
,” J. Q. Tech.
, 46
(4
), pp. 285
–301
. 24.
Plumlee
, M.
, and Apley
, D. W.
, 2017
, “Lifted Brownian Kriging Models
,” Technometrics
, 59
(2
), pp. 165
–177
. 25.
Paulo
, R.
, 2005
, “Default Priors for Gaussian Processes
,” Ann. Stat.
, 33
(2
), pp. 556
–582
. 26.
Brunton
, S. L.
, Proctor
, J. L.
, and Kutz
, J. N.
, 2016
, “Discovering Governing Equations From Data by Sparse Identification of Nonlinear Dynamical Systems
,” Proc. Natl. Acad. Sci. USA
, 113
(15
), pp. 3932
–7
. 27.
Long
, Z.
, Lu
, Y.
, Ma
, X.
, and Dong
, B.
, 2018
, “PDE-NET: Learning PDEs from Data
,” Proceedings of the 35th International Conference on Machine Learning
, Stockholmsmässan, Stockholm, Sweden
, July 10–15
.28.
Somacal
, A.
, Boechi
, L.
, Jonckheere
, M.
, Lefieux
, V.
, Picard
, D.
, and Smucler
, E.
, 2020
, Uncovering Differential Equations from Data With Hidden Variables.29.
Schaeffer
, H.
, 2017
, “Learning Partial Differential Equations Via Data Discovery and Sparse Optimization
,” Proc. Math. Phys. Eng. Sci.
, 473
(2197
), p. 20160446
.30.
Rudy
, S. H.
, Brunton
, S. L.
, Proctor
, J. L.
, and Kutz
, J. N.
, 2017
, “Data-Driven Discovery of Partial Differential Equations
,” Sci. Adv.
, 3
(4
), p. e1602614
. 31.
Martius
, G.
, and Lampert
, C. H.
, 2016
, Extrapolation and Learning Equations.32.
Kim
, S.
, Lu
, P. Y.
, Mukherjee
, S.
, Gilbert
, M.
, Jing
, L.
, Čeperić
, V.
, and Soljačić
, M.
, 2020
, “Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
,” IEEE Trans. Neural Netw. Learn. Syst.
, 31
(8
), pp. 1
–12
. 33.
Schmidt
, M.
, and Lipson
, H.
, 2010
, Symbolic Regression of Implicit Equations
, Springer
, Ithaca, NY
, pp. 73
–85
.34.
Chollet
, F.
, 2017
, Deep Learning With Python
, Manning Publications Co
, New York, NY
.35.
Bostanabad
, R.
, Kearney
, T.
, Tao
, S. Y.
, Apley
, D. W.
, and Chen
, W.
, 2018
, “Leveraging the Nugget Parameter for Efficient Gaussian Process Modeling
,” Int. J. Numer. Methods Eng.
, 114
(5
), pp. 501
–516
. 36.
Zhang
, W. Z.
, Bostanabad
, R.
, Liang
, B.
, Su
, X. M.
, Zeng
, D.
, Bessa
, M. A.
, Wang
, Y. C.
, Chen
, W.
, and Cao
, J.
, 2019
, “A Numerical Bayesian-Calibrated Characterization Method for Multiscale Prepreg Preforming Simulations With Tension-Shear Coupling
,” Compos. Sci. Technol.
, 170
(3
), pp. 15
–24
. 37.
Xu
, H.
, 2020
, “Constructing Oscillating Function-Based Covariance Matrix to Allow Negative Correlations in Gaussian Random Field Models for Uncertainty Quantification
,” ASME J. Mech. Des.
, 142
(7
), p. 074501
. 38.
Gramacy
, R. B.
, and Apley
, D. W.
, 2015
, “Local Gaussian Process Approximation for Large Computer Experiments
,” J. Computat. Graphical Stat.
, 24
(2
), pp. 561
–578
. 39.
MacDonald
, B.
, Ranjan
, P.
, and Chipman
, H.
, 2015
, “GPfit: An R Package for Fitting a Gaussian Process Model to Deterministic Simulator Outputs
,” J. Stat. Soft.
, 64
(12
), pp. 1
–23
. 40.
Ranjan
, P.
, Haynes
, R.
, and Karsten
, R.
, 2011
, “A Computationally Stable Approach to Gaussian Process Interpolation of Deterministic Computer Simulation Data
,” Technometrics
, 53
(4
), pp. 366
–378
. 41.
Sacks
, J.
, Schiller
, S. B.
, and Welch
, W. J.
, 1989
, “Designs for Computer Experiments
,” Technometrics
, 31
(1
), pp. 41
–47
. 42.
Toal
, D.
, Bressloff
, N.
, and Keane
, A.
, 2008
, “Kriging Hyperparameter Tuning Strategies
,” AIAA. J.
, 46
(5
), pp. 1240
–1252
. 43.
Audet
, C.
, and Dennis Jr
, J. E.
, 2002
, “Analysis of Generalized Pattern Searches
,” SIAM J. Optim.
, 13
(3
), pp. 889
–903
. 44.
Zhao
, L.
, Choi
, K. K.
, and Lee
, I.
, 2011
, “Metamodeling Method Using Dynamic Kriging for Design Optimization
,” AIAA. J.
, 49
(9
), pp. 2034
–2046
. 45.
Toal
, D.
, Bressloff
, N. W.
, Keane
, A. J.
, and Holden
, C. M. E.
, 2011
, “The Development of a Hybridized Particle Swarm for Kriging Hyperparameter Tuning. Engineering Optimization
,” Eng. Optim.
, 43
(6
), pp. 675
–699
. 46.
Tao
, S.
, Shintani
, K.
, Bostanabad
, R.
, Chan
, Y.-C.
, Yang
, G.
, Meingast
, H.
, and Chen
, W.
, 2017
, “Enhanced Gaussian Process Metamodeling and Collaborative Optimization for Vehicle Suspension Design Optimization
,” ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Cleveland, OH
.47.
Koza
, J. R.
, 1994
, “Genetic Programming As a Means for Programming Computers by Natural-selection
,” Stat. Comput.
, 4
(2
), pp. 87
–112
.48.
Fortin
, F. A.
, De Rainville
, F. M.
, Gardner
, M. A.
, Parizeau
, M.
, and Gagne
, C.
, 2012
, “Deap: Evolutionary Algorithms Made Easy
,” J. Mach. Learn. Res.
, 13
(Jul.
), pp. 2171
–2175
.49.
Koller
, D.
, and Friedman
, N.
, 2009
, Probabilistic Graphical Models: Principles and Techniques
, MIT Press
, London, UK
.50.
Izzo
, D.
, Biscani
, F.
, and Mereta
, A.
, 2017
, Differentiable Genetic Programming
, J.
McDermott
, M.
Castelli
, L.
Sekanina
, I.
Wolf
, and P.
García-Sánchez
, eds., Springer
, Amsterdam, The Netherlands
, pp. 35
–51
.51.
Xiong
, J.
, Shi
, S.-Q.
, and Zhang
, T.-Y.
, 2020
, “A Machine-Learning Approach to Predicting and Understanding the Properties of Amorphous Metallic Alloys
,” Mater. Des.
, 187
(6
), p. 108378
. 52.
Guyon
, I.
, and Elisseeff
, A.
, 2003
, “An Introduction to Variable and Feature Selection
,” J. Mach. Learning Res.
, 3
(Mar.
), pp. 1157
–1182
.53.
Sobol’
, I. M.
, 1967
, “On the Distribution of Points in a Cube and the Approximate Evaluation of Integrals
,” Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki
, 7
(4
), pp. 784
–802
.54.
Sobol
, I. M.
, 1998
, “On Quasi-monte Carlo Integrations
,” Math. Comput. Simul.
, 47
(2–5
), pp. 103
–112
. 55.
Arruda
, E. M.
, and Boyce
, M. C.
, 1993
, “A Three-Dimensional Constitutive Model for the Large Stretch Behavior of Rubber Elastic Materials
,” J. Mech. Phys. Solids.
, 41
(2
), pp. 389
–412
. 56.
Bostanabad
, R.
, Chen
, W.
, and Apley
, D. W.
, 2016
, “Characterization and Reconstruction of 3d Stochastic Microstructures Via Supervised Learning
,” J. Microsc.
, 264
(3
), pp. 282
–297
. 57.
Bessa
, M. A.
, Bostanabad
, R.
, Liu
, Z.
, Hu
, A.
, Apley
, D. W.
, Brinson
, C.
, Chen
, W.
, and Liu
, W. K.
, 2017
, “A Framework for Data-Driven Analysis of Materials Under Uncertainty: Countering the Curse of Dimensionality
,” Comput. Methods. Appl. Mech. Eng.
, 320
, pp. 633
–667
. 58.
Belytschko
, T.
, Liu
, W. K.
, Moran
, B.
, and Elkhodary
, K.
, 2013
, Nonlinear Finite Elements for Continua and Structures
, John Wiley & sons
.59.
Friedman
, J. H.
, and Stuetzle
, W.
, 1981
, “Projection Pursuit Regression
,” J. Am. Stat. Assoc.
, 76
(376
), pp. 817
–823
. 60.
Friedman
, J. H.
, 1991
, “Multivariate Adaptive Regression Splines
,” Ann. Stat.
, 19
(1
), pp. 1
–67
.61.
Shan
, S.
, and Wang
, G. G.
, 2010
, “Metamodeling for High Dimensional Simulation-Based Design Problems
,” ASME J. Mech. Des.
, 132
(5
), p. 051009
. Copyright © 2021 by ASME
You do not currently have access to this content.
