Abstract

Scientific and engineering problems often require the use of artificial intelligence to aid understanding and the search for promising designs. While Gaussian processes (GP) stand out as easy-to-use and interpretable learners, they have difficulties in accommodating big data sets, categorical inputs, and multiple responses, which has become a common challenge for a growing number of data-driven design applications. In this paper, we propose a GP model that utilizes latent variables and functions obtained through variational inference to address the aforementioned challenges simultaneously. The method is built upon the latent-variable Gaussian process (LVGP) model where categorical factors are mapped into a continuous latent space to enable GP modeling of mixed-variable data sets. By extending variational inference to LVGP models, the large training data set is replaced by a small set of inducing points to address the scalability issue. Output response vectors are represented by a linear combination of independent latent functions, forming a flexible kernel structure to handle multiple responses that might have distinct behaviors. Comparative studies demonstrate that the proposed method scales well for large data sets with over 104 data points, while outperforming state-of-the-art machine learning methods without requiring much hyperparameter tuning. In addition, an interpretable latent space is obtained to draw insights into the effect of categorical factors, such as those associated with “building blocks” of architectures and element choices in metamaterial and materials design. Our approach is demonstrated for machine learning of ternary oxide materials and topology optimization of a multiscale compliant mechanism with aperiodic microstructures and multiple materials.

References

1.
Forrester
,
A.
,
Sobester
,
A.
, and
Keane
,
A.
,
2008
,
Engineering Design via Surrogate Modelling: A Practical Guide
,
John Wiley & Sons
,
Hoboken, NJ
.
2.
Tao
,
S.
,
Shintani
,
K.
,
Bostanabad
,
R.
,
Chan
,
Y.-C.
,
Yang
,
G.
,
Meingast
,
H.
, and
Chen
,
W.
, 2017, “
Enhanced Gaussian Process Metamodeling and Collaborative Optimization for Vehicle Suspension Design Optimization
,”
ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Cleveland, OH
,
Aug. 6–9
,
American Society of Mechanical Engineers Digital Collection
, Paper No. DETC2017-67976, p.
V02BT03A039
.
3.
Gardner
,
P.
,
Rogers
,
T. J.
,
Lord
,
C.
, and
Barthorpe
,
R. J.
,
2021
, “
Learning Model Discrepancy: A Gaussian Process and Sampling-Based Approach
,”
Mech. Syst. Signal Process
,
152
, p.
107381
.
4.
Bostanabad
,
R.
,
Liang
,
B.
,
Gao
,
J.
,
Liu
,
W. K.
,
Cao
,
J.
,
Zeng
,
D.
,
Su
,
X.
,
Xu
,
H.
,
Li
,
Y.
, and
Chen
,
W.
,
2018
, “
Uncertainty Quantification in Multiscale Simulation of Woven Fiber Composites
,”
Comput. Methods Appl. Mech. Eng.
,
338
, pp.
506
532
.
5.
Wang
,
L.
,
Tao
,
S.
,
Zhu
,
P.
, and
Chen
,
W.
,
2021
, “
Data-Driven Topology Optimization With Multiclass Microstructures Using Latent Variable Gaussian Process
,”
ASME J. Mech. Des.
,
143
(
3
), p.
031708
.
6.
Bauer
,
J.
,
Meza
,
L. R.
,
Schaedler
,
T. A.
,
Schwaiger
,
R.
,
Zheng
,
X.
, and
Valdevit
,
L.
,
2017
, “
Nanolattices: An Emerging Class of Mechanical Metamaterials
,”
Adv. Mater.
,
29
(
40
), p.
1701850
.
7.
Momeni
,
K.
,
Mofidian
,
S. M. M.
, and
Bardaweel
,
H.
,
2019
, “
Systematic Design of High-Strength Multicomponent Metamaterials
,”
Mater. Des.
,
183
, p.
108124
.
8.
Liu
,
H.
,
Ong
,
Y.-S.
,
Shen
,
X.
, and
Cai
,
J.
,
2020
, “
When Gaussian Process Meets Big Data: A Review of Scalable GPs
,”
IEEE Trans. Neural Netw. Learn. Syst.
,
31
(
11
), pp.
4405
4423
.
9.
Bostanabad
,
R.
,
Chan
,
Y.-C.
,
Wang
,
L.
,
Zhu
,
P.
, and
Chen
,
W.
,
2019
, “
Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111402
.
10.
Chalupka
,
K.
,
Williams
,
C. K.
, and
Murray
,
I.
,
2013
, “
A Framework for Evaluating Approximation Methods for Gaussian Process Regression
,”
J. Mach. Learn. Res.
,
14
(
1
), pp.
333
350
.
11.
Gneiting
,
T.
,
2002
, “
Compactly Supported Correlation Functions
,”
J. Multivar. Anal.
,
83
(
2
), pp.
493
508
.
12.
Wilson
,
A.
, and
Nickisch
,
H.
,
2015
, “
Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
,”
International Conference on Machine Learning
,
Lille, France
,
July 6–11
,
PMLR
, pp.
1775
1784
.
13.
Gramacy
,
R. B.
, and
Apley
,
D. W.
,
2015
, “
Local Gaussian Process Approximation for Large Computer Experiments
,”
J. Comput. Graph. Stat.
,
24
(
2
), pp.
561
578
.
14.
Deng
,
X.
,
Lin
,
C. D.
,
Liu
,
K.-W.
, and
Rowe
,
R.
,
2017
, “
Additive Gaussian Process for Computer Models With Categorical and Quantitative Factors
,”
Technometrics
,
59
(
3
), pp.
283
292
.
15.
Qian
,
P. Z. G.
,
Wu
,
H.
, and
Wu
,
C. J.
,
2008
, “
Gaussian Process Models for Computer Experiments With Categorical and Quantitative Factors
,”
Technometrics
,
50
(
3
), pp.
383
396
.
16.
Alvarez
,
M. A.
,
Rosasco
,
L.
, and
Lawrence
,
N. D.
,
2011
, “Kernels for Vector-Valued Functions: A Review,” arXiv preprint arXiv:1106.6251.
17.
Fricker
,
T. E.
,
Oakley
,
J. E.
, and
Urban
,
N. M.
,
2013
, “
Multivariate Gaussian Process Emulators With Nonseparable Covariance Structures
,”
Technometrics
,
55
(
1
), pp.
47
56
.
18.
Gelfand
,
A. E.
,
Schmidt
,
A. M.
,
Banerjee
,
S.
, and
Sirmans
,
C.
,
2004
, “
Nonstationary Multivariate Process Modeling Through Spatially Varying Coregionalization
,”
Test
,
13
(
2
), pp.
263
312
.
19.
Higdon
,
D.
,
2002
, “Space and Space-Time Modeling Using Process Convolutions,”
Quantitative Methods for Current Environmental Issues
,
C. W.
Anderson
,
V.
Barnett
,
P. C.
Chatwin
, and
A. H.
El-Shaarawi
, eds.,
Springer
,
London, UK
, pp.
37
56
.
20.
van der Wilk
,
M.
,
Dutordoir
,
V.
,
John
,
S.
,
Artemev
,
A.
,
Adam
,
V.
, and
Hensman
,
J.
,
2020
, “A Framework for Interdomain and Multioutput Gaussian Processes,” arXiv preprint arXiv:2003.01115.
21.
Barber
,
D.
,
2012
,
Bayesian Reasoning and Machine Learning
,
Cambridge University Press
.
22.
Zhang
,
Y.
,
Apley
,
D. W.
, and
Chen
,
W.
,
2020
, “
Bayesian Optimization for Materials Design With Mixed Quantitative and Categorical Variables
,”
Sci. Rep.
,
10
(
1
), pp.
1
13
.
23.
Zhang
,
Y.
,
Tao
,
S.
,
Chen
,
W.
, and
Apley
,
D. W.
,
2020
, “
A Latent Variable Approach to Gaussian Process Modeling With Categorical and Quantitative Factors
,”
Technometrics
,
62
(
3
), pp.
291
302
.
24.
Hensman
,
J.
,
Fusi
,
N.
, and
Lawrence
,
N. D.
,
2013
, “Gaussian Processes for Big Data,” arXiv preprint arXiv:1309.6835.
25.
Chen
,
T.
,
He
,
T.
,
Benesty
,
M.
,
Khotilovich
,
V.
,
Tang
,
Y.
, and
Cho
,
H.
,
2015
, “
Xgboost: Extreme Gradient Boosting
,”
R package version 0.4-2
,
1
(
4
).
26.
Karniadakis
,
G. E.
,
Kevrekidis
,
I. G.
,
Lu
,
L.
,
Perdikaris
,
P.
,
Wang
,
S.
, and
Yang
,
L.
,
2021
, “
Physics-Informed Machine Learning
,”
Nat. Rev. Phys.
,
3
, pp.
1
19
.
27.
Liu
,
Z.
,
Wu
,
C.
, and
Koishi
,
M.
,
2019
, “
A Deep Material Network for Multiscale Topology Learning and Accelerated Nonlinear Modeling of Heterogeneous Materials
,”
Comput. Methods Appl. Mech. Eng.
,
345
, pp.
1138
1168
.
28.
Yucesan
,
Y. A.
, and
Viana
,
F.
,
2022
, “
A Hybrid Model for Main Bearing Fatigue Prognosis Based on Physics and Machine Learning
,”
AIAA Scitech 2020 Forum
,
Orlando, FL
,
Jan. 6–10
, p.
1412
.
29.
Zhang
,
Z.
,
Rai
,
R.
,
Chowdhury
,
S.
, and
Doermann
,
D.
,
2021
, “
MIDPhyNet: Memorized Infusion of Decomposed Physics in Neural Networks to Model Dynamic Systems
,”
Neurocomputing
,
428
, pp.
116
129
.
30.
Ghassemi
,
P.
,
Behjat
,
A.
,
Zeng
,
C.
,
Lulekar
,
S. S.
,
Rai
,
R.
, and
Chowdhury
,
S.
,
2020
, “
Physics-Aware Surrogate-Based Optimization With Transfer Mapping Gaussian Processes: For Bio-Inspired Flow Tailoring
,”
AIAA Aviation 2020 Forum
,
Virtual Online
,
June 15–19
, p.
3183
.
31.
Chen
,
J.
, and
Liu
,
Y.
,
2021
, “
Probabilistic Physics-Guided Machine Learning for Fatigue Data Analysis
,”
Expert Syst. Appl.
,
168
, p.
114316
.
32.
Viana
,
F. A.
, and
Subramaniyan
,
A. K.
,
2021
, “
A Survey of Bayesian Calibration and Physics-Informed Neural Networks in Scientific Modeling
,”
Arch. Comput. Meth. Eng.
,
28
(
12
), pp.
3801
3830
.
33.
Rasmussen
,
C. E.
, and
Williams
,
C. K. I.
,
2006
,
Gaussian Processes for Machine Learning
,
MIT Press
,
Cambridge, MA
.
34.
Cook
,
R. D.
, and
Ni
,
L.
,
2005
, “
Sufficient Dimension Reduction via Inverse Regression: A Minimum Discrepancy Approach
,”
J. Am. Stat. Assoc.
,
100
(
470
), pp.
410
428
.
35.
Li
,
K.-C.
,
1991
, “
Sliced Inverse Regression for Dimension Reduction
,”
J. Am. Stat. Assoc.
,
86
(
414
), pp.
316
327
.
36.
Zhou
,
Q.
,
Qian
,
P. Z.
, and
Zhou
,
S.
,
2011
, “
A Simple Approach to Emulation for Computer Models With Categorical and Quantitative Factors
,”
Technometrics
,
53
(
3
), pp.
266
273
.
37.
Wang
,
Y.
,
Iyer
,
A.
,
Chen
,
W.
, and
Rondinelli
,
J. M.
,
2020
, “
Featureless Adaptive Optimization Accelerates Functional Electronic Materials Design
,”
Appl. Phys. Rev.
,
7
(
4
), p.
041403
.
38.
Alvarez
,
M. A.
, and
Lawrence
,
N. D.
,
2008
, “
Sparse Convolved Gaussian Processes for Multi-output Regression
,”
NIPS
,
Vancouver, Canada
,
Dec. 12–13
, pp.
57
64
.
39.
LeCun
,
Y.
,
Bengio
,
Y.
, and
Hinton
,
G.
,
2015
, “
Deep Learning
,”
Nature
,
521
(
7553
), pp.
436
444
.
40.
Lippmann
,
R.
,
1987
, “
An Introduction to Computing With Neural Nets
,”
IEEE ASSP Mag.
,
4
(
2
), pp.
4
22
.
41.
Bentéjac
,
C.
,
Csörgő
,
A.
, and
Martínez-Muñoz
,
G.
,
2021
, “
A Comparative Analysis of Gradient Boosting Algorithms
,”
Artif. Intell. Rev.
,
54
(
3
), pp.
1937
1967
.
42.
Chen
,
T.
, and
Guestrin
,
C.
,
2016
, “
Xgboost: A Scalable Tree Boosting System
,”
Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining
,
San Francisco, CA
,
Aug. 13–17
, pp.
785
794
.
43.
Matthews
,
A. G. D. G.
,
Van Der Wilk
,
M.
,
Nickson
,
T.
,
Fujii
,
K.
,
Boukouvalas
,
A.
,
León-Villagrá
,
P.
,
Ghahramani
,
Z.
, and
Hensman
,
J.
,
2017
, “
GPflow: A Gaussian Process Library Using Tensor Flow
,”
J. Mach. Learn. Res.
,
18
(
40
), pp.
1
6
.
44.
Honkela
,
A.
,
Raiko
,
T.
,
Kuusela
,
M.
,
Tornio
,
M.
, and
Karhunen
,
J.
,
2010
, “
Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes
,”
J. Mach. Learn. Res.
,
11
(
106
), pp.
3235
3268
.
45.
Kingma
,
D. P.
, and
Ba
,
J.
,
2014
, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980.
46.
Hensman
,
J.
,
Rattray
,
M.
, and
Lawrence
,
N. D.
,
2012
, “Fast Variational Inference in the Conjugate Exponential Family,”
arXiv preprint
. https://arxiv.org/abs/1206.5162
47.
Salimbeni
,
H.
,
Eleftheriadis
,
S.
, and
Hensman
,
J.
,
2018
, “
Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models
,”
International Conference on Artificial Intelligence and Statistics
,
Playa Blanca, Lanzarote
,
Apr. 9–11
,
PMLR
, pp.
689
697
.
48.
Swiler
,
L. P.
,
Hough
,
P. D.
,
Qian
,
P.
,
Xu
,
X.
,
Storlie
,
C.
, and
Lee
,
H.
,
2014
, “Surrogate Models for Mixed Discrete-Continuous Variables,”
Constraint Programming and Decision Making
,
M.
Ceberio
, and
V.
Kreinovich
, eds.,
Springer
,
Cham, Switzerland
, Vol. 539, pp.
181
202
.
49.
Conti
,
S.
,
Gosling
,
J. P.
,
Oakley
,
J. E.
, and
O'Hagan
,
A.
,
2009
, “
Gaussian Process Emulation of Dynamic Computer Codes
,”
Biometrika
,
96
(
3
), pp.
663
676
.
50.
Kailkhura
,
B.
,
Gallagher
,
B.
,
Kim
,
S.
,
Hiszpanski
,
A.
, and
Han
,
T. Y.-J.
,
2019
, “
Reliable and Explainable Machine-Learning Methods for Accelerated Material Discovery
,”
Npj Comput. Mater.
,
5
(
1
), pp.
1
9
.
51.
Kirklin
,
S.
,
Saal
,
J. E.
,
Meredig
,
B.
,
Thompson
,
A.
,
Doak
,
J. W.
,
Aykol
,
M.
,
Rühl
,
S.
, and
Wolverton
,
C.
,
2015
, “
The Open Quantum Materials Database (OQMD): Assessing the Accuracy of DFT Formation Energies
,”
Npj Comput. Mater.
,
1
(
1
), pp.
1
15
.
52.
Wang
,
L.
,
Chan
,
Y.-C.
,
Ahmed
,
F.
,
Liu
,
Z.
,
Zhu
,
P.
, and
Chen
,
W.
,
2020
, “
Deep Generative Modeling for Mechanistic-Based Learning and Design of Metamaterial Systems
,”
Comput. Methods Appl. Mech. Eng.
,
372
, p.
113377
.
53.
Zhu
,
B.
,
Zhang
,
X.
,
Zhang
,
H.
,
Liang
,
J.
,
Zang
,
H.
,
Li
,
H.
, and
Wang
,
R.
,
2020
, “
Design of Compliant Mechanisms Using Continuum Topology Optimization: A Review
,”
Mech. Mach. Theory
,
143
, p.
103622
.
54.
Svanberg
,
K.
,
1987
, “
The Method of Moving Asymptotes—A New Method for Structural Optimization
,”
Int. J. Numer. Methods Eng.
,
24
(
2
), pp.
359
373
.
You do not currently have access to this content.