Abstract

Bayesian optimization (BO) is an efficient and flexible global optimization framework that is applicable to a very wide range of engineering applications. To leverage the capability of the classical BO, many extensions, including multi-objective, multi-fidelity, parallelization, and latent-variable modeling, have been proposed to address the limitations of the classical BO framework. In this work, we propose a novel multi-objective BO formalism, called srMO-BO-3GP, to solve multi-objective optimization problems in a sequential setting. Three different Gaussian processes (GPs) are stacked together, where each of the GPs is assigned with a different task. The first GP is used to approximate a single-objective computed from the multi-objective definition, the second GP is used to learn the unknown constraints, and the third one is used to learn the uncertain Pareto frontier. At each iteration, a multi-objective augmented Tchebycheff function is adopted to convert multi-objective to single-objective, where the regularization with a regularized ridge term is also introduced to smooth the single-objective function. Finally, we couple the third GP along with the classical BO framework to explore the convergence and diversity of the Pareto frontier by the acquisition function for exploitation and exploration. The proposed framework is demonstrated using several numerical benchmark functions, as well as a thermomechanical finite element model for flip-chip package design optimization.

References

1.
Allmendinger
,
R.
,
Emmerich
,
M. T.
,
Hakanen
,
J.
,
Jin
,
Y.
, and
Rigoni
,
E.
,
2017
, “
Surrogate-Assisted Multicriteria Optimization: Complexities, Prospective Solutions, and Business Case
,”
J. Multi-Criteria Decis. Anal.
,
24
(
1–2
), pp.
5
24
.
2.
Rojas-Gonzalez
,
S.
, and
Van Nieuwenhuyse
,
I.
,
2019
, “
A Survey on Kriging-Based Infill Algorithms for Multi-Objective Simulation Optimization
”.
FEB Research Report KBI_1907
.
3.
Jeong
,
S.
, and
Obayashi
,
S.
,
2005
, “
Efficient Global Optimization (EGO) for Multi-Objective Problem and Data Mining
,”
2005 IEEE Congress on Evolutionary Computation
,
Edinburgh, Scotland
, Vol.
3
,
IEEE
, pp.
2138
2145
.
4.
Knowles
,
J.
,
2006
, “
ParEGO: a Hybrid Algorithm With On-Line Landscape Approximation for Expensive Multiobjective Optimization Problems
,”
IEEE Trans. Evol. Comput.
,
10
(
1
), pp.
50
66
.
5.
Davins-Valldaura
,
J.
,
Moussaoui
,
S.
,
Pita-Gil
,
G.
, and
Plestan
,
F.
,
2017
, “
ParEGO Extensions for Multi-Objective Optimization of Expensive Evaluation Functions
,”
J. Global Optim.
,
67
(
1–2
), pp.
79
96
.
6.
Dächert
,
K.
,
Gorski
,
J.
, and
Klamroth
,
K.
,
2012
, “
An Augmented Weighted Tchebycheff Method With Adaptively Chosen Parameters for Discrete Bicriteria Optimization Problems
,”
Comput. Oper. Res.
,
39
(
12
), pp.
2929
2943
.
7.
Zhang
,
Q.
,
Liu
,
W.
,
Tsang
,
E.
, and
Virginas
,
B.
,
2009
, “
Expensive Multiobjective Optimization by MOEA/D with Gaussian Process Model
,”
IEEE Trans. Evol. Comput.
,
14
(
3
), pp.
456
474
.
8.
Li
,
M.
,
Li
,
G.
, and
Azarm
,
S.
,
2008
, “
A Kriging Metamodel Assisted Multi-Objective Genetic Algorithm for Design Optimization
,”
J. Mech. Des.
,
130
(
3
), p.
031401
.
9.
Li
,
G.
,
Li
,
M.
,
Azarm
,
S.
,
Al Hashimi
,
S.
,
Al Ameri
,
T.
, and
Al Qasas
,
N.
,
2009
, “
Improving Multi-Objective Genetic Algorithms With Adaptive Design of Experiments and Online Metamodeling
,”
Struct. Multidiscipl. Optim.
,
37
(
5
), pp.
447
461
.
10.
Feliot
,
P.
,
Bect
,
J.
, and
Vazquez
,
E.
,
2017
, “
A Bayesian Approach to Constrained Single-and Multi-Objective Optimization
,”
J. Global Optim.
,
67
(
1–2
), pp.
97
133
.
11.
Yang
,
K.
,
Emmerich
,
M.
,
Deutz
,
A.
, and
Bäck
,
T.
,
2019
, “
Multi-Objective Bayesian Global Optimization Using Expected Hypervolume Improvement Gradient
,”
Swarm Evol. Comput.
,
44
, pp.
945
956
.
12.
Valladares
,
H.
, and
Tovar
,
A.
,
2020
, “
A Simple and Effective Methodology to Perform Multi-Objective Bayesian Optimization: An Application in the Design of Sandwich Composite Armors for Blast Mitigation
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Vol.
84003
,
Virtual, Online
,
American Society of Mechanical Engineers
, p.
V11AT11A056
.
13.
Gupta
,
S.
,
Shilton
,
A.
,
Rana
,
S.
, and
Venkatesh
,
S.
,
2018
, “
Exploiting Strategy-Space Diversity for Batch Bayesian Optimization
,”
International Conference on Artificial Intelligence and Statistics
,
Playa Blanca, Lanzarote, Canary Islands
,
PMLR
, pp.
538
547
.
14.
Shu
,
L.
,
Jiang
,
P.
,
Shao
,
X.
, and
Wang
,
Y.
,
2020
, “
A New Multi-Objective Bayesian Optimization Formulation With the Acquisition Function for Convergence and Diversity
,”
ASME J. Mech. Des.
,
142
(
9
), p.
091703
.
15.
Hernández-Lobato
,
D.
,
Hernández-Lobato
,
J.
,
Shah
,
A.
, and
Adams
,
R.
,
2016
, “
Predictive Entropy Search for Multi-Objective Bayesian Optimization
,”
International Conference on Machine Learning
,
PMLR
, pp.
1492
1501
.
16.
Garrido-Merchán
,
E. C.
, and
Hernández-Lobato
,
D.
,
2019
, “
Predictive Entropy Search for Multi-Objective Bayesian Optimization With Constraints
,”
Neurocomputing
,
361
, pp.
50
68
.
17.
Abdolshah
,
M.
,
Shilton
,
A.
,
Rana
,
S.
,
Gupta
,
S.
, and
Venkatesh
,
S.
,
2019
, “
Cost-Aware Multi-Objective Bayesian Optimisation
.”
arXiv preprint
.
18.
Gaudrie
,
D.
,
Le Riche
,
R.
,
Picheny
,
V.
,
Enaux
,
B.
, and
Herbert
,
V.
,
2019
, “
Targeting Solutions in Bayesian Multi-Objective Optimization: Sequential and Batch Versions
,”
Ann. Math. Artif. Intell.
,
88
(
1
), pp.
1
26
.
19.
Palar
,
P. S.
,
Zuhal
,
L. R.
,
Chugh
,
T.
, and
Rahat
,
A.
,
2020
, “
On the Impact of Covariance Functions in Multi-Objective Bayesian Optimization for Engineering Design
,”
AIAA Scitech 2020 Forum
,
AIAA
, p.
1867
.
20.
Golovin
,
D.
, and
Zhang
,
Q.
,
2020
, “
Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization
.”
arXiv preprint
.
21.
Han
,
D.
, and
Zheng
,
J.
,
2020
, “
A Kriging Model-Based Expensive Multi-Objective Optimization Algorithm Using R2 Indicator of Expectation Improvement
,”
Math. Probl. Eng.
,
2020
.
22.
Zhao
,
G.
,
Arroyave
,
R.
, and
Qian
,
X.
,
2018
, “
Fast Exact Computation of Expected Hypervolume Improvement.
arXiv preprint
.
23.
Beume
,
N.
,
Fonseca
,
C. M.
,
Lopez-Ibanez
,
M.
,
Paquete
,
L.
, and
Vahrenhold
,
J.
,
2009
, “
On the Complexity of Computing the Hypervolume Indicator
,”
IEEE Trans. Evol. Comput.
,
13
(
5
), pp.
1075
1082
.
24.
Tran
,
A.
,
Eldred
,
M.
,
Wang
,
Y.
, and
McCann
,
S.
,
2020
, “
srMO-BO-3GP: A Sequential Regularized Multi-Objective Constrained Bayesian Optimization for Design Applications
,”
Proceedings of the ASME 2020 IDETC/CIE, Vol. Volume 1: 40th Computers and Information in Engineering Conference of International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Virtual, Online
,
American Society of Mechanical Engineers
.
25.
Lin
,
X.
,
Zhen
,
H.-L.
,
Li
,
Z.
,
Zhang
,
Q.
, and
Kwong
,
S.
,
2018
, “
A Batched Scalable Multi-Objective Bayesian Optimization Algorithm
.”
arXiv preprint
.
26.
Shahriari
,
B.
,
Swersky
,
K.
,
Wang
,
Z.
,
Adams
,
R. P.
, and
de Freitas
,
N.
,
2016
, “
Taking the Human out of the Loop: A Review of Bayesian Optimization
,”
Proc. IEEE
,
104
(
1
), pp.
148
175
.
27.
Ishibuchi
,
H.
,
Sakane
,
Y.
, and
Nojima
,
Y.
,
2010
, “
Use of Multiple Grids with Different Scalarizing Functions in MOEA/D
,”
In SCIS & ISIS SCIS & ISIS 2010
,
Okayama, Japan
,
Japan Society for Fuzzy Theory and Intelligent Informatics
, pp.
898
903
.
28.
Trivedi
,
A.
,
Srinivasan
,
D.
,
Sanyal
,
K.
, and
Ghosh
,
A.
,
2016
, “
A Survey of Multiobjective Evolutionary Algorithms Based on Decomposition
,”
IEEE Trans. Evol. Comput.
,
21
(
3
), pp.
440
462
.
29.
Bentley
,
J. L.
,
1975
, “
Multidimensional Binary Search Trees Used for Associative Searching
,”
Commun. ACM
,
18
(
9
), pp.
509
517
.
30.
Hastie
,
T.
,
Rosset
,
S.
,
Zhu
,
J.
, and
Zou
,
H.
,
2009
, “
Multi-Class AdaBoost
,”
Stat. Its Interface
,
2
(
3
), pp.
349
360
.
31.
Breiman
,
L.
,
2001
, “
Random Forests
,”
Mach. Learn.
,
45
(
1
), pp.
5
32
.
32.
Hearst
,
M. A.
,
Dumais
,
S. T.
,
Osuna
,
E.
,
Platt
,
J.
, and
Scholkopf
,
B.
,
1998
, “
Support Vector Machines
,”
IEEE Intell. Syst. Appl.
,
13
(
4
), pp.
18
28
.
33.
Suykens
,
J. A.
, and
Vandewalle
,
J.
,
1999
, “
Least Squares Support Vector Machine Classifiers
,”
Neural Process. Lett.
,
9
(
3
), pp.
293
300
.
34.
LeCun
,
Y.
,
Bengio
,
Y.
, and
Hinton
,
G.
,
2015
, “
Deep Learning
,”
Nature
,
521
(
7553
), p.
436
.
35.
Rasmussen
,
C. E.
,
2004
, “
Gaussian Processes in Machine Learning
,”
Advanced Lectures on Machine Learning
,
Springer
, pp.
63
71
.
36.
Hansen
,
N.
,
Müller
,
S. D.
, and
Koumoutsakos
,
P.
,
2003
, “
Reducing the Time Complexity of the Derandomized Evolution Strategy With Covariance Matrix Adaptation (CMA-ES)
,”
Evol. Comput.
,
11
(
1
), pp.
1
18
.
37.
Hansen
,
N.
, and
Kern
,
S.
,
2004
, “
Evaluating the CMA Evolution Strategy on Multimodal Test Functions
,”
International Conference on Parallel Problem Solving from Nature
,
Birmingham, UK
,
Springer
, pp.
282
291
.
38.
Huband
,
S.
,
Hingston
,
P.
,
Barone
,
L.
, and
While
,
L.
,
2006
, “
A Review of Multiobjective Test Problems and a Scalable Test Problem Toolkit
,”
IEEE Trans. Evol. Comput.n
,
10
(
5
), pp.
477
506
.
39.
Zitzler
,
E.
,
Deb
,
K.
, and
Thiele
,
L.
,
2000
, “
Comparison of Multiobjective Evolutionary Algorithms: Empirical Results
,”
Evol. Comput.
,
8
(
2
), pp.
173
195
.
40.
Deb
,
K.
,
Thiele
,
L.
,
Laumanns
,
M.
, and
Zitzler
,
E.
,
2005
, “
Scalable Test Problems for Evolutionary Multiobjective Optimization
,”
Evolutionary Multiobjective Optimization
,
Springer
, pp.
105
145
.
41.
Jin
,
Y.-F.
,
Yin
,
Z.-Y.
,
Zhou
,
W.-H.
, and
Huang
,
H.-W.
,
2019
, “
Multi-Objective Optimization-Based Updating of Predictions During Excavation
,”
Eng. Appl. Artif. Intell.
,
78
, pp.
102
123
.
42.
Ishibuchi
,
H.
,
Masuda
,
H.
, and
Nojima
,
Y.
,
2015
, “
Pareto Fronts of Many-Objective Degenerate Test Problems
,”
IEEE Trans. Evol. Comput.
,
20
(
5
), pp.
807
813
.
43.
Cox
,
W.
, and
While
,
L.
,
2016
, “
Improving the IWFG Algorithm for Calculating Incremental Hypervolume
,”
2016 IEEE Congress on Evolutionary Computation (CEC)
,
Vancouver, Canada
,
IEEE
, pp.
3969
3976
.
44.
While
,
L.
,
2005
, “
A New Analysis of the LebMeasure Algorithm for Calculating Hypervolume
,”
In International Conference on Evolutionary Multi-Criterion Optimization
,
Guanajuato, Mexico
,
Springer
, pp.
326
340
.
45.
Michael Waskom, and the seaborn development team
,
2020
. mwaskom/seaborn, Sep.
46.
Tran
,
A.
,
He
,
L.
, and
Wang
,
Y.
,
2018
, “
An Efficient First-Principles Saddle Point Searching Method Based on Distributed Kriging Metamodels
,”
ASCE-ASME J. Risk Uncertainty in Eng. Syst., Part B: Mech. Eng.
,
4
(
1
), p.
011006
.
47.
Bostanabad
,
R.
,
Chan
,
Y.-C.
,
Wang
,
L.
,
Zhu
,
P.
, and
Chen
,
W.
,
2019
, “
Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111402
.
48.
Tran
,
A.
,
Tran
,
M.
, and
Wang
,
Y.
,
2019
, “
Constrained Mixed-Integer Gaussian Mixture Bayesian Optimization and Its Applications in Designing Fractal and Auxetic Metamaterials
,”
Struct. Multidiscipl. Optim.
,
59
, pp.
1
24
.
49.
Tran
,
A.
,
Sun
,
J.
,
Furlan
,
J. M.
,
Pagalthivarthi
,
K. V.
,
Visintainer
,
R. J.
, and
Wang
,
Y.
,
2019
, “
pBO-2GP-3B: A Batch Parallel Known/unknown Constrained Bayesian Optimization With Feasibility Classification and Its Applications in Computational Fluid Dynamics
,”
Comput. Methods. Appl. Mech. Eng.
,
347
, pp.
827
852
.
50.
Tran
,
A.
,
McCann
,
S.
,
Furlan
,
J. M.
,
Pagalthivarthi
,
K. V.
,
Visintainer
,
R. J.
,
Wildey
,
T.
, and
Eldred
,
M.
,
2020
, “
aphBO-2GP-3B: A Budgeted Asynchronous-Parallel Multi-Acquisition for Known/Unknown Constrained Bayesian Optimization on High-Performing Computing Architecture
.”
arXiv preprint
.
51.
Tran
,
A.
,
Wildey
,
T.
, and
McCann
,
S.
,
2019
, “
sBF-BO-2CoGP: A Sequential Bi-Fidelity Constrained Bayesian Optimization for Design Applications
,”
Proceedings of the ASME 2019 IDETC/CIE, Vol. Volume 1: 39th Computers and Information in Engineering Conference of International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
,
American Society of Mechanical Engineers
, p.
V001T02A073
.
52.
Tran
,
A.
,
Wildey
,
T.
, and
McCann
,
S.
,
2020
, “
sMF-BO-2CoGP: A Sequential Multi-fidelity Constrained Bayesian Optimization for Design Applications
,”
ASME J. Comput. Inf. Sci. Eng.
,
20
(
3
), p. 031007.
53.
Travaglino
,
S.
,
Murdock
,
K.
,
Tran
,
A.
,
Martin
,
C.
,
Liang
,
L.
,
Wang
,
Y.
, and
Sun
,
W.
,
2020
, “
Computational Optimization Study of Transcatheter Aortic Valve Leaflet Design Using Porcine and Bovine Leaflets
,”
ASME J. Biomech. Eng.
,
142
(
1
), p.
011007
.
54.
Tran
,
A.
,
Furlan
,
J. M.
,
Pagalthivarthi
,
K. V.
,
Visintainer
,
R. J.
,
Wildey
,
T.
, and
Wang
,
Y.
,
2019
, “
WearGP: A Computationally Efficient Machine Learning Framework for Local Erosive Wear Predictions Via Nodal Gaussian Processes
,”
Wear
,
422
, pp.
9
26
.
55.
Tran
,
A.
,
Wang
,
Y.
,
Furlan
,
J.
,
Pagalthivarthi
,
K. V.
,
Garman
,
M.
,
Cutright
,
A.
, and
Visintainer
,
R. J.
,
2020
, “
WearGP: A UQ/ML Wear Prediction Framework for Slurry Pump Impellers and Casings
,”
ASME 2020 Fluids Engineering Division Summer Meeting
,
Virtual, Online
,
American Society of Mechanical Engineers
.
56.
Miettinen
,
K.
,
2012
,
Nonlinear Multiobjective Optimization
, Vol.
12
.
Springer Science & Business Media
,
New York
.
57.
Bull
,
A. D.
,
2011
, “
Convergence Rates of Efficient Global Optimization Algorithms
,”
J. Mach. Learn. Res.
,
12
, pp.
2879
2904
.
You do not currently have access to this content.