Abstract

The purpose of this work is to compare learning algorithms to identify which is the fastest and most accurate for training mechanical neural networks (MNNs). MNNs are a unique class of lattice-based artificial intelligence (AI) architected materials that learn their mechanical behaviors with repeated exposure to external loads. They can learn multiple behaviors simultaneously in situ and re-learn desired behaviors after being damaged or cut into new shapes. MNNs learn by tuning the stiffnesses of their constituent beams similar to how artificial neural networks (ANNs) learn by tuning their weights. In this work, we compare the performance of six algorithms (i.e., genetic algorithm, full pattern search, partial pattern search, interior point, sequential quadratic progression, and Nelder–Mead) applied to MNN leaning. A computational model was created to simulate MNN learning using these algorithms with experimentally measured noise included. A total of 3900 runs were simulated. The results were validated using experimentally collected data from a physical MNN. We identify algorithms like Nelder–Mead that are both fast and able to reject noise. Additionally, we provide insights into selecting learning algorithms based on the desired balance between accuracy and speed, as well as the general characteristics that are favorable for training MNNs. These insights will promote more efficient MNN learning and will provide a foundation for future algorithm development.

References

1.
LeCun
,
Y.
,
Bengio
,
Y.
, and
Hinton
,
G.
,
2015
, “
Deep Learning
,”
Nature
,
521
(
7553
), pp.
436
444
.
2.
Gao
,
H.
,
Cheng
,
B.
,
Wang
,
J.
,
Li
,
K.
,
Zhao
,
J.
, and
Li
,
D.
,
2018
, “
Object Classification Using CNN-Based Fusion of Vision and LIDAR in Autonomous Vehicle Environment
,”
IEEE Trans. Ind Inform.
,
14
(
9
), pp.
4224
4231
.
3.
Abiodun
,
O. I.
,
Jantan
,
A.
,
Omolara
,
A. E.
,
Dada
,
K. V.
,
Umar
,
A. M.
,
Linus
,
O. U.
,
Arshad
,
H.
,
Kazaure
,
A. A.
,
Gana
,
U.
, and
Kiru
,
M. U.
,
2019
, “
Comprehensive Review of Artificial Neural Network Applications to Pattern Recognition
,”
IEEE Access
,
7
(
1
), pp.
158820
158846
.
4.
McCulloch
,
W. S.
, and
Pitts
,
W.
,
1943
, “
A Logical Calculus of the Ideas Immanent in Nervous Activity
,”
Bull. Math. Biophys.
,
5
(
4
), pp.
115
133
.
5.
Rosenblatt
,
F.
,
1958
, “
The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain
,”
Psychol. Rev.
,
65
(
6
), pp.
386
408
.
6.
Hornik
,
K.
,
Stinchcombe
,
M.
, and
White
,
H.
,
1989
, “
Multilayer Feedforward Networks Are Universal Approximators
,”
Neural Netw.
,
2
(
5
), pp.
359
366
.
7.
Rumelhart
,
D. E.
,
Hinton
,
G. E.
, and
Williams
,
R. J.
,
1986
, “
Learning Representations by Back-Propagating Errors
,”
Nature
,
323
(
6088
), pp.
533
536
.
8.
Du
,
S.
,
Lee
,
J.
,
Li
,
H.
,
Wang
,
L.
, and
Zhai
,
X.
,
2019
, “
Gradient Descent Finds Global Minima of Deep Neural Networks
,”
Proceedings of the 36th International Conference on Machine Learning
,
Long Beach, CA
,
June 10
, PMLR, pp.
1675
1685
.
9.
Fernando
,
C.
, and
Sojakka
,
S.
,
2003
, “Pattern Recognition in a Bucket,”
Advances in Artificial Life
,
W.
Banzhaf
,
J.
Ziegler
,
T.
Christaller
,
P.
Dittrich
, and
J. T.
Kim
, eds.,
Springer
,
Berlin, Heidelberg
, pp.
588
597
.
10.
Lv
,
Z.
,
Liu
,
P.
, and
Pei
,
Y.
,
2020
, “
Temporal Acoustic Wave Computational Metamaterials
,”
Appl. Phys. Lett.
,
117
(
13
), p.
131902
.
11.
Zuo
,
S.
,
Wei
,
Q.
,
Tian
,
Y.
,
Cheng
,
Y.
, and
Liu
,
X.
,
2018
, “
Acoustic Analog Computing System Based on Labyrinthine Metasurfaces
,”
Sci. Rep.
,
8
(
1
), p.
10103
.
12.
Hughes
,
T. W.
,
Williamson
,
I. A. D.
,
Minkov
,
M.
, and
Fan
,
S.
, “
Wave Physics as an Analog Recurrent Neural Network
,”
Sci. Adv.
,
5
(
12
), p.
eaay6946
.
13.
Coulombe
,
J. C.
,
York
,
M. C. A.
, and
Sylvestre
,
J.
,
2017
, “
Computing With Networks of Nonlinear Mechanical Oscillators
,”
PLoS One
,
12
(
6
), p.
e0178663
.
14.
Wright
,
L. G.
,
Onodera
,
T.
,
Stein
,
M. M.
,
Wang
,
T.
,
Schachter
,
D. T.
,
Hu
,
Z.
, and
McMahon
,
P. L.
,
2022
, “
Deep Physical Neural Networks Trained With Backpropagation
,”
Nature
,
601
(
7894
), pp.
549
555
.
15.
Stern
,
M.
,
Arinze
,
C.
,
Perez
,
L.
,
Palmer
,
S. E.
, and
Murugan
,
A.
,
2020
, “
Supervised Learning Through Physical Changes in a Mechanical System
,”
Proc. Natl. Acad. Sci. U. S. A.
,
117
(
26
), pp.
14843
14850
.
16.
Furuhata
,
G.
,
Niiyama
,
T.
, and
Sunada
,
S.
,
2021
, “
Physical Deep Learning Based on Optimal Control of Dynamical Systems
,”
Phys. Rev. Appl.
,
15
(
3
), p.
034092
.
17.
Widrow
,
B.
,
Greenblatt
,
A.
,
Kim
,
Y.
, and
Park
,
D.
,
2013
, “
The No-Prop Algorithm: A New Learning Algorithm for Multilayer Neural Networks
,”
Neural Netw.
,
37
(
1
), pp.
182
188
.
18.
Boyd
,
S.
, and
Chua
,
L.
,
1985
, “
Fading Memory and the Problem of Approximating Nonlinear Operators With Volterra Series
,”
IEEE Trans. Circuits Syst.
,
32
(
11
), pp.
1150
1161
.
19.
Nakajima
,
K.
,
Hauser
,
H.
,
Li
,
T.
, and
Pfeifer
,
R.
,
2015
, “
Information Processing via Physical Soft Body
,”
Sci. Rep.
,
5
(
1
), p.
10487
.
20.
Hauser
,
H.
,
Ijspeert
,
A. J.
,
Füchslin
,
R. M.
,
Pfeifer
,
R.
, and
Maass
,
W.
,
2012
, “
The Role of Feedback in Morphological Computation With Compliant Bodies
,”
Biol Cybern
,
106
(
10
), pp.
595
613
.
21.
Stern
,
M.
,
Hexner
,
D.
,
Rocks
,
J. W.
, and
Liu
,
A. J.
,
2021
, “
Supervised Learning in Physical Networks: From Machine Learning to Learning Machines
,”
Phys. Rev. X
,
11
(
2
), p.
021045
.
22.
Dillavou
,
S.
,
Stern
,
M.
,
Liu
,
A. J.
, and
Durian
,
D. J.
,
2022
, “
Demonstration of Decentralized, Physics-Driven Learning
,”
Phys. Rev. Appl.
,
18
(
1
), p.
014040
.
23.
Kaveh
,
A.
, and
Bakhshpoori
,
T.
,
2019
,
Metaheuristics: Outlines, MATLAB Codes and Examples
,
Springer Nature
,
Switzerland AG
.
24.
Lee
,
R. H.
,
Mulder
,
E. A. B.
, and
Hopkins
,
J. B.
,
2022
, “
Mechanical Neural Networks: Architected Materials That Learn Behaviors
,”
Sci. Robot.
,
7
(
71
), p.
eabq7278
.
25.
Paterni
,
P.
,
Vitet
,
S.
,
Bena
,
M.
, and
Yokoyama
,
A.
,
1999
, “
Optimal Location of Phase Shifters in the French Network by Genetic Algorithm
,”
IEEE Trans. Power Syst.
,
14
(
1
), pp.
37
42
.
26.
Lambora
,
A.
,
Gupta
,
K.
, and
Chopra
,
K.
,
2019
, “
Genetic Algorithm—A Literature Review
,”
2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon)
,
Faridabad, Haryana, India
,
Feb. 14
, pp.
380
384
.
27.
Katoch
,
S.
,
Chauhan
,
S. S.
, and
Kumar
,
V.
,
2021
, “
A Review on Genetic Algorithm: Past, Present, and Future
,”
Multimed. Tools Appl.
,
80
(
5
), pp.
8091
8126
.
28.
Hooke
,
R.
, and
Jeeves
,
T. A.
,
1961
, “
“Direct Search” Solution of Numerical and Statistical Problems
,”
J. ACM
,
8
(
2
), pp.
212
229
.
29.
Güneş
,
F.
, and
Tokan
,
F.
,
2010
, “
Pattern Search Optimization With Applications on Synthesis of Linear Antenna Arrays
,”
Expert Syst. Appl.
,
37
(
6
), pp.
4698
4705
.
30.
Findler
,
N. V.
,
Lo
,
C.
, and
Lo
,
R.
,
1987
, “
Pattern Search for Optimization
,”
Math. Comput. Simul.
,
29
(
1
), pp.
41
50
.
31.
Lewis
,
R. M.
, and
Torczon
,
V.
,
2002
, “
A Globally Convergent Augmented Lagrangian Pattern Search Algorithm for Optimization With General Constraints and Simple Bounds
,”
SIAM J. Optim.
,
12
(
4
), pp.
1075
1089
.
32.
Torczon
,
V.
, and
Trosset
,
M. W.
,
1998
, “
From Evolutionary Operation to Parallel Direct Search: Pattern Search Algorithms for Numerical Optimization
,”
Comput. Sci. Stat.
,
29
(
1
), pp.
396
401
.
33.
MathWorks
, “
How Pattern Search Polling Works—MATLAB & Simulink
,” https://www.mathworks.com/help/gads/how-pattern-search-polling-works.html, Accessed October 12, 2022.
34.
Jarre
,
F.
,
Kocvara
,
M.
, and
Zowe
,
J.
,
1998
, “
Optimal Truss Design by Interior-Point Methods
,”
SIAM J. Optim.
,
8
(
4
), pp.
1084
1107
.
35.
Fiacco
,
A. V.
, and
McCormick
,
G. P.
,
1990
,
Nonlinear Programming: Sequential Unconstrained Minimization Techniques
,
SIAM
,
Philadelphia, PA
.
36.
Bussotti
,
P.
,
2003
, “
On the Genesis of the Lagrange Multipliers
,”
J. Optim. Theory Appl.
,
117
(
3
), pp.
453
459
.
37.
Montoya
,
O. D.
,
Gil-González
,
W.
, and
Garces
,
A.
,
2019
, “
Sequential Quadratic Programming Models for Solving the OPF Problem in DC Grids
,”
Electr. Power Syst. Res.
,
169
(
4
), pp.
18
23
.
38.
Vanderbei
,
R. J.
, and
Shanno
,
D. F.
,
1999
, “
An Interior-Point Algorithm for Nonconvex Nonlinear Programming
,”
Comput. Optim. Appl.
,
13
(
1
), pp.
231
252
.
39.
MathWorks
, “
Constrained Nonlinear Optimization Algorithms—MATLAB & Simulink
,” https://www.mathworks.com/help/optim/ug/constrained-nonlinear-optimization-algorithms.html, Accessed October 12, 2022.
40.
Hager
,
W. W.
,
1999
, “Stabilized Sequential Quadratic Programming,”
Computational Optimization
,
J.-S.
Pang
, ed.,
Springer US
,
Boston, MA
, pp.
253
273
.
41.
Nelder
,
J. A.
, and
Mead
,
R.
,
1965
, “
A Simplex Method for Function Minimization
,”
Comput. J.
,
7
(
4
), pp.
308
313
.
42.
Olsson
,
D. M.
, and
Nelson
,
L. S.
,
1975
, “
The Nelder-Mead Simplex Procedure for Function Minimization
,”
Technometrics
,
17
(
1
), pp.
45
51
.
43.
Marandi
,
P. J.
,
Mansooriazdeh
,
M.
, and
Charkari
,
N. M.
,
2008
, “The Effect of Re-Sampling on Incremental Nelder-Mead Simplex Algorithm: Distributed Regression in Wireless Sensor Networks,”
Wireless Algorithms, Systems, and Applications
,
Y.
Li
,
D. T.
Huynh
,
S. K.
Das
, and
D.-Z.
Du
, eds.,
Springer
,
Berlin, Heidelberg
, pp.
420
431
.
44.
MathWorks
, “
Fminsearchbnd, Fminsearchcon
,” https://www.mathworks.com/matlabcentral/fileexchange/8277-fminsearchbnd-fminsearchcon, Accessed December 12, 2022.
You do not currently have access to this content.