This paper describes a new learning algorithm for time-varying recurrent neural networks whose weights are functions of time instead of scalars. First, an objective functional that is a function of the weight functions quantifying the discrepancies between the desired outputs and the network’s outputs is formulated. Then, dynamical optimization is used to derive the necessary conditions for the extreme of the functional. These necessary conditions result in a two-point boundary-value problem. This two-point boundary-value problem is subsequently solved by the Hilbert function space BFGS quasi-Newton algorithm, which is obtained by using the dyadic operator to extend the Euclidean space BFGS method into an infinite-dimensional, real Hilbert space. Finally, the ability of the network and the learning algorithm is demonstrated in the identification of three simulated nonlinear systems and a resistance spot welding process.

1.
Broyden
C. G.
,
1970
, “
The Convergence of a Class of Double Rank Minimization Algorithm. Paris I and II
,”
Journal Institute of Mathematics and Its Application
, Vol.
6
, pp.
76
90
.
2.
Chen
F. C.
,
1990
, “
Backpropagation Neural Networks for Nonlinear Self-Turning Adaptive Control
,”
IEEE Control Magazine
, Vol.
10
, No.
3
, pp.
44
48
.
3.
Edge
E. R.
, and
Powers
W. F.
,
1976
, “
Function-Space Quasi-Newton Algorithms for Optimal Control Problems with Bounded Controls and Singular Arcs
,”
Journal of Optimization and Appreciation
, Vol.
20
, No.
4
, pp.
455
477
.
4.
Fletcher
R.
,
1970
, “
A New Approach to Variable Metric Algorithms
,”
Computer Journal
, Vol.
8
, pp.
317
322
.
5.
Fortmann, T. E., and Hitz, K. L., 1977, “The Kalman-Bucy Filter in An Introduction to Linear Control Systems,” Marcel Dekker, NY and Basel, pp. 550–561.
6.
Goldfarb
D.
,
1970
, “
A Family of Variable Metric Methods Derived by Variational Means
,”
Mathematics of Computation
, Vol.
24
, pp.
23
26
.
7.
Hopfield
J. J.
,
1982
, “
Neural Networks and Physical Systems with Emergent Collective Computational Abilities
,”
Proceedings National Academy Science
, Vol.
79
, pp.
2554
2558
.
8.
Hopfield
J. J.
,
1984
, “
Neurons with Graded Response Have Collective Computation Properties Abilities
,”
Proceedings National Academy Science
, Vol.
8
, pp.
3088
3092
.
9.
Jou, M., Messler, R. W. Jr., and Li, C. J., 1994, “A Fuzzy Logic Control System for Resistance Spot Welding Based on a Neural Network Model,” Proc. of Sheet Metal Welding Conference VI, American Welding Society, Novi, MI, Oct. 12–14.
10.
Kirk, D. E., 1970, Optimal Control Theory, Prentice-Hall, NJ.
11.
Lasdon
L. S.
,
1970
, “
Conjugate Direction Methods for Optimal Control
,”
IEEE Transitions on Automatic Control
, Vol.
AC-15
, No.
2
, pp.
267
268
.
12.
Li, J. C., and Yan, L., 1992, “Recurrent Neural Network Learning Based on Quasi-Newton and Quasi-Newton With Scaling Methods,” Proceedings of Symposium on Neural Networks in Manufacturing and Robotics, ASME, WAM, pp. 53–64.
13.
Mayorga
R. V.
, and
Quintana
V. H.
,
1980
, “
A Family of Variable Metric Methods in Function Space, Without Exact Line Searches
,”
Journal of Optimization and Application
, Vol.
31
, No.
3
, pp.
303
329
.
14.
Pearlmutter, B. A., 1989, “Learning State Space Trajectories in Recurrent Neural Networks,” International Joint Conference on Neural Networks, II, pp. 365–372.
15.
Pineda
F. J.
,
1989
, “
Recurrent Backpropagation and the Dynamical Approach to Adaptive Neural Computation
,”
Neural Computation
, Vol.
1
, pp.
161
172
.
16.
Pontryagin, L. S., Boltyanskii, V., Gamkrelidze, R., and Mishchenko, E., 1962, The Mathematical Theory of Optimal Processes, Interscience Publishers, NY.
17.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J., 1986, “Learning Internal Representations by Error Propagation,” D. E. Rumelhart and J. L. McClelland,” (Eds.), Parallel Distributed Processing I, MIT Press, Cambridge.
18.
Sato
M. A.
,
1990
, “
A Real Time Learning Algorithm for Recurrent Analog Neural Networks
,”
Biological Cybernetics
, Vol.
62
, pp.
237
241
.
19.
Sayama
K. Oi
, and
Takamatsu
T.
,
1973
, “
Computational Schemes of the Davidon-Flectcher-Powell Method in Infinite-Dimensional Space
,”
Journal of Optimization and Application
, Vol.
12
, No.
5
, pp.
447
458
.
20.
Shanno
D. F.
,
1970
, “
Conditioning of Quasi-Newton Methods for Function Minimization
,”
Mathematics of Computation
, Vol.
24
, pp.
647
656
.
21.
Werbos
P.
,
1988
, “
Generalization of Backpropagation with Application to a Recurrent Gas Market Model
,”
Neural Networks
, Vol.
1
, pp.
339
356
.
22.
Williams, R. J., 1990, “Adaptive State Representation and Estimation Using Recurrent Connectionist Networks,” W. T. Miller, III, R. S. Sutton, and P. J. Werbos, eds. Neural Networks for Control, MIT press, Cambridge.
23.
Williams
R. J.
, and
Zipser
D.
,
1989
, “
A Learning Algorithm for Continually Running Fully Recurrent Networks
,”
Neural Computation
, Vol.
1
, pp.
270
280
.
This content is only available via PDF.
You do not currently have access to this content.