This paper makes use of long short-term memory (LSTM) neural networks for forecasting probability distributions of time series in terms of discrete symbols that are quantized from real-valued data. The developed framework formulates the forecasting problem into a probabilistic paradigm as hΘ: X × Y → [0, 1] such that yYhΘ(x,y)=1, where X is the finite-dimensional state space, Y is the symbol alphabet, and Θ is the set of model parameters. The proposed method is different from standard formulations (e.g., autoregressive moving average (ARMA)) of time series modeling. The main advantage of formulating the problem in the symbolic setting is that density predictions are obtained without any significantly restrictive assumptions (e.g., second-order statistics). The efficacy of the proposed method has been demonstrated by forecasting probability distributions on chaotic time series data collected from a laboratory-scale experimental apparatus. Three neural architectures are compared, each with 100 different combinations of symbol-alphabet size and forecast length, resulting in a comprehensive evaluation of their relative performances.

References

References
1.
Montgomery
,
D. C.
,
Jennings
,
C. L.
, and
Kulahci
,
M.
,
2015
,
Introduction to Time Series Analysis and Forecasting
,
Wiley
, Hoboken, NJ.
2.
Hauser
,
M.
,
Fu
,
Y.
,
Li
,
Y.
, and
Ray
,
A.
,
2017
, “
Probabilistic Forecasting of Symbol Sequences With Deep Neural Networks
,”
American Control Conference
(
ACC
), Seattle, WA, May 24–26, pp.
3147
3152
.
3.
Zhang
,
G.
,
Patuwo
,
B. E.
, and
Hu
,
M. Y.
,
1998
, “
Forecasting With Artificial Neural Networks: The State of the Art
,”
Int. J. Forecasting
,
14
(
1
), pp.
35
62
.
4.
Hornik
,
K.
,
Stinchcombe
,
M.
, and
White
,
H.
,
1989
, “
Multilayer Feedforward Networks are Universal Approximators
,”
Neural Networks
,
2
(
5
), pp.
359
366
.
5.
Gneiting
,
T.
,
2008
, “
Editorial: Probabilistic Forecasting
,”
J. R. Stat. Soc. Ser. A
, 171(
2
), pp.
319
321
.
6.
Gneiting
,
T.
, and
Katzfuss
,
M.
,
2014
, “
Probabilistic Forecasting
,”
Annu. Rev. Stat. Appl.
,
1
(
1
), pp.
125
151
.
7.
Box
,
G. E.
,
Jenkins
,
G. M.
,
Reinsel
,
G. C.
, and
Ljung
,
G. M.
,
2015
,
Time Series Analysis: Forecasting and Control
,
Wiley
, Hoboken, NJ.
8.
Dupont
,
P.
,
Denis
,
F.
, and
Esposito
,
Y.
,
2005
, “
Links Between Probabilistic Automata and Hidden Markov Models, Probability Distributions, Learning Models and Induction Algorithms
,”
Pattern Recognit.
,
38
(
9
), pp.
1349
1371
.
9.
Rozenberg
,
G.
, and
Salomaa
,
A.
,
1997
,
Handbook of Formal Languages: Beyonds Words
, Vol.
3
,
Springer Science & Business Media
, Berlin.
10.
Wen
,
Y.
,
Mukherjee
,
K.
, and
Ray
,
A.
,
2013
, “
Adaptive Pattern Classification for Symbolic Dynamic Systems
,”
Signal Process.
,
93
(
1
), pp.
252
260
.
11.
Ray
,
A.
,
2004
, “
Symbolic Dynamic Analysis of Complex Systems for Anomaly Detection
,”
Signal Process.
,
84
(
7
), pp.
1115
1130
.
12.
Mukherjee
,
K.
, and
Ray
,
A.
,
2014
, “
State Splitting and Merging in Probabilistic Finite State Automata for Signal Representation and Analysis
,”
Signal Process.
,
104
, pp.
105
119
.
13.
Darema
,
F.
,
2005
, “
Dynamic Data Driven Applications Systems: New Capabilities for Application Simulations and Measurements
,”
Fifth International Conference on Computational Science
(
ICCS
), Atlanta, GA, May 22–25, pp.
610
615
.
14.
Hochreiter
,
S.
, and
Schmidhuber
,
J.
,
1997
, “
Long Short-Term Memory
,”
Neural Comput.
,
9
(
8
), pp.
1735
1780
.
15.
Graves
,
A.
,
2012
, “
Supervised Sequence Labelling
,”
Supervised Sequence Labelling With Recurrent Neural Networks
,
Springer
,
New York
, pp.
5
13
.
16.
Gers
,
F. A.
,
Schmidhuber
,
J.
, and
Cummins
,
F.
,
2000
, “
Learning to Forget: Continual Prediction With LSTM
,”
Neural Comput.
,
12
(
10
), pp.
2451
2471
.
17.
Li
,
Y.
,
Chattopadhyay
,
P.
, and
Ray
,
A.
,
2015
, “
Dynamic Data-Driven Identification of Battery State-of-Charge Via Symbolic Analysis of Input–Output Pairs
,”
Appl. Energy
,
155
, pp.
778
790
.
18.
Hauser
,
M.
,
Li
,
Y.
,
Li
,
J.
, and
Ray
,
A.
,
2016
, “
Real-Time Combustion State Identification Via Image Processing: A Dynamic Data-Driven Approach
,”
American Control Conference
(
ACC
), Boston, MA, July 6–8, pp.
3316
3321
.
19.
Abarbanel
,
H.
,
2012
,
Analysis of Observed Chaotic Data
,
Springer Science & Business Media
,
New York
.
20.
Cover
,
T. M.
, and
Thomas
,
J. A.
,
2012
,
Elements of Information Theory
,
Wiley
,
Hoboken, NJ
.
21.
Nasr
,
G. E.
,
Badr
,
E.
, and
Joun
,
C.
,
2002
, “
Cross Entropy Error Function in Neural Networks: Forecasting Gasoline Demand
,”
Fifteenth International Florida Artificial Intelligence Research Society Conference
(
FLAIRS
), Pensacola, FL, May 14–16, pp.
381
384
.
22.
Zeiler
,
M. D.
,
2012
, “Adadelta: An Adaptive Learning Rate Method,” preprint
arXiv:1212.5701
.
23.
Duchi
,
J.
,
Hazan
,
E.
, and
Singer
,
Y.
,
2011
, “
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
,”
J. Mach. Learn. Res.
,
12
, pp.
2121
2159
.
24.
Bastien
,
F.
,
Lamblin
,
P.
,
Pascanu
,
R.
,
Bergstra
,
J.
,
Goodfellow
,
I.
,
Bergeron
,
A.
,
Bouchard
,
N.
,
Warde-Farley
,
D.
, and
Bengio
,
Y.
,
2012
, “Theano: New Features and Speed Improvements,” preprint
arXiv:1211.5590
.
25.
Bergstra
,
J.
,
Breuleux
,
O.
,
Bastien
,
F.
,
Lamblin
,
P.
,
Pascanu
,
R.
,
Desjardins
,
G.
,
Turian
,
J.
,
Warde-Farley
,
D.
, and
Bengio
,
Y.
,
2010
, “
Theano: A CPU and GPU Math Compiler in Python
,”
Ninth Python in Science Conference
, Austin, TX, June 28–July 3, pp.
1
7
.
26.
Theano Development Team
,
2016
, “Theano: A Python Framework For Fast Computation Of Mathematical Expressions,” e-print
arXiv:1605.02688
.
27.
Sarkar
,
S.
,
Chakravarthy
,
S.
,
Ramanan
,
V.
, and
Ray
,
A.
,
2016
, “
Dynamic Data-Driven Prediction of Instability in a Swirl-Stabilized Combustor
,”
Int. J. Spray Combust.
,
8
(
4
), pp.
235
253
.
28.
Graben
,
P. B.
,
2001
, “
Estimating and Improving the Signal-to-Noise Ratio of Time Series by Symbolic Dynamics
,”
Phys. Rev. E
,
64
(
5
), p.
051104
.
29.
Thompson
,
J.
, and
Stewart
,
H.
,
1986
,
Nonlinear Dynamics and Chaos
,
Wiley
,
Chichester, UK
.
30.
Cheng
,
L.
,
Liu
,
W.
,
Hou
,
Z.-G.
, and
Yu
,
J.
,
2015
, “
Neural Network Based Nonlinear Model Predictive Control for Piezoelectric Actuators
,”
IEEE Trans. Ind. Electron.
,
62
(
12
), pp.
7717
7727
.
You do not currently have access to this content.