This tutorial paper presents the expositions of stochastic optimal feedback control theory and Bayesian spatiotemporal models in the context of robotics applications. The presented material is self-contained so that readers can grasp the most important concepts and acquire knowledge needed to jump-start their research. To facilitate this, we provide a series of educational examples from robotics and mobile sensor networks.

References

References
1.
Bertsekas
,
D. P.
,
1995
,
Dynamic Programming and Optimal Control
, Vol.
1
,
Athena Scientific
,
Belmont
, MA.
2.
Bryson
,
A. E.
, and
Ho
,
Y.-C.
,
1975
,
Applied Optimal Control: Optimization, Estimation, and Control
,
Taylor & Francis
, New York.
3.
Pontryagin
,
L. S.
,
1962
,
The Mathematical Theory of Optimal Processes
, Vol.
4
,
Interscience Publishers, New York.
4.
Young
,
J.
, and
Zhou
,
X. Y.
,
1999
,
Stochastic Controls: Hamiltonian Systems and HJB Equations
, Vol.
43
,
Springer
, New York.
5.
Kushner
,
H. J.
, and
Dupuis
,
P.
,
2001
,
Numerical Methods for Stochastic Control Problems in Continuous Time
, Vol.
24
,
Springer
, New York.10.1007/978-1-4613-0007-6
6.
Stengel
,
R. F.
,
1986
,
Optimal Control and Estimation
,
Dover
, New York.
7.
Oksendal
,
B.
,
2003
,
Stochastic Differential Equations: An Introduction With Applications
,
Springer Verlag
,
New York
.
8.
Davis
,
M. H.
,
1993
,
Markov Models and Optimization
, Vol.
49
,
Chapman & Hall/CRC, Boca Raton, FL
.
9.
Koutsoukos
,
X. D.
,
2004
, “
Optimal Control of Stochastic Hybrid Systems Based on Locally Consistent Markov Decision Processes
,” Proceedings of the 2005 IEEE International Symposium on Intelligent Control, Limassol, Cyprus, pp. 435–440.
10.
Fleming
,
W. H.
, and
Soner
,
H. M.
,
2006
,
Controlled Markov Processes and Viscosity Solutions
, Vol.
25
,
Springer
,
New York.
11.
Engel
,
Y.
,
Mannor
,
S.
, and
Meir
,
R.
,
2005
, “
Reinforcement Learning With Gaussian Processes
,”
Proceedings of the 22nd International Conference on Machine Learning
, University of Bonn, Germany, ACM, pp.
201
208
.10.1145/1102351.1102377
12.
Engel
,
Y.
,
Mannor
,
S.
, and
Meir
,
R.
,
2003
, “
Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning
,” Proceedings of the 20th International Conference on Machine Learning (ICML-2003), Washington, DC, pp.
154
161
.
13.
Deisenroth
,
M. P.
,
Rasmussen
,
C. E.
, and
Peters
,
J.
,
2009
, “
Gaussian Process Dynamic Programming
,”
Neurocomputing
,
72
(
7
), pp.
1508
1524
.10.1016/j.neucom.2008.12.019
14.
Leonard
,
N. E.
,
Paley
,
D. A.
,
Lekien
,
F.
,
Sepulchre
,
R.
,
Fratantoni
,
D. M.
, and
Davis
,
R.
,
2007
, “
Collective Motion, Sensor Networks, and Ocean Sampling
,”
Proc. IEEE
,
95
(
1
), pp.
48
74
.10.1109/JPROC.2006.887295
15.
Lynch
,
K. M.
,
Schwartz
,
I. B.
,
Yang
,
P.
, and
Freeman
,
R. A.
,
2008
, “
Decentralized Environmental Modeling by Mobile Sensor Networks
,”
IEEE Trans. Rob.
,
24
(
3
), pp.
710
724
.10.1109/TRO.2008.921567
16.
Choi
,
J.
,
Oh
,
S.
, and
Horowitz
,
R.
,
2009
, “
Distributed Learning and Cooperative Control for Multi-Agent Systems
,”
Automatica
,
45
(
12
), pp.
2802
2814
.10.1016/j.automatica.2009.09.025
17.
Xu
,
Y.
,
Choi
,
J.
, and
Oh
,
S.
,
2011
, “
Mobile Sensor Network Navigation Using Gaussian Processes With Truncated Observations
,”
IEEE Trans. Rob.
,
27
(
6
), pp.
1118
1131
.10.1109/TRO.2011.2162766
18.
Xu
,
Y.
,
Choi
,
J.
,
Dass
,
S.
, and
Maiti
,
T.
,
2013
, “
Efficient Bayesian Spatial Prediction With Mobile Sensor Networks Using Gaussian Markov Random Fields
,”
Automatica
,
49
(
12
), pp.
3520
3530
.10.1016/j.automatica.2013.09.008
19.
Le Ny
,
J.
, and
Pappas
,
G.
,
2013
, “
Adaptive Deployment of Mobile Robotic Networks
,”
IEEE Trans. Autom. Control
,
58
(
3
), pp.
654
666
.10.1109/TAC.2012.2215512
20.
Jadaliha
,
M.
, and
Choi
,
J.
,
2013
, “
Environmental Monitoring Using Autonomous Aquatic Robots: Sampling Algorithms and Experiments
,”
IEEE Trans. Control Syst. Technol.
,
21
(
3
), pp.
899
905
.10.1109/TCST.2012.2190070
21.
Cao
,
Y.
,
Yu
,
W.
,
Ren
,
W.
, and
Chen
,
G.
,
2013
, “
An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination
,”
IEEE Trans. Ind. Inf.
,
9
(
1
), pp.
427
438
.10.1109/TII.2012.2219061
22.
Kalnay
,
E.
,
2003
,
Atmospheric Modeling, Data Assimilation, and Predictability
,
Cambridge University
, New York.10.1017/CBO9780511802270
23.
Cortés
,
J.
,
2009
, “
Distributed Kriged Kalman Filter for Spatial Estimation
,”
IEEE Trans. Autom. Control
,
54
(
12
), pp.
2816
2827
.10.1109/TAC.2009.2034192
24.
Graham
,
R.
, and
Cortés
,
J.
,
2009
, “
Cooperative Adaptive Sampling of Random Fields With Partially Known Co-Variance
,”
Int. J. Rob. Nonlinear Control
,
22
(5), pp.
504
534
.
25.
Graham
,
R.
, and
Cortés
,
J.
,
2012
, “
Adaptive Information Collection by Robotic Sensor Networks for Spatial Estimation
,”
IEEE Trans. Autom. Control
,
57
(
6
), pp.
1404
1419
.10.1109/TAC.2011.2178332
26.
Xu
,
Y.
,
Choi
,
J.
,
Dass
,
S.
, and
Maiti
,
T.
,
2012
, “
Sequential Bayesian Prediction and Adaptive Sampling Algorithms for Mobile Sensor Networks
,”
IEEE Trans. Autom. Control
,
57
(
8
), pp.
2078
2084
.10.1109/TAC.2012.2190181
27.
Xu
,
Y.
, and
Choi
,
J.
,
2012
, “
Spatial Prediction With Mobile Sensor Networks Using Gaussian Processes With Built-In Gaussian Markov Random Fields
,”
Automatica
,
48
(
8
), pp.
1735
1740
.10.1016/j.automatica.2012.05.029
28.
Xu
,
Y.
, and
Choi
,
J.
,
2012
, “
Stochastic Adaptive Sampling for Mobile Sensor Networks Using Kernel Regression
,”
Int. J. Control Autom. Syst.
,
10
(
4
), pp.
778
786
.10.1007/s12555-012-0414-5
29.
Varagnolo
,
D.
,
Pillonetto
,
G.
, and
Schenato
,
L.
,
2012
, “
Distributed Parametric and Nonparametric Regression With On-Line Performance Bounds Computation
,”
Automatica
,
48
(
10
), pp.
2468
2481
.10.1016/j.automatica.2012.06.080
30.
Samson
,
C.
, and
Ait-Abderrahim
,
K.
,
1990
, “
Mobile Robot Control, Part 1: Feedback Control of a Nonholo-Nomic Wheeled Cart in Cartesian Space
,” Institut National de Recherche en Informatique et en Automatique, Report 1288, Le Chesnay, France.
31.
Aicardi
,
M.
,
Casalino
,
G.
,
Bicchi
,
A.
, and
Balestrino
,
A.
,
1995
, “
Closed Loop Steering of Unicycle Like Vehicles via Lyapunov Techniques
,”
IEEE Rob. Autom. Mag.
,
2
(
1
), pp.
27
35
.10.1109/100.388294
32.
Ren
,
W.
, and
Beard
,
R.
,
2004
, “
Trajectory Tracking for Unmanned Air Vehicles With Velocity and Heading Rate Constraints
,”
IEEE Trans. Control Syst. Technol.
,
12
(
5
), pp.
706
716
.10.1109/TCST.2004.826956
33.
Anderson
,
R.
,
Bakolas
,
E.
,
Milutinović
,
D.
, and
Tsio-tras
,
P.
,
2013
, “
Optimal Feedback Guidance of a Small Aerial Vehicle in a Stochastic Wind
,”
J. Guidance Control Dyn.
,
36
(
4
), pp.
975
985
.10.2514/1.59512
34.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
1998
,
Reinforcement Learning: An Introduction
, Vol.
1
,
The MIT Press, Cambridge, MA
.
35.
Powell
,
W. B.
,
2007
,
Approximate Dynamic Programming: Solving the Curses of Dimensionality
, Vol.
703
,
John Wiley & Sons
, Hoboken, NJ.10.1002/9781118029176
36.
Long
,
A. W.
,
Wolfe
,
K. C.
,
Mashner
,
M. J.
, and
Chirikjian
,
G. S.
,
2012
, “
The Banana Distribution is Gaussian: A Localization Study With Exponential Coordinates
,”
Proceedings of Robotics: Science and Systems
,
Syndey, Australia
, pp. 265–272.
37.
Thrun
,
S.
,
Burgard
,
W.
, and
Fox
,
D.
,
2005
,
Probabilistic Robotics
,
The MIT Press, Cambridge
, MA.
38.
Anderson
,
R. P.
, and
Milutinović
,
D.
,
2011
, “
A Stochastic Approach to Dubins Feedback Control for Target Tracking
,”
Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
San Francisco, CA
, pp. 3917–3922.
39.
Jadaliha
,
M.
,
Lee
,
J.
, and
Choi
,
J.
,
2012
, “
Adaptive Control of Multiagent Systems for Finding Peaks of Uncertain Static Fields
,”
J. Dyn. Syst. Meas. Contr.
,
134
(
5
), p.
051007
.10.1115/1.4006369
40.
Cressie
,
N.
,
1986
, “
Kriging Nonstationary Data
,”
J. Am. Stat. Assoc.
,
81
(
395
), pp.
625
634
.10.1080/01621459.1986.10478315
41.
Rasmussen
,
C. E.
, and
Williams
,
C. K. I.
,
2006
,
Gaussian Processes for Machine Learning
,
The MIT Press, Cambridge
,
MA
.
42.
Choi
,
J.
,
Lee
,
J.
, and
Oh
,
S.
,
2008
, “
Biologically-Inspired Navigation Strategies for Swarm Intelligence Using Spatial Gaussian Processes
,”
Proceedings of the 17th International Federation of Automatic Control (IFAC) World Congress
, Seoul, Korea.
43.
Choi
,
J.
,
Lee
,
J.
, and
Oh
,
S.
,
2008
, “
Swarm Intelligence for Achieving the Global Maximum Using Spatio-Temporal Gaussian Processes
,”
Proceedings of the 27th American Control Conference (ACC)
, Seattle, WA, pp. 135–140.
44.
Shi
,
J. Q.
, and
Choi
,
T.
,
2011
,
Gaussian Process Regression Analysis for Functional Data
,
CRC
, Boca Raton, FL.
45.
Abrahamsen
,
P.
,
1997
,
A Review of Gaussian Random Fields and Correlation Functions
,
Norsk Regnesentral/Norwegian Computing Center
, Oslo, Norway.
46.
Snelson
,
E. L.
,
2007
, “
Flexible and Efficient Gaussian Process Models for Machine Learning
,” Ph.D. thesis, University College, London, UK.
47.
Zhang
,
B.
, and
Sukhatme
,
G.
,
2007
, “
Adaptive Sampling for Estimating a Scalar Field Using a Robotic Boat and a Sensor Network
,” 2007
IEEE
International Conference on Robotics and Automation, Rome, Italy, Apr. 10–14, pp.
3673
3680
.10.1109/ROBOT.2007.364041
48.
Laut
,
J.
,
Henry
,
E.
,
Nov
,
O.
, and
Porfiri
,
M.
,
2014
, “
Development of a Mechatronics-Based Citizen Science Platform for Aquatic Environmental Monitoring
,”
IEEE/ASME Trans. Mechatron.
,
19
(
5
), pp.
1541
1551
.10.1109/TMECH.2013.2287705
49.
Xu
,
Y.
, and
Choi
,
J.
,
2011
, “
Adaptive Sampling for Learning Gaussian Processes Using Mobile Sensor Networks
,”
Sensors
,
11
(
3
), pp.
3051
3066
.10.3390/s110303051
50.
Bishop
,
C. M.
,
2006
,
Pattern Recognition and Machine Learning
,
Springer
,
New York.
51.
Xu
,
Y.
,
Choi
,
J.
,
Dass
,
S.
, and
Maiti
,
T.
,
2011
, “
Bayesian Prediction and Adaptive Sampling Algorithms for Mobile Sensor Networks
,”
Proceedings of the 2011 American Control Conference (ACC)
, San Francisco, CA, pp.
4095
4200
.
52.
Xu
,
Y.
,
2011
, “
Environmental Adaptive Sampling for Mobile Sensor networks using Gaussian processes
,” Ph.D. thesis, Michigan State University, East Lansing, MI.
53.
Gaudard
,
M.
,
Karson
,
M.
,
Linder
,
E.
, and
Sinha
,
D.
,
1999
, “
Bayesian Spatial Prediction
,”
Environ. Ecol. Stat.
,
6
(
2
), pp.
147
171
.10.1023/A:1009614003692
54.
Rue
,
H.
, and
Tjelmeland
,
H.
,
2002
, “
Fitting Gaussian Markov Random Fields to Gaussian Fields
,”
Scand. J. Stat.
,
29
(
1
), pp.
31
49
.10.1111/1467-9469.00058
55.
Cressie
,
N.
, and
Verzelen
,
N.
,
2008
, “
Conditional-Mean Least-Squares Fitting of Gaussian Markov Random Fields to Gaussian Fields
,”
Comput. Stat. Data Anal.
,
52
(
5
), pp.
2794
2807
.10.1016/j.csda.2007.10.008
56.
Hartman
,
L.
, and
Hössjer
,
O.
,
2008
, “
Fast Kriging of Large Data Sets With Gaussian Markov Random Fields
,”
Comput. Stat. Data Anal.
,
52
(
5
), pp.
2331
2349
.10.1016/j.csda.2007.09.018
57.
Le Ny
,
J.
, and
Pappas
,
G. J.
,
2009
, “
On Trajectory Optimization for Active Sensing in Gaussian Process Models
,”
Proceedings of the 48th IEEE Conference on Decision and Control
, Shanghai, China, pp.
6286
6292
.
58.
Rasmussen
,
C. E.
, and
Nickisch
,
H.
,
2010
, “
Gaussian Processes for Machine Learning (GPML) Toolbox
,”
J. Mach. Learn. Res.
,
11
, pp.
3011
3015
.
59.
Minka
,
T. P.
,
2001
, “
Expectation Propagation for Approximate Bayesian Inference
,”.
Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence
,
Morgan Kaufmann
, pp.
362
369
.
60.
Williams
,
C. K.
, and
Barber
,
D.
,
1998
, “
Bayesian Classification With Gaussian Processes
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
20
(
12
), pp.
1342
1351
.10.1109/34.735807
You do not currently have access to this content.