This paper describes a stochastic predictive control algorithm for partially observable Markov decision processes (POMDPs) with time-joint chance constraints. We first present the algorithm as a general tool to treat finite space POMDP problems with time-joint chance constraints together with its theoretical properties. We then discuss its application to autonomous vehicle control on highways. In particular, we model decision-making/behavior-planning for an autonomous vehicle accounting for safety in a dynamic and uncertain environment as a constrained POMDP problem and solve it using the proposed algorithm. After behavior is planned, we use nonlinear model predictive control (MPC) to execute the behavior commands generated from the planner. This two-layer control framework is shown to be effective by simulations.

References

References
1.
Anderson
,
J. M.
,
Nidhi
,
K.
,
Stanley
,
K. D.
,
Sorensen
,
P.
,
Samaras
,
C.
, and
Oluwatola
,
O. A.
,
2014
,
Autonomous Vehicle Technology: A Guide for Policymakers
,
Rand Corporation
,
Santa Monica, CA
.
2.
Shladover
,
S. E.
,
Desoer
,
C. A.
,
Hedrick
,
J. K.
,
Tomizuka
,
M.
,
Walrand
,
J.
,
Zhang
,
W.-B.
,
McMahon
,
D. H.
,
Peng
,
H.
,
Sheikholeslam
,
S.
, and
McKeown
,
N.
,
1991
, “
Automated Vehicle Control Developments in the Path Program
,”
IEEE Trans. Veh. Technol.
,
40
(
1
), pp.
114
130
.
3.
Buehler
,
M.
,
Iagnemma
,
K.
, and
Singh
,
S.
,
2009
,
The DARPA Urban Challenge: Autonomous Vehicles in City Traffic
, Vol.
56
,
Springer
,
Berlin
.
4.
Lefèvre
,
S.
,
Vasquez
,
D.
, and
Laugier
,
C.
,
2014
, “
A Survey on Motion Prediction and Risk Assessment for Intelligent Vehicles
,”
Robomech J.
,
1
(
1
), p.
1
.
5.
Li
,
N.
,
Oyler
,
D. W.
,
Zhang
,
M.
,
Yildiz
,
Y.
,
Kolmanovsky
,
I.
, and
Girard
,
A. R.
,
2018
, “
Game Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems
,”
IEEE Trans. Control Syst. Technol.
,
26
(
5
), pp.
1782
1797
.
6.
Carvalho
,
A.
,
Lefévre
,
S.
,
Schildbach
,
G.
,
Kong
,
J.
, and
Borrelli
,
F.
,
2015
, “
Automated Driving: The Role of Forecasts and Uncertainty—A Control Perspective
,”
Eur. J. Control
,
24
, pp.
14
32
.
7.
Hrovat
,
D.
,
Di Cairano
,
S.
,
Tseng
,
H. E.
, and
Kolmanovsky
,
I. V.
,
2012
, “
The Development of Model Predictive Control in Automotive Industry: A Survey
,”
IEEE International Conference on Control Applications
(
CCA
),
Dubrovnik, Croatia
,
Oct. 3–5
, pp.
295
302
.
8.
Gray
,
A.
,
Ali
,
M.
,
Gao
,
Y.
,
Hedrick
,
J. K.
, and
Borrelli
,
F.
,
2012
, “
Integrated Threat Assessment and Control Design for Roadway Departure Avoidance
,”
15th International IEEE Conference on Intelligent Transportation Systems
(
ITSC
),
Anchorage, AK
,
Sept. 16–19
, pp.
1714
1719
.
9.
Gray
,
A.
,
Ali
,
M.
,
Gao
,
Y.
,
Hedrick
,
J.
, and
Borrelli
,
F.
,
2012
, “
Semi-Autonomous Vehicle Control for Road Departure and Obstacle Avoidance
,”
13th IFAC Symposium on Control in Transportation Systems, Sofia, Bulgaria
, Sept. 12–14, pp.
1
6
.https://pdfs.semanticscholar.org/a628/e50e4ab0b84a0a5c01e46eee519f41865814.pdf
10.
Gray
,
A.
,
Gao
,
Y.
,
Hedrick
,
J. K.
, and
Borrelli
,
F.
,
2013
, “
Robust Predictive Control for Semi-Autonomous Vehicles With an Uncertain Driver Model
,”
IEEE Intelligent Vehicles Symposium
(
IV
),
Gold Coast, Australia
,
June 23–26
, pp.
208
213
.
11.
Falcone
,
P.
,
Borrelli
,
F.
,
Asgari
,
J.
,
Tseng
,
H. E.
, and
Hrovat
,
D.
,
2007
, “
Predictive Active Steering Control for Autonomous Vehicle Systems
,”
IEEE Trans. Control Syst. Technol.
,
15
(
3
), pp.
566
580
.
12.
Falcone
,
P.
,
Eric Tseng
,
H.
,
Borrelli
,
F.
,
Asgari
,
J.
, and
Hrovat
,
D.
,
2008
, “
MPC-Based Yaw and Lateral Stabilisation Via Active Front Steering and Braking
,”
Veh. Syst. Dyn.
,
46
(
Suppl. 1
), pp.
611
628
.
13.
Mesbah
,
A.
,
2016
, “
Stochastic Model Predictive Control: An Overview and Perspectives for Future Research
,”
IEEE Control Syst.
,
36
(
6
), pp.
30
44
.
14.
Bemporad
,
A.
, and
Morari
,
M.
,
1999
, “
Robust Model Predictive Control: A Survey
,”
Robustness in Identification and Control
,
Springer
,
London, UK
, pp.
207
226
.
15.
Bellman
,
R.
,
2013
,
Dynamic Programming
,
Dover Publications
,
Mineola, NY
.
16.
Gray
,
A.
,
Gao
,
Y.
,
Lin
,
T.
,
Hedrick
,
J. K.
, and
Borrelli
,
F.
,
2013
, “
Stochastic Predictive Control for Semi-Autonomous Vehicles With an Uncertain Driver Model
,”
16th International IEEE Conference on Intelligent Transportation Systems
(
ITSC
),
The Hague, The Netherlands
,
Oct. 6–9
, pp.
2329
2334
.
17.
Cesari
,
G.
,
Schildbach
,
G.
,
Carvalho
,
A.
, and
Borrelli
,
F.
,
2017
, “
Scenario Model Predictive Control for Lane Change Assistance and Autonomous Driving on Highways
,”
IEEE Intell. Transp. Syst. Mag.
,
9
(
3
), pp.
23
35
.
18.
Schwarting
,
W.
,
Alonso-Mora
,
J.
, and
Rus
,
D.
,
2018
, “
Planning and Decision-Making for Autonomous Vehicles
,”
Annu. Rev. Control, Rob., Auton. Syst.
,
1
, pp.
187
210
.
19.
Feinberg
,
E. A.
, and
Shwartz
,
A.
,
2012
,
Handbook of Markov Decision Processes: Methods and Applications
, Vol.
40
,
Springer Science & Business Media
,
New York
.
20.
Lusch
,
B.
,
Kutz
,
J. N.
, and
Brunton
,
S. L.
,
2018
, “
Deep Learning for Universal Linear Embeddings of Nonlinear Dynamics
,”
Nat. Commun.
,
9
(
1
), p.
4950
.
21.
Filev
,
D. P.
, and
Kolmanovsky
,
I.
,
2012
, “
Markov Chain Modeling and on-Board Identification for Automotive Vehicles
,”
Identification for Automotive Systems
,
Springer
,
London
, pp.
111
128
.
22.
Filev
,
D. P.
, and
Kolmanovsky
,
I.
,
2014
, “
Generalized Markov Models for Real-Time Modeling of Continuous Systems
,”
IEEE Trans. Fuzzy Syst.
,
22
(
4
), pp.
983
998
.
23.
Altman
,
E.
,
1999
,
Constrained Markov Decision Processes
, Vol.
7
,
CRC Press
,
Boca Raton, FL
.
24.
Feyzabadi
,
S.
, and
Carpin
,
S.
,
2014
, “
Risk-Aware Path Planning Using Hierarchical Constrained Markov Decision Processes
,”
IEEE International Conference on Automation Science and Engineering
(
CASE
),
New Taipei, Taiwan
,
Aug. 18–22
, pp.
297
303
.
25.
Li
,
N.
,
Kolmanovsky
,
I.
, and
Girard
,
A.
,
2018
, “
Tractable Stochastic Predictive Control for Partially Observable Markov Decision Processes With Time-Joint Chance Constraints
,”
57th IEEE Conference on Decision and Control
(
CDC
),
Miami, FL
,
Dec. 17–19
, pp.
3276
3282
.
26.
Cannon
,
M.
,
Kouvaritakis
,
B.
,
Rakovic
,
S. V.
, and
Cheng
,
Q.
,
2011
, “
Stochastic Tubes in Model Predictive Control With Probabilistic Constraints
,”
IEEE Trans. Autom. Control
,
56
(
1
), pp.
194
200
.
27.
Cannon
,
M.
,
Cheng
,
Q.
,
Kouvaritakis
,
B.
, and
Raković
,
S. V.
,
2012
, “
Stochastic Tube MPC With State Estimation
,”
Automatica
,
48
(
3
), pp.
536
541
.
28.
Schildbach
,
G.
,
Fagiano
,
L.
,
Frei
,
C.
, and
Morari
,
M.
,
2014
, “
The Scenario Approach for Stochastic Model Predictive Control With Bounds on Closed-Loop Constraint Violations
,”
Automatica
,
50
(
12
), pp.
3009
3018
.
29.
Sehr
,
M. A.
, and
Bitmead
,
R. R.
,
2017
, “
Stochastic Model Predictive Control: Output-Feedback, Duality and Guaranteed Performance
,” preprint
arXiv:1706.00733
.https://arxiv.org/abs/1706.00733
30.
Schwarm
,
A. T.
, and
Nikolaou
,
M.
,
1999
, “
Chance-Constrained Model Predictive Control
,”
AIChE J.
,
45
(
8
), pp.
1743
1752
.
31.
Blackmore
,
L.
, and
Ono
,
M.
,
2009
, “
Convex Chance Constrained Predictive Control Without Sampling
,”
AIAA
Paper No. 2009-5876.
32.
Paulson
,
J. A.
,
Buehler
,
E. A.
,
Braatz
,
R. D.
, and
Mesbah
,
A.
,
2017
, “
Stochastic Model Predictive Control With Joint Chance Constraints
,”
Int. J. Control
, (accepted).
33.
Lin
,
Z.
, and
Bai
,
Z.
,
2011
,
Probability Inequalities
,
Springer Science & Business Media
,
New York
.
34.
Särkkä
,
S.
,
2013
,
Bayesian Filtering and Smoothing
, Vol.
3
,
Cambridge University Press
,
Cambridge, UK
.
35.
Claussmann
,
L.
,
Carvalho
,
A.
, and
Schildbach
,
G.
,
2015
, “
A Path Planner for Autonomous Driving on Highways Using a Human Mimicry Approach With Binary Decision Diagrams
,”
European Control Conference
(
ECC
),
Linz, Austria
,
July 15–17
, pp.
2976
2982
.
36.
Li
,
N.
,
Chen
,
H.
,
Kolmanovsky
,
I.
, and
Girard
,
A.
,
2017
, “
An Explicit Decision Tree Approach for Automated Driving
,”
ASME
Paper No. DSCC2017-5099.
37.
Kumar
,
P. R.
, and
Varaiya
,
P.
,
2015
,
Stochastic Systems: Estimation, Identification, and Adaptive Control
,
Society for Industrial and Applied Mathematics
,
Philadelphia, PA
.
38.
Rudin
,
W.
,
1976
,
Principles of Mathematical Analysis
, Vol.
3
,
McGraw-Hill
,
New York
.
39.
Bazaraa
,
M. S.
,
Sherali
,
H. D.
, and
Shetty
,
C. M.
,
2013
,
Nonlinear Programming: Theory and Algorithms
,
Wiley
,
Hoboken, NJ
.
40.
Potra
,
F. A.
, and
Wright
,
S. J.
,
2000
, “
Interior-Point Methods
,”
J. Comput. Appl. Math.
,
124
(
1–2
), pp.
281
302
.
41.
Boyd
,
S.
, and
Vandenberghe
,
L.
,
2004
,
Convex Optimization
,
Cambridge University Press
,
Cambridge, UK
.
42.
Lee
,
J.
,
2004
,
A First Course in Combinatorial Optimization
, Vol.
36
,
Cambridge University Press
,
Cambridge, UK
.
43.
Adler
,
I.
,
Resende
,
M. G.
,
Veiga
,
G.
, and
Karmarkar
,
N.
,
1989
, “
An Implementation of Karmarkar's Algorithm for Linear Programming
,”
Math. Program.
,
44
(
1–3
), pp.
297
335
.
44.
Hoy
,
M.
,
Matveev
,
A. S.
, and
Savkin
,
A. V.
,
2015
, “
Algorithms for Collision-Free Navigation of Mobile Robots in Complex Cluttered Environments: A Survey
,”
Robotica
,
33
(
3
), pp.
463
497
.
45.
Lorenzen
,
M.
,
Allgöwer
,
F.
,
Dabbene
,
F.
, and
Tempo
,
R.
,
2015
, “
An Improved Constraint-Tightening Approach for Stochastic MPC
,”
American Control Conference
(
ACC
),
Chicago, IL
,
July 1–3
, pp.
944
949
.
46.
Lorenzen
,
M.
,
Allgöwer
,
F.
,
Dabbene
,
F.
, and
Tempo
,
R.
,
2015
, “
Scenario-Based Stochastic MPC With Guaranteed Recursive Feasibility
,”
54th IEEE Conference on Decision and Control
(
CDC
),
Osaka, Japan
,
Dec. 15–18
, pp.
4958
4963
.
47.
Kolmanovsky
,
I.
, and
Gilbert
,
E. G.
,
1998
, “
Theory and Computation of Disturbance Invariant Sets for Discrete-Time Linear Systems
,”
Math. Problems Eng.
,
4
(
4
), pp.
317
367
.
48.
Rungger
,
M.
, and
Tabuada
,
P.
,
2017
, “
Computing Robust Controlled Invariant Sets of Linear Systems
,”
IEEE Trans. Autom. Control
,
62
(
7
), pp.
3665
3670
.
49.
Brackstone
,
M.
, and
McDonald
,
M.
,
1999
, “
Car-Following: A Historical Review
,”
Transp. Res., Part F: Traffic Psychol. Behav.
,
2
(
4
), pp.
181
196
.
50.
Toledo
,
T.
, and
Zohar
,
D.
,
2007
, “
Modeling Duration of Lane Changes
,”
Transp. Res. Rec.
,
1999
(
1
), pp.
71
78
.
51.
FHWA
,
2012
, “Manual on Uniform Traffic Control Devices,” Federal Highway Administration, Washington, DC, accessed Mar. 19, 2019, https://mutcd.fhwa.dot.gov/pdfs/2009r1r2/mutcd2009r1r2edition.pdf
52.
Di Cairano
,
S.
,
Bernardini
,
D.
,
Bemporad
,
A.
, and
Kolmanovsky
,
I. V.
,
2014
, “
Stochastic MPC With Learning for Driver-Predictive Vehicle Control and Its Application to HEV Energy Management
,”
IEEE Trans. Control Syst. Technol.
,
22
(
3
), pp.
1018
1031
.
53.
Kurniawati
,
H.
,
Hsu
,
D.
, and
Lee
,
W. S.
,
2008
, “
SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces
,”
Robotics: Science and Systems
, Vol.
2008
,
MIT Press
,
Cambridge, MA
.
54.
Altman
,
Y. M.
,
2014
,
Accelerating MATLAB Performance: 1001 Tips to Speed Up MATLAB Programs
,
CRC Press
,
Boca Raton, FL
.
55.
Wang
,
Y.
, and
Boyd
,
S.
,
2010
, “
Fast Model Predictive Control Using Online Optimization
,”
IEEE Trans. Control Syst. Technol.
,
18
(
2
), pp.
267
278
.
56.
Gill
,
P. E.
,
Murray
,
W.
,
Saunders
,
M. A.
, and
Wright
,
M. H.
,
1984
, “
Sparse Matrix Methods in Optimization
,”
SIAM J. Sci. Stat. Comput.
,
5
(
3
), pp.
562
589
.
57.
Henriksson
,
D.
, and
Åkesson
,
J.
,
2004
, “
Flexible Implementation of Model Predictive Control Using Sub-Optimal Solutions
,” Department of Automatic Control, Lund Institute of Technology (LTH), Lund, Sweden, Technical Report No. TFRT-7610.
58.
Dontchev
,
A. L.
,
Huang
,
M.
,
Kolmanovsky
,
I. V.
, and
Nicotra
,
M. M.
,
2018
, “
Inexact Newton-Kantorovich Methods for Constrained Nonlinear Model Predictive Control
,”
IEEE Trans. Autom. Control
(epub).
59.
Canale
,
M.
,
Fagiano
,
L.
, and
Milanese
,
M.
,
2009
, “
Fast Nonlinear Model Predictive Control Via Set Membership Approximation: An Overview
,”
Nonlinear Model Predictive Control
,
Springer
,
Berlin
, pp.
461
470
.
60.
Grancharova
,
A.
, and
Johansen
,
T. A.
,
2012
,
Explicit Nonlinear Model Predictive Control: Theory and Applications
, Vol.
429
,
Springer Science & Business Media
,
New York
.
You do not currently have access to this content.