Abstract

This paper addresses the constrained H optimal control problem for nonlinear active vehicle suspension systems, with a focus on deriving an approximate solution through data-driven reinforcement learning in the context of differential games. A dynamic model of the half-car active suspension system with constraints is first established, where the constrained control forces and external road disturbances are formulated as a zero-sum game between two players. This leads to the Hamilton–Jacobi–Isaacs (HJI) equation, with a Nash equilibrium as the desired solution. To efficiently solve the HJI equation and mitigate the impact of model parameter uncertainties, an actor-critic neural network framework is employed to approximate both the control policy and the value function of the system. A reinforcement learning algorithm based on the input-output data of the suspension system is subsequently derived. Numerical examples are provided to demonstrate the effectiveness of the proposed approach for time-invariant suspension systems. Under varying control force constraints, the active suspension system consistently exhibits excellent vibration reduction performance.

References

1.
Yu
,
M.
,
Evangelou
,
S. A.
, and
Dini
,
D.
,
2024
, “
Advances in Active Suspension Systems for Road Vehicles
,”
Engineering
,
33
, pp.
160
177
.10.1016/j.eng.2023.06.014
2.
Liu
,
Z.
,
Si
,
Y.
, and
Sun
,
W
,
2024
, “
Ride comfort oriented integrated design of preview active suspension control and longitudinal velocity planning
,”
Mech. Syst. Signal Process.
,
208
, p.
110992
.10.1016/j.ymssp.2023.110992
3.
Zhou
,
Z.
,
Zhang
,
M.
,
Liu
,
H.
, and
Jing
,
X.
,
2023
, “
Fixed-Time Safe-by-Design Control for Uncertain Active Vehicle Suspension Systems With Nonlinear Reference Dynamics
,”
IEEE/ASME Trans. Mechatronics
, 29(5), pp.
1
12
.10.1109/TMECH.2023.3342013
4.
Kim
,
Y.
,
Kim
,
M. W.
,
Kanno
,
M.
, and
Kim
,
T. H.
,
2024
, “
Meta-heuristic optimization-based robust H controller design for active suspension systems subject to actuator saturation
,”
Alex. Eng. J.
,
105
, pp.
523
537
.10.1016/j.aej.2024.08.032
5.
Zhang
,
C.
, and
Xiao
,
J.
,
2018
, “
Chaotic behavior and feedback control of magnetorheological suspension system with fractional-order derivative
,”
ASME J. Comput. Nonlin. Dyn.
,
13
(
2
), p.
021007
.10.1115/1.4037931
6.
Du
,
G.
,
Yang
,
T.
,
Chen
,
G.
,
Xie
,
X.
, and
Xia
,
J.
,
2023
, “
Multi-Rate Sampled-Data Controller Design For Vehicle Active Suspension Systems
,”
IEEE Contr. Syst. Lett.
,
7
, pp.
3349
3354
.10.1109/LCSYS.2023.3327717
7.
Li
,
W.
,
Xie
,
Z.
,
Zhao
,
J.
,
Wong
,
P. K.
,
Wang
,
H.
, and
Wang
,
X.
,
2020
, “
Static-Output-Feedback Based Robust Fuzzy Wheelbase Preview Control for Uncertain Active Suspensions With Time Delay and Finite Frequency Constraint
,”
IEEE/CAA J. Autom. Sin.
,
8
(
3
), pp.
664
678
.10.1109/JAS.2020.1003183
8.
Moradi
,
S. M.
,
Akbari
,
A.
, and
Mirzaei
,
M.
,
2019
, “
An Offline LMI-Based Robust Model Predictive Control of Vehicle Active Suspension System With Parameter Uncertainty
,”
Trans. Inst. Meas. Control
,
41
(
6
), pp.
1699
1711
.10.1177/0142331218787599
9.
Hong
,
C.
, and
Konghui
,
G.
,
2005
, “
Constrained H Control of Active Suspensions: An LMI Approach
,”
IEEE Trans. Control Syst. Technol.
,
13
(
3
), pp.
412
421
.10.1109/TCST.2004.841661
10.
Li
,
W.
,
Du
,
H.
,
Ning
,
D.
,
Li
,
W.
,
Sun
,
S.
, and
Wei
,
J.
,
2021
, “
Event-Triggered H Control for Active Seat Suspension Systems Based on Relaxed Conditions for Stability
,”
Mech. Syst. Signal Process.
,
149
, p.
107210
.10.1016/j.ymssp.2020.107210
11.
Sun
,
W.
,
Gao
,
H.
, and
Kaynak
,
O.
,
2011
, “
Finite Frequency H Control for Vehicle Active Suspension Systems
,”
IEEE Trans. Control Syst. Technol.
,
19
(
2
), pp.
416
422
.10.1109/TCST.2010.2042296
12.
Zhang
,
Z.
,
Li
,
H.
,
Wu
,
C.
, and
Zhou
,
Q.
,
2018
, “
Finite Frequency H Control for Vehicle Active Suspension Systems
,”
IEEE/CAA J. Autom. Sin.
,
5
(
4
), pp.
777
786
.10.1109/JAS.2018.7511132
13.
Zhang
,
Z.
,
Qin
,
A.
,
Zhang
,
J.
,
Zhang
,
B.
,
Fan
,
Q.
, and
Zhang
,
N.
,
2021
, “
Fuzzy Sampled-Data H Sliding-Mode Control for Active Hysteretic Suspension of Commercial Vehicles with Unknown Actuator-Disturbance
,”
Control Eng. Pract.
,
117
, p.
104940
.10.1016/j.conengprac.2021.104940
14.
Viadero-Monasterio
,
F.
,
Boada
,
B. L.
,
Boada
,
M. J. L.
, and
Díaz
,
V.
,
2022
, “
H Dynamic Output Feedback Control for a Networked Control Active Suspension System Under Actuator Faults
,”
Mech. Syst. Signal Process.
,
162
, p.
108050
.10.1016/j.ymssp.2021.108050
15.
Chen
,
S. A.
,
Jiang
,
X. D.
,
Yao
,
M.
,
Jiang
,
S. M.
,
Chen
,
J.
, and
Wang
,
Y. X.
,
2020
, “
A Dual Vibration Reduction Structure-Based Self-Powered Active Suspension System With PMSM-Ball Screw Actuator Via an Improved H2/H Control
,”
Energy
,
201
, p.
117590
.10.1016/j.energy.2020.117590
16.
Jing
,
H.
,
Wang
,
R.
,
Li
,
C.
, and
Bao
,
J.
,
2019
, “
Robust Finite-Frequency H Control of Full-Car Active Suspension
,”
J. Sound. Vib.
,
441
, pp.
221
239
.10.1016/j.jsv.2018.06.047
17.
Yang
,
W.
,
Li
,
S.
, and
Luo
,
X.
,
2024
, “
Data Driven Vibration Control: A Review
,”
IEEE/CAA J. Autom. Sin.
,
11
(
9
), pp.
1898
1917
.10.1109/JAS.2024.124431
18.
Qin
,
Z. C.
, and
Xin
,
Y.
,
2023
, “
Data-Driven H Vibration Control Design and Verification for an Active Suspension System with Unknown Pseudo-Drift Dynamics
,”
Commun. Nonlinear Sci. Numer. Simul.
,
125
, p.
107397
.10.1016/j.cnsns.2023.107397
19.
Wang
,
G.
,
Li
,
K.
,
Liu
,
S.
, and
Jing
,
H.
,
2023
, “
Model-Free H Output Feedback Control of Road Sensing in Vehicle Active Suspension Based on Reinforcement Learning
,”
ASME J. Dyn. Sys. Meas. Control
,
145
(
6
), p.
061003
.10.1115/1.4062342
20.
Liang
,
Y.
,
Zhang
,
H.
,
Zhang
,
J.
, and
Luo
,
Y.
,
2021
, “
Integral Reinforcement Learning-Based Guaranteed Cost Control for Unknown Nonlinear Systems Subject to Input Constraints and Uncertainties
,”
Appl. Math. Comput.
,
408
, p.
126336
.10.1016/j.amc.2021.126336
21.
Pang
,
H.
,
Wang
,
Y.
,
Zhang
,
X.
, and
Xu
,
Z.
,
2019
, “
Robust State-Feedback Control Design for Active Suspension System With Time-Varying Input Delay and Wheelbase Preview Information
,”
J. Franklin Inst.
,
356
(
4
), pp.
1899
1923
.10.1016/j.jfranklin.2019.01.011
22.
Sun
,
W.
,
Zhao
,
Z.
, and
Gao
,
H.
,
2013
, “
Saturated Adaptive Robust Control for Active Suspension Systems
,”
IEEE Trans. Ind. Electron.
,
60
(
9
), pp.
3889
3896
.10.1109/TIE.2012.2206340
23.
Shao
,
X.
,
Naghdy
,
F.
,
Du
,
H.
, and
Li
,
H.
,
2019
, “
Output feedback H control for active suspension of in-wheel motor driven electric vehicle with control faults and input delay
,”
ISA Trans.
,
92
, pp.
94
108
.10.1016/j.isatra.2019.02.016
24.
Wang
,
D.
,
Gao
,
N.
,
Liu
,
D.
,
Li
,
J.
, and
Lewis
,
F. L.
,
2024
, “
Recent progress in reinforcement learning and adaptive dynamic programming for advanced control applications
,”
IEEE/CAA J. Autom. Sin.
,
11
(
1
), pp.
18
36
.10.1109/JAS.2023.123843
25.
Dridi
,
I.
,
Hamza
,
A.
, and
Ben, Yahia
,
N.
,
2023
, “
A New Approach to Controlling an Active Suspension System Based on Reinforcement Learning
,”
Adv. Mech. Eng.
,
15
(
6
), pp.
1
21
.10.1177/16878132231180480
26.
Wang
,
C.
,
Cui
,
X.
,
Zhao
,
S.
,
Zhou
,
X.
,
Song
,
Y.
,
Wang
,
Y.
, and
Guo
,
K.
,
2024
, “
A Deep Reinforcement Learning-Based Active Suspension Control Algorithm Considering Deterministic Experience Tracing for Autonomous Vehicle
,”
Appl. Soft Comput.
,
153
, p.
111259
.10.1016/j.asoc.2024.111259
27.
Kimball
,
J, B.
,
DeBoer
,
B.
, and
Bubbar
,
K.
,
2024
, “
Adaptive Control and Reinforcement Learning for Vehicle Suspension Control: A Review
,”
Ann. Rev. Control
,
58
, p.
100974
.10.1016/j.arcontrol.2024.100974
28.
Liu
,
S.
,
Liu
,
L.
, and
Yu
,
Z.
,
2023
, “
Safe Reinforcement Learning for Affine Nonlinear Systems With State Constraints and Input Saturation Using Control Barrier Functions
,”
Neurocomputing
,
518
, pp.
562
576
.10.1016/j.neucom.2022.11.006
29.
Luo
,
B.
,
Wu
,
H. N.
, and
Huang
,
T.
,
2015
, “
Off-Policy Reinforcement Learning for H Control Design
,”
IEEE Trans. Cybern.
,
45
(
1
), pp.
65
76
.10.1109/TCYB.2014.2319577
30.
Yang
,
Y.
,
Ding
,
D. W.
,
Xiong
,
H.
,
Yin
,
Y.
, and
Wunsch
,
D. C.
,
2020
, “
Online Barrier-Actor-Critic Learning for H Control with Full-State Constraints and Input Saturation
,”
J. Franklin Inst.
,
357
(
6
), pp.
3316
3344
.10.1016/j.jfranklin.2019.12.017
31.
Wu
,
H. N.
, and
Luo
,
B.
,
2012
, “
Neural Network Based Online Simultaneous Policy Update Algorithm for Solving the HJI Equation in Nonlinear H Control
,”
IEEE Trans. Neural Network Learn. Syst.
,
23
(
12
), pp.
1884
1895
.10.1109/TNNLS.2012.2217349
32.
Kiumarsi
,
B.
,
Lewis
,
F. L.
, and
Jiang
,
Z. P.
,
2017
, “
H Control of Linear Discrete-Time Systems: Off-Policy Reinforcement Learning
,”
Automatica
,
78
, pp.
144
152
.10.1016/j.automatica.2016.12.009
33.
Jiang
,
H.
,
Zhang
,
H.
,
Luo
,
Y.
, and
Cui
,
X.
,
2017
, “
H Control with Constrained Input for Completely Unknown Nonlinear Systems Using Data-Driven Reinforcement Learning Method
,”
Neurocomputing
,
237
, pp.
226
234
.10.1016/j.neucom.2016.11.041
34.
Li
,
P.
,
Lam
,
J.
, and
Cheung
,
K. C.
,
2014
, “
Multi-Objective Control for Active Vehicle Suspension With Wheelbase Preview
,”
J. Sound. Vib.
,
333
(
21
), pp.
5269
5282
.10.1016/j.jsv.2014.06.017
35.
Xue
,
W.
,
Li
,
K.
,
Chen
,
Q.
, and
Liu
,
G.
,
2019
, “
Mixed FTS/H Control of Vehicle Active Suspensions With Shock Road Disturbance
,”
Vehicle Syst. Dyn.
,
57
(
6
), pp.
841
854
.10.1080/00423114.2018.1490023
36.
Kim
,
J.
, and
Yim
,
S.
,
2024
, “
Design of Static Output Feedback Suspension Controllers for Ride Comfort Improvement and Motion Sickness Reduction
,”
Processes
,
12
(
5
), p.
968
.10.3390/pr12050968
37.
Xu
,
Y.
,
Xie
,
Z.
,
Zhao
,
J.
,
Li
,
W.
,
Li
,
P.
, and
Wong
,
P. K.
,
2021
, “
Robust Non-Fragile Finite Frequency H Control for Uncertain Active Suspension Systems with Time-Delay Using TS Fuzzy Approach
,”
J. Franklin Inst.
,
358
(
8
), pp.
4209
4238
.10.1016/j.jfranklin.2021.03.019
You do not currently have access to this content.