The linear quadratic regulator (LQR) is a classical optimal control approach that can regulate gait dynamics about target kinematic trajectories. Exoskeletons to restore gait function have conventionally utilized time-varying proportional-derivative (PD) control of leg joints. But, these PD parameters are not uniquely optimized for whole-body (full-state) performance. The objective of this study was to investigate the effectiveness of LQR full-state feedback compared to PD control to maintain bipedal walking of a sagittal-plane computational model against force disturbances. Several LQR controllers were uniquely solved with feedback gains optimized for different levels of tracking capability versus control effort. The main implications to future exoskeleton control systems include (1) which LQR controllers out-perform PD controllers in walking maintenance and effort, (2) verifying that LQR desirably produces joint torques that oppose rapidly growing joint state errors, and (3) potentially equipping accurate sensing systems for nonjoint states such as hip-position and torso orientation. The LQR controllers capable of longer walk times than respective PD controllers also required less control effort. During sudden leg collapse, LQR desirably behaved like PD by generating feedback torques that opposed the direction of leg-joint errors. Feedback from nonjoint states contributed to over 50% of the LQR joint torques and appear critical for whole-body LQR control. While LQR control poses implementation challenges, such as more sensors for full-state feedback and operation near the desired trajectories, it offers significant performance advantages over PD control.

## Introduction

The linear quadratic regulator (LQR) is a classical optimal control approach to regulating the dynamics of a linear system, but LQR may also be utilized on nonlinear dynamics that are effectively approximated as a linear time-varying system. Applying LQR to nonlinear biomechanical systems such as bipedal human gait can provide insight into basic operating considerations for optimal control of powered gait devices such as exoskeletons [1,2]. Previous studies for autonomous robot or humanoid biped walking have emulated natural gait utilizing LQR for online planning and control of optimal motions according to desired center of mass and center of pressure coordinates during walking [37]. Localized formulations may be sufficient given minimal deviations from desired trajectories. LQR tree methods have been developed to combine locally valid LQR control with a nonlinear feedback policy that evaluates regions of attraction to stabilize larger system deviations [8]. For robust autonomous walking, additional levels of control, such as fuzzy identification [9], model prediction [10,11], and sinusoidal oscillator [11], can be required to make sufficiently fast corrections against stronger perturbations. Since fundamental operating control by LQR has been demonstrated with robot walking, it may have potential for powered assistive devices such as exoskeletons as well. Exoskeletons, to restore or augment gait function, have previously utilized time-varying linear control [2]. In this case, gains and setpoints for closed-loop joint control were fit to empirical human angle-moment data for each individual joint across finite states. Such an approach for time-varying control with full-state feedback, where each joint control would be dependent upon all states of the system, would require examination across a large controller design space. Alternatively, an optimal control approach could identify unique and time-continuous parameters for full-state control that explicitly minimizes a specified cost function. Furthermore, proportional and derivative gain values were always designated to produce negative feedback torques to locally oppose errors at each joint. Such control presuppositions are not necessarily optimal for global gait objectives such as achieving target walking speeds and minimizing effort across the entire system.

Given highly refined sensorimotor integration capabilities of humans, natural gait may be better reflected in an LQR formulation with full-state feedback and generating controller parameters according to whole-body objectives rather than basic proportional-derivative (PD) control at each individual joint. For control of gait, LQR computes optimal feedback gains that minimize a specific cost function during motion [8]. The cost function can be formulated in terms of controller ability to track desired kinematic state trajectories versus the controller effort to do so [12]. After specifying factors to weigh the relative importance of tracking versus effort, the LQR control solution provides the respective optimal time-varying gains. The LQR formulation indicates the fundamental control required to stabilize locomotion against basic system errors as a function of energy cost and movement time-course. LQR models have been previously proposed for identifying optimal sensorimotor control of locomotion such as gait [13,14]. An LQR structure that modulates gain values over time for a cyclic function may explain motor control phenomena for gait. The relative gain values during different phases of gait (i.e., single-support, double-support, swing) may suggest which feedback states are most critical to executing the motor objectives of those phases with minimal effort. Time-varying control systems for exoskeletons that assist gait can be better designed with these considerations in mind.

Time-varying feedback control is evident in observable natural locomotion with physiological mechanisms whose properties modulate with time and task execution. Central pattern generators produce open-loop motions while reflexes use closed-loop feedback to regulate activations against internal errors and external perturbations [14,15]. Previous investigations have suggested that reflex modulation may act to optimize feedback gains across walking motion phases [1618]. Feedback gains to a leg may be lower during swing to facilitate energy savings but higher during ground contact to readily reject disturbances while weight bearing and propelling the entire system forward. Torques and muscle activations estimated from experimental walking data are consistently shown to be elevated during stance phase progression [19,20]. The particular values of these time-varying gains may be optimal with respect to stability and an energy cost for mechanical work [21]. Thus, the intrinsic features of LQR including time-varying control and gain optimization with performance and effort considerations make it a sensible choice for the restoration of natural locomotion by assistive devices such as exoskeletons.

Investigating an optimal control approach for human walking should provide insight into how to develop an improved exoskeleton controller for more effective and efficient gait. To this end, we formulate LQR control systems that minimize errors about kinematic trajectories that are optimal for a desired walking speed with minimal joint torques. This will facilitate identification of feedback gains that optimally vary with the gait cycle according to tracking versus effort specifications. Performance of these LQR control systems to maintain walking against force perturbations will be directly compared to conventional constant gain PD control. While LQR systems are explicitly designed for optimal performance regarding better tracking of states and reduced control effort, this is done explicitly for linear systems. The relative performance of LQR to PD when applied to nonlinear gait dynamics should provide better specific quantitative insight on the additive features and shortcomings of both approaches. The specific objectives of this simulation study are to (1) design optimal linear control systems for bipedal human walking that considers desired gait performance against effort and (2) test the performance of these LQR systems in simulation against external disturbances and compare to performance of analogous PD controllers. The implications of these results in designing exoskeleton control systems include (1) which LQR controllers out-perform PD controllers in terms of walking maintenance and effort, (2) verifying that LQR desirably produces joint torques that oppose rapidly growing joint state errors, and (3) considerations for equipping a measurement system of feedback states, including nonjoint states, based on their relative contribution to LQR torques.

## Methods

In this section, we review the development for our simulation platform to perform analysis of LQR control of a gait model and produce results that provide insight into better exoskeleton control. First, the bipedal sagittal-plane walking model is presented along with procedures to generate the optimal (minimal) joint-torque trajectories required for specified normal walking data [22]. Next, the linearization and discretization procedures of the gait dynamics are outlined. Finally, the formulation and testing of the various (tracking versus effort) LQR solutions relative to comparable PD controllers are described. The resulting optimal gain values and various test evaluation parameters indicate the potential performance advantages of LQR over the conventional PD control employed for controlling exoskeletons that restore gait.

### Walking Model.

Walking was simulated with a planar sagittal-plane model, consisting of seven rigid bodies, connected by six revolute joints [23,24]. The mass, length, and inertia parameters for the segments are based on Ref. [22]. In the present work, the model was actuated by six joint torques and muscles were not used. Points of contact between each foot and ground were modeled with spring-damper elements as described in Ref. [23] located at the heel and toe. Passive joint moments were included to provide realistic joint motions and limits. There was basic stiffness (1 N·m/rad), about the zero position for each joint, and damping (1 N·m s/rad) through normal ranges of motion and high stiffness (1000 N·m/rad) at specified maximum limits for each joint.

There are nine kinematic degrees-of-freedom in the model: global X (anterior–posterior) and Y (superior–inferior) coordinates of the hip joint, torso orientation, right hip angle, right knee angle, right ankle angle, left hip angle, left knee angle, and left ankle angle. When the generalized coordinates are zero, the model is in an upright standing position. Positive angle values correspond to counterclockwise rotation of the torso or the distal segment of a joint. Equations of motion were generated by Autolev (Online Dynamics Inc., Sunnyvale, CA), exported as C code and compiled into an MEX function for matlab (Mathworks, Inc., Natick, MA) [25]. The MEX function computes the right hand side of the system dynamics in the form
$x˙=f(x,u)$
(1)

where x is the state, consisting of the nine generalized coordinates and the corresponding velocities, and u are the six joint torques. The MEX function and documentation can be downloaded from Ref. [25]. The model dynamics equations were formulated to be twice differentiable, facilitating the use of implicit differential equation solvers, gradient-based optimization, and linearization. Foot-ground contact was differentiable with inclusion of a very weak linear elastic term for vertical forces when the foot was above the ground.

### Optimal Open-Loop Controlled Walking Trajectory.

An open-loop controlled cyclic walking trajectory was generated by solving an optimal control problem on the dynamic model. The cost function to be minimized was a weighted sum of two terms. The first term was the integral of squared tracking deviations from typical human joint angles and ground reaction forces [22]. Data for a normal walking speed (1.325 m/s, cycle time 1.1396 s) were obtained from Ref. [22]. The second term represented the required effort, defined as the sum of the integrated squared joint-torque controls. The weighting of the two terms was chosen such that smooth, efficient, and realistic trajectories were generated. Optimal state and torque trajectories, xo(t) and uo(t), were found such that the cost function was minimized, the system dynamics (Eq. (1)) was satisfied, and the trajectories were periodic with the required forward translation for the full gait cycle. The direct collocation method [23] was used, and the gait cycle was represented by 100 time-nodes. Mesh refinement was done from 400 nodes to verify that 100 nodes demonstrated sufficient detail. The body posture and hip-position trajectory at selected time points during the gait cycle are shown in Fig. 1. The result of the trajectory optimization was a time series of states x1,…, x100 and a time series of controls u1,…, u100, which will represent the discrete-time trajectory for which a controller will be developed in Sec. 2.3.

### Linear Quadratic Regulator for Closed-Loop Feedback Controller.

The LQR feedback controllers were developed from linearization about the optimal gait trajectories. These controllers utilized feedback from all 18 states (nine positions, nine velocities) to drive torque control of the 6 joints. These controllers were tested in simulation against random force perturbations in the fore-aft direction applied to the hip. The performance of the LQR full-state feedback controllers were then compared to that of conventional proportional-derivative joint feedback controllers with comparable gain magnitudes. Since the LQR controller incorporates full-state feedback, the optimal gain matrix is not diagonal as each of the 6 joint-torque controls receives input from all 18 states.

#### Linearizing Gait Dynamics.

Feedback control for resisting the effects of basic perturbations was applied to the nonlinear gait dynamics of the model, $x˙=f(x,u)$. LQR feedback controllers were designed by linearizing about the optimal walking trajectories at each time-node. Variables for a surrogate perturbation control model that regulates deviations from the optimal state and control trajectories () were defined as $y(t)=x(t)−xo(t)$ and $v(t)=u(t)−uo(t)$ for the states and controls, respectively. For compact notation, the (t) to indicate time-varying systems is dropped for the remainder of this section. Substituting in the surrogate variables expresses the gait dynamics as
$x˙o+y˙=f(xo+y,uo+v)$
(2)
First-order Taylor series expansion at $xo$ and $uo$ yields the linearization
$x˙o+y˙=f(xo,uo)+fxy+fuv$
(3)

where $fx=(∂f/∂x)xo,uo$ and $fu=(∂f/∂u)xo,uo$

Since $xo$ and $uo$ satisfies dynamics of the system where , the linearized system dynamics can be reduced to
$y˙=fxy+fuv$
(4)

#### Discretizing Gait Dynamics.

The gait dynamics were then discretized according to the backward Euler method where . Applying this discretization to the system dynamics yields
$yi+1=yi+h(∂f∂x)xo,uoyi+1+h(∂f∂u)xo,uovi+1$
(5)
A re-arrangement of terms produces a discretized state space form of a linear system model for control of states from time node i to i + 1 as follows:
$yi+1=Aiyi+Bivi+1$
(6)
Jacobian system dynamics are readily extracted to populate the values for system (A) and control (B) matrices at each time node as follows:
$Ai=[I−h(∂f∂x)xo(ti+1),uo(ti+1)]−1$
(7)

$Bi=[I−h(∂f∂x)xo(ti+1),uo(ti+1)]−1h(∂f∂u)xo(ti+1),uo(ti+1)$
(8)

#### Solving Optimal LQR Feedback Gains From Riccati Equation.

For the linear system in general state-space form in discrete time: $yi+1=Aiyi+Bivi+1$, an LQR formulation provided an optimal state feedback solution. A control law of $vi+1=−Kiyi$ was found to minimize the quadratic cost function of as the optimal control problem. The diagonal weighting matrices $Q$ and $R$ quantify how to relatively minimize deviations from desired states and controls, respectively. Essentially, $Q$ and $R$ serve as controller design parameters that produce relative desired effects for better tracking (smaller errors in states) versus less effort (lower joint torques).

Each diagonal term corresponds to a state (i) or control (j) to normalize its contribution to the objective function by the variance of the respective optimal trajectory. The matrix diagonal terms are explicitly specified from the values of the standard deviation (σ) from the mean of the optimal trajectory for each state or control as computed from Sec. 2.2. This σ value indicates a magnitude for each state and control from which to intuitively perform quadratic normalization of each term for units and typical variations within the respective optimal trajectories as follows:
$Qii=φ(1σxi)2, where i=1–18 states$
(9)

$Rjj=(1σuj)2, where j=1–6 joint−torque controls$
(10)

To vary weighting toward tracking over effort, only the $Qii$ terms were subsequently and uniformly multiplied by a specific Q/R ratio, $φ$. This ratio serves as our single controller design parameter. It was assumed with $φ$  = 1, there is equal consideration of tracking and effort in controller performance.

The optimal feedback gain matrix, K, was then solved using a discrete, periodic Riccati equation solver (dpre.m, Matlab®) based on Refs. [26,27]. Optimal feedback gains by LQR were computed for the following 12 Q/R ratios: 0.001, 0.01, 0.1, 1, 2, 5, 10, 20, 50, 100, 1000, and 10,000. From pilot simulations, these ratios were shown to produce a sufficient range of walking capabilities for evaluating controller performance against perturbations. For each LQR controller, a comparably localized PD controller was also created for subsequent testing and evaluation. The proportional and derivative gains for each joint were set equal to the maximum absolute value of the respective leg-joint position and velocity state gains from the respective LQR controller across all times. From pilot testing, it was observed that assigning the highest respective joint values from the LQR feedback gain matrix as the proportional and derivative gain values produced the best PD control performance. Therefore, utilizing maximum gain values yielded the highest standard of comparison for LQR control.

### Basic Kinematic Analysis of Controller Behavior.

To identify basic operating characteristics of LQR control, the feedback torques to be generated during a series of leg collapse postures were examined at a single instance within the gait cycle. Leg collapse, as would happen with knee buckle, is an interesting test case given it produces drastic concurrent changes in several state variables. This analysis can verify whether the LQR controller, despite complex time-varying and full-state dependent gain profiles, would still generate intuitive joint torques that directly oppose quickly emerging joint errors as would be done with PD control. The collapse postures were determined by first prescribing downward vertical hip displacements between 0 and 3 cm onto a select instance of single-leg stance during the gait cycle. A time instance during single-leg stance was selected as the test case to serve as a worst-case scenario with no support assistance from the other leg. Inverse kinematics were then applied to determine the corresponding changes in the hip, knee, and ankle joint angles for each stance leg posture assuming fixed placement of the foot and fixed orientation of the torso. A range of vertical displacement rates (0–100 cm/s) were also applied for each posture to generate corresponding changes in joint angular velocities. The torques generated by an LQR controller for each set of positional and velocity state changes were then computed to verify controller actions in the presence of leg collapse.

### Perturbation Simulations.

Each simulation had the model executing ten full gait-cycles about the optimal trajectories under random force perturbation of one of two types. The two perturbation types (see Fig. 2) were specified for testing two respective controller performance parameters, namely, our ad hoc definitions for stability and efficiency. The first perturbation type (type#1) for assessing time-to-fall entailed the bound of the perturbation magnitude growing with time over the course of the simulation at 10 N/s. Given the selected growth magnitude of 10 N/s, it was determined that ten gait cycles provided a sufficiently long time-course from which to ascertain various time-to-fall performance results across the wide range of controllers being tested. The force for type#1 perturbations was always directed forward to ensure falling occurred within ten gait cycles for all controllers tested. The stability parameter, time-to-fall, was computed as the simulation time at which the walking model was unable to recover from falling while under perturbation. A fall was defined as the instance the hip Y-position deviates from its optimal trajectory by more than 20 standard deviations from the mean. This deviation threshold was justified in that the model never recovers toward optimal trajectories in any simulation when this threshold was exceeded.

The force for the second perturbation type (type#2) to assess root-mean-squared (RMS) of the torques had its bound being fixed at 5 N and directed in either the forward or backward directions since falling was not desired to evaluate efficiency. To further gauge efficiency against type#2 perturbation, the RMS value of the state error signals and the relative contributions of each feedback state to all torques generated were also computed. The efficiency parameter was computed as the RMS of the torque signal at each joint across an entire simulation, and then summed across all six leg joints. Efficiency was evaluated for both the feedback torque and total torque (feedback + optimal open-loop control torque) signals.

Both type#1 and type#2 perturbations were applied as piecewise constant loads in either the fore or aft direction over 0.1 s intervals. The perturbation force magnitude at each interval was a uniform random value within defined bounds. The perturbation force was applied at the hip. Each LQR and PD controller was evaluated across a total of 20 simulations for each perturbation type.

## Results

Optimal feedback gains as a function of time across the full gait-cycle were computed for all Q/R ratios specified. An example (Q/R = 1) of the gain profiles from all 18 feedback states to generate corrective torques onto the right leg joints are shown in Fig. 3. By design of the system (18 states onto six controls), there exists 108 gain profiles, but given left–right symmetry, only 72 profiles are unique. All gain profiles observe the periodicity constraint and demonstrate smooth transitions between single- and double-leg support phases. All gain profiles also show time-dependent patterns and can assume positive or negative values.

There are notable variations in the range (maximum–minimum) of gain values for each of the feedback states. Among state-control gain profiles with common units (e.g., degrees for joint position states onto join torques, always N-m), larger ranges would indicate greater time-varying control modulation. This, in turn, would suggest potentially greater dependence of a given joint torque from the given feedback state for basic execution of gait. The smallest range in position (Kp) and velocity (Kd) gain values among joint feedback states were both observed for feedback from the right ankle angle onto right knee torque and right hip torque, respectively. Among joint feedback states, the largest range in position (Kp) and velocity (Kd) gain values were both observed for feedback from the right hip angle onto right ankle torque. For hip-position feedback gains to any joint torque, the range in Y-coordinate Kp is greater than that of the X-coordinate, but the range in Y-coordinate Kv is less than that of the X-coordinate. There is similarity in the ranges of feedback gain values onto all three joints (hip, knee, ankle) for both the torso angle Kp gain (1100, 1350, 1830) and the torso angle Kv gain (131, 135, 101).

The mean and maximum absolute values for the optimal LQR feedback gains computed across three feedback groups (leg-joint angles, hip position, torso angle) for each Q/R ratio tested are seen in Table 1. With higher Q/R, the gain values increase monotonically for both Kp and Kv across all feedback groups, presumably because they are optimized to generate greater tracking performance relative to control effort. Relative to their respective maximum values, the mean Kp and Kv are smallest among joint feedbacks (4.5%, 8.6%, respectively) and largest for torso feedbacks (20.3%, 34.6%).

The feedback torques on the stance leg under LQR control (Q/R = 1) from changes in position and velocity states following prescribed downward vertical displacements at the hip are shown in Fig. 4. The resulting feedback torques were always directed in opposition to the angular state errors incurred from leg collapse as would be typically observed with conventional PD control. The mean torque was proportional to the hip displacement during collapse for all three joints, but torque magnitudes and rate of increase (slope) of torque to displacement were clearly largest at the knee.

The stability performance of each LQR controller in terms of the time to resist falling against a growing perturbation (type#1) is shown in Fig. 5. The mean time-to-fall monotonically improves with higher Q/R for Q/R from 0.001 to 2. Following a peak mean time-to-fall of 9.32 s at Q/R = 2, the mean time-to-fall generally decreases with higher Q/R, despite these controllers having higher gain values on average (see Table 1).

The efficiency performance of each LQR controller in terms of the RMS of the torque applied to resist a fixed bound perturbation is shown in Fig. 6. For the fixed bound perturbation, the two LQR controllers with the lowest Q/R ( = 0.001 and 0.01) incurred relatively large total torques because of falling prior to the completion of 10 full gait-cycles. For controllers with no falling under perturbation type#2, i.e., successful controllers, the total torque RMS nonmonotonically varied between 173 and 182 N·m. When observing RMS of only the feedback portions of the joint torques, the closed-loop control effort monotonically increased with higher Q/R and ranged between 20 and 65 N·m.

The RMS value of the error signal in each feedback state and the average (across right and left sides) RMS value of the respective joint feedback torque generated from LQR control are shown in Table 2. The relative contributions, defined as a percentage, for each feedback state to all torques generated are also shown. The feedback state generating the largest torque at the hip and knee joints was hip angle position. The feedback state generating the largest RMS torque value at the ankle joint was the Y-coordinate position of the hip. The two feedback states with largest relative contribution to all torques generated was the hip and knee angle positions. Results for summing RMS torque values within feedback state “groups” are shown in Table 3. The three groups consist of (1) the leg-joint angles, (2) two dimensions of hip position, and (3) global torso orientation. The relative contributions for each group to all torques generated are also shown. While the group consisting of leg-joint angle kinematics contributed the most (46.37%) to all torques generated, all three feedback groups each contributed over 20% of all torques generated.

The stability and efficiency results of the tested LQR controllers versus comparable PD controllers are concurrently shown in Fig. 7. The two LQR and two PD controllers with the lowest gains, denoted by the thinnest standard deviation lines, were clear outliers. These controllers were unable to prevent falling under perturbation type#2, which was used to establish efficiency. Among successful controllers, the two PD controllers with highest gains produced the best times-to-fall but at the expense of notably greater feedback torque (RMS > 80 N·m). All LQR controllers exhibited feedback torque RMS of less than 60 N·m with all but one between 20 and 40 N·m. Only LQR controllers simultaneously achieved time-to-fall of greater than 5 s and feedback torque RMS of less than 40 N·m.

## Discussion

This simulation study demonstrates fundamental operating characteristics of LQR feedback control of leg-joint torques during sagittal-plane walking. LQR control of walking considers mechanical coupling between all feedback states and all joint-torque controls. From these established relationships, the optimal gain values may be computed in accordance to a single controller design parameter (Q/R) that defines the relative consideration of minimizing tracking error (Q) versus minimizing control effort (R). Higher Q/R expectedly produced correspondingly higher feedback gain values on average. In designing real-world gait control systems, such as exoskeletons, it is desirable to employ lower feedback gains for stable operation. LQR controllers produced better performance across both stability and efficiency parameters over conventional PD control at the lower Q/R ratios. Therefore, stabilization of real-world exoskeleton gait by LQR may be more promising given its effectiveness at lower feedback gain values.

Profiles of the time-varying gain values were observed to be smooth but suggested complex cross-relationships between state feedback inputs to joint-torque outputs. Motor control and neurophysiology studies for walking have indicated higher gains during stance and lower gains during swing [1621,24]. Several of the LQR gain profiles suggest a similar trend, but this was not universal. Furthermore, time-varying gains switching between positive and negative values demonstrated the dependence on instantaneous gait dynamics in determining how state errors from the target gait trajectories can be optimally attenuated. These complex gain results are indicative of a holistic controller that considers whole-body dynamics over the full-gait cycle and not only kinematic tracking at individual joints. These results suggest that while challenging to implement for exoskeleton applications, optimal full-state feedback operates on principles notably different from simple PD control. As such, PD control approaches for exoskeleton gait may be inherently limited in their potential to achieve natural responses to disturbance during gait.

Transient existence of positive values for feedback gains of joint states are potentially destabilizing in contributing joint torques in the same direction as the errors in those states. But, the leg collapse analysis yielded net torques that are directed in the opposite direction of those errors just as classical PD joint control with negative feedback. This demonstrates that the synergy from the full-state control system could produce intuitive actions such as joint torques that directly oppose the fast-emerging joint errors as would occur with collapse. The observation of highest feedback torques being generated at the knee against buckling is also consistent with clinical intuition. In clinical settings, device developers want to ensure these types of fail-safe actions with applications such as exoskeleton-assisted human locomotion.

The modulation, or range, in gain values being relatively large for hip angle and hip position strongly suggest the importance in hip state feedback on maintaining stability of the entire system. The relatively smaller gain ranges of ankle state feedbacks infer that even ankle torques may be generated more in accordance to hip rather than ankle feedback. Exoskeleton developers may need to consider deployment of sensors systems that ensure greater measurement accuracy of hip feedback states and create actuation systems, including ankle torques, that are more dependent on those states.

The primary metrics observed in this study to assess performance of LQR control as a function of Q/R were ad hoc stability and efficiency. These metrics were positively indicated by greater time prior to falling and lower feedback torque generation against growing and fixed bound random perturbations, respectively. The greatest time-to-fall values were observed with Q/R between 1 and 10. Larger LQR feedback gains with Q/R greater than 10 may be producing torques that are counterproductive to resisting falls. This observation is surprising based on the controller design specification, whereby higher Q/R should produce a controller with better tracking capabilities at the expense of greater effort. The observed performance reduction may be explained by linear controller design being applied to nonlinear system dynamics where assumptions of linear behavior are breaking down with gains too high. Higher gains coupled with full-state feedback to each control torque may amplify applied torques in ways that are unstable. Furthermore, it is possible that control limits on purely kinematic controls are being reached with this approach, and the need for additional strategy (e.g., center of pressure monitoring) is required. Another possible contributor to reduced tracking performance despite higher Q/R may be due to the underactuation of the biped walking system [28]. However, better LQR performance both absolutely and relative to PD at the lower gains would still be acceptable for exoskeleton applications that would desirably operate at lower gains for the sake of safe and stable human–device interaction.

Consistent with the linear control law, greater controller effort was generally observed to be positively related to larger gains given higher Q/R. The total torque response largely increased with higher Q/R but may not have been directly proportional to Q/R due to system nonlinearity. In any case, controller operation optimized about local linearized representations of the system dynamics inherently limits the ability to reject larger perturbations. In this study, a random, bidirectional (fore-aft) perturbation force at the hip that is bounded to be within 5 N in magnitude could be rejected for all LQR controllers tested except those with Q/R below 0.1. We selected the 5 N value from preliminary observations indicating we could study a large range of controllers within this force level. Although not addressed in this study, larger perturbations would be rejected by LQR controllers with higher Q/R within specific limits. Since LQR is formulated about dynamics local to the desired gait trajectories, observed deviations from desired state trajectories for successful walking were small. Mean errors for position and angle states were below 5 cm and 0.02 rad, respectively, as shown in Table 2. Other controller modes [811,29] would still be necessary when perturbations cause deviations in the feedback states to be too large as could be expected with exoskeleton applications.

While proportional-derivative joint feedback control is often employed for bipedal walking applications [1,2,3032], including exoskeletons, inclusion of LQR operation in control of walking [3,4,912,3336] offers important insights and distinct performance advantages. PD controllers are intuitive and more simply tuned given only 12 gain parameters (six proportional, six derivative) associated with the six lower-extremity joints to be controlled [31]. However, PD control may not proficiently consider the natural mechanical synergies that emerge across the entire system during walking. For the combined objective of stability and efficiency, the family of successful LQR controllers tested greatly outperformed comparable PD controllers in this study. The LQR controllers could produce the best time-to-fall values while also generating feedback torque RMS values below 40 N·m. The PD controller that achieved similar time-to-fall values generated feedback torque RMS of over 130 N·m on average. It appears that PD controllers with even higher gains than those tested may achieve better time-to-fall values. But, this improvement in performance comes at the expense of generating even larger torque efforts, which may be impractical and unstable to implement with exoskeletons that are limited in torque actuation due in part to design features of being light-weight or fitting within standard wheelchairs. Another consideration for practical implementation of LQR-based control is that strict time-dependent control may be insufficient for reliable operation. Exoskeleton control modulation based on gait-phase variables may produce time-varying operation that is more robust [37].

This simulation analysis provided inferences as to which state variables contribute most to feedback torque. In real-world applications, it may be critical to ensure feedback measurements from these states are especially accurate. Hip and knee angular positions had the greatest individual state contributions to closed-loop torque generation and are conventionally utilized for walking applications [32]. However, the next state with largest contribution was the hip position in the superior–inferior (Y-coordinate) dimension, which can be challenging to measure with conventional instrumentation. Among velocity feedbacks, the global torso orientation had the largest contribution. Overall, feedback from states associated with hip-position and torso orientation contributed over 50% of the feedback torque being generated. Quantifying these novel states may reduce the need to accurately assess other states, but may require implementing innovative modes of measurement. While feedback states may be coupled such that some state estimation is redundant, the LQR control solution generating such high feedback contributions from hip-position and torso orientation suggest their intrinsic value toward optimal control. These findings suggest that more efficient exoskeleton control of walking may require sensor-based measurement of kinematic variables beyond those of the leg joints. Inertial sensing units for estimating torso orientation are available in commercial exoskeletons (e.g., within hip component of INDEGO by Parker Hannifin). While these sensors are used to reliably trigger stepping by forward leaning, the measured information can also be utilized for feedback about global torso orientation to potentially drive LQR-like optimal controllers. However, additional instrumentation with better distribution across the whole torso may be needed for better measurement assessment of global torso orientation. Information about forward progression of the system, such as with hip position, may be inferred from expected coupling of the other states or perhaps measured via high-accuracy global positioning system [38].

In developing exoskeleton-assisted walking applications for individuals with spinal cord injury, future work would include deriving controller feedback estimates of system states from physical sensors and incorporating actuation from functional electrical stimulation of muscles. Due to limitations in joint-torque magnitudes generated by powered orthoses, identifying optimal gait trajectories for walking models that include constraints on leg-joint torques and models that describe the upper-body support actions of the user with crutches or walkers may also be needed. While time-varying LQR approaches can provide a strong template for improved gait performance for applications such as exoskeletons compared to conventional and practical PD control, integration of the human operator must also be more fully considered in future studies. It may be necessary to map user intent, perhaps via upper-body action or other physiological (electromyography [40], electroencephalography [41]) command sources, to trigger the desired reference trajectory online. As such, it would be necessary to formulate a continuum of reference trajectories from which the exoskeleton controller would utilize to best support the intent and actions of the user.

The main conclusions of this simulation study of LQR control of bipedal gait to be considered for exoskeleton controller design are as follows:

1. (1)

Relative to comparable PD controllers, LQR controllers could better maintain walking and at reduced effort against persistent random perturbations.

2. (2)

Linear quadratic regulator solutions with relative cost function weighting ratios of tracking over effort (Q/R) between 1 and 10 produced the best resistance to falling.

3. (3)

Despite gain values occasionally being positive, the net actions of the full-state controller desirably produced negative feedback torques against leg collapse.

4. (4)

For optimal full-state control, hip position and torso angle states generated notable contributions to feedback torques suggesting their fundamental control value.

## Acknowledgment

We would like to thank Parker Hannifin Corporation for their support of the Human Motion Control Laboratory at Cleveland State University.

## Funding Data

• National Science Foundation (Grant No. NSF 1344954).

## References

References
1.
Farris
,
R. J.
,
Quintero
,
H. A.
, and
Goldfarb
,
M.
,
2012
, “
Performance Evaluation of a Lower Limb Exoskeleton for Stair Ascent and Descent With Paraplegia
,”
IEEE International Conference on Engineering in Medicine and Biology Society (EMBC)
, San Diego, CA, Aug. 28–Sept. 1, pp. 1908–1911.
2.
Farris
,
R. J.
,
Quintero
,
H. A.
,
Murray
,
S. A.
,
Ha
,
K. H.
,
Hartigan
,
C.
, and
Goldfarb
,
M.
,
2014
, “
A Preliminary Assessment of Legged Mobility Provided by a Lower Limb Exoskeleton for Persons With Paraplegia
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
22
(
3
), pp.
482
490
.
3.
Kajita
,
S.
,
Kanehiro
,
F.
,
Kaneko
,
K.
,
Fujiwara
,
K.
,
,
K.
,
Yokoi
,
K.
, and
Hirukawa
,
H.
,
2003
, “
Biped Walking Pattern Generation by Using Preview Control of Zero-Moment Point
,”
IEEE International Conference on Robotics and Automation
(
ICRA
), Taipei, Taiwan, Sept. 14–19, pp.
1620
1626
.
4.
Shih
,
C.-L.
,
1996
, “
The Dynamics and Control of a Biped Walking Robot With Seven Degrees of Freedom
,”
ASME J. Dyn. Syst. Meas. Control
,
118
(
4
), pp.
683
690
.
5.
Mason
,
S.
,
Righetti
,
L.
, and
Schaal
,
S.
,
2014
, “
Full Dynamics LQR Control of a Humanoid Robot: An Experimental Study on Balancing and Squatting
,”
14th IEEE-RAS International Conference on Humanoid Robots
(
Humanoids
), Madrid, Spain, Nov. 18–20, pp.
374
379
.
6.
Mason
,
S.
,
Rotella
,
N.
,
Schaal
,
S.
, and
Righetti
,
L.
,
2016
, “
Balancing and Walking Using Full Dynamics LQR Control With Contact Constraints
,”
IEEE-RAS 16th International Conference on Humanoid Robots
(
Humanoids
), Cancun, Mexico, Nov. 15–17, pp.
63
68
.
7.
Posa
,
M.
,
Kuindersma
,
S.
, and
Tedrake
,
R.
,
2016
, “
Optimization and Stabilization of Trajectories for Constrained Dynamical Systems
,”
IEEE International Conference on Robotics and Automation
(
ICRA
), Stockholm, Sweden, May 16–21, pp.
1366
1373
.
8.
Tedrake
,
R.
,
Manchester
, I
. R.
,
Tobenkin
,
M.
, and
Roberts
,
J. W.
,
2010
, “
LQR-Trees: Feedback Motion Planning Via Sums-of-Squares Verification
,”
Int. J. Rob. Res.
,
29
(
8
), pp.
1038
1052
.
9.
Kubica
,
E.
,
Wang
,
D.
, and
Winter
,
D.
,
2001
, “
Feedforward and Deterministic Fuzzy Control of Balance and Posture During Human Gait
,”
IEEE International Conference on Robotics and Automation
(
ICRA
), Seoul, South Korea, May 21–26, pp.
2293
2298
.
10.
Herdt
,
A.
,
Diedam
,
H.
,
Wieber
,
P.-B.
,
Dimitrov
,
D.
,
Mombaur
,
K.
, and
Diehl
,
M.
,
2010
, “
Online Walking Motion Generation With Automatic Footstep Placement
,”
,
24
(
5–6
), pp.
719
737
.
11.
Morimoto
,
J.
,
Endo
,
G.
,
Nakanishi
,
J.
,
Hyon
,
S.
,
Cheng
,
G.
,
Bentivegna
,
D.
, and
Atkeson
,
C. G.
,
2006
, “
Modulation of Simple Sinusoidal Patterns by a Coupled Oscillator Model for Biped Walking
,”
IEEE International Conference on Robotics and Automation
(
ICRA
), Orlando, FL, May 15–19, pp.
1579
1584
.
12.
,
M. J.
,
Taherkhorsandi
,
M.
, and
Bagheri
,
A.
,
2014
, “
Pareto Design of State Feedback Tracking Control of a Biped Robot Via Multiobjective PSO in Comparison With Sigma Method and Genetic Algorithms: Modified NSGAII and MATLAB's Toolbox
,”
Sci. World J.
,
2014
, p.
303101
.
13.
Kuo
,
A. D.
,
2005
, “
An Optimal State Estimation Model of Sensory Integration in Human Postural Balance
,”
J. Neural Eng.
,
2
(
3
), pp.
S235
S249
.
14.
Loeb
,
G. E.
,
Levine
,
W. S.
, and
He
,
J.
,
1990
, “
Understanding Sensorimotor Feedback Through Optimal Control
,”
Cold Spring Harbor Symp. Quant. Biol.
,
55
, pp.
791
803
.
15.
He
,
J.
,
Levine
,
W. S.
, and
Loeb
,
G. E.
,
1991
, “
Feedback Gains for Correcting Small Perturbations to Standing Posture
,”
IEEE Trans. Autom. Control
,
36
(
3
), pp.
322
332
.
16.
Geyer
,
H.
, and
Herr
,
H.
,
2010
, “
A Muscle-Reflex Model That Encodes Principles of Legged Mechanics Produces Human Walking Dynamics and Muscle Activities
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
18
(
3
), pp.
263
273
.
17.
Nielsen
,
J. B.
, and
Sinkjaer
,
T.
,
2002
, “
Afferent Feedback in the Control of Human Gait
,”
J. Electromyography Kinesiology
,
12
(
3
), pp.
213
217
.
18.
Sinkjaer
,
T.
,
Andersen
,
J. B.
,
,
M.
,
Christensen
,
L. O. D.
, and
Nielsen
,
J. B.
,
2000
, “
Major Role for Sensory Feedback in Soleus EMG Activity in the Stance Phase of Walking in Man
,”
J. Physiol.
,
523
(
3
), pp.
817
827
.
19.
Anderson
,
F. C.
, and
Pandy
,
M. G.
,
2001
, “
Dynamic Optimization of Human Walking
,”
ASME J. Biomech. Eng.
,
123
(
5
), pp.
381
390
.
20.
Thelen
,
D. G.
, and
Anderson
,
F. C.
,
2006
, “
Using Computed Muscle Control to Generate Forward Dynamic Simulations of Human Walking From Experimental Data
,”
J. Biomech.
,
39
(
6
), pp.
1107
1115
.
21.
Kuo
,
A. D.
,
2002
, “
Energetics of Actively Powered Locomotion Using the Simplest Walking Model
,”
ASME J. Biomech. Eng.
,
124
(
1
), pp.
113
120
.
22.
Winter
,
D. A.
,
2005
,
Biomechanics and Motor Control of Human Movement
,
Wiley
,
New York
.
23.
Ackermann
,
M.
, and
van den Bogert
,
A. J.
,
2010
, “
Optimality Principles for Model-Based Prediction of Human Gait
,”
J. Biomech.
,
43
(
6
), pp.
1055
1060
.
24.
Gerritsen
,
K. G. M.
,
van den Bogert
,
A. J.
,
Hulliger
,
M.
, and
Zernicke
,
R. F.
,
1998
, “
Intrinsic Muscle Properties Facilitate Locomotor Control—A Computer Simulation Study
,”
Mot. Control
,
2
(
3
), pp.
206
220
.
25.
van den Bogert
,
A. J.
,
2011
, “
gait2dem Model
,” arker Hannifin Human Motion Control Laboratory, Cleveland, OH, accessed Aug. 16, 2017, http://hmc.csuohio.edu/resources/musculoskeletal-modeling-and-simulation
26.
Hench
,
J. J.
, and
Laub
,
A. J.
,
1994
, “
Numerical Solution of the Discrete-Time Periodic Riccati Equation
,”
IEEE Trans Autom. Control
,
39
(
6
), pp.
1197
1210
.
27.
Varga
,
A.
,
2005
, “
On Solving Discrete-Time Periodic Riccati Equations
,”
16th IFAC World Congress
, Prague, Czech Republic, July 3–8, pp.
354
359
.http://ac.els-cdn.com/S1474667016366423/1-s2.0-S1474667016366423-main.pdf?_tid=e1879424-8272-11e7-be20-00000aab0f6c&acdnat=1502881738_89ea9b104df19c47362d173065909cc3
28.
Braun
,
D. J.
, and
Goldfarb
,
M.
,
2009
, “
A Control Approach for Actuated Dynamic Walking in Biped Robots
,”
IEEE Trans. Rob.
,
25
(
6
), pp.
1292
1303
.
29.
Moore
,
J.
,
Cory
,
R.
, and
Tedrake
,
R.
,
2014
, “
Robust Post-Stall Perching With a Simple Fixed-Wing Glider Using LQR-Trees
,”
Bioinspiration Biomimetics
,
9
(
2
), p.
025013
.
30.
Coros
,
S.
,
Beaudoin
,
P.
, and
van de Panne
,
M.
,
2010
, “
Generalized Biped Walking Control
,”
ACM Trans. Graphics
,
29
(
4
), p.
130
.
31.
Pratt
,
J.
, and
Pratt
,
G.
,
1998
, “
Intuitive Control of a Planar Bipedal Walking Robot
,”
IEEE International Conference on Robotics and Automation
(
ICRA
), Leuven, Belgium, May 16–20, pp.
2014
2021
.https://doi.org/10.1109/ROBOT.1998.680611
32.
Yin
,
K.
,
Loken
,
K.
, and
van de Panne
,
M.
,
2007
, “
Simbicon: Simple Biped Locomotion Control
,”
ACM Trans. Graphics
,
26
(
3
), p.
105
.
33.
Czarnetzki
,
S.
,
Kerner
,
S.
, and
Urbann
,
O.
,
2009
, “
Observer-Based Dynamic Walking Control for Biped Robots
,”
Rob. Auton. Syst.
,
57
(
8
), pp.
839
845
.
34.
Dimitrov
,
D.
,
Wieber
,
P. B.
,
Stasse
,
O.
,
Ferreau
,
H. J.
, and
Diedam
,
H.
,
2009
, “
An Optimized Linear Model Predictive Control Solver for Online Walking Motion Generation
,”
IEEE International Conference on Robotics and Automation
(
ICRA
), Kobe, Japan, May 12–17, pp.
1171
1176
.
35.
Homich
,
A. J.
,
Doerzbacher
,
M. A.
,
Tschantz
,
E. L.
,
Piazza
,
S. J.
,
Hills
,
E. C.
, and
Moore
,
J. Z.
,
2015
, “
Minimizing Human Tracking Error for Robotic Rehabilitation Device
,”
ASME J. Med. Dev.
,
9
(
4
), p.
041003
.
36.
Kim
,
M.
, and
Collins
,
S. H.
,
2015
, “
Once-Per-Step Control of Ankle-Foot Prosthesis Push-Off Work Reduces Effort Associated With Balance During Walking
,”
J. Neuroeng. Rehabil.
,
12
(
1
), p.
43
.
37.
Gregg
,
R. D.
,
Rouse
,
E. J.
,
Hargrove
,
L. J.
, and
Sensinger
,
J. W.
,
2014
, “
Evidence for a Time-Invariant Phase Variable in Human Ankle Control
,”
PLoS One
,
9
(
2
), p.
e89163
.
38.
Terrier
,
P.
, and
Schutz
,
Y.
,
2005
, “
How Useful Is Satellite Positioning System (GPS) to Track Gait Parameters? A Review
,”
J. Neuroeng. Rehabil.
,
2
(
1
), p.
28
.
39.
Todorov
,
E.
,
2005
, “
Stochastic Optimal Control and Estimation Methods Adapted to the Noise Characteristics of the Sensorimotor System
,”
Neural Comput.
,
17
(
5
), pp.
1084
1108
.
40.
Kiguchi
,
K.
,
Tanaka
,
T.
, and
Fukuda
,
T.
,
2004
, “
Neuro-Fuzzy Control of a Robotic Exoskeleton With EMG Signals
,”
IEEE Trans. Fuzzy Syst.
,
12
(
4
), pp.
481
490
.
41.
Veneman
,
J. F.
,
Kruidhof
,
R.
,
Hekman
,
E. E.
,
Ekkelenkamp
,
R.
,
Van Asseldonk
,
E. H.
, and
Van Der Kooij
,
H.
,
2007
, “
Design and Evaluation of the LOPES Exoskeleton Robot for Interactive Gait Rehabilitation
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
15
(
3
), pp.
379
386
.