A Wireless Human Motion Monitoring System for Smart Rehabilitation

The world is experiencing an unprecedent, enduring, and pervasive aging process [1]. One major challenge that aging brings about is the degenerative conditions of the musculoskeletal system (such as osteoporosis and arthritis) and nervous system (such as Alzheimer's disease [2], stroke [3], and Parkinson's disease [4]). As a result, an increasing number of people have suffered from walking difficulties, and the demand for gait physical therapy has increased rapidly over the years.

The current gait physical therapy is provided by therapists who manually stimulate patients' reflexes and rotate their lower limbs to retrain their central nervous systems with the correct gait patterns, as shown in Fig. 1. This approach can be physically demanding for both the patient and the therapist. Supervised therapy is also expensive and can be time-consuming. Moreover, in the clinical setting, assessments of gait abnormalities are based on the visual observation therapists, video analysis, physical performance tests (strength, range of motion, balance, gait speed, endurance, and symmetry), as well as patient self-reports (confidence, resilience, pain, fatigue, freezing, and falls). Therapists usually do not have access to the objective measurements of ground contact forces (GCFs), joint rotations, step length, or movement of the CoP. As a consequence, gait evaluations can include subjective information which may not be accurate.

It is also observed in a clinical environment that patients with neurological impairments often have a difficult time understanding their gait abnormalities. Usually patients simply try to follow the commands from physical therapists without knowing how serious their problems are or how much progress they have made in the training. Therefore, it is very important to provide useful and intuitive visual feedback to patients so that they can be better informed of their walking behaviors and rehabilitation training progress.

As the first step of gait rehabilitation therapy, it is very important to accurately capture the human motion. Motivated by the aforementioned requirements from clinical physical therapy, different types of human motion capture systems have been developed. To name a few, pressure sensors and force plates have been adopted to measure the GCFs for gait inference [5,6]; electromyogram sensors have been used to analyze electrical activities of the contracting muscles [7]; inertial sensors have been widely used to estimate joint kinematics of the lower extremities during walking [8,9]; and vision-based sensors have also been employed for reconstruction of the human motion [10,11]. The sensors mentioned above have been widely used in clinical environment, and researchers have also proposed to use those sensors at home for long-term health monitoring [12,13].

Despite all the human motion capture systems mentioned above, there are still some problems that need to be solved before those systems can be used in clinical rehabilitation, which include:

• Feedback: although quite a few sensors can measure the users' walking kinematics, many of them are just used to record the data instead of providing useful feedback to the users, which is not adequate to perform active intervention on the patients.

• Visual feedback design: it is clearly not a good idea to display all the measurement data to patients as they may not be able to understand and digest the overwhelming messages. Moreover, too much visual feedback will distract the patients' attention and make it difficult for them to focus on correcting their gait patterns. As a result, only selected useful feedback should be displayed in real-time to patients.

• Overall mobility: despite the wireless technology being widely used in human motion capture systems, the receiver of the sensor signals is usually a computer. However, the patient cannot look at the computer screen all the time when they perform selected rehabilitation exercise. Therefore, a wireless mobile display needs to be developed for the patients.

• Usability: patients will easily get bored if they conduct the rehabilitation exercise for a long period, so it is important to make the training process engaging and informative so that they can be motivated to continue the rehabilitation training.

In this paper, a wireless human motion monitoring system is designed to address the problems mentioned above and bridge the gap between academic development and clinical use. An inertial measurement unit (IMU) is employed to estimate the joint rotation in three dimensions. Four pressure sensors are embedded in each smart shoe to measure the GCFs for gait phase inference. Raw data from the sensors can be used to calculate the gait phase and kinematic information. Wireless modules are integrated into the system so that it could work independently without wiring to a local computer. User interfaces are developed on both a laptop and an iPad to provide visual feedback to the patients. The novelty of the system lies in the following aspects:

• Instead of just using the raw sensor data, systematic approaches are developed to process the data and calculate the clinical gait parameters to provide more insights.

• The user interface is able to provide intuitive information so that patients can easily understand and integrate the feedback into their gait exercise.

• The system is tested on both healthy subjects and patients with gait disorders. Moreover, clinical effectiveness of the system is further verified in a clinical study with 24 patients [14].

The remainder of the paper is organized as follows: Section 2 gives an overview of the wireless human motion monitoring system and discusses the importance of visual feedback. Section 3 introduces the sensory devices of the system. Gait inference and kinematic analysis are performed in Sec. 4. Section 5 introduces the visual feedback design on a laptop and an iPad. Section 6 sketches the design of the clinical study and details the experimental results from one healthy subject, one stroke patient, and one Parkinson's disease patient. Conclusions and future work are presented in Sec. 7.

In this section, advantages of visual feedback in a rehabilitation system are briefly introduced. Structure of the proposed wireless human motion monitoring system is introduced as well.

A block diagram of the rehabilitation training is shown in Fig. 2. In the traditional rehabilitation training, a physical therapist observes a patient's walking patterns and provides auditory feedback to the patient. The patient then tries to correct his or her gait patterns. In many cases, the physical therapist also provides physical assistance when the patient cannot perform the designed tasks, as shown in Fig. 1. It is clear that feedback to the patient is heavily based on the therapists' knowledge and experience. However, a patient's abnormal gait patterns, especially incorrect force distributions on the two feet, may not be very obvious for a therapist to observe. Furthermore, it is difficult to evaluate the progress of rehabilitation based on observation only.

To improve the performance and efficiency of the rehabilitation training, new feedback is introduced to patients and therapists by some wireless motion sensors. Selected measurement data from the sensors are shown on a laptop as well as an iPad so that patients can get visual feedback regarding their gait abnormalities and training progresses they have made. The visual feedback, together with the command signals and physical assistance from therapists, helps the patients to stimulate their central nervous systems for gait correction. Moreover, the measurement data can be stored in a database to systematically study the effectiveness of the intervention and develop some metrics for disease diagnosis.

Structure of the proposed system is shown in Fig. 3. In the system, a user walks with a pair of smart shoes and several joint angle sensors. The smart shoes and joint angle sensors send measurement data to a laptop wirelessly. A labview program is developed in the laptop to read the sensor signals through serial ports, and the different sensor readings are synchronized in the program. The program then processes the raw data from the sensors and sends selected data to an iPad for visual feedback. Bluetooth low energy (BLE, Bluetooth v4.0) protocol is employed for communication between the laptop and iPad. Alternatively, the user can also choose to open an application program in this laptop to receive the visual feedback. All the raw and processed data are stored in the laptop for further study.

In this section, design of the two measurement devices, wireless joint angle sensors and smart shoes, is introduced.

The wireless joint angle sensors are developed to measure the joint rotations in three dimensions. A nine degrees-of-freedom IMU sensor stick is employed as the main measurement device. The IMU sensor stick includes an accelerometer, magnetometer, and gyroscope, each with three degrees-of-freedom. A microcontroller is employed to read the raw sensor data from the IMU sensor stick through the serial peripheral interface bus. A wireless module is connected to the microcontroller through serial ports to send out the measurement packets wirelessly. The sensor node is powered by a Li–Po battery and it can work continuously for 90 mins. The dimension of the joint angle sensor node is 2 in. × 1.4 in. × 0.6 in. and its weight is around 0.15 lbs including the battery. The joint angle sensors are attached to human body using velcros, and it can also be firmly screwed to exoskeletons or prostheses.

It is well known that the integration of the angular rate measurement from a gyroscope results in drift of angle estimate, and acceleration measurement from the accelerometer is often contaminated with high-frequency noise. Both phenomena will make the angle estimate inaccurate. To deal with this problem, a time-varying complementary filter (TVCF) is implemented in the microcontroller for onboard signal processing. The idea of the TVCF is to pass vector measurements from the accelerometer and magnetometer through a low-pass filter to cancel the sensor noise, and pass the rate measurements from the gyroscope through a high-pass filter to deal with the drift effect. Cutoff frequency of the TVCF is tuned automatically based on fuzzy logic. More details of the TVCF are available in Ref. [15].

After $q2→1$ is obtained, it can be converted to the Euler angles for intuitive visualization [17].

In order to better analyze patients' gaits during walking, a pair of smart shoes is developed to measure the GCFs on two feet. Four barometric sensors are employed to measure the GCFs on the toe, the first and second metatarsophalangeal joint (Meta12), the fourth and fifth metatarsophalangeal joint (Meta45), and the heel. Silicone tubes are wound into air bladders to connect barometric sensors with a measurement range from 0 to 250 mbar. Each sensor can measure weight up to 200 lbs with a resolution of 0.2 lbs. The air bladders and barometric sensors are calibrated with a load cell [5]. Figure 4(a) shows the air bladders on a shoe pad.

The pressure sensor outputs are read by an Arduino Pro Mini 328 microcontroller through analog input channels, and the sensor signals are sent out through a bluetooth module. The bluetooth module can smoothly and reliably transmit signals within 200 feet to the receiver, which is enough for normal clinical use. A 9-V alkaline battery is used to power the smart shoes, and it can work consecutively for 90 mins. The sampling rate of the smart shoes is set as 50 Hz with the bluetooth module. A pair of wireless smart shoes is shown in Fig. 4(b).

Based on the raw measurements, some metrics can be calculated to provide insights for therapists to diagnose patients' walking problems and develop appropriate training plans.

Gait phases reveal human walking patterns, and they are frequently used in clinical gait analysis [18,19]. There are eight gait phases within a normal gait cycle: initial contact (IC), loading response (LR), midstance, terminal stance (TS), preswing (PS), initial swing, midswing, and terminal swing. In this paper, the last three swing phases are combined as a swing phase (S). In a healthy gait, all the gait phases should be detected and time allocation in each phase within one gait cycle should follow a predefined pattern to maintain walking stability. Gait phases can be detected by the GCF patterns from smart shoe measurements. Fuzzy logic [5] and a hidden Markov model (HMM) [20] have been proposed to infer gait phases during walking.

where $xCoP(t)$ and $yCoP(t)$ are the coordinates in the directions of foot width and foot length.

Stride length is the distance between two successive placements of the same foot [19], as shown in Fig. 5. It consists of two step lengths, left and right, each of which is the distance by which the named foot moves forward to the other one. The two step lengths are similar in a normal gait, but they can be very different in a pathological gait [22].

Inertial sensors have been widely used in estimating step length, which is often modeled as a linear combination of several parameters related, such as body height, walking frequency, variance of the accelerometer signals, and so on [23,24]. However, this technique requires precise identification of the model coefficients, which vary significantly for different patients and for one patient in different training stages.

An alternative method is used to calculate the step length based on the geometry and lengths of the lower limbs. As shown in Fig. 6, the step length is calculated at the moment when one foot is touching the ground (heel strike phase) while the other foot is about to leave the ground (PS phase). In Fig. 6, this moment is captured when the right hip joint rotation reaches the maximum.

where l_f is the foot length, and l_tl, l_sl, l_tr, and l_sr are the lengths of the thigh and shank on the left and right sides, respectively. The left step length can be calculated in a similar way. Based on the joint angle measurements, toe-out angles and stride widths can be calculated as well. Due to the space limit, they will not be emphasized in this paper but briefly introduced in Sec. 4.4.

The toe-out describes the angle between the direction of progression and a reference line on the sole of the foot [19]. The toe-out angle can be extracted from the ankle joint rotation in the transverse plane. The toe-out angle can affect the force distribution on the foot, and it is therefore very important to keep this angle in the normal range [25].

The stride width is the side-to-side distance between the lines of the two feet [19]. The stride width can be measured either at the back of the heel or the center of the ankle. The stride width can be calculated in a similar way to step length. The only difference is one should use joint angles in the frontal plane for calculation.

An independent application is developed based on the Windows operation system. After opening this file, a user interface is created, which demonstrates the GCF on each sensing point and the CoP on each side in real-time. An animation is also developed based on the open-source graphics rendering engines (OGRE) [26] so that patients can easily visualize their walking behaviors. Figure 7 shows a subject wearing smart shoes and the wireless joint angle sensors, together with the user interface on the laptop monitor.

When a patient walks on the ground, it is inconvenient and unsafe to track his or her walking behaviors by looking at the laptop monitor all the time. To make the system more user-friendly, an application program is developed on the iOS. The processed sensor signals are transmitted from the laptop to the iPad using the BLE protocol. Therefore, the user is able to see the visual feedback on the iPad in real-time. The user interface of the application is shown in Fig. 8, where the force distribution, stride length, toe-out angle, and stride width are displayed and updated in real-time. A score given by the physical therapist is also shown on the user interface, and the score is given after each training session based on the progress of patient. In addition to the gait parameters, some customized messages are also displayed to patients based on the gait parameters. For example, if a patient has uneven step lengths on the left and right sides, a warning message will be displayed, as shown in Fig. 8. Similarly, a message can be designed to remind the patient for stronger heel strike based on the smart shoe measurement.

In order to make it easier for patients to get visual feedback from the iPad program, an iPad holder is developed on a belt so that patients or therapists do not need to hold the iPad during the training, as shown in Fig. 9. Patients can rotate the iPad in the holder to find their best view of the screen.

The experimental results are presented in this section. Performance of the wireless joint angle sensors is first examined. The wireless human motion monitoring system has been tested with healthy subjects and then applied to a clinical study with 24 stroke and Parkinson's disease patients. Experimental results with one healthy subject, one stroke patient, and one Parkinson's disease patient are illustrated in this section to verify the performance of the system.

In this section, a PhaseSpace IMPULSE motion capture system with ten calibrated cameras [27] was employed to verify the accuracy of joint angle estimation. Each camera had 3600 × 3600 pixels and 480 frame-per-second (FPS) sampling rate. There were at least three markers and one IMU sensor for the torso and each lower limb in order to estimate the joint rotations, as shown in Fig. 10. Specifically, in the system there was one IMU sensor on each side of the waist, one on each thigh, and one on each shank. The three markers were placed to be parallel with the IMU sensor on one limb to minimize the error due to misalignment.

In the experiment, the subject was asked to perform lower limb lifting, forward walking, side walking, and squat. Hip and knee joint rotations in squat are compared from the motion capture system and IMU sensors, as shown in Fig. 11. For left hip, right hip, left knee, and right knee, the mean absolute difference values between the two measurements are 2.56 deg, 2.87 deg, 6.78 deg, and 6.44 deg, respectively. It can be confirmed that the estimation results from the two measurement techniques are quite close, which confirms the accuracy of the IMU sensors. It is also clear that the setup of IMU sensors is much simpler than that of the camera-based motion capture system and such advantage plays an important role in a clinical environment.

In order to examine the performance of the system, experiments were conducted on both healthy subjects without known walking problems and patients with stroke and Parkinson's disease. Experiments with healthy subjects were conducted in the Mechanical Systems Control Laboratory at the University of California, Berkeley, as shown in Fig. 9. The clinical study with patients was conducted in the William J. Rutter Center at the University of California, San Francisco (UCSF). The purpose of the clinical study was to examine whether patients could use visual feedback to direct their rehabilitation training and how was the training performance compared to traditional rehabilitation training directed by a physical therapist. The Committee on Human Research (CHR) at UCSF reviewed and approved this study. Twenty-four stroke and Parkinson's disease patients participated in the study and they were randomly assigned to either the control (traditional rehabilitation training) or experimental (training with visual feedback from both the laptop and iPad) group. Both groups demonstrated statistically significant progress during the gait training, but no statistically significant difference was observed in the clinical training performance between the control and experimental groups, which suggests the use of this system for in-home gait rehabilitation. Detailed experimental design and statistical analysis of the clinical outcomes are available in Ref. [14].

In this section, the experimental results from one healthy subject, one stroke patient, and one Parkinson's disease patient are demonstrated. The healthy subject is male, weighs 140 lbs, and is 5 ft 8 in. tall. The stroke patient is male, weighs 176 lbs, and is 5 ft 10 in. tall. The Parkinson's disease patient is female, weighs 155 lbs, and is 5 ft 5 in. tall. In this section, experimental data are presented for the three subjects with their normal walking speeds.

The left-side GCFs and detected gait phases of the healthy subject are shown in Fig. 12. From Fig. 12(a) one can observe that at each step, the heel pressed the ground first, then, Meta45 and Meta12 touched the ground, and finally, the toe pushed the ground before the swing phase. In this paper, the fuzzy logic technique [5] is implemented to infer gait phases based on the measurement data from smart shoes. Figure 12(b) shows the fuzzy membership value (FMV), where the FMV of a specific gait phase indicates the likelihood that the user is currently in that phase [5]. It can be confirmed that all the six phases are successfully detected by the fuzzy logic, and the detected phases are sequential because the subject followed a normal gait pattern. Based on the pressure sensor measurements, CoP on the left foot in each gait cycle is calculated and shown in Fig. 13(a). It is observed that the CoP rolls from the heel all the way to the forefoot, and the CoP trace is closer to the medial side of the foot to maintain balance.

Left hip and knee joint rotations of the healthy subject are shown in Fig. 14. Due to the space limit, only measurements in the sagittal plane are shown in this paper. The statistics come from 50 consecutive steps from the healthy subject. For mean values, the hip joint reaches a maximum flexion of around 50 deg and the knee joint reaches a maximum flexion of approximately 55 deg. The standard deviation is around 2 deg at each point in a gait cycle, indicating a very consistent gait pattern for the healthy subject. Note that there are two local minima of knee flexion in a gait cycle. When the hip joint reaches the maximum flexion, the knee joint reaches the first minimum flexion, which corresponds to the IC phase in a gait cycle (start of the gait cycle). The knee flexion reaches the second minimum when the hip joint flexion is around 20 deg (40% of the gait cycle), which corresponds to TS phase in a gait cycle. Those joint angles and force distributions change periodically to generate stable and smooth walking dynamics in a normal gait.

The right-side GCFs and detected gait phases of the stroke patient are shown in Fig. 15. The right side was the affected side of the patient, which means he suffered abnormal gait patterns mainly on this side due to stroke. As shown in Fig. 15(a), although the patient's gait was also repetitive, he tended to press all the four sensors simultaneously instead of rolling from the heel to the forefoot. Moreover, some pressure was detected by the toe sensor before heel strike, which indicates that the patient does not fully release his toe in the swing phase. The gait phase detection result in Fig. 15(b) is consistent with the GCF pattern. Since the subject cannot support his weight with his heel only, there is no IC phase. The lack of PS phase results from the fact that he cannot shift his body weight to the toe right before lifting his foot off the ground. The CoP on the right foot is shown in Fig. 13(b). Comparing with Fig. 13(a), the range of CoP is much smaller and the CoP does not start from the heel. This further verifies that one significant problem of the patient is the lack of strong and independent heel support.

Right hip and knee joint rotations of the stroke patient in the sagittal plane are shown in Fig. 16. While the walking speed of the stroke patient is similar to that of the healthy subject, ranges of both hip and knee joint rotations are much smaller. Moreover, the knee joint rotation pattern is also different because it has an inflection point (60–70% in the gait cycle) in Fig. 16(b). Last but not least, the standard deviations of the hip and knee joint angles are larger than those of the healthy subject, which indicates more variations in his gait patterns between steps.

Figure 17(a) shows the GCF signals and the detected gait phases from the left (affected) side of the Parkinson's disease patient. Three problems can be clearly identified from the smart shoe signals, with the first one being weak heel strike. The maximum heel strike force is around 50 lbs, which is much less than the force on the support force from the forefoot. The second abnormality lies in the improper lateral boarder support. Figure 12(a) indicates that a healthy subject tends to put more body weight on the medial boarder during forefoot support. However, this patient did in an opposite manner, which leads to imbalance and fall. Moreover, the patient failed to put enough force on the toe, which yielded a poor push-off before swing phase. As a result of the three problems mentioned above, three gait phases (IC, LR, and PS) are missing, respectively, as shown in Fig. 17(b).

Left hip and knee joint rotations in the sagittal plane are shown in Fig. 18. While this patient presents a large range of motion in the hip joint, there is an inflection point after 60% of the gait cycle and large variations also occur here, which is clearly abnormal by comparing with the healthy subject data in Fig. 14(a). The knee joint rotation of this patient is almost normal except for the small range of motion.

Finally, step lengths of both sides for the three subjects are calculated based on the consecutive 50 steps and they are summarized in Table 1. While the absolute values of the step lengths cannot be compared across different subjects due to different lengths of lower limbs, they can still reveal some abnormal gait behaviors. It is clear that the step lengths of the healthy subject are much larger than those of the stroke patient for both sides. Furthermore, the standard deviations of the patients are larger than those of the healthy subject, indicating they have more inconsistent gait patterns between steps. Last but not least, for the two patients, the standard deviations of the affected sides are much larger than those of the other sides.

In this paper, design of a wireless human motion monitoring system was presented for smart gait rehabilitation. Joint angle sensors were employed to estimate the joint rotations in three dimensions. Smart shoes were developed to measure the GCFs at four points on the shoe pad during walking. Gait phases and step lengths were calculated using raw sensor measurements. User interfaces were developed on a laptop and an iPad to provide visual feedback to the patients. The experimental results were shown for healthy subjects and patients to verify the performance of the proposed system. Based on the measurement from the system, it would be easy to detect abnormal gait and evaluate the training progress. The system was applied in a clinical study, which consisted of patients with post-stroke and Parkinson's disease, and clinical outcomes suggested that the patients might use the system to conduct high-quality and independent gait training at home. This in-home rehabilitation paradigm would make the gait training more convenient, accessible, and effective.

Despite its good performance, the system will be further improved to prolong the battery life, reduce the size and weight of the sensor box, and achieve better time synchronization of different sensor signals. The proposed system will be integrated into a networked rehabilitation system with both sensors and rehabilitation robotics [28,29]. The system will enable patients to reinforce gait training with in-home rehabilitation with a physical therapist monitoring the process remotely.

This work was supported by the National Science Foundation under Grant No. CMMI-1013657. The authors want to thank Dr. Gregorij Kurillo for his help in the verification of the IMU sensors. The authors want to thank Sophia Coo for her assistance in the clinical trials, and Chen-Yu Chan, Jianyu Chen, Taoyuanmin Zhu, Lauren Farrell, Maung Soe, and Daniel Hsu for their contributions to the design of user interfaces and fabrication of smart shoes.