Exploring the Influence of Feature Selection Methods on a Random Forest Model for Gait Time Series Prediction Using Inertial Measurement Units

The insights provided by three-dimensional (3D) gait analysis are necessary to assess gait abnormality and facilitate informed decision-making in medical care [1]. Machine learning models provide an accurate and cost-effective solution for estimating gait time series by mapping raw inertial measurement unit (IMU) data to optical motion capture (OMC)-generated gait data [2–4]. This approach enables analysis outside clinical settings and overcomes the time-increasing drift errors that affect the accuracy of IMU-only methods [5]. Prior research has used feature engineering to enhance a model's performance [6–10]. However, none have investigated how the feature selection method can affect the model's performance. In certain studies [6,7,9,10], a random forest (RF) regressor [11] was employed to rank features using Gini importance [12]. This approach streamlined the feature space by identifying the most representative features for the intended target. Another study [8] proposed a deep belief network method for extracting an optimal feature set from multichannel surface electromyography signals. Their method indicated superior performance compared to using principal component analysis. Despite these insights, a comparative exploration of feature selection methods specifically for gait time series prediction remains unexplored.

An effective feature selection method enhances model performance and reduces computational costs by identifying the most relevant characteristics of the input data and minimizing the dimensionality. Moreover, a generic feature set that remains consistent across various training datasets contributes to a more robust and generalized machine learning model [13].

In our previous investigations, we highlighted that RF models offer lower kinematics prediction error than traditional convolutional neural networks [14]. Consequently, our current emphasis is on using RF model, with an objective to assess the impact of varying feature selection methods. Our focus in this study centers on four filter methods (Spearman, Kendall's Tau, mutual information (MI), and RReliefF) and four embedded methods (RF, extreme gradient boosting (XGBoost), least absolute shrinkage and selection operator (Lasso), and ElasticNet).

This research aims to explore the influence of different feature selection methods on the performance of personalized (trained and tested on the same subject) and generalized (trained on data from 17 participants and tested on six new participants) RF models for pelvis, hip, knee, and ankle joints kinematics prediction. Personalized models are tailored to individual subjects, requiring only a single visit to the gait clinic, while generalized models can predict gait kinematics for new patients without the need for any clinic visits. To ensure robust performance across unseen participants, a generic feature set was derived from shared features identified in personalized models. These shared features capture common patterns across participants while retaining predictive relevance. We pursued two primary objectives: (1) identify the most suitable feature selection method based on its ability to uncover a general feature set, time efficiency, and the impact on personalized RF models' accuracy for predicting joint kinematics during gait. (2) Assessing the performance of generalized RF models using generic feature sets for joint kinematics prediction on a new dataset not employed for feature selection or model training.

Each child's legal guardian provided informed consent and each child provided assent prior to data collection. The research strictly adhered to ethical principles outlined in the Helsinki Declaration and received approval from the University of Auckland (New Zealand) human participant ethics committee (reference number 021615).

The first dataset used in this study was previously outlined in Ref. [14]. In summary, 17 typically developed (TD) children (nine females and eight males, age: 10.5 ± 2.8 yr, weight: 37.1 ± 11.7 kg, and height: 147 ± 17 cm) underwent overground walking (n = 15 trials) at self-selected speed. The 37 marker trajectories from OMC (VICON, Oxford, UK) and two IMUs placed on the feet (VICON IMeasureU) were recorded and synchronized with nexus (VICON). This dataset included 73,364 time points. The pelvis (3DOF), hip (3DOF), knee flexion/extension, and ankle inversion/eversion and dorsi/plantarflexion kinematics were computed from the markers' trajectories in opensim (gait2392) [14]. A combination of raw IMU data (acceleration (x3) and angular velocity (x3)) placed on each foot and an RF machine learning model [15] was used to predict the pelvis, hip, knee, and ankle joints' kinematics (15 targets) during gait.

The second dataset included six TD children (five females and one male, age: 13.3 ± 2.5 yr, weight: 46.3 ± 5.6 kg, and height: 158 ± 13 cm). The same data collection process was performed on this second dataset (see details above and in Ref. [14]). This second dataset was used as a testing set to evaluate the model's performance on participants who were not included in the feature extraction, selection, and random forest model training set. This dataset included 27,009 time points.

We identified and extracted all features from raw IMU data by using an open-source python package called tsfresh [16] as described in our previous paper [14]. Then, the most important features were selected using eight different feature selection methods.

• Spearman correlation coefficient investigates the monotonic association between two variables [17]. The top 30 features with the highest absolute correlation with each target variable (kinematics and kinetics for each joint and plane of motion) were selected.

• Kendall's Tau is a rank-based correlation coefficient [18]. The top 30 features with the highest absolute correlation with each target variable were retained.

• MI, also known as information gain [19], measures the shared information between features and each target variable [20]. Top 30 features with the highest mutual information were kept.

• RReliefF assigns scores to features based on their ability to predict each target variable [21]. The top 30 features with higher scores were retained.

• RF feature selection method uses the Gini index to assess feature importance [11]. The top 30 features with higher Gini importance were maintained, while those with lower importance were discarded. This method was also employed in our previous study [14].

• XGBoost calculates the “gain” of each feature, with higher gain indicating greater importance [22]. The top 30 features with the highest gain were kept.

• Lasso applies L1 regularization, eliminating features with coefficients shrunk to zero [23]. The top 30 features with nonzero coefficients were maintained.

• ElasticNet combines L1 and L2 regularization, grouping correlated features and assigning them shared coefficients [24]. Most relevant features (top 30) were selected, while less important ones were removed.

The RF models developed in this study used data from the two IMUs. We designed single-output models, each customized to predict a specific target, utilizing the 30 most relevant features associated with that specific target. The experiments were conducted on a system with 32 AMD EPYC 73F3 processors, 2048 GB memory, three Nvidia Tesla A100 GPUs (80 GB each), and Red Hat 9 OS.

Each personalized model was trained using 70% of the walking trials and tested on the remaining 30% of the same participant, using the same data for feature extraction and selection (Fig. 1). We constructed 15 personalized models for each of the first dataset participants to predict all the desired targets (one model per target with 15 lower limbs joint kinematics). This process was repeated for all feature selection methods (Sec. 2.1).

To derive the generic feature set, the most important features for predicting each target were identified from the training dataset (first dataset, 17 participants) by analyzing their frequency of selection across personalized models. Features consistently ranked highly across participants were considered shared features. The top 30 shared features were selected as the generic feature set for each target. This process was repeated for all eight feature selection methods under investigation.

For the generalized RF models, the entire first dataset, comprising the gait cycles of 17 participants, was used for training, while the second dataset (six participants) served as the testing set (Fig. 1). The models were constructed using the generic feature sets derived as described above (Sec. 2.2). The entire procedure was repeated for each feature selection method to assess their influence on generalized model performance.

We computed the root-mean-square error (RMSE) to quantify the differences between the predicted and the opensim computed joint angles (pelvis 3DOF, hip 3DOF, knee flexion/extension, and ankle 2DOF) across all participants. These differences were quantified for each feature selection method. To gauge the efficacy of each method in identifying the generic features, we noted the frequency with which each top feature appeared across all participants. Additionally, we recorded the time required for the feature selection process across all methods to ascertain the most computationally and time-efficient approach.

Among all target variables and participants, the MI method consistently unveiled the highest number of shared top features. In contrast, the XGBoost method exhibited the lowest frequency of top features (Fig. 2). All feature selection methods consistently yielded the highest frequency of shared top features for predicting the hip, knee, and ankle angles in the sagittal plane. However, only MI was able to keep consistency in the other plane of motion.

The Relief method was the most time-consuming (>100 s), while the ElasticNet method was the quickest in selecting the most important features (Fig. 3). ElasticNet, Lasso, Spearman, and Kendall methods required less than 1 s, and RF, XGBoost, and MI needed between 1 and 10 s to extract the most important features.

The choice of feature selection method did not influence the performance of the RF model in the personalized models (Fig. 4). RMSE distribution remained consistent across different feature selection algorithms for the personalized models. The mean RMSE values, along with their corresponding standard deviation (SD), for the prediction of joint kinematics using personalized RF models are illustrated in Appendix Table 1 for the different feature selection methods.

While the generalized RF model exhibited higher errors compared to the personalized ones, the impact of the feature selection method on the results was low (Fig. 5), except for the pelvis tilt and pelvis rotation, where XGBoost showed the highest RMSE. The MI method consistently delivered the lowest average RMSE in the sagittal plane for hip, knee, and ankle angles (Fig. 5). Furthermore, MI exhibited the lowest quartiles and fewer outliers in predicting most targets. The RF feature selection produced the lowest errors for pelvis angles and hip adduction/abduction, while the Relief method was best in predicting ankle inversion/eversion. Table 2 represents mean RMSE (SD) values for generalized models.

This study aimed to compare the performance of RF models by employing filter and embedded feature selection methods for gait time series prediction. First, we compared different feature selection methods based on their robustness in selecting features, time efficiency, and their impact on personalized RF models. Second, we investigated how different feature selection methods impacted the accuracy of generalized RF models using generalized feature sets in predicting joint kinematics.

The MI method stood out for its capacity to select the same features across different participants. Its information-theoretic approach proved effective in identifying relevant features, demonstrating its suitability for predicting joint kinematics in all plane of motion not just in the sagittal plane. On the other hand, XGBoost exhibited the lowest frequency of shared top features, indicating potential limitations in capturing common features across participants. In situations where the selection of personalized feature sets is impractical due to computational resource limitations, employing a robust feature selection method to build a generic feature set becomes imperative.

The consideration of time efficiency in feature selection is crucial for real-world applications. The study evaluated the average time required by each feature selection method, with ElasticNet emerging as the quickest and Relief as the most time-consuming. Efficient feature selection methods such as ElasticNet and Lasso, which require less than 1 s, may be preferred in scenarios where real-time predictions are essential. Alternative feature selection methods presented similar levels of computational cost-effectiveness.

The study employed a diverse set of eight feature selection methods, including both filter and embedded techniques. We found that the choice of feature selection method exerts minimal influence on the performance of an RF model, regardless of whether it is applied in personalized or generalized modeling contexts. Nevertheless, the RF and MI feature selection methods demonstrated superior performance, consistently yielding the lowest prediction errors across most targets, making them highly viable options for inclusion in predictive modeling frameworks. The MI method not only displayed reduced prediction errors but also demonstrated lower quartiles and fewer outliers when predicting most targets, demonstrating its robust applicability in this specific context of 3D gait analysis in TD children. As demonstrated previously [14], TD children do not always have a repetitive gait cycle and show a diverse set of gait patterns depending on age. Therefore, this study shows that using MI feature selection might be more robust in populations with altered gait kinematics on not as repeatable as healthy adults.

Several factors contributed to the observed convergence in performance across different feature selection methods. One possible explanation is the existence of highly correlated features, causing different feature selection methods to converge on similar subsets that yield comparable performance. Additionally, the inherent robustness of the RF model to less informative features may enable it to accommodate variations in feature selection choices. Finally, the RF model inherently introduces randomness through bootstrapping and feature selection at each tree's split. This randomness can mitigate the impact of feature selection variations [12].

It is worth mentioning that the results of this study are specific to the dataset and RF model architecture used. While RF models were selected for their interpretability and robustness, the feature selection method could be adapted to other machine learning models, such as deep learning, which may handle high-dimensional data differently. Additionally, although our dataset includes healthy participants, further validation is required to extend these findings to clinical populations or other datasets with different gait characteristics. While our previous study demonstrated that two IMUs placed on the feet are sufficient to accurately predict lower limb joint kinematics and kinetics [6], this may not be the case for clinical populations. In these populations, additional IMUs may be required to capture movements across all body segments, which could result in a different set of relevant features and potentially alter the outcomes of feature selection methods.

The results of the study shed light on the robustness of RF models in predicting lower limbs joint kinematics from IMUs across different feature selection methods in children during gait. MI and RF methods stood out for their robustness, consistently selecting generic feature sets across participants, and effectively predicting joint kinematics. The selection of feature extraction methods had minimal impact on the overall accuracy of the RF models. This suggests that opting for the most time-efficient method could be the optimal choice, especially in scenarios where computational resources are constrained. Therefore, time-efficient methods like ElasticNet are preferable for real-time applications.

Shima Mohammadi Moghadam: Conceptualization, methodology, validation, investigation, writing original draft, and visualization. Julie Choisne: Supervision, funding acquisition, project administration, resources, investigation, conceptualization, and writing—review and editing.

• Health Research Council NZ (No. 19/652; Funder ID: 10.13039/501100001505).

• Aotearoa Fellowship (Funder ID: 10.13039/501100021732).

• Friedlander Foundation.

• Science for Technological Innovation NZ (No. UOAX2005).

The authors declare no conflict of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.