This paper presents a long short-term memory (LSTM)-based ensemble learning approach for time-dependent reliability analysis. An LSTM network is first adopted to learn system dynamics for a specific setting with a fixed realization of time-independent random variables and stochastic processes. By randomly sampling the time-independent random variables, multiple LSTM networks can be trained and leveraged with the Gaussian process (GP) regression to construct a global surrogate model for the time-dependent limit state function. In detail, a set of augmented data is first generated by the LSTM networks and then utilized for GP modeling to estimate system responses under time-dependent uncertainties. With the GP models, the time-dependent system reliability can be approximated directly by sampling-based methods such as the Monte Carlo simulation (MCS). Three case studies are introduced to demonstrate the efficiency and accuracy of the proposed approach.
Reliability is the probability that an engineered system successfully performs its intended functionality with the consideration of various uncertainties. Reliability analysis is of critical importance for engineering product development, and research efforts have been devoted to accurately estimating system reliability. Most probable point (MPP)-based methods such as the first-order reliability method (FORM) and second-order reliability method (SORM) [1–5] have been utilized for analytically estimating the reliability, where the limit state function is transformed into the standard normal space and the MPP is defined as the point on the failure surface that is closest to the origin. With the iterative MPP searching process, reliability can be approximated based on the first- or second-order Tayler expansion. To reduce the computational cost for high-dimensional problems, dimension reduction methods decompose the high-dimensional limit state function into multiple low-dimensional component functions to form a global surrogate model. Sampling methods such as Monte Carlo simulation (MCS) [6–8] and importance sampling [9–11] have been developed to enhance the accuracy of reliability estimations, which are commonly integrated with surrogate modeling techniques [12–15] for alleviating the computational burden.
Conventional reliability analysis methods are not applicable for time-dependent problems, in which the limit state function involves time-dependent uncertainties such as system performance degradation, stochastic operation conditions, and system aging. With the time parameter and stochastic processes, performing time-dependent reliability analysis is much more complex and difficult compared with estimating reliability for static problems. The most challenging part is to efficiently yet accurately account for the time-dependent correlation of the system responses. In the literature, different approaches have been investigated [16–19], including extreme value and outcrossing rate based approaches, for handling time-dependent reliability analysis problems. Extreme value-based approaches focus on extracting the worst performance of an engineered system over a certain period. Failure occurs if the extreme value is greater than or less than the threshold within a time interval. By identifying the extreme system performances, a time-dependent reliability problem can be transformed into a time-independent case, and then conventional reliability analysis methods can be applied for assessing the time-dependent reliability. However, it is intractable to obtain the probabilistic characterization of the extreme value analytically for practical problems. Though simulation-based methods can be used to estimate the extreme value distribution, the required computational resources are still significant due to the inevitable large number of function evaluations. In the outcrossing rate-based approaches, the outcrossing event occurs when the response reaches its limit state and the outcrossing rate is then defined as the rate of change in the probability with respect to time. The time-dependent reliability can be approximated by the integration of the outcrossing rate. However, the existing approaches such as Rice’s formula  can only applied to obtain the outcrossing rate for Gaussian random processes. In addition, Rice’s formula assumes that all the outcrossing events are independent. Based on Rice’s formula, different methods have been developed for time-dependent reliability analysis. Hu and Du  extended the joint outcrossing rate method to limit state functions with both random variables and stochastic processes, then used both the single and joint outcrossing rates to approximate the time-dependent probability of failure. In the PHI2 method, Andrieu-Renaud et al.  estimated the outcrossing rate based on the correlation coefficient function of the limit state at different time instants and point-in-time reliability indices.
Most of the aforementioned methods rely on the first- or second-order approximation of the limit state, which may introduce significant errors for time-dependent problems with highly nonlinearity. Therefore, surrogate model methods have been investigated to improve the accuracy of the time-dependent reliability analysis. Wang and Wang  proposed a nested extreme value response method for efficient time-dependent reliability analysis. The Gaussian process (GP) modeling technique was utilized to predict extreme system responses, and a double-loop sampling scheme was developed to iteratively update the GP model. Instead of using an optimization loop for identifying the global extreme responses, Hu and Mahadevan  proposed a single-loop Kriging modeling approach to sequentially refine surrogate models by identifying sample points for both random variables and time at the same level. Shi et al.  proposed a multiple-Kriging-surrogate method to determine the best regression trend. The adaptive sampling strategy was established to identify new training samples and determine the ensemble of surrogate models.
Inspired by the biological neural network of the human brain, deep learning utilizes a layered structure of algorithms to make informed decisions based on the knowledge learned from data. In recent years, deep learning techniques have been successfully applied in a variety of applications such as image processing [26,27], natural language processing , medical information processing , and robotics and control [30,31]. Recurrent neural networks (RNNs) [32,33] are a class of neural networks that allow previous outputs to be used as inputs while having hidden states. To deal with issues associated with gradient exploding and vanishing for the training of RNN, long short-term memory (LSTM) [34,35] has gained tremendous successes to make predictions based on time-series data. In this paper, an LSTM-based ensemble learning approach is developed for time-dependent reliability analysis involving stochastic processes. With the capability of modeling the time-dependent relationship, LSTM networks are employed to learn the time-dependent behavior of the system response with respect to the stochastic processes while fixing the random variables. The benefit of constructing the LSTM models lies in that they can accurately predict system responses given any new random realizations of the stochastic processes. As a result, a set of augmented data can be collected based on multiple LSTM models. The Gaussian process regression technique is then adopted for modeling the time-dependent system response. With specified stochastic processes and time instant, GP models can be constructed to predict the system response. By employing the MCS, the proposed approach can be utilized to estimate the time-dependent reliability. The remainder of this paper is organized as follows. Section 2 introduces the background of time-dependent reliability analysis and realization of stochastic processes. Section 3 introduces the details of the LSTM-based ensemble learning approach. The effectiveness of the proposed approach is demonstrated using three examples in Sec. 4, while the conclusions are presented in Sec. 5.
3 LSTM-Based Ensemble Learning
As shown in Fig. 1, multiple LSTM networks are first trained to learn conditional limit state functions based on a set of random realizations of the input variables and the stochastic processes. To handle the time-independent random variables, a set of Gaussian process models are leveraged with the LSTM models to predict the global time-dependent system response under uncertainty. With the GP models, time-dependent reliability can be approximated using the MCS method. The LSTM-based conditional limit state modeling is introduced in Sec. 3.1. Section 3.2 presents the global surrogate modeling using GP regression, and the numerical procedure of employing the proposed approach for time-dependent reliability analysis is explained in Sec. 3.3.
3.1 LSTM-Based Conditional Limit State Modeling.
The RNN has been developed to learn insights from time-series data, which has a feed-back loop to store prior information. A major limitation remains that it cannot provide accurate predictions if the time-series data have long-term dependencies. All RNNs have the form of a chain of repeating modules. To remember information for long periods of time, three gated units in LSTM are used in the repeating modules, including an input gate, output gate, and forget gate. The core idea of LSTM lies in that the information flow is represented by the cell state, which is shown as the top horizontal line in Fig. 2.
3.2 Gaussian Process-Based Global Surrogate Modeling.
The LSTMs constructed in Sec. 3.1 can provide accurate predictions for the conditional limit state functions at certain local locations. In other words, these LSTMs lack the capability of modeling the global limit state function with the time-independent random variables. To address this difficulty, Gaussian process regression is adopted for global surrogate modeling, which aims at predicting the time-dependent system responses given any random realizations of time-independent variables x′ and stochastic processes z′(t).
3.3 Time-Dependent Reliability Analysis.
The procedure of employing the proposed ensemble learning framework is summarized in a flowchart as shown in Fig. 3. According to the statistical information of the input variables and the stochastic processes, N random realizations are generated as the MCS samples, denoted as Umcs = [Xmcs, Zmcs(t)], where Xmcs = [x1m, x2m, …, xNm] and Zmcs(t) = [z1m(t), z2m(t), …, zNm(t)]. To prepare the data for training multiple LSTM models, Latin hypercube sampling (LHS) is employed to generate n random samples of the time-independent variables as X = [x1, x2, …, xn] and thus lead to n conditional limit state functions. With n random realizations of the stochastic processes Z = [z1(t), z2(t), …, zn(t)], the time-dependent responses are directly evaluated based on the conditional limit state functions. Following the procedure introduced in Sec. 3.1, n LSTM models can be constructed while the ith LSTM model is built based on the input [zi(t), t] and output yi. In the proposed approach, the LSTM models are constructed based on the “Keras” library in python 3.6. The MSE is used as the loss function, where the “Adam” optimizer with default learning rate 0.001 is adopted for the training process of all LSTM models.
4 Case Studies
In this section, three examples are used to demonstrate the effectiveness of the proposed approach for solving time-dependent reliability analysis problems.
4.1 Case Study I: A Mathematical Problem.
The first step of employing the proposed approach is to prepare the training data for the LSTM models. As introduced in Sec. 3.3, the LHS is employed to generate ten samples of random variables x. Each LHS sample is combined with a random realization of the stochastic processes z(t), then the corresponding time-dependent responses are evaluated based on Eq. (22). Given the training data sets, ten LSTM models can be constructed, respectively. For each LSTM model, 105 MCS samples of stochastic processes are provided for predicting the time-dependent system responses of the conditional limit state functions. Based on the achieved augmented data, GP models can be constructed by specifying a time node with the realization of the stochastic processes. Eventually, the time-dependent responses corresponding to the 105 MCS samples can be obtained based on the GP models, which are further utilized for estimating the time-dependent reliability.
Ten random realizations of the stochastic processes that are used to train multiple LSTM models are shown in Figs. 4(a)–4(c), respectively, which plot the random realizations for each stochastic process, and the resultant time-dependent responses are depicted in Fig. 4(d). As a result, ten LSTM models are trained based on these training data, where the number of neurons for each LSTM model is set to 40. All the LSTM models are well trained by using Adam optimizer to minimize the MSE with 3000 epochs. To demonstrate that the LSTM is capable of accurately capturing the relationship between the stochastic processes and the time-dependent responses. The comparison of the actual and predicted time-dependent responses is depicted in Fig. 5, where Fig. 5(a) shows the response comparison of the first random realization z1m(t) using the first LSTM, and Fig. 5(b) shows the response comparison of the tenth random realization z10m(t) using the tenth LSTM. The results demonstrate that the LSTM models have been well trained, and thus are capable of accurately predicting the time-dependent responses of the conditional limit state functions for any realizations of the stochastic processes.
To demonstrate the accuracy of time-dependent response predictions using GP models, Fig. 6 shows the comparison of the accurate and predicted time-dependent responses for the 25th and 75th MCS samples, respectively. The results demonstrate that the constructed GP models can effectively handle the randomness of the time-independent variables, and they can provide accurate response predictions for the whole time-series data. For comparison purposes, the “equivalent stochastic process transformation (eSPT)” method  is adopted for approximating the time-dependent reliability of the mathematical example. To validate the accuracy of reliability estimation, the actual time-dependent system responses for the 105 MCS samples are calculated, and the resultant accurate reliability 0.9754 is treated as a reference. The reliability approximations achieved by the proposed approach and eSPT are given as 0.9732 and 0.9726, respectively. By converting the stochastic processes into time-independent variables, the eSPT method requires 88 time-series data to obtain the reliability estimation. Note that the time interval consists of 60 time nodes. In the proposed approach, we constructed multiple LSTM models by using only ten time-series data. Therefore, the computational cost required by the proposed approach is much less than the eSPT method. By specifying different time intervals within [0, 1], time-dependent reliability given different time periods can be approximated based on the predicted responses. The comparison between the reliability approximations using the proposed approach and accurate results computed by direct MCS is summarized in Table 2, which indicates that the proposed approach is capable of accurately capturing the variation of time-dependent reliability with respect to time.
4.2 Case Study II: A Corroded Beam Problem.
The time interval is given as [1, 30] month, which is evenly divided into 59 time nodes. Following the numerical procedure, the proposed approach is employed to solve the corroded beam problem. For reliability analysis, 105 MCS samples are generated according to the statistical properties of the random variables and the stochastic process. Fifteen training data sets are utilized for training the LSTM models, and the augmented data are obtained by using the LSTM models to predict the time-dependent responses given the 105 random realizations of the stochastic process. Accordingly, GP models can be constructed to model the time-dependent responses corresponding to each random realization of F(t). As a result, time-dependent response predictions for the MCS samples are obtained based on the GP models.
For comparison purposes, the eSPT method is employed for time-dependent reliability analysis with a predefined cumulative confidence level 0.999. The reliabilities calculated by direct MCS, proposed approach, and the eSPT are given as 0.9756, 0.9748, and 0.9778, respectively. The results show that both the eSPT and the proposed approach can achieve an accurate time-dependent reliability estimation. In the eSPT method, 84 time-series data are utilized for updating process while the number of time-series data required by the proposed approach is 15. The results indicate that the proposed approach can achieve accurate time-dependent reliability estimations in an efficient manner.
In this study, the LSTM models are trained with a maximum epochs 3000, and the convergence of the MSE loss function for the first LSTM is shown in Fig. 8. It has been observed that all the LSTM models have similar convergence curves. To validate the effectiveness of the LSTM, the comparison of the actual and predicted responses using LSTM models is shown in Fig. 9, where Fig. 9(a) shows the response comparison of the first random realization F1m(t) using the first LSTM, and Fig. 9(b) shows the response comparison of the sixth random realization F6m(t) using the sixth LSTM. The results demonstrate that the LSTM models can make accurate response predictions for the conditional limit state functions with any random realization of the stochastic process. Thought the augmented data are not collected by directly evaluating the conditional limit state functions, the accuracy is guaranteed due to the benefits of using the LSTM.
Once all the augmented data have been collected from the multiple LSTM models, GP models for modeling the time-dependent response can be constructed as introduced in Sec. 3.3. As a result, the time-dependent responses corresponding to each MCS sample can be predicted based on the GP models. The actual minimum responses for the first 50 MCS samples are compared with the minimum responses extracted from the GP predictions. As shown in Fig. 10, the estimated minimum responses are ensured to be accurate, which almost overlap the actual ones. Considering all the 105 MCS samples, Fig. 11 shows the comparison between the PDFs of the actual and estimated minimum responses. The results reveal that the proposed ensemble learning framework can provide accurate predictions of the minimum value of the time-dependent responses within the time interval, thus lead to accurate time-dependent reliability approximation. By specifying different time intervals, time-dependent reliability can be approximated based on the overall time-dependent response predictions. The comparison between the reliability approximations and accurate results computed by direct MCS is summarized in Table 4, which indicates that the proposed approach is capable of accurately predicting the time-dependent reliability within any time intervals [0, TI], TI ≤ T.
To test the robustness of the proposed approach, time-dependent reliability estimations are performed for the corroded beam problem with different modifications. Four different scenarios are introduced according to the detailed configurations presented in Table 5. With the identical settings and procedure as previous, the proposed LSTM-based ensemble learning framework is employed for estimating the time-dependent reliability for each scenario. To demonstrate the accuracy of the proposed approach, the direct MCS method is also adopted for accurate reliability analysis, and all the results are presented in Table 5. For different scenarios, the performances of the proposed approach are quite stable as accurate reliability approximations can always be achieved with small relative errors. The results demonstrate that the proposed approach is capable of solving time-dependent reliability analysis problems involving stochastic processes.
4.3 Case Study III: A Truss Example.
The proposed approach is employed for time-dependent reliability analysis of the truss example, where eight training data sets are utilized for training the LSTM models. The maximum epochs are set to 3000 during the LSTM training processes. To validate the accuracy of the LSTM models, the predicted responses for the first random realization F1m(t) using the first LSTM model are compared with the actual responses as shown in Fig. 13. After collecting the augmented data from all LSTM models, GP models are constructed to model the global time-dependent responses. As shown in Fig. 14, accurate time-dependent response predictions can be achieved for the MCS samples. In Fig. 15(a), the actual minimum responses for the first 50 MCS samples are compared with the predicted minimum responses that are extracted from the GP estimations, while Fig. 15(b) depicts the PDF comparison between the actual and estimated minimum responses of all MCS samples. With eight time-series data, the proposed approach provides a reliability estimation of 0.9467, while the actual reliability evaluated by direct MCS is given as 0.9408. The results demonstrate that the proposed approach is applicable to time-dependent problems involving non-stationary processes.
To investigate the performance when dealing with higher reliability levels, the mean value of the random variable A is modified to 4.7. The accurate reliability evaluated by 105 MCS samples is given as 0.99118. With eight time-series data, the reliability approximation using the proposed approach is given as 0.99486. The estimated minimum responses for the first 50 MCS samples are compared with the actual values as shown in Fig. 16(a), and the overall PDF comparison is depicted in Fig. 16(b). The accuracy of the proposed approach is demonstrated through the comparison study, and the results show that the proposed approach can provide a reasonable reliability approximation when handling higher reliability levels.
In this paper, an LSTM-based ensemble learning framework has been established, where MCS is adopted for estimating the time-dependent reliability based on the combination of LSTM networks and the GP modeling technique. To employ the LSTM network for learning the relationship between stochastic processes and time-dependent system responses, conditional limit state functions are introduced by fixing the time-independent random variables. Based on one time-series data, an LSTM model can be constructed for modeling a specific conditional limit state function, which can provide accurate response predictions given any random realizations of the stochastic processes. The time-dependent response predictions collected from multiple LSTM models are reorganized according to the realizations of stochastic processes and time instant. Gaussian process models are constructed to specifically model the time-dependent response with respect to random variables. As a result, time-dependent response predictions for the MCS samples can be achieved based on the GP models. The results from three case studies demonstrate that the proposed approach can handle complex problems involving multiple stochastic processes. Moreover, the proposed approach is capable of accurately predicting the overall time-dependent responses, which ensures the accuracy of reliability estimation and enables the capability of depicting the change of reliability within different time intervals.
Conflict of Interest
There are no conflicts of interest.