A New Evaluation Metric for Demand Response-Driven Real-Time Price Prediction Towards Sustainable Manufacturing

With the growing concerns about increasing energy demand and environmental pollution, special attention has been recently focused on promoting sustainability of the industry sector [1–3]. Among various sustainable management techniques, demand response is considered a critical approach since it can contribute positively to peak power demand alleviation [4], operational reliability improvement in power systems [5], and greenhouse gas emissions reduction [6] without vast investment. Especially, the industrial demand response customers are anticipated to contribute to around 46% of the total peak demand reduction [7]. Therefore, the implementation of demand response in the industry sector has attracted tremendous attention [8–10].

Considering that the real-time electricity price (RTP) directly reflects the underlying demand–supply relationship in the electricity market [11], the RTP-based demand response can potentially provide the maximum benefits for both power grids [12] and manufacturers [13,14]. In the current literature, different approaches have been proposed to achieve RTP-based demand response management for manufacturing systems. For example, Dababneh et al. developed a simulation-based scheduling method to implement demand response in manufacturing systems [15]. Cui and Zhou formulated an analytical model for industrial power load scheduling under demand response [16]. Recently, Yun et al. proposed a data-driven demand response scheduling and rescheduling method considering the uncertainties in manufacturing systems [17]. Regardless of the techniques used to obtain the production schedule under demand response, the RTP prediction is always the foundation since the future RTP is unknown at the scheduling state. As shown in Fig. 1, the demand response schedule is optimized based on price prediction, but the manufacturers need to pay their electricity bills using actual RTP. Therefore, the RTP prediction quality can essentially affect the effectiveness of demand response.

In the current literature, various methods have been applied to improve the RTP prediction quality. For example, Huang et al. used an artificial neural network (ANN)-based price forecasting model for one-step ahead RTP prediction [18]. In addition, Lu et al. adopted the long short-term memory (LSTM) recurrent neural network for one-step ahead RTP prediction considering the autocorrelation in the input price data [19]. Recently, Keles et al. presented a deep neural network-based method for multi-step ahead RTP prediction, which is essential for demand response scheduling [20]. In the current literature, the main objective of the prediction is to provide a price forecast that is similar to the actual RTP, i.e., minimizing the mean squared error (MSE) or mean absolute error (MAE). However, a good prediction indicated by these metrics does not necessarily lead to a good demand response schedule (a detailed discussion of this problem is provided in Sec. 2). Since the ultimate goal of price prediction is to obtain a high-quality demand response schedule, as shown in Fig. 1, a suitable RTP prediction evaluation metric should be highly related to the effectiveness of demand response but does not depend on the detailed information of manufacturing systems for generality purposes. Such a metric could help improve the quality of RTP prediction and eventually improve the effectiveness of demand response toward sustainable manufacturing. To the best of the authors’ knowledge, this problem has not been thoroughly investigated in the current literature, and whether the existing metrics are highly correlated to demand response performance remains unknown.

To fill the abovementioned research gap, a new evaluation metric, i.e., k-peak distance (KPD), is proposed in this study considering the characteristics of demand response problems. Specifically, the contributions of this study lie in:

1. A comprehensive assessment of the commonly used existing evaluation metrics is conducted, which reveals the correlation between these metrics and the effectiveness of demand response scheduling for manufacturing systems.

2. A new evaluation metric, i.e., KPD, is proposed to function as an indicator for price prediction quality that is highly correlated to the demand response performance, and its effectiveness is validated by comparing it with other existing metrics.

3. An encoder-decoder LSTM-based RTP prediction algorithm with KPD is proposed to provide better price forecasts specifically for demand response problems. In addition, the suitability of integrating KPD into other commonly used prediction algorithms is investigated.

The outcomes of this study can improve the RTP prediction quality, help manufacturers reduce energy costs through demand response, and contribute positively to sustainable manufacturing.

The rest of this paper is organized as follows. The motivation of this study is presented in detail in Sec. 2. The metrics comparison and discussion of the proposed KPD are presented in Sec. 3, followed by the explanation of the proposed encoder-decoder LSTM with KPD in Sec. 4. Case studies evaluating the effectiveness of the proposed method and the suitability of the proposed KPD in various prediction algorithms are shown in Sec. 5. Finally, the conclusions and future research directions are discussed in Sec. 6.

In RTP-based demand response, the dynamic price is the driving force in production scheduling. The manufacturers who enrolled in the RTP-based demand response program could achieve energy cost reduction by reducing their electricity consumption during peak hours to avoid high electricity prices. As shown in Fig. 2, suppose that the highest RTP appears at around 18:00. The optimal production schedule should reduce the power demand during these periods to minimize the energy cost. Meanwhile, the manufacturing system should keep its maximum productivity during the other off-peak periods to satisfy the daily production throughput requirement. Although Fig. 2 only shows a simplified demonstration of demand response scheduling results for manufacturing systems, similar scheduling strategies are already proven in the current literature [21,22].

In practice, the RTP of a certain period is not known until after that period has passed. Therefore, price prediction, instead of actual RTP, is applied in the production scheduling stage for demand response management. The purpose of price prediction is to provide guidance for production scheduling in response to dynamic electricity prices. Therefore, the quality of a price prediction should be judged by whether the production schedule obtained using that price prediction can help manufacturers reduce energy costs under actual RTP. However, in the current literature, commonly used metrics, such as MSE and MAE [16,17], can only represent the general similarity between price prediction and actual RTP. Since neither MSE nor MAE can reflect any characteristics in demand response scheduling problems, these metrics may encounter issues in assessing the RTP prediction quality.

For example, the Prediction 1 in Fig. 3(a) results in a smaller MSE since it is closer to the actual RTP than the Prediction 2 in Fig. 3(b). However, the optimal schedule obtained using Prediction 2 is more similar to the optimal schedule under the actual RTP (as shown in Fig. 2). This example shows that an apparently good price prediction indicated by MSE does not necessarily lead to an ideal production schedule in demand response problems. Therefore, it is critical to find a metric that can better represent the relationships between price prediction and the effectiveness of demand response in manufacturing systems.

A straightforward solution to the abovementioned challenge is directly applying the effectiveness of demand response, e.g., energy cost reduction, as the price prediction evaluation metric. However, calculating the energy cost reduction of a certain manufacturing system requires detailed manufacturing information, such as production line layout, machine rated power, and buffer capacity, which may not be available at the price prediction stage. Therefore, an ideal metric for price prediction in demand response problems is expected to be highly correlated to the effectiveness of demand response with no need of detailed manufacturing system information. In addition, it should be able to be integrated with a price prediction algorithm to provide better RTP predictions for demand response scheduling in manufacturing systems.

In addition to the assessment of existing metrics, a new metric KPD is proposed considering the characteristics of demand response problems. The correlation between DC and KPD results is tested and compared to the other six metrics to identify the best metric for price prediction in demand response problems.

In order to fairly compare different evaluation metrics, an identical demand response scheduling method for manufacturing systems should be applied to find the optimal production schedule under a given price, i.e., the first step in Fig. 4. In this study, a serial production line is applied as an example of manufacturing systems, and its schematic diagram is shown in Fig. 5. A total of N machines are placed in the production line, where the ith machine is denoted by M_i. A work-in-process buffer, denoted by B_i, is placed between machines M_i and M_i+1 to alleviate the impact on production throughput due to the machine downtime during the peak period under the demand response schedule.

In the objective function (2), C_t is the electricity price at time t, P_i is the power demand when machine M_i is turned on, Δt is the length of a time interval, T is the index of time interval at the end of day, and x_i,t is decision variable for machine M_i at time t. More specifically, x_i,t = 1 if machine M_i is turned on at time t, and x_i,t = 0 otherwise. Equation (3) represents the production throughput constraint, where PT is the production throughput of the entire production line at the end of the day, and PA is the daily production throughput target.

To calculate the energy cost under demand response management, the price prediction is first used as C_t in the objective function, and the optimal production schedule $xi,t*$ can be obtained by solving the optimization problem. Then, the energy cost can be calculated by applying the optimal schedule $xi,t*$ and the actual RTP using (2).

Some commonly used metrics to represent the similarity between two time series, i.e., the price prediction and RTP in this study, are briefly presented as follows:

Value matching metrics:

Pattern matching metrics:

Rank matching metrics:

According to the discussions in Sec. 2, two characteristics can be observed in the relationships between price prediction and production schedule.

1. To achieve the minimum energy cost, the optimal production schedule should avoid the highest electricity prices during peak periods. This means that it is critical to predict the starting time and duration of peak periods.

2. The production system should maintain its maximum productivity during the off-peak period to ensure the production throughput requirement. This means that the prediction accuracy during the off-peak periods does not affect the production schedule.

Since none of the metrics presented in Sec. 3.2 consider the above two characteristics in demand response problems, a new price prediction evaluation metric KPD is proposed as follows.

In this section, a data-driven RTP prediction algorithm with the proposed KPD is introduced. Unlike the traditional one-step ahead time-series prediction algorithm, an RTP prediction algorithm for demand response should have the following properties:

1. Since the demand response mechanism is load shifting over a period of time, the prediction algorithm should predict a sequence of RTP instead of one price data, i.e., it should be capable of multi-step ahead prediction.

2. According to the discussion of KPD in Sec. 3.3, the parameters in the algorithm should be optimized based on the evaluation of the entire output sequence instead of a single prediction point.

Considering these two properties, the encoder-decoder LSTM is adopted and integrated with the proposed KPD as the loss function for RTP prediction. Specifically, the LSTM is a type of recurrent neural network that can remember important events with lags of unknown duration in a time series. Therefore, it is specialized in processing sequence data. Developed from the classic LSTM, the encoder-decoder LSTM is capable of sequence-to-sequence learning (Seq2Seq), i.e., it can predict a sequence of RTP in the following period of time and optimize the weights and biases in the neural network based on the evaluation of the entire prediction sequence.

The framework of encoder-decoder LSTM with KPD proposed in this study is shown in Fig. 6. The RTP record with a length of t (i.e., from x₁ to x_t) is used as the input sequence to the first encoder LSTM layer. The encoder LSTM can summarize and memorize the information in the input sequence. After processing the entire input sequence, the encoder LSTM generates an encoder state (i.e., the hidden state and cell state of the LSTM) as its memory is learned from the input sequence. The encoder state and the last data x_t before the prediction period are then used as initial input to the second decoder LSTM layer. The decoder LSTM can output a sequence of price predictions with the length τ, i.e., from $y^1$ to $y^τ$⁠. Each output price is predicted based on the previous prediction and the relationships within the output sequence learned by the decoder LSTM. During the neural network training stage, the sequence of RTP predictions (i.e., $y^1$ to $y^τ$⁠) is compared with the actual RTP (i.e., y₁ to $yτ$⁠) through the proposed KPD metric. The evaluation result is treated as the value of the loss function and used to optimize the parameters in the encoder-decoder LSTM neural network. Since calculating KPD involves the non-differentiable sort(*) operation, commonly used gradient descent-based optimizers, such as stochastic gradient descent and Adam, cannot be used to optimize the network parameters. Therefore, the genetic algorithm (GA) is adopted as a derivative-free optimizer in this study to search for the optimal set of parameters.

The RTP prediction procedure through the proposed encoder-decoder LSTM with KPD is summarized in Algorithm 1. The prediction starts with RTP data collection from the utility company. The original data are cleaned and scaled through normalization. Then the 1D time-series price data are reshaped into a 2D matrix, where each row is one time series with input and output sequences. For example, suppose the length of input sequence iw = 3 and the length of output sequence ow = 2, the windowing of the time series is shown in Fig. 7. After formulating the matrix, the data are separated into training, validation, and testing sets. The neural network is trained by minimizing the KPD on the training set

through GA, while the KPD of the validation set is also calculated in each epoch. The training ends when it reaches the maximum epochs, or it ends earlier to avoid overfitting if the validation loss increases. The well-trained model is then used to predict the output on the testing set, and the RTP prediction can be obtained after unscaling the output sequence. Finally, the effectiveness of demand response based on the predicted RTP is tested using the same procedure shown in Fig. 4.

In the case studies, a three-machine-two-buffer production line is adopted as the test system to calculate the optimal demand response schedules and energy costs under various RTP predictions. The power demands P_i for three machines are set to [20 kW, 25 kW, 20 kW]. The length of each time interval Δt is set to 1 h, and thus the number of time intervals T is 24 per day. The daily production throughput requirement PA is assumed to be 15.

To investigate the relationships between different evaluation metrics and demand response effectiveness, the RTP prediction provided by the utility companies, i.e., day-ahead hourly price (DAHP), is adopted in this study. The daily DAHP and RTP from November 1, 2020, to October 31, 2021, collected from ComEd, are adopted in this study [25]. The optimization problem is solved by particle swarm optimization (PSO), where both swarm size and maximum iteration are set to 600. The PSO runs three times using both DAHP and RTP to find the daily optimal production schedules and associated energy costs. The minimum costs are treated as EC^PRE and EC^RTP to calculate DC. The parameter k in KPD is set to 9 based on (21).

The same RTP dataset is used for encoder-decoder LSTM prediction. Specifically, the one-year RTP data are divided into eight months, two months, and two months for training, validation, and testing sets. The length of input sequence iw equals 3 × 24, and the length of output sequence ow equals 24. The sizes of hidden and cell states of LSTM are set to 32, the number of generations in GA is set to 250, and k in KPD is also set to 9. The encoder-decoder LSTM is built with PyTorch, and the GA is implemented with PyGAD.

The DC obtained by solving the optimization problem using the one-year DAHP and RTP data is presented in Fig. 8. The figure shows that DC ranges from 0 to over 16, and in most cases, the DC is lower than 5.

The correlations between DC and six metrics presented in Sec. 3.2 are shown in Fig. 9. The results show no clear correlations between DCs and these metrics. More specifically, some points have small metric values but large DC values. In this case, the metric suggests a high RTP prediction quality. However, the corresponding production schedule obtained based on that price prediction is far from optimum. On the other hand, some points have large metric values but relatively low DC values. In this case, the metric shows that the price prediction is not good enough and needs further improvement. However, the corresponding schedule already leads to the near minimum energy cost, and additional efforts for prediction quality improvement cannot significantly reduce the energy cost of the manufacturing system. In summary, these metrics are not highly correlated with the effectiveness of demand response DC. Hence, applying them in the price prediction stage does not guarantee that the obtained good price prediction can necessarily lead to an optimal production schedule.

The correlation between DC and the proposed KPD is shown in Fig. 10. Compared to the results in Fig. 9, the proposed KPD can better represent the changes in DC. The correlation coefficients between DC and seven metrics are shown in Fig. 11(a), and the standard errors for the linear regressions are shown in Fig. 11(b). The correlation coefficient between KPD and DC is 0.91, which is around 1.8–3.6 times higher than the correlation coefficients of the other six metrics (from 0.25 to 0.50). In addition, the standard error for linear regression is only 1.10 for KPD, which is only about half of the standard errors for the other metrics (from 2.34 to 2.61).

The results in Fig. 11 show that the proposed KPD can better represent the changes in DC, and thus, it is more suitable to evaluate and improve the quality of RTP prediction in demand response problems.

In this section, the effectiveness of demand response based on the RTP prediction obtained by the proposed algorithm is evaluated. First, two series of price predictions are obtained, one is obtained from the proposed encoder-decoder LSTM with KPD and the other one is obtained from an encoder-decoder LSTM with the same configuration but using MSE as the loss function. Then, the demand response scheduling method discussed in Sec. 3.1 is applied based on these two sequences of RTP predictions to find two optimal schedules. Finally, manufacturing energy costs under these two schedules are calculated using the actual RTP, and the DC associated with two demand response schedules are compared. Since the prediction results from November 2020 to June 2021 are used to train the neural networks, the comparisons of demand response effectiveness are conducted based on two RTP predictions from July 2021 to October 2021, and the results are shown in Fig. 12.

As indicated in Fig. 12, in all four months, the demand response schedules obtained based on predictions from the proposed algorithm outperform their counterparts. More specifically, although the schedules obtained using both KPD and MSE as loss functions can lead to zero DC on some days, the 75th percentile, median, and 25th percentile of KPD results are generally 20.0%, 29.8%, and 48.6% smaller than their counterparts of MSE results. The differences in DCs indicate that the schedules obtained based on the proposed algorithm are more similar to the ideal schedules obtained under actual RTP, and the proposed algorithm can lead to lower energy costs.

In addition, since the proposed encoder-decoder LSTM with KPD needs a hyperparameter k, a sensitivity analysis is conducted to test its impact on the demand response effectiveness. As shown in Fig. 13, the value of k is tested within the range of 5–13. The shaded area represents the MSE result distribution from the 25th percentile to the 75th percentile. As shown in Fig. 13, under the case study setting, k = 9 provides the smallest DC and best result. However, the performance of the encoder-decoder LSTM with KPD is relatively robust to the value of k, and the results obtained when k is between 8 and 12 are all smaller than the result obtained using MSE. When k is too small, the prediction algorithm may overlook some important peak periods in the RTP time series. For example, the DC significantly increases when k ≤ 7. On the contrary, when k is too large, the underlying assumption is that the manufacturing resources can be turned off for a long period of time without affecting the achievement of production target. Since this assumption is not true, the result when k = 13 shows a negative effect on the effectiveness of demand response DC.

The suitableness of using the proposed KPD as loss functions in price prediction for demand response problems is also tested with other prediction algorithms. Specifically, two RTP prediction algorithms applied in the recent literature, i.e., ANN [18,20] and vanilla LSTM [19], are trained with and without KPD, and the results are compared with that of the encoder-decoder LSTM approach. The ANN has two hidden layers with 48 and 24 neurons. It treats the RTP data in the previous three days as 72 independent features and has 24 outputs representing the hourly price predictions for the following day. The vanilla LSTM has 32 neurons and uses the previous three days’ RTP data to predict the price for the next hour, and it is recursively applied 24 times to obtain the price predictions for the following day. All three algorithms are tested using KPD and MSE as loss functions. The results of average daily DC, stand deviation (std) of daily DC, and cumulative DC on the testing dataset are shown in Table 1.

The results show that the encoder-decoder LSTM with KPD provides the best price prediction in demand response problems, where the cumulative DC over the two months of the testing dataset is only 117.66. The result indicates that the demand response schedules obtained based on these price predictions are very similar to the theoretical optimal schedules. When using encoder-decoder LSTM as the prediction algorithm, the cumulative DC is reduced by 23.4% when switching the loss function from MSE to KPD. In addition, the ANN also has better performance with KPD, and its cumulative DC is reduced by 50.2%. It can also be observed that the use of KPD in Vanilla LSTM is the least desirable among the three methods. Given that the proposed KPD is calculated based on a sequence of predictions, it can work with prediction algorithms that are capable of learning relationships between sequences of inputs and outputs. On the contrary, KPD is not suitable to be used as the loss function for one-step ahead prediction algorithms, such as Vanilla LSTM, since it cannot clearly evaluate the quality of a single price prediction.

This study explores the relationships between RTP prediction evaluation metrics and the effectiveness of demand response in manufacturing systems. In addition, based on the unique characteristics of the demand response problem, a new price prediction evaluation metric KPD is proposed and integrated with encoder-decoder LSTM to provide better RTP prediction for demand response scheduling. The case studies show that the proposed metric has a 1.8–3.6 times higher correlation with demand response effectiveness and only about half of the standard error compared to other commonly used metrics. The demand response effectiveness can improve by 23.4% on average based on the RTP prediction obtained from the proposed algorithm. The results indicate that the proposed metric can better represent the RTP prediction quality in demand response problems, and the proposed algorithm can provide better price prediction for production scheduling under RTP-based demand response.

In this study, the RTP records are used to train the proposed neural network. To further extend this work, more related data, such as forecasting of energy demand and temperature, can be collected and used as inputs to further improve the RTP prediction accuracy. In addition, the robustness of the proposed metric and algorithm in different manufacturing systems under demand response problems should be evaluated.

The authors sincerely appreciate the funding support from the U.S. Department of Energy under Grant No. DE-EE0009714.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.