A Systematic and Comparative Study of Distinct Recurrent Neural Networks for Lithium-Ion Battery State-of-Charge Estimation in Electric Vehicles

Abstract

The precise estimation of the state of charge (SOC) is fundamental to the reliable operation of lithium-ion batteries. The development of deep learning techniques makes it possible to employ advanced methods to estimate a battery’s SOC. In order to better utilize a recurrent neural network (RNN) for battery SOC estimation, this paper conducts a comparative study of SOC estimation methods based on different RNN models. First, a general framework for deep-learning-based SOC estimation is undertaken, followed by the description of four kinds of RNNs employed in the estimation. Then, the estimation performances of these RNN models are compared under three scenarios, including the SOC estimation accuracy, the adaptability against different battery aging statuses, and the robustness against measurement uncertainties, in which the estimation performances of different RNN models are quantitively evaluated. Finally, a multiple-criteria decision-making method based on the analytic hierarchy process (AHP) is utilized to reflect the comprehensive performance of each RNN model, and the model with the highest score could be chosen for online SOC estimation during actual applications. This paper provides an in-depth analysis of RNN models in battery SOC estimation and could help battery management engineers develop the most appropriate estimation methods.

1. Introduction

According to the Paris Agreement, to limit global warming, countries should aim to reach the global peak of greenhouse gas emissions as soon as possible to achieve a climate-neutral world by mid-century. In the search to reduce carbon dioxide emissions, environmentally friendly energy storage devices have become a hot research point, such as lithium-ion batteries, fuel cells, and supercapacitors [1,2]. In the automotive industry, battery electric vehicles (BEVs) are being actively developed [3,4,5,6]. The SOC is defined as the ratio of the residual capacity to the nominal capacity, representing the remaining charge within the battery pack. Unlike the amount of oil, the battery’s SOC cannot be measured directly due to the non-linear relationship between the SOC and external measurements such as the current, voltage, and temperature. Additionally, the mapping relationships vary with different operating conditions and battery statuses. The wide temperature range during operation and battery attenuation is a challenge for SOC estimations. Despite the above difficulties, the accurate determination of the SOC is essential for ensuring the safe and efficient operation of BEVs. By maintaining safe charging and discharging limits, optimizing energy usage, and extending the battery’s lifespan through effective battery management, the SOC plays a vital role in ensuring the optimal performance and reliability of the overall system. Therefore, as a crucial state of a battery management system (BMS), the SOC should be estimated accurately and reliably.

Numerous approaches have been investigated to give accurate and reliable lithium-ion battery SOC estimations, which are generally categorized into conventional methods, model-based methods, and data-driven methods.

Many conventional methods have already been applied for SOC estimation, including the Coulomb counting method and the open-circuit voltage (OCV) method. The Coulomb counting method [7], also known as the ampere-hour integral method, is based on the definition of the battery’s SOC. Due to its ease of implementation, it is widely used to estimate a battery’s SOC during actual applications. However, it is an open-loop method, and its estimation accuracy is affected by factors such as the initial SOC, accumulated current measurement error, and battery capacity. The OCV method uses the non-linear relationship between the OCV and the SOC of the battery for its estimates. However, the measurement of the battery OCV requires a long rest time, which makes its use challenging for real-time estimations. A statistical analysis method was also designed to improve the estimation accuracy using the OCV [8]. These two methods are usually combined to give full play to their advantages—the OCV method for initialization and the Coulomb counting method for real-time estimation.

The model-based methods track the battery’s SOC by establishing a battery model, constructing the battery state space equation, and designing the observer. Battery models mainly include electrochemical models (EMs) and equivalent circuit models (ECMs). EMs attract researchers due to their ability to provide additional information regarding the battery’s internal operating conditions. However, the process of establishing a model is complex, and identifying the enormous number of parameters online is difficult, as it is time-consuming and requires a large number of calculations. In contrast, ECMs are much simpler and more suitable for real-time battery SOC estimation. They use closed-loop feedback control, which overcomes the errors accumulated over time by the Coulomb counting method and makes the model-based method available for SOC estimations with an unknown initial state. The research in this field focuses on the Kalman filter and its extension algorithm [9,10,11,12,13], as well as the particle filter and sliding mode observer [14,15]. Additionally, SOC estimations can be figured out using electrochemical impedance spectroscopy (EIS) [16,17]. Nevertheless, the deficiency of the model-based method is that the estimation performance primarily depends on the accuracy of the battery model and the parameter identification process. Since the battery model will change with temperature and aging, SOC estimations in different environments require different battery models or additional parameters.

The data-driven methods aim to establish a mapping relationship between the measurements and SOC, which requires little knowledge of the battery’s internal mechanism or additional parameter identification, and has become an area of concern. The machine learning method approaches the SOC estimation as a regression problem, and the goal is to predict the continuous value of the SOC based on the inputs, such as the voltage, current, temperature, and other relevant factors. Traditional machine learning models such as support vector machine (SVM) [18,19,20] and Gaussian process regression (GPR) [21] models are often used in SOC estimation due to their ability to effectively model non-linear relationships between inputs and outputs. With the improved computing capacity brought about by the graphics processing unit (GPU), using deep learning algorithms to estimate battery SOC values has become a field of interest for researchers. Deep learning is a subfield of machine learning that leverages artificial neural networks with multiple layers. Each layer learns to extract features from the data and then passes the results to the next layer. The ability of the deep learning method to capture valuable new features makes it distinct from other neural networks. The deep architecture of these neural networks enables them to learn complex and non-linear relationships between the inputs and outputs. When performing SOC estimations, deep learning networks make predictions based on input data such as the current, terminal voltage, and temperature, by adjusting the weights and biases of the connections between the layers through the training process. Convolutional neural networks (CNNs) [22] are common deep learning methods that can be used for SOC estimation, as they enable variable-length input and output data. However, with a limited ability to handle time series of data, the above data-driven methods may not be the most suitable for SOC estimation.

An RNN [23] is a deep learning method that has a memory, which enables it to capture and store information about previous inputs. This property makes RNNs able to effectively process sequences of battery data, where the SOC at any given time is influenced not only by current measurements but also by historical inputs and their corresponding SOC values. Therefore, RNNs are highly suitable for modeling the complex dynamic relationships between the battery’s inputs and its SOC. Additionally, RNN models can be trained on large amounts of data, allowing the models to generalize well to new situations, making them ideal for battery state estimations. In recent years, much work has been done for RNNs to make accurate SOC estimations and achieve adequate generalization ability. These studies and improvements can be classified into studies on the following categories: the structure of the neural network, the input features, and parameter optimization in training processes. Various structures have been proposed to achieve accurate SOC estimations, including the long short-term memory (LSTM) [24], gated recurrent unit (GRU) [25], bidirectional GRU (BGRU) [26], bidirectional LSTM (BLSTM) encoder–decoder architecture [27,28], and other structures. Feng et al. [29] used a clockwork RNN for SOC estimation, which divides hidden layers into several modules, capturing long-term and high-frequency information separately to capture the battery’s dynamic characteristics better. Wang et al. [30] proposed an SOC estimation method for small target sample sets using GRU-based transfer learning. In order to obtain better SOC estimation results using more correlated input features, Wu et al. [31] used the random forests method to select the aging and thermoelectric characteristic parameters with high correlations as input features of the LSTM. Chen et al. [32] introduced a denoising autoencoder neural network (DAE-NN) to reduce the effect of the measurement noise and combined a DAE with a GRU to achieve better estimation results. To study the impacts of the parameters in the training process, Xi et al. [33] built a TD-RNN model, examined each hidden neuron’s output, and tried to use a time–frequency analysis to quantitatively identify the ‘overexcited’ neurons. Ren et al. [34] used particle swarm optimization (PSO) to obtain the proper learning rate and epoch, which were then set as the hyperparameters to train the LSTM model. Xiao et al. [35] studied the effects of different optimizers, such as Adam and SGD.

So far, a variety of research studies have proved that RNN-based models can achieve accurate SOC estimations, and the current researchers are mainly focused on the further improvement of the SOC estimation accuracy. Nevertheless, with numerous deep learning-based estimation methods available, it can be challenging for battery management engineers to choose the most appropriate one for SOC estimations. As a result, it is important to evaluate the performance of the representative deep-learning-based estimation methods in different scenarios to determine the model with the most comprehensive performance. In addition, RNNs can be complex models that require significant computational resources to train and run. This can be a challenge in BMS, where the systems may have limited computational resources. Therefore, it is also crucial to compare the computational efficiency, including the running speed and memory consumption, of these data-driven models. Based on this analysis, it will be possible to choose appropriate models for real-world SOC estimations under various conditions.

To fill the gaps and provide guidance for engineering implementation, a framework for the comprehensive assessment of the RNN-based SOC estimation model is proposed in this paper. The main contributions of this work are summarized as follows: (1) a general framework for deep learning-based SOC estimation is summarized and constructed, mainly including the battery data acquisition, data processing, RNN-based model building and training, and SOC estimation and evaluation processes; (2) four kinds of RNNs employed in the SOC estimations are investigated, and the estimation performances of these RNN models are compared under three scenarios; (3) a multiple-criteria decision-making method based on the AHP is utilized to reflect the comprehensive performance of each RNN model, and finally to obtain the most suitable model for actual application.

The rest of the paper is organized as follows. Section 2 presents the methodologies of the deep-learning models for battery SOC estimations. Then, the battery tests and data collection process are elucidated in Section 3. Section 4 discusses the SOC estimation results and the corresponding comprehensive performance results under various scenarios. The conclusions are drawn in Section 5.

2. RNN-Based SOC Estimation and Evaluation

2.1. The Framework for RNN-Based SOC Estimation

The overall framework of the proposed RNN-based SOC estimation is illustrated in Figure 1, which includes the following four main steps.

Step 1: Battery data acquisition. For this paper, the battery data were acquired from the battery test experiment, which will be introduced in detail in Section 3. The battery current and voltage were recorded during experimentation, constituting the dataset’s inputs. The vector of inputs is defined as ${\mathit{x}}_t=\left[I_t, V_t\right]$, where $I_t$ and $V_t$ are the battery current and voltage at timestep t, respectively. The actual SOC is calculated according to the definition of the battery’s SOC:

$$y_t=SO{C}_t=SO{C}_{t-1}-\frac{I_t\Delta t}{Q},$$

(1)

where $SO{C}_t$ represents the battery’s SOC at timestep t, $\Delta t$ is the sampling time, and Q is the battery capacity.

Step 2: Data processing. The dataset used to train the networks is given by $\mathrm{D}=\left\{{\mathit{x}}_{t-T},\dots , {\mathit{x}}_{t-1},{\mathit{x}}_t, y_t\right\}$, where T is defined as the timestep, i.e., the length of data we use to predict the SOC at timestep t. In this study, the data are normalized with a Min-Max scaler, whose formula is as follows:

$$x^{\prime }=\frac{x-x_{min}}{x_{max}-x_{min}},$$

(2)

where x′ is the data after normalization and x represents the raw data; $x_{max}$ and $x_{min}$ are the maximum and minimum values in the raw data, respectively. The normalized data are divided into three separate sets, i.e., the training set, validation set, and test set. The training set is fed to the network when training the SOC estimation models, and the validation set is used for parameter optimization during training. In contrast, the test set is employed for model performance evaluations.

Step 3: RNN-based model building and training. Four RNN-based models were built, and detailed information is provided in Section 4. All parameters are randomly initialized, and then weights and biases are iterated with an optimizer until the loss is reduced to an acceptable range. The optimizer uses Adam or SGD; the processing and setting of optimizer parameters occurs with reference to [36,37]. Loss is denoted by the mean absolute error (MAE) or mean squared error (MSE). The validation set calculates the validation loss in each epoch, and the RNN model will be saved if the loss decreases.

Step 4: SOC estimation and model evaluation. To evaluate the RNN model for lithium-ion battery SOC estimation precisely, three metrics are calculated based on the test set, including the MAE, root mean squared error (RMSE), and maximum error (MAX). The calculation formulas are as follows:

$$MAE=\frac{1}{n}{\displaystyle \sum }_{t=1}^n\left|{\widehat{y}}_t-y_t\right|,$$

(3)

$$RMSE=\sqrt{\frac{1}{n}{\displaystyle \sum }_{t=1}^n{\left({\widehat{y}}_t-y_t\right)}^2},$$

(4)

$$MAX=\underset{t=1,\dots ,n}{\max }\left|{\widehat{y}}_t-y_t\right|,$$

(5)

where ${\widehat{y}}_t$ is the SOC estimation made by the RNN-based models.

2.2. Description of Different RNNs

The basic RNN [38] provides an elegant way of addressing time series prediction due to its ability to capture long-term dependencies of data with historically measured information. As shown in Figure 2a, the RNN consists of a hidden state $\mathit{h}$ and an optional output $\mathit{y}$, operating on a variable-length sequence $\mathit{x}$. At each time step t, the hidden state ${\mathit{h}}_{〈t〉}$ of the RNN is updated by:

$${\mathit{h}}_{〈t〉}=f\left({\mathit{h}}_{〈t-1〉}, x_t\right),$$

(6)

where $f$ is a non-linear activation function.

When choosing a large timestep for SOC estimation, it is difficult to train the RNN due to the gradient vanishing and exploding problem caused by the increase in power and decrease in the error gradient; that is to say, it is difficult for RNNs to use long-term memory. To tackle this issue, much work has been done to improve the model structure. For example, Lin et al. proposed variants of time-delay networks called NARX networks [39], which can solve the long-term dependency problems only using the output neuron feedback. Hochreiter et al. [40] and Cho et al. [41] upgraded the traditional RNN and proposed LSTM and RNN models separately, which are the two famous RNN structures.

The structure of the LSTM model can be depicted in Figure 2b. Unlike the basic RNN, where the gradient changes exponentially during backpropagation, the LSTM keeps the gradient constant by using cell memory. The key component of the LSTM unit is the state unit $C_j^{〈t〉}$, which is controlled by a forget gate unit $f_j^{〈\mathsf{t}〉}$ and an input gate unit $i_j^{〈\mathsf{t}〉}$. The output gate $o_j^{〈\mathsf{t}〉}$ is used to control whether the output $h_j^{〈t〉}$ of the LSTM cell is shut off or not. The actual activation of the unit $h_j$ at time t is computed by:

$$h_j^{〈t〉}=o_j\odot \tanh C_j^{〈t〉},$$

(7)

in which:

$$f_j^{〈\mathsf{t}〉}=\mathsf{\sigma }\left({\left[{\mathit{W}}_f{\mathit{x}}_{〈t〉}\right]}_j+{\left[{\mathit{U}}_f{\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(8)

$$i_j^{〈\mathsf{t}〉}=\mathsf{\sigma }\left({\left[{\mathit{W}}_i{\mathit{x}}_{〈t〉}\right]}_j+{\left[{\mathit{U}}_i{\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(9)

$$o_j^{〈\mathsf{t}〉}=\mathsf{\sigma }\left({\left[{\mathit{W}}_o{\mathit{x}}_{〈t〉}\right]}_j+{\left[{\mathit{U}}_o{\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(10)

$${\tilde{C}}_j^{〈t〉}=\tan \mathrm{h}\left({\left[\mathit{W}{\mathit{x}}_{〈t〉}\right]}_j+[\mathit{U}{\left({\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(11)

$$C_j^{〈t〉}=f_j^{〈\mathsf{t}〉}{C}_j^{〈t-1〉}+i_j^{〈\mathsf{t}〉}{\tilde{C}}_j^{〈t〉},$$

(12)

where x is the input, $h_{〈t-1〉}$ is the previous hidden state, and $\odot$ is an element-wise multiplication; W and U are weight matrices learned in the training process. The $\tan \mathrm{h}\left(·\right)$ is a hyperbolic tangent function and $\sigma \left(·\right)$ is a logistic sigmoid function, which have the following expressions:

$$\tanh \left(x\right)=\frac{2}{1+e^{-x}}-1,$$

(13)

$$\mathsf{\sigma }\left(x\right)=\frac{1}{1+e^x}.$$

(14)

However, some pieces of the LSTM architecture are unnecessary and lead to parameter redundancy. To address this question, a less computationally intensive gated RNN, the GRU model, was designed. The main difference between the GRU and LSTM models is that a single gating unit controls the forgetting factor and the decision to update the state unit simultaneously. A hidden GRU consists of the update gate and the reset gate, as shown in Figure 2c. The update gate z selects whether to completely ignore the state vector by updating the hidden state with a new hidden state $\tilde{h}$ or to copy it. The reset gate r decides which parts of the previous hidden state get used to compute the next target; when the reset gate is close to 0, the previous hidden state will be ignored and the hidden state will be reset using only the current input. In a GRU unit, the actual activation of the unit $h_j$ at time t is computed by:

$$h_j^{〈t〉}=z_j{h}_j^{〈t-1〉}+\left(1-z_j\right){\tilde{h}}_j^{〈t〉},$$

(15)

in which:

$${\tilde{h}}_j^{〈t〉}=\tan \mathrm{h}\left({\left[\mathit{W}{\mathit{x}}_{〈t〉}\right]}_j+[\mathit{U}{\left(\mathit{r}\mathbf{\odot }{\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(16)

$$z_j^{〈\mathsf{t}〉}=\mathsf{\sigma }\left({\left[{\mathit{W}}_z{\mathit{x}}_{〈t〉}\right]}_j+{\left[{\mathit{U}}_z{\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(17)

$$r_j^{〈\mathsf{t}〉}=\mathsf{\sigma }\left({\left[{\mathit{W}}_r{\mathit{x}}_{〈t〉}\right]}_j+{\left[{\mathit{U}}_r{\mathit{h}}_{〈t-1〉}\right]}_j\right),$$

(18)

The RNNs mentioned above have a “causal” structure, meaning that the state at time t only captures information from the past and the present inputs, ${\mathit{x}}_{〈1〉}, \dots ,{\mathit{x}}_{〈t〉}$, or the past output $\mathit{y}$. However, in some situations, a prediction of ${\mathit{y}}_{〈\mathit{t}〉}$ that depends on the whole input sequence is required. As illustrated in Figure 2d, the bidirectional RNN (BRNN) [42] makes it possible for the output layer to acquire information from both past and future states by splitting the regular RNN neurons into two directions, i.e., forward states and backward states. According to the basic unit adopted in the network, the BRNN can be further divided into the BLSTM and BGRU.

Several typical RNNs are introduced above. According to the characteristics of the network and the demand for SOC estimation, we finally selected the LSTM, GRU, BLSTM, and BGRU approaches as the network models for the comprehensive comparison.

3. Experimental Data Description

In this study, a commercial 18650-type battery was utilized, along with a Li(NiCoMn)O₂ cathode and graphite anode. The nominal capacity of the tested battery was 2.75 Ah, and the charging cut-off voltage and discharging cut-off voltage were 4.2 V and 2.5 V, respectively. To provide a sophisticated battery dataset and verify the RNN-based SOC estimation method, a battery test bench was established, as shown in Figure 3a, which consisted of a battery test system (Chroma 17011), a thermal chamber, and a host computer. The Chroma 17011 system was responsible for testing the charging and discharging of the battery under different conditions, and transmitting the battery-related information to the host computer. The host computer communicated with the Chroma 17011 system to record the battery information. The thermal chamber provided a stable operating environment for the battery.

The experimental scheme is presented in Figure 3b, and the specific steps were as follows. First, the battery capacity was calibrated using the standard test profile. The battery was charged with a constant current rate of 0.5 C until the cut-off voltage of 4.2 V was reached and charged with a constant 4.2 V until the current dropped to below 0.01 C. After standing for 1 h to make the battery reach a steady state, a full discharging process with a constant current of 0.5 C and a cut-off voltage of 2.5 V was employed. Subsequently, the OCV-SOC test and dynamic charge–discharge test, including NEDC, WLTP, and UDDS, were conducted. Then, an aging test for 100 charging cycles with a constant current of 0.5 C and discharging with 1 C was implemented. The above process was repeated six times to obtain the current and voltage data for battery discharging under dynamic conditions during each aging cycle. The whole aging test was conducted at a constant ambient temperature of 25 ± 2 °C.

Based on the above testing procedures, a sophisticated battery dataset covering different dynamic conditions and various battery aging statuses can be obtained. The obtained datasets are shown in

Table 1. The datasets obtained from the battery experiments.

Dynamic Test	Cycle
	0	100	200	300	400	500	600
NEDC	○	○	○	×	○	○	○
WLTP	○	○	○	○	○	○	○
UDDS	×	○	○	○	○	○	○

. Besides measuring the battery voltage and current, the reference SOC was calculated using the Coulomb counting method.

4. Comparative Analysis of Battery SOC Estimation

As shown in Figure 1, the neural network was constructed with four layers, including an input layer, an RNN layer, a fully connected layer, and an output layer. The four RNN models involved in the comparison are the GRU, LSTM, BGRU, and BLSTM models. To make the comparison results more reasonable and reliable, it is necessary to determine the appropriate hyperparameters offline before online estimation. The number of RNN nodes and timesteps chosen and suggested in part of the previous work are listed in

Table 2. The numbers of RNN nodes and timesteps chosen and suggested.

RNN	Input	RNN Nodes	Timesteps	Reference
LSTM	[V, I, T]	300 (50, 100, 200, 300, 400, 500)	-	[43]
LSTM	[V, I, T]	32 (32/50)	30	[44]
LSTM	[V, I, T]	300 (200–500)	20	[45]
LSTM	[V, I, T]	500	1000 (250, 500, 1000)	[24]
LSTM	[V, I, T]	100 (20–500)	-	[46]
LSTM	[V, I]	238 (150–400)	-	[34]
GRU	[V, I]	30	-	[47]
GRU	[V, I, T]	100	100	[48]
GRU	[V, I, T]	260	-	[35]
GRU	[V, I, T]	1000	1000	[25]
GRU	[V, I, T]	32	15	[30]
BGRU	[V, I, T]	128 (32, 64, 96, 128, 160, 192, 224, 256)	-	[26]
BLSTM	[V, I, T]	64 (16, 32, 64, 128, 256)	-	[28]

.

On this basis, taking the goals of real-time estimation and less storage space occupation into consideration, the timestep was set as 300 in this setup, which meant historical information over 5 min was fed into the network. After comparing the test results and model size, 300 RNN nodes were adopted. The other hyperparameters included the batch size, sampling interval, total training epochs, and loss function, which were set to 64, 1 s, 100, and “MAE”, respectively.

To assess the comprehensive performance of the RNN-based models, the test time and the model size were utilized to evaluate the computationally expensive performance beside the three accuracy indicators mentioned in Section 2. The model size is related to the sophistication of the RNN models and will affect the test time.

4.1. Scenario 1: The Estimation Accuracy

The datasets for the battery after 100 aging cycles were utilized in this part, and four models with the lowest validation losses were extracted after training with the NEDC, UDDS, and WLTP datasets. When training each model, one of the three datasets was used for testing and the other two for training, where 1% of the data were split for validation. The horizontal comparison of Figure 4 shows that the deviation between the SOC estimation and the reference SOC varies with the dynamic test condition, which is smaller under NEDC and WLTP conditions compared to the UDDS condition. This is maybe because of the low dynamic characteristics under UDDS conditions compared to NEDC and WLTP conditions, making it difficult for the RNN to extract effective features. The vertical comparison in Figure 4 presents shows these four RNN-based models led to similar SOC estimation results, which shows the inherent consistencies of RNN algorithms. Figure 4m–o intuitively illustrates the inaccuracy of the SOC estimation process, which calculates the error between the estimated SOC and reference SOC.

The statistical results in

Table 3. The statistical results of scenario 1.

Model	MAE	RMSE	MAX	Test Time (ms)	Model Size (KB)
	(NEDC100/UDDS100/WLTP100)
GRU	2.46/7.82/4.45	4.33/8.82/5.72	15.38/16.34/15.61	15.19	4318
LSTM	1.72/8.03/3.53	2.67/8.98/5.39	14.28/16.66/16.71	19.61	5376
BGRU	1.30/7.81/4.51	1.98/8.79/5.54	11.64/16.44/15.95	22.10	8607
BLSTM	1.05/7.81/1.81	1.58/8.77/3.00	10.44/16.69/15.12	27.36	10,724

Test time calculated from the average of the four models.

show that the BLSTM-based model functioned best in this case, with MAE values of 1.05%, 7.81%, and 1.81% under NEDC, UDDS, and WLTP conditions, respectively, followed by the BGRU-based model with a slight deviation. However, the memory consumption will rise along with the high precision, which can be inferred from the test time and model size. Questions of how to balance this conflict and objectively evaluate the differences between these models based on various indicators need to be discussed.

4.2. Scenario 2: The Estimation Adaptability Evaluation against Different Battery Statuses

Battery aging is a significant factor in the capacity decline, affecting SOC estimations. Thus, the SOC estimation accuracy under different battery statuses is an important index to evaluate the performance of the model. In this part, the datasets of the battery after 500 aging cycles were employed for validation and testing. The datasets of the battery after 100, 400, and 600 aging cycles were used for training.

As seen in Figure 5, the SOC curves estimated by the RNN models are an excellent approximation of the reference curve when the discharge processes begin and are about to end. However, the curves start to deviate from the reference curve when the battery discharges to a SOC of about 70%. It is apparent that the estimation results of the NEDC dataset outperform the other two datasets. The models embody inferior estimation performance under UDDS and WLTP conditions, whereas the estimation curves of the four models for each test set are pretty similar. On this basis, it can be inferred that the estimated performance of the RNN models is highly correlated with the operating conditions, which requires further study.

The estimated results of the three test sets (NEDC500, UDDS500, and WLTP500) are shown in

Table 4. The statistical results of scenario 2.

Model	MAE	RMSE	MAX
	(NEDC500/UDDS500/WLTP500)
GRU	2.75/7.45/4.92	4.62/8.48/6.09	16.23/16.05/16.01
LSTM	2.08/7.67/3.77	3.04/8.63/5.48	15.00/16.22/17.33
BGRU	1.43/7.45/4.99	2.20/8.46/5.91	12.52/16.09/16.31
BLSTM	1.04/7.44/2.28	1.69/8.43/3.30	11.19/16.26/15.55

. Similar to scenario 1, the BRNN-based models perform better in terms of the estimation error, and the difference between the BGRU and BLSTM models is pretty close. Although the MAE and RMSE of the GRU-based model are larger than the others, its MAX performance is reasonable.

4.3. Scenario 3: The Estimation Robustness Evaluation against Different Measurement Uncertainties

There are far more disturbances in BEVs than in labs, and the sensors in BEVs are less accurate than in the laboratory. Hence, random noise rates of 0.1 A and 0.01 V were added to the current and voltage, respectively, in the test datasets. Figure 6a–l shows the SOC estimation results of the models for test datasets containing noise. Figure 6m–o presents the estimation deviation compared to the SOC estimation on noise-free datasets (the result in Scenario 1), whose minor differences illustrate the excellent robustness of the four RNN-based models.

The statistical results of robustness in

Table 5. The statistical results of scenario 3.

Model	MAE	RMSE	MAX
	(NEDC100/UDDS100/WLTP100)
GRU	0.12/−0.05/0.11	0.07/−0.04/0.06	0.06/−0.02/0.03
LSTM	0.21/−0.05/0.07	0.14/−0.04/0.02	0.05/−0.03/0.04
BGRU	0.10/−0.05/0.11	−0.05/−0.04/0.07	0.11/−0.02/0.02
BLSTM	−0.01/−0.05/0.26	0.07/−0.04/0.10	0.07/−0.03/0.03

were calculated as follows:

$$R_r=\frac{E_i-E_{initial}}{E_{initial}},$$

(19)

where $E_i$ is the SOC estimation error in scenario 3 and $E_{initial}$ is the SOC estimation error in scenario 1. Here, $R_r$ represents the variation in the samples, for which positive values indicate the relative degradation of the accuracy and negative values indicate a relative improvement, so it is expected to be as small as possible.

The robustness performance was scattered. The LSTM-based model showed the lowest MAE and RMSE values under the WLTP conditions and the lowest MAX error under the NEDC conditions, while the BGRU-based model had the lowest RMSE under the NEDC conditions and lowest MAX value under the WLTP conditions. The BLSTM-based model had the slightest MAE deviation under NEDC conditions. Additionally, it is interesting to note that the datasets with noise showed even better SOC estimation results than the noise-free datasets under some conditions, especially the UDDS conditions, which can be explained by the fact that in some cases the presence of noise compensates for the model estimation bias.

4.4. Comprehensive Comparison of Different RNNs

So far, much work has been done on the RNN-based method to estimate battery SOC values, which has given satisfactory results. Here, we would like to provide a comparison to select the most suitable model for BEVs in the real world. Figure 7 shows the comprehensive performances of the different RNNs, considering their SOC estimation accuracy, adaptability, robustness, running efficiency, and memory usage. Since accuracy and adaptability indicators are expressed as errors, a lower value indicates better performance. As for the model size and memory occupation rate, they are also supposed to be smaller.

The performance values are normalized to reduce the divergence caused by the scale of evaluation indicators. Meanwhile, the assessment index of the model is expressed as a positive score $S_i$, for which the larger the value the better, to make the data more intuitive, which is calculated as follows:

$$S_i=1-\frac{In{d}_i-In{d}_{min}}{In{d}_{max}-In{d}_{min}},$$

(20)

where $In{d}_i$ is the normalized performance value, and $In{d}_{min}$ and $In{d}_{max}$ are the minimum and maximum values of the normalized performance, respectively.

As can be seen, the GRU-based model has the smallest model size and fastest speed with pretty good adaptability; however, the accuracy performance results are diverse under various dynamic driving conditions. Notably, the GRU-based model shows balanced performance under the NEDC conditions. The LSTM-based model performs well in terms of robustness with small storage occupation rates but behaves inferiorly in terms of accuracy and adaptability. The BGRU-based model shows good accuracy and adaptability under NEDC and UDDS conditions but with poor robustness performance. The BLSTM-based model has high accuracy but requires the most significant storage space, with superior robustness under the UDDS conditions and good adaptability under the WLTP conditions.

It is clear from the above results that the three evaluation metrics cannot yield a model that is unanimously the best. For instance, the BLSTM-based model gives the most precise SOC estimation results, but its adaptability to the battery status is not the strongest. Moreover, the model with the best robustness is not the most adaptable one either. Therefore, an evaluation scheme that can comprehensively considers various factors is required. To tackle this issue, it might be a solution to allocate weights for each index. This paper adopts a multiple-criteria decision-making method called the AHP, whose implementation process is illustrated in Figure 8a (for the detailed process and parameters, refer to [49]).

In terms of the analysis in the last section, various index results were obtained, including MAE, RMSE, and MAX values for the three scenarios under each dynamic cycle condition. In addition, the estimation model was supposed to calculate results fast and occupy less memory space, which is highly related to the neural network’s structure. Hence, the efficiency and the model size are also taken into account.

The pairwise comparison matrix (PCM), a significant concept in AHP, is constructed by comparing the importance of every pair of objects at the same level. Scenario 1 is supposed to be the most critical, and scenario 2 and scenario 3 are considered equally important because of the complicated operating conditions, while the efficiency and the model size are less important. Thus, the PCM of the first level is as follows:

$$PCM1=\begin{array}{c}accuracy\\ adaptability\\ robustness\\ model size\\ efficiency\end{array}\left[\begin{array}{ccccc}1& 2& 3& 3& 5\\ 1/2& 1& 3& 3& 5\\ 1/3& 1/3& 1& 3& 5\\ 1/3& 1/3& 1/3& 1& 3\\ 1/5& 1/5& 1/5& 1/3& 1\end{array}\right],$$

(21)

From the calculation, the maximum eigenvalue ${\lambda }_{max}$ of PCM1 is 5.30, and the criteria weights are 0.39, 0.29, 0.17, 0.10, and 0.05, respectively. The consistency index (CI) is 0.074 and the consistency ratio (CR) is 0.067, conforming to the consistency requirements.

The MAE, RMSE, and MAX are regarded as being of equal importance; thus, the weights of these three indexes are equivalent to 1/3 in the three scenarios of the second level and scenario 2 of the third level. NEDC, UDDS, and WLTP are also considered of equal importance; therefore, the weights of these three indexes are equivalent to 1/3 on the second level.

To minimize the differences among data for various dimensions for the evaluation indexes, the performance values are normalized and the data are confined to the range of [0, 1].

The final weights and calculated scores for each model are shown in Figure 8b. It is clear that the GRU-based model achieves the highest score of 0.67, which indicates its all-around performance, striking a balance among the estimation accuracy, adaptability against the battery status, robustness, model size, and computation efficiency.

5. Conclusions

In this paper, a comprehensive performance evaluation method based on AHP was proposed to select an appropriate RNN model to estimate battery SOC. To this end, four RNN-based models were trained to obtain relevant parameters. Then, the results in terms of estimation accuracy, adaptability to the battery aging state, and robustness to measurement interferences of different RNNs were compared, and the model size and operating efficiency of the different RNNs were also assessed. The model with the best comprehensive performance was determined by further applying the multi-objective analysis decision method, the AHP, and the normalized score. The evaluation results showed that the GRU-based model gave the most comprehensive estimation performance. Therefore, among the four RNN-based models trained with the dataset in this paper, the GRU-based model is recommended for online SOC estimations.

Although only four RNN-based models were compared in this paper, the proposed method can be used to comprehensively compare more SOC estimation models and obtain a more suitable choice. Additionally, in the work here, we drew our conclusions based on experiments performed at the same temperature. However, the temperature is a significant factor leading to SOC changes in battery systems. In addition, the proposed AHP method in this paper requires manual assignment, which is inconvenient for real-time selection. As the model selection plays a crucial role in SOC estimations during battery operation, more intelligent and considerable evaluation methods are worth further study.