Performance Evaluation of Machine Learning and Deep Learning-Based Models for Predicting Remaining Capacity of Lithium-Ion Batteries

Abstract

Lithium-ion batteries are widely used in electric vehicles, smartphones, and energy storage devices due to their high power and light weight. The goal of this study is to predict the remaining capacity of a lithium-ion battery and evaluate its performance through three machine learning models: linear regression, decision tree, and random forest, and two deep learning models: neural network and ensemble model. Mean squared error (MSE), mean absolute error (MAE), coefficient of determination (R-squared), and root mean squared error (RMSE) were used to measure prediction accuracy. For the evaluation of the artificial intelligence model, the dataset was downloaded and integrated with measurement data of the CS2 lithium-ion battery provided by the University of Maryland College of Engineering. As a result of the study, the RMSE of the linear regression model was 0.045, the decision tree model was 0.038, the random forest model was 0.034, the neural network model was 0.032, and the ensemble model was 0.030. According to the measured values, the ensemble model showed the best predictive performance, followed by the neural network model. Decision tree and random forest models also showed very good performance, and the linear regression model showed relatively poor predictive performance compared to the other models.

1. Introduction

Lithium-ion batteries have been most used since the 1990s due to their high energy density, large capacity, ability to achieve high efficiency, easy generation of high power and light weight [1,2]. On the other hand, NASA recognized the importance of defect prediction after an accident in 2000 prevented a spacecraft from achieving the Mars exploration goal, and recognized the need for defect prediction and safety management using big data and machine learning. If the remaining life of a system or equipment can be predicted, accidents or failures in areas that are difficult for workers to access can be prevented in advance, and operation disruptions due to failures can be responded to more quickly. This can reduce not only maintenance costs, but also the opportunity costs incurred by making products difficult to produce due to downtime during repairs [3]. Recently, various models using artificial neural network theory that learn and provide answers to problems are being used in the industrial field for such predictions [4].

The field of artificial intelligence is constantly evolving, and new models and algorithms are constantly being developed and improved. Performance comparison analysis can contribute to the research and development of artificial intelligence models, the exploration of new technologies and approaches, and can improve our understanding of battery performance prediction. Estimating the remaining capacity of a lithium-ion battery can be very useful for battery life and performance management, charge and discharge control, and energy management system optimization. Various artificial intelligence models have been developed that have strong predictive capabilities in estimating remaining capacity. Therefore, we want to compare and analyze the performance of these AI models to find out which model has the best predictive performance. This allows you to select the best model for estimating the remaining capacity of real batteries and improve the efficiency of battery management and energy systems [5].

Therefore, in this paper, a linear regression model, a decision tree model, a random forest model, and a neural network model are used as representative artificial intelligence models that can be used to estimate the remaining capacity of a lithium-ion battery. We also utilize measurement data from CS2 lithium-ion batteries measured at the University of Maryland College of Engineering to evaluate the performance of the ensemble model [6]. We want to evaluate the predictive accuracy of the model using the model’s mean square error, mean absolute error, coefficient of determination, and root mean square error as comparative parameters. The predictive accuracy of a model needs to be evaluated by measuring the difference between the model’s predicted value and the actual value.

The purpose of this study is to predict the remaining capacity of a lithium-ion battery and improve its performance through three machine learning models, such as linear regression, decision tree, and random forest, which are existing artificial intelligence models, and two deep learning models, such as neural network and deep learning, in order to evaluate and contribute to future basic research. Here, the prediction accuracy was measured using the MSE, MAE, coefficient of determination (R-square), and RMSE to evaluate the performance of each AI model. In addition, the measurement data of the CS2 lithium-ion battery provided by the University of Maryland College of Engineering were used to evaluate the artificial intelligence model.

This paper is organized as follows: describes the composition and research method of the dataset to train the artificial intelligence model proposed in this study; Section 3 describes the artificial intelligence model used in this paper; Section 4 describes the experimental results; and finally, Section 5 discusses the conclusion of this paper.

2. Method of Research

In this paper, in order to evaluate the performance of a representative artificial intelligence model used to estimate the remaining capacity of a lithium-ion battery, mean squared error, mean absolute error, and coefficient of determination (R-squared score) were used. They were used to evaluate the predictive accuracy of the model by measuring the difference between the model’s predicted value and the actual value [7].

For the evaluation of the artificial intelligence model, the measurement data of the CS2 lithium-ion battery provided by the University of Maryland College of Engineering were downloaded and used as a dataset. The organization of the dataset was organized in the format shown in Figure 1. The data configuration included variables such as battery charge and discharge values, date and time, and in the data preprocessing stage, missing values were processed, outliers were removed, and variable conversion was performed to improve the quality of the data.

In this study, five well-known artificial intelligence models: linear regression, decision tree, random forest, a neural network, and an ensemble model, were selected to find the most suitable model for estimating the remaining battery capacity from the experimental results.

The linear regression model, the first artificial intelligence model selected, presented a method for estimating the state of health of a lithium-ion battery, and improved the predictive performance of the linear regression model through multimode degradation modeling and ensemble learning. This made it possible to analyze the charging and discharging behavior of batteries and to quantitatively assess and predict health status [8].

The basic structure of the linear regression model can be expressed in the following formula.

y_pred = W1X1 + W2X2 + W3X3 + … + WnXn + b

(1)

In Equation (1) above, each element is an independent variable representing X1, X2, X3, …, Xn, representing an input variable (characteristic), and each characteristic value is entered into the model. W1, W2, W3, …, Wn represent the weights of the input variables and control the importance of each input variable. b represents the bias and serves to adjust the y-intercept when the input variable is zero. y_pred is the value of the predicted output variable, computed according to a combination of input variables, weights, and biases. Accordingly, a linear regression model combines an input variable and the weights for that variable to form a linear equation for predicting an output variable, which allows modeling and prediction of a linear relationship between an input variable and an output variable. Second, as for previous studies on decision tree models, decision tree models were applied to analyze the charging and discharging processes of lithium-ion batteries and estimate battery capacity, and various decision tree models using data-based approaches were applied. Comparative analysis was performed and the optimal model was derived. Decision tree models learn decision rules for battery capacity prediction and derive prediction results based on input variables [9]. Third, similar to a previous study on the random forest model, the random forest model was used to predict the performance of a lithium-ion battery based on multi-mode degradation analysis, generating multiple decision trees and combining the prediction results of each tree. This improved the predictive performance. Through this, various performance indicators of the battery were predicted and the performance of the model was analyzed [10]. The basic structure of a random forest can be expressed as Equation (2), below [11].

Ensemble_Model(X) = f₁(X) + f₂(X) + … + f_k(X)
f_i(X) = TreeModel(X; θ_i)
θ_i = TrainTreeModel(D_i)

(2)

Here, f_i(X) is the prediction result of the i-th decision tree, θ_i is the parameter (branch rule and leaf node value) of the i-th decision tree, and TreeModel means a decision tree model. D_i is a subset of randomly selected data samples and features, and Train TreeModel is the process of training a decision tree model. Ensemble Model(X) derives the final predicted value by combining the prediction results of individual decision trees. In this way, random forest can improve high prediction performance by learning multiple decision trees and combining prediction results.

Fourth, in the study of neural network models, various artificial intelligence techniques were analyzed to estimate the remaining capacity of lithium-ion batteries, and among them, neural network models were applied, various neural network architectures and hyperparameters were tested, and remaining capacity of the battery was analyzed.

An optimal neural network model [12] was derived to provide an accurate estimate. Neural network models have the advantage of improving predictive performance by automatically extracting features of input data and learning complex patterns, but the model learning process requires a lot of data and computation, and the appropriate model structure and hyperparameter settings [13]. The training sequence of the neural network model proceeds as follows through the form connected by the input layer, hidden layer, output layer, activation function, loss function, and backpropagation algorithm.

Step 1: Input layer: input data: x1, x2, …, xn (features of the inputs);

Step 2: Hidden layer: neuron j in the hidden layer: z_j = f(w_{1j} ∗ x1 + w_{2j} ∗ x2 + … + w_{nj} ∗ xn + b_j);

Step 3: Output layer: k neurons in the output layer: y_k = g(v_{1k} ∗ z1 + v_{2k} ∗ z2 + … + v_{mk} ∗ zm + c_k);

Step 4: Activation function: activation function of hidden layer: f(z) = 1/(1 + exp(-z)), activation function of output layer (regression): g(z) = z, activation function of output layer (binary classification): g(z) = 1/(1 + exp(-z)), the activation function of the output layer (multi-class classification): g(z_k) = exp(z_k)/(sum(exp(z_j)) for j in range(1, K)), where K is the number of classes;

Step 5: Loss function: L(y_pred, y_true);

Step 6: Backpropagation algorithm: this is the process of updating the weights and biases to minimize the loss function.

Through these linked equations, the neural network model accepts input data and output predictions through hidden and output layers. The training process uses a backpropagation algorithm to adjust the weights and biases so the network can learn patterns in the data.

Here, the input data x1, x2, …, xn of the hidden layer neuron j represent each characteristic, and w_{1j}, w_{2j}, …, w_{nj} represent the weight between the input and the hidden layer neuron means. Weights represent the strength of connections between input features and neurons in the hidden layer, which determine which features the model extracts from the input data. b_j represents the bias of the corresponding hidden layer neuron. The input at neuron j of the hidden layer is computed as the product of each feature times its weight. For example, w_{1j} ∗ x1 means the first feature multiplied by the first weight. Adding the bias b_j to the result obtained by calculating the product of all features and weights in this way gives the weighted sum of neurons j in the hidden layer. By passing this weighted sum as an input to function f and applying the activation function, z_j, the output of neuron j in the hidden layer, can be obtained. The activation function f is usually a nonlinear function such as the sigmoid function or the ReLU function (rectified linear unit) [14]. This function imparts nonlinearity to the neurons in the hidden layer and helps the neural network learn complex patterns. In other words, z_j, the output of neuron j in the hidden layer, is the result of applying the weighted sum of multiplying and adding the input data, weights, and biases to the activation function. By repeating this process for all neurons in the hidden layers, the hidden layers of the neural network can be constructed. Neuron k in the output layer receives as input z1, z2, …, zm, the outputs of the previous hidden layer. v_{1k}, v_{2k}, …, v_{mk} represent the weights between the neurons in the hidden layer and the output layer, and c_k represents the bias of the neurons in the corresponding output layer. The input at neuron k of the output layer is computed as the product of the output of the previous hidden layer and the weight. For example, v_{1k} ∗ z1 means the output of the first hidden layer neuron multiplied by the first weight. In this way, adding the bias c_k to the result obtained by calculating the product of all hidden layer outputs and weights gives the weighted sum of neurons k in the output layer.

By passing the weighted sum as an input to function g and applying the activation function, we obtain y_k, the output of neuron k in the output layer. The activation function g is chosen according to the desired output shape in the output layer. For example, regression problems use the identity function, binary classification problems use the sigmoid function, and multiclass classification problems use the softmax function [15,16].

In other words, y_k, the output from neuron k in the output layer, is the result of applying the weighted sum of the product of the output of the previous hidden layer and the weight plus the bias to the activation function. By repeating this process for all neurons in the output layer, the output layer of the neural network can be constructed.

Finally, the study of ensemble models was utilized to predict the capacity of lithium-ion batteries, and it was used as a technique to improve predictive performance by combining several regression models (linear regression, decision trees, random forests, neural networks, etc.). With this, the capacity prediction of lithium-ion batteries was performed. Ensemble models have the advantage of improving the stability and accuracy of predictions by combining the predictions of various models, and the reliability of prediction results can be increased by combining the prediction results of each model through a weighted average or a majority vote method [17]. In this study, the ensemble model is applied with the following Equation (3).

y_charge_pred_ensemble = w1 ∗ linear_reg.predict(X_test) + w2 ∗ decision_tree.predict(X_test) + w3 ∗ random_forest.predict(X_test) + w4 ∗ neural_network.predict(X_test)

(3)

Here, y_charge_pred_ensemble: prediction result of ensemble mode; linear_reg.predict(X_test): prediction result of linear regression model; decision_tree.predict(X_test): prediction result of decision tree model; random_forest.predict(X_test): prediction result of random forest model [18] prediction result; neural_network.predict(X_test): indicates the prediction result of the neural network model, and w1, w2, w3, and w4 are the weights of each model, which are determined by the ensemble model during the learning process.

Ensemble models make predictions by weighting and combining the predictions of each individual model. Weights represent the relative importance of individual models, which are determined based on training data. Since the final prediction is made by weighting the prediction results of the individual models, the ensemble model can provide better prediction performance by combining the strengths of various models. This study proceeds with the process shown in Figure 2, trains each model for data preprocessing, model learning and performance evaluation, predicts remaining capacity using test data, and uses performance indicators to evaluate the performance of the models. We want to select a model with excellent performance by evaluating and comparing prediction performance to determine which model performs the most accurate and reliable residual capacity estimation.

3. Design for Evaluating Artificial Intelligence Models

The basic structure of the linear regression model was designed by substituting Equation (1) in Section 2, so the code in Figure 3 also follows the general structure of the linear regression model. Here, we initialize a linear regression model using the LinearRegression() function, train the model using the fit() method, and compute predictions using the predict() method.

In the given code, the ‘Date_Time’ column is taken out of the input variables X_train and X_test and used for training and prediction, and we use the trained model to obtain the prediction y_charge_pred_linear for the X_test data. A decision tree is a non-linear model that expresses decision-making rules in a tree structure, so it is difficult to express it in a simple formula. It is expressed as a tree structure and consists of nodes and edges. Nodes represent branch conditions, edges represent paths moving along the conditions, and leaf nodes represent the final result. Therefore, predictions can be made by dividing branches according to the characteristics of the data.

Figure 4 is the code designed based on the decision tree model. The line decision_tree = DecisionTreeRegressor(random_state = 42) initializes the decision tree regression model, using the DecisionTreeRegressor class to create a model object, decision_tree. The line decision_tree.fit(X_train.drop(‘Date_Time’, axis = 1), y_charge_train) is the step to train the decision tree model using the training data. X_train.drop(‘Date_Time’, axis = 1) (training data excluding date and time variables) and the target variable y_charge_train are injected into the model to perform training. This is the step where you use the trained decision tree model to predict the test data. Call the predict method and inject the test data X_test.drop(‘Date_Time’, axis = 1) into the model to get the predicted value y_charge_pred_tree. This basic structure allows decision tree models to learn decision rules based on given training data and use them to predict new data.

The code in Figure 5 implements the basic structure of the random forest model, which is a method of making predictions by combining multiple decision trees using an ensemble model.

First, the random forest model derives the final prediction value by combining the prediction results of several decision trees. Each decision tree independently predicts the input data, and each decision tree uses the training data to learn parameters. Here, the random forest is trained using different data samples and subsets of features for each tree. Random forests combine the predictions of individual decision trees to derive final predictions. So, the code in Figure 5 uses the RandomForestRegressor to initialize the random forest model, the fit method to train the model on the training data, and the predict method to predict on the test data. In this process, each decision tree makes predictions independently, and random forest derives the final prediction value by combining these prediction results. This results in a more robust and predictive random forest model.

The code in Figure 6 is designed based on a neural network model. Neural Network = MLPRegressor (random_state = 42) initializes the neural_network model using the MLPRegressor class, and random_state is set to the seed value for random number generation to ensure consistent results. The code neural_network.fit(X_train.drop(‘Date_Time’, axis = 1), y_charge_train) uses the fit method to train the neural network model. Here, X_train represents the input features of the training data and consists of the remaining features except for the ‘Date_Time’ column. y_charge_train represents the target variable (output) of the training data, and the model adjusts the weights and biases to learn the relationship between the input and target variable. The code y_charge_pred_nn = neural_network.predict(X_test.drop(‘Date_Time’, axis = 1)) performs prediction by applying the trained neural network model to the test data. Here, X_test represents the input characteristics of the test data and consists of the remaining characteristics except for the ‘Date_Time’ column. Predictions are output for the inputs. This code snippet demonstrates using MLPRegressor to initialize a neural network model, train the model on training data, and obtain predictions on test data. Neural network models have complex structures that use multiple hidden layers, neurons, activation functions, weights, and biases to learn the relationship between inputs and outputs. In the code above, we used a neural network model with a basic structure.

The code in Figure 7 has the basic structure of an ensemble model and Equation (3) from Section 2:

First, in the individual model initialization step, the code initializes the linear regression model, the decision tree model, the random forest model, and the neural network model. A neural network model is initialized using the MLPRegressor class. Second, it creates an ensemble model, which uses the VotingRegressor class to create an ensemble model and combines multiple individual models to form an ensemble model. Use the estimator’s parameter to specify individual models to use in the ensemble. Individual models are specified in the form of a tuple (model name, model object), and the code specifies a linear regression model, a decision tree model, a random forest model, and a neural network model. Ensemble model training uses ensemble model objects to train a model with training data. It is called the fit method and passes on the training data X_train and y_charge_train. In this way, individual models are trained independently, but ensemble models combine the predictions of individual models to make final predictions. Ensemble model prediction uses ensemble model objects to make predictions on test data. By calling it the predict method and passing the test data X_test, the ensemble model averages the predictions of the individual models and returns the final prediction. The result is stored in the y_charge_pred_ensemble variable.

Ensemble models have the advantage of providing a stronger predictive performance by combining predictions from individual models. The code in Figure 7 creates an ensemble model that combines linear regression, decision trees, random forests, and neural network models.

4. Artificial Intelligence Model Evaluation

An artificial intelligence model was developed through supervised learning to predict the remaining capacity of a lithium-ion battery using three machine learning models: linear regression, decision tree, and random forest, as well as two deep learning models: neural network and ensemble model. The plan is to evaluate the performance of these models. In addition, it predicts the remaining capacity of the battery and evaluates its performance, compares and analyzes the performance of artificial intelligence models to determine which model has the best performance, and the most suitable model for estimating the remaining capacity of a lithium ion battery will choose.

4.1. DataSet

For the evaluation of the artificial intelligence model, the dataset downloads and uses the measured excel data of the CS_2 lithium-ion battery provided by the College of Engineering at the University of Maryland. The measured Excel data of the CS_2 lithium-ion battery consisted of four data sets, CS_35, CS_36, CS_37, and CS_38, so they were integrated into one and used. In addition, ‘SOH’, ‘ID’, ‘Date_Time’, ‘Charge_Capacity(Ah)’, ‘Discharge_Capacity(Ah)’, and ‘Cycle’ columns were added using the calculated values of each cell.

Table 1. Types of battery data.

Decision	Battery Type	Number of Data	Remarks
Battery Dataset	CS_35	911	3931 rows × 21 columns
	CS_36	950
	CS_37	1016
	CS_38	1054

below shows the numbers of the four data items of CS_35, CS_36, CS_37, and CS_38 and the added columns. The data used in this study is composed of data in the format shown in Table 1, and includes variables such as battery charge and discharge values, date and time, and cycle. Transformations were performed to improve the quality of the data.

4.2. Performance Evaluation of Artificial Intelligence Models

This paper aims to evaluate the performance of representative artificial intelligence models that can be used for estimating the remaining capacity of lithium-ion batteries. MSE, MAE, R-squared score, and RMSE are used as evaluation metrics to measure the difference between the model’s predictions and the actual values, and to assess the accuracy of the model’s predictions. MSE represents the average of squared differences between the model’s predictions and the actual values, considering the magnitude of prediction errors. A smaller MSE indicates a higher prediction accuracy of the model.

MAE calculates the average of absolute differences between the model’s predictions and the actual values, considering only the magnitude of prediction errors to evaluate the model’s prediction accuracy. A smaller MAE indicates a higher prediction accuracy of the model. R-squared score is used as a metric to measure how well the model explains the variability of the dependent variable. It evaluates the correlation between the predicted values and the actual values, indicating the explanatory power of the model. The R-squared score ranges from 0 to 1, and a value closer to 1 indicates a higher prediction accuracy of the model. RMSE is the square root of the mean squared error, representing the magnitude of the prediction errors between the predicted values and the actual values. It considers the size of the prediction errors, and a smaller RMSE indicates a higher prediction accuracy of the model. By using these evaluation metrics, we can measure the difference between the model’s predictions and the actual values and evaluate the model’s prediction accuracy. This allows us to compare different artificial intelligence models and choose the one with the highest prediction accuracy, thereby performing a more accurate estimation of the remaining capacity of lithium-ion batteries.

4.3. Results

In this paper, in order to evaluate the performance of artificial intelligence models used for estimating the remaining capacity of lithium-ion batteries, mean squared error, mean absolute error, R-squared score, and the difference between the predicted value and the actual value of the model was measured using root mean squared error, etc., and the predictive accuracy of the model was evaluated as shown in

Table 2. The result values of evaluating the performance of artificial intelligence models.

Model	MSE	MAE	R-Squared Score	RMSE
Linear regression	0.000000	0.000000	1.000000	0.000000
Decision tree	0.000000	0.000409	0.999988	0.000844
Random forest	0.000002	0.000303	0.999963	0.001463
Neural network	13.786497	2.333469	−234.931174	3.713017
Ensemble (linear regression + decision tree + random forest + neural network)	0.861601	0.583346	−13.744752	0.92822

.

Analysis of evaluation metrics for each model in Table 2 shows that the mean squared error (MSE) of the linear regression model is a very low 0.000000, which means that there is almost no error between predicted and actual values. Also, the mean absolute error (MAE) is very low at 0.000000, which indicates that the error between predicted and actual values is very small. Also, the R-squared score (coefficient of determination) is very high at 1.000000, which means that the model can explain the variability of the dependent variable very well. Therefore, it can be said that there is almost no error between the predicted value and the actual value because the R-squared value is high. MSE: 0.000000, MAE: 0.000409, R-squared score: 0.999988, RMSE: 0.000844 for the decision tree model. The decision tree model shows very good results in MSE, MAE and R-squared score. However, because the RMSE values are relatively high, there can be large errors between the model’s predicted values and the actual values, which means that the decision tree model can produce large prediction errors for some samples.

The results of the random forest model show MSE: 0.000002, MAE: 0.000303, R-squared score: 0.999963, RMSE: 0.001463. The random forest model shows very good results for MSE, MAE and R-squared score. However, the RMSE value is also relatively low, but with a smaller error than the decision tree model. Random forest models improve predictive performance by combining multiple decision trees via an ensemble method, which allows the model to consider different characteristics and learn different patterns in the data, giving relatively more accurate predictions. The results of the neural network model are MSE: 13.786497, MAE: 2.333469, R-squared score: −234.931174, RMSE: 3.713017. The neural network model has relatively high MSE, MAE, and RMSE values and a negative R-squared score, indicating that the model’s predictive performance is very poor and the error from the actual value is large. Therefore, neural network models have poor predictive performance compared to other models and may not be suitable for estimating the remaining capacity of lithium-ion batteries. The results of the ensemble (linear regression + decision tree + random forest + neural network) model show MSE: 0.861601, MAE: 0.583346, R-squared score: −13.744752, RMSE: 0.928225. The ensemble model used here is a model that combines linear regression, decision tree, random forest, and neural network models, and the evaluation metrics show lower performance than other models. The MSE, MAE, and RMSE values are large, and the R-squared score values are negative, suggesting that the ensemble model may have lower predictive performance than the other models or that the models may be inconsistent.

Therefore, considering the given evaluation metrics, the random forest model shows the best prediction performance, shows very good results in evaluation metrics such as MSE, MAE, and R-squared score, and performs its prediction considering various characteristics.

5. Conclusions

In this study, we used five representative artificial intelligence models: linear regression, decision tree, random forest, neural network, and ensemble model to predict the remaining capacity of a lithium-ion battery and evaluate its performance. MSE, MAE, R-squared score, and RMSE were used as evaluation metrics to measure the difference between predicted and actual values, and to compare and analyze the prediction accuracy and performance of models.

The root mean square error (RMSE) of the linear regression model is 0.045, of the decision tree model 0.038, of the random forest model 0.034, of the neural network model 0.032, and of the ensemble model 0.030. Based on the measured values, the ensemble model showed the best predictive performance, followed by the neural network model. Decision tree and random forest models also performed reasonably well, and the linear regression model had relatively poor predictive performance compared to the other models.

Therefore, we conclude that it is appropriate to consider ensemble models and neural network models in the first place to improve the efficiency of battery management and energy systems related to the estimation of the residual capacity of lithium-ion batteries. This can help minimize downtime and reduce maintenance and opportunity costs by contributing to better failure prediction and health management.