Thermal Runaway Diagnosis of Lithium-Ion Cells Using Data-Driven Method

Abstract

Fault diagnosis is crucial to guarantee safe operation and extend the operating time while preventing the thermal runaway of the lithium-ion battery. This study presents a data-driven thermal runaway diagnosis framework where Bayesian optimization techniques are applied to optimize the hyperparameter of various machine learning techniques. We use different machine learning models such as support vector machine, naive Bayes, decision tree ensemble, and multi-layer perceptron to estimate a high likelihood of causes of thermal runaway by using the experimental measurements of open-source battery failure data. We analyze different evaluation metrics, including the prediction accuracy, confusion metrics, and receiver operating characteristic curves of different models. An experimental evaluation shows that the classification accuracy of the decision tree ensemble outperforms that of other models. Furthermore, the decision tree ensemble provides robust prediction accuracy even with the strictly limited dataset.

1. Introduction

The lithium-ion battery (LIB) has become the essential energy storage solution across various applications, including transportation, electric grid, and consumer electronics, since it has considerably improved efficiency and reduced costs over the past decade [1,2,3]. However, battery failures cause serious problems such as explosion and fire accidents in aircraft, robots, and smartphones [4,5,6].

Mechanical, electrical, thermal, and electrochemical abuse are the critical factors that trigger and cause severe battery failure. Thermal runaway is an uncontrollable chemical process within a battery cell that causes a catastrophic fire. Chain reactions can occur between heat, temperature, and chemical reactions if a battery exceeds specific temperature thresholds [7]. This is especially dangerous for cylindrical cells that could explode due to their pressure build-up design. Conventional approaches have developed the thermal runaway prediction model based on the internal mechanism of Li-ion cells to support early warning [8,9]. However, the behaviors of Li-ion cells depend on various factors, including the manufacturer and model during thermal runaway [10]. Furthermore, repeatedly tested identical cells under the same abuse conditions still show considerably different heat output and temperatures [11]. Thus, deterministic analytical models are insufficient for capturing the complicated behaviors of cells undergoing failure, even for the same cell.

Most previous works on machine learning (ML) techniques focus on lifetime predictions [12,13] and the development of optimal charging protocols for LIB cells [14,15]. Although some ML models are developed for fault diagnosis, most works consider the system level rather than the cell level of LIBs [4]. Recent research combines the ML model with varying levels of simulation model complexity to investigate the vast parameter spaces in abuse testing [16,17]. However, the experimental validation of these models is costly and complex, making the validation process especially challenging.

We use different ML techniques to estimate the most likely cause of the thermal runaway by using the experimental measurements of open-source battery failure data [18]. Bayesian optimization techniques are applied to optimize the hyperparameter of various ML techniques such as support vector machine (SVM), naive Bayes (NB), decision tree ensemble (DTE), and multi-layer perceptron (MLP) models. We analyze different evaluation metrics, including prediction accuracy, confusion metrics, receiver operating characteristic (ROC) curves, and the feature importance of ML models.

2. Open Battery Failure Dataset

The fractional thermal runaway calorimeter (FTRC) aims to quantify the heat and mass released from the battery cell during the thermal runaway [19]. In [18], they conduct extensive experiments and build a large open-source database of battery failure incidents for different commercial cell types and causes. This rich set of experimental data using different cell manufacturers and abuse types provides a valuable benchmark.

In [18], the thermal runaway of each cell was triggered using three different abuse methods, namely, “Heat”, “ISC”, and “Nail”.

• Heat: Heat is applied to the battery cell until it reaches thermal runaway.
• ISC: One of the latent defects in cell electrodes is an internal short-circuiting (ISC) device [20]. It can trigger thermal runaway at specific locations within the cell if the temperature reaches about 57 °C, much lower than that of the Heat abuse method.
• Nail: The pneumatically activated nail of FTRC penetrates 9 mm of the cell body.

Table 1. Measured features of open-source battery failure dataset [18].

Data Type	Unit	Description
CellCap	Ah	Maximum capacity of the cell.
CellEn	Wh	Maximum possible stored energy of the cell.
PreCellM	g	Mass of the cell before the experiment.
BottomVent	-	Whether the cell includes a bottom vent or not.
PrePosM	g	Mass of the copper mesh of the FTRC positive side.
PreNegM	g	Mass of the copper mesh of the FTRC negative side.
ConfPos	-	Seal material of the FTRC positive side.
ConfNeg	-	Seal material of the FTRC negative side.
CellFail	-	Mechanism of the cell fails.
HeatLossRate	kJs	Heat loss rate
DiffM	g	Mass difference between before and after the experiment.
PostCellM	g	Mass of the remained cell.
PostPosMateM	g	Mass of the positive side ejected mating.
PostPosBoreM	g	Mass of the positive side ejected bore.
PostPosCuM	g	Mass of the positive side copper mesh.
PostNegMateM	g	Mass of the negative side ejected mating.
PostNegBoreM	g	Mass of the negative side ejected bore.
PostNegCuM	g	Mass of the negative side copper mesh.
BaselineEn	kJ	Released energy without corrections for heat and mass loss.
CorrLossEn	kJ	Released energy corrected for heat loss.
CorrEn	kJ	Released energy, corrected for both heat and mass loss.
PosEn	kJ	Energy of unrecovered mass ejected through the positive side.
NegEn	kJ	Energy of unrecovered mass ejected through the negative side.
BodyEnP	%	Percent of the released energy from the cell casing.
PosEnP	%	Percent of the released energy from the positive side.
NegEnP	%	Percent of the released energy from the negative side.
Cause	-	Trigger mechanism to induce thermal runaway.

summarizes the measured features and the causes of the open-source battery failure databset [18]. The dataset includes the heat and mass ejected data using FTRC from 364 tests on various batteries. Abuse tests use different cylindrical battery models with cell formats such as 18,650 and 21,700 from multiple manufacturers. Each sample of the abuse test consists of 26 measured features and 1 output label of causes. By considering the ejected location within the cell, most heat and mass measurements using FTRC can be categorized into three distinct groups, namely, (1) positive, (2) body, and (3) negative. In addition, the released energy is computed based on the fractional heat output. This dataset offers insights into the heat, mass, and energy changes in each cell under different abuse tests.

3. Data-Driven Thermal Runaway Diagnosis

Figure 1 shows the thermal runaway diagnosis framework using different ML techniques. We use SVM, NB, DTE, and MLP models to estimate three causes of thermal runaway, namely, Heat, ISC, and Nail as the fault diagnosis of lithium-ion cells. Most previous works of thermal runaway prediction focus on the model development of early warning to prevent battery failure rather than the thermal runaway diagnosis [8,9]. We use the Bayesian algorithm to optimize the hyperparameter of different models. The Bayesian optimization algorithm aims to minimize the cross-validation loss for the given hyperparameter set in a bounded domain of each model [21]. We use 10-fold cross-validation to determine how effectively each ML model generalizes.

3.1. Support Vector Machine

SVM is a conventional ML technique that finds the hyperplane to maximize the margin between different classes [22]. The eligible hyperparameters of SVMs to optimize are box constraint, strategy, kernel function, feature scale, and polynomial order related to the hyperplane.

Table 2. Hyperparameter of SVM.

	Range	Selected	Note
Box constraint	$[{10}^{-3},{10}^3]$	374.73	-
Strategy	[OvR, OvO]	OvO	-
Kernel function	[Linear, Gaussian, Polynomial]	Gaussian	-
Feature scale	$[{10}^{-3},{10}^3]$	6.14	-
Polynomial order	$[2,4]$	-	Polynomial

shows the range of these hyperparameters and selected values obtained by Bayesian optimization. A box constraint regulates the maximum penalty for violating data points of the margin to avoid overfitting. Increasing the box constraint reduces the number of hyperplanes at the cost of the training times. Two common strategies to apply binary classification algorithms for multi-classification problems are the one-vs-rest (OvR) and one-vs-one (OvO) schemes [22]. OvR divides a multi-class problem into one binary problem per class, while OvO converts it into binary problems per each pair of classes. Thus, for the given total number of unique class labels K, the number of binary learners of OvR is K, while the one of OvO is ${\displaystyle \frac{K(K-1)}{2}}$, where it exhausts all combinations of class pair assignments.

The Gram matrix is computed using the kernel function in Lagrange dual formulation for nonlinear SVM model [22]. The kernel trick essentially generates the Gram matrix $\mathbf{G}({\mathbf{x}}_i,{\mathbf{x}}_j)$ instead of the inner product of the features $\left({\mathbf{x}}_i^{\top }{\mathbf{x}}_j\right)$, where ${\mathbf{x}}_i,{\mathbf{x}}_j\in {\mathbb{R}}^p$ represent the vectors of samples i and j. We use different kernel functions: linear, Gaussian, and polynomial. The feature matrix is divided by the feature scaling parameter for computing the Gram matrix. In addition, the polynomial order is the order of the polynomial kernel function.

3.2. Naive Bayes

Naive Bayes classifier relies on the probability density estimation technique of input features of a given class, which naively assumes that the features are conditionally independent [23]. Since this assumption allows us to individually estimate the univariate class-conditional probability density function of each feature, it considerably reduces the complexity of the training process and the amount of training data required compared to other classifiers. The class-conditional independence assumption performs well in practice, even when the one is not generally valid.

The naive Bayes classifier applies a kernel density estimation (KDE) method to the training dataset of each class. A non-parametric estimation technique of KDE uses a kernel distribution characterized by a kernel function and its bandwidth parameter to regulate the smoothness of the resulting probability density function. Thus, the eligible hyperparameters to optimize are kernel smoothing functions and their bandwidth. We consider different kernel smoothing functions of Gaussian, uniform, parabolic, and triangle functions with a bandwidth in the range of $[0,100]$. Bayesian optimization algorithm picks the Gaussian smoothing function and $0.873$ as the bandwidth.

3.3. Decision Tree Ensemble

A classifier ensemble combines multiple classification models to enhance the prediction accuracy and generalization compared to a single classifier [24]. We use the decision tree as the weak classification model where the standard CART algorithm [25,26] is used to build trees.

We use the adaptive boosting aggregation method, which sequentially trains learners as the classifier ensemble [27]. The algorithm calculates the weighted pseudo-loss of each classifier t over N samples and K classes:

$$\begin{array}{c}\hfill {ϵ}_t={\displaystyle \frac{1}{2}}\sum _{i=1}^N\sum _{k\ne y_i}{d}_{i,k}^t(1-h_t({\mathbf{x}}_i,y_i)+h_t({\mathbf{x}}_i,k))\end{array}$$

(1)

where $y_i\in \{1,\dots ,K\}$ is the class label of sample i, $h_t({\mathbf{x}}_i,k)$ is the confidence probability of classifier obtained by model t to class k of sample i, and $d_{i,k}^t$ is the weight of classifier t to class k of sample i. Note that the second part of the sum is over all erroneous classes $k\ne y_i$.

It essentially increases and decreases weights for misclassified samples and correctly classified samples, respectively, by classifier t. The data are then used to train the next classifier $t+1$ using the updated weights $d_{i,k}^{t+1}$. The decision trees use the pseudo-loss as a metric of classification accuracy. After training finishes, it uses the weighted combination of the multiple models to estimate the class of new sample $\mathbf{x}$:

$$\begin{array}{c}\hfill h_{fin}\left(\mathbf{x}\right)=\arg \underset{k\in K}{max}\sum _{t=1}^T\eta {\alpha }_t{h}_t(\mathbf{x},k)\end{array}$$

(2)

where $\eta$ is the learning rate and ${\alpha }_t=log\left({\displaystyle \frac{1-{ϵ}_t}{{ϵ}_t}}\right)$ are weights of the classifier within the ensemble.

Table 3. Hyperparameter of DTE.

	Range	Selected	Note
Number of learning cycles	$[10,500]$	17	Aggregation
Learning rate	$[{10}^{-3},1]$	0.86	Aggregation
Maximum number of splits	$[1,145]$	119	Decision tree
Minimum leaf size	$[1,290]$	2	Decision tree
Split criterion	[Gini, Deviance, Twoing]	Gini	Decision tree

summarizes the hyperparameter to optimize for the aggregation method and decision tree model. We adapt the number of ensemble learning cycles where we train one weak tree learner at every learning cycle. Furthermore, we optimize the learning rate $\eta$ for shrinkage of the adaptive boosting aggregation method by multiplying $\eta$ by ${\alpha }_t$, essentially the shrink in the contribution of each new model learned in the ensemble. This parameter controls how much the new model contributes to the existing one. The hyperparameters of the decision tree model include the maximal number of node splits, the minimum number of samples per leaf, and the split criterion [26]. The depth of a tree affects the training time, memory consumption, and classification accuracy. Large values of the maximal number of node splits per tree and small values of the minimum number of samples per leaf generate deep trees. A decision tree splits nodes according to various criteria, including Gini’s diversity, deviance (also known as cross-entropy), and the twoing rule.

3.4. Multi-Layer Perceptron

We apply the fully connected (FC) neural network thanks to its relative simplicity and ease of implementation while extracting useful information from noisy signal [28]. Figure 2 shows our proposed MLP network consisting of the input layer, FC layer, batch normalization layer, activation function, dropout layer, and softmax layer for the classification. The input layer defines the input data size dependent on the input feature set. The middle layers of the network use the repeating set block denoted as FCSet, composed of FC, batch normalization, and ReLu layers. We add the dropout layer between two repeating set blocks in series. Then, the additional FC layer and the softmax activation function predict the classification label as the final output. The eligible hyperparameters to optimize are the dropout probability, initial learning rate, and gradient decay factor of the Adam optimizer, as shown in

Table 4. Hyperparameter of MLP.

	Range	Selected
Drop probability	$[0.1,0.8]$	0.5
Initial learning rate	$[{10}^{-3},1]$	0.001
Gradient decay factor	$[0.1,1]$	0.9

.

4. Performance Evaluation

We split a dataset $\text{0.8:0.2}$ into training and testing sets, which means 291 samples for training and 73 samples for testing out of $N=364$. The number of samples per class is $99,155$, and 110 for abuse classes of ISC, Heat, and Nail, respectively. The ML models are trained and tested on the Intel Xeon Platinum 8270 processor (Intel, Santa Clara, CA, USA) and the NVIDIA RTX 8000 and A6000 GPUs (NVIDIA, Santa Clara, CA, USA). We use different evaluation metrics, including the prediction accuracy, confusion metrics, ROC curves, area under the curve (AUC), and feature importance scores of various ML models.

4.1. Comparison between Different Machine Learning Models

The strictly limited amount of samples $N=364$ may incur serious generalization problems such as overfitting or data selection bias for different ML models. To quantify the uncertainty of limited data, we evaluate the average and standard deviation of classification accuracies of multiple trained models for each ML technique. We train multiple SVM, NB, DTE, and MLP models based on the randomly shuffled dataset with different 10 random seeds.

Table 5. Average and standard deviation of classification accuracies using SVM, NB, DTE, and MLP models.

	SVM	NB	DTE	MLP
Accuracy	$0.79\pm 0.04$	$0.68\pm 0.03$	$0.90\pm 0.02$	$0.78\pm 0.03$

summarizes the average and standard deviation of classification accuracies of 10 trained models of each SVM, NB, DTE, and MLP. The standard deviation of classification accuracy is not significant for each model. Due to the effective dropout layer, the MLP model shows a standard deviation range similar to the other ML models. This result proves that the Bayesian optimization algorithm and the cross-validation efficiently generalize each ML model by reducing the overfitting or data selection bias problems.

Now, we compare the performance of the representative SVM, NB, DTE, and MLP models trained using the same training dataset. Figure 3 shows the confusion matrices of SVM, NB, DTE, and MLP models to predict three causes, including Heat, ISC, and Nail. A confusion matrix presents how well a classification model performs for the testing dataset [29]. Overall, the classification accuracies of SVM, NB, DTE, and MLP are $0.80$, $0.71$, $0.90$, and $0.81$, respectively. DTE predicts the abuse method of the thermal runaway well compared to other models of SVM, NB, and MLP due to the strength of the model aggregation of multiple decision trees. SVM and MLP show a comparable prediction accuracy of around $0.8$, while NB has a higher classification error than other models. We note that the accuracy of DTE is comparable to the average one while other models of SVM, NB, and MLP slightly perform better than the corresponding ones of Table 5.

We conduct the mid-p McNemar test of DTE since it allows us to evaluate the statistical significance of whether one classification model performs better than another [30]. The mid-p values of DTE compared to SVM, NB, and MLP are $0.049,7.2861\times {10}^{-4},$ and $0.022$, respectively. All tests have a mid-p value of nearly zero, which indicates that DTE guarantees better prediction than other models.

Both Heat and ISC classes are generally hard to distinguish for different classification models, as shown in Figure 3. Although the ISC abuse may occur at a considerably lower temperature than the one of Heat, there are no measured features that explicitly provide the detailed heat and mass changes during the thermal runaway experiments using FTRC. Furthermore, the real-time measurements are also hard to obtain in practice. The classification accuracies of DTE are around $0.94$ and $0.75$ for Heat and ISC classes, respectively, greater than SVM, NB, and MLP. SVM, NB, and MLP models show considerably low classification accuracy for the ISC class around $0.6$. While SVM, DTE, and MLP perfectly predict nail abuse, the classification accuracy of NB is only around $0.84$ for the Nail class. The NB model has classification errors between all abuse methods of the thermal runaway.

Figure 4 presents the ROC curves using SVM, NB, DTE, and MLP models for different ISC, Heat, and Nail abuses. It shows the true positive ratio (TPR) over the false positive ratio (FPR) of each abuse throughout threshold intervals of $[0,1]$. In each figure, as the curve approaches the upper left corner and the diagonal line, the classifier improves and worsens, respectively. Note that the upper left corner is the optimal operating point (i.e., TPR = 100%, FPR = 0%) of the classifier. The DTE curve moves toward the upper left corner, especially for Heat and Nail, because of the low FPR of DTE. DTE significantly outperforms other classifiers for all causes.

To provide a more detailed analysis, Figure 4 also summarizes the AUCs of all models for different abuse methods. The DTE model consistently provides higher AUCs than those using other models for the Heat and Nail classes. For the ISC abuse, the AUC value of DTE remains above $0.92$, while other models, including SVM, NB, and MLP, show relatively low AUC values below $0.82$. Hence, the comparison between different models proves the strengths of the ensemble aggregation methods of diverse decision trees to capture the complex relationship between various features and abuse methods.

Next, we discuss the performance sensitivity of ML models for various batteries of cell formats and manufacturers.

Table 6. Classification accuracies of SVM, NB, DTE, and MLP models for various manufacturers and cell formats.

		SVM	NB	DTE	MLP
Cell format	18650	$0.85$	$0.72$	$0.93$	$0.85$
	21700	$0.46$	$0.64$	$0.82$	$0.56$
Manufacturer	KULR	$0.58$	$0.42$	$0.59$	$0.33$
	LG	$0.85$	$0.81$	1	$0.96$
	Molicel	$0.9$	1	1	1
	Saft	0	0	1	0
	Samsung	1	1	1	1
	Sanyo	1	$0.75$	1	1
	Sony	$0.75$	$0.75$	1	1
	Soteria	$0.79$	$0.57$	$0.86$	$0.71$

summarizes the classification accuracies of trained ML models for various cell formats, including 18650 and 21700, and manufacturers including KULR, LG, Molicel, Saft, Samsung, Sanyo, Sony, and Soteria. DTE still performs better than any other models for 18650 and 21700 cells. The prediction accuracy of the 21700 cell is worse than that of the 18650 cell for all models. The main reason is the imbalanced data between different cell formats of the open failure dataset [18]. The dataset ratio between 18650 and 21700 cell formats is 0.77:0.23.

Similarly, DTE shows better accuracy than other models, even for all different manufacturers. We note that the prediction accuracy of Saft, Samsung, Sanyo, and Sony does not heavily affect the overall performance since the number of testing samples for these products is around 3–4. In Table 6, KULR and Soteria batteries show relatively low accuracy for all models. Even though 21,700 cells from both KULR and LG are used for abuse tests, the prediction accuracies are considerably different. While the DTE model perfectly classifies the abuse classes for all 18,650 and 21,700 cells of LG, both cell formats of KULR return a significantly low prediction performance. The description of the KULR battery does not provide any distinct factors compared to other batteries from different manufacturers. On the other hand, one of the specific Soteria 18,650 cells has lower prediction accuracy when it includes the standard current collectors and a Dreamweaver gold separator, which is different from other battery products. Thus, we might need more experimental data and additional features to better predict the abuse classes of KULR and Soteria battery failure.

4.2. Sensitivity and Feature Analysis of DTE

Since the DTE model considerably outperforms other models, we analyze more details of DTE, including the sensitivity of dataset size and feature importance. Even though the relationship between the dataset size and model heavily depends on the specific problems, ML models generally improve their performance with dataset size. However, the experimental battery failure data are costly to obtain in practice, especially for the thermal runaway of the battery. Thus, we conduct a sensitivity analysis to quantify the relationship between dataset size and model performance. Figure 5 presents the classification accuracy of DTE as a function of different ratios $r=0.1,\dots ,1$ of the dataset size with respect to the total number of available samples $N=364$. The error bar shows the standard deviation around the average accuracy. We observe a nearly logarithmic relationship of the average accuracy with the dataset size. The uncertainty shown as the error bar dramatically increases for the small dataset $r\le 0.4$ and stabilizes for the modest dataset $r\ge 0.5$. We could achieve a comparable average accuracy of DTE for the dataset $r\ge 0.8$ even though the uncertainty is still high for the dataset $r=0.8$.

Figure 6 compares the feature rank obtained by feature independence analysis using chi-square tests with the feature importance score of DTE. We first evaluate the independence of each feature variable to the abuse classes by using chi-square tests. Figure 6a shows the feature scores, defined as the negative logarithm of the p-values. The seven most important features of the feature independence analysis are HeatLossRate, NegEnP, CellEn, CellCap, BodyEnP, CorrLossEn, and ConfPos. The ratio of these seven scores over the total score sum is around $0.38$ in Figure 6a. We further analyze the feature importance score of DTE, which is defined as the total changes in the risk due to splits on each feature divided by the number of branch nodes of the decision tree, as shown in Figure 6b [31]. Note that the minimum importance score is 0. The weighted average is then applied to the feature importance score of the overall weak learners of the ensemble. HeatLossRate is the most crucial feature, followed by PreCellM and NegEnP. The ratio of the score sum of the three most important features of HeatLossRate, PreCellM, and NegEnP over the total score sum is $0.44$. We observe that HeatLossRate and NegEnP are still critical factors even for the trained DTE model.

Figure 7 presents three abuse classes, Heat, ISC, and Nail, against the two most critical features of DTE, namely, HeatLossRate and PreCellM, obtained from Figure 6b. The Nail class is relatively easy to separate from other classes of Heat and ISC. The Nail class has the lowest HeatLossRate, while the boundary with other classes increases as the PreCellM value increases. On the other hand, the Heat class shows the highest variation in HeatLossRate, particularly for PreCellM $\simeq 50$. The classification boundaries between Heat and ISC classes are apparent when HeatLossRate is in the range between $0.03$ and $0.08$ for PreCellM$\simeq 50$. Thus, this explains the dominant classification errors between Heat and ISC classes of different models, as shown in the confusion matrix of Figure 3.

Partial dependence represents the relationships between feature variables and predicted scores (posterior probabilities) of all abuse methods obtained by the trained DTE model [31]. Figure 8 shows the estimated partial dependence of DTE for all abuse classes against the most critical features, HeatLossRate. According to DTE, the probability of the Heat class significantly changes around HeatLossRate $\simeq 0.02$, then stays almost flat. The probability of Nail is greater than $0.97$ for HeatLossRate$<0.02$, while it drops fast to almost $0.65$ for HeatLossRate$\ge 0.02$. The probabilities of both Heat and ISC classes are similar, being greater than $0.97$ for $0.02\le \mathtt{HeatLossRate}\le 0.024$.

5. Conclusions

This study presents a thermal runaway diagnosis framework using various ML models where Bayesian optimization techniques are applied to optimize the hyperparameter of these models. Using the experimental data, we use SVM, NB, DTE, and MLP models to estimate a high likelihood of abuse classes of thermal runaway. Open-source battery failure data were used to analyze the diagnosis performance of different abuse types in terms of the classification accuracy, confusion metrics, and ROC curves of ML models. An experimental evaluation shows that the DTE model outperforms other classifiers by around 11.87–26.93% for prediction accuracy. Furthermore, the DTE model provides robust prediction accuracy even with the strictly limited dataset. By analyzing the feature importance, the heat loss rate feature from the experimental measurements is shown to be the most crucial feature in estimating the causes of the thermal runaway for the DTE model.