IIP-Mixer: Intra–Inter-Patch Mixing Architecture for Battery Remaining Useful Life Prediction

Abstract

Accurately estimating the Remaining Useful Life (RUL) of lithium-ion batteries is crucial for maintaining the safe and stable operation of rechargeable battery management systems. However, this task is often challenging due to the complex temporal dynamics. Recently, attention-based networks, such as Transformers and Informer, have been the popular architecture in time series forecasting. Despite their effectiveness, these models with abundant parameters necessitate substantial training time to unravel temporal patterns. To tackle these challenges, we propose a straightforward MLP-Mixer-based architecture named “Intra–Inter Patch Mixer” (IIP-Mixer), which leverages the strengths of multilayer perceptron (MLP) models to capture both local and global temporal patterns in time series data. Specifically, it extracts information using an MLP and performs mixing operations along both intra-patch and inter-patch dimensions for battery RUL prediction. The proposed IIP-Mixer comprises parallel dual-head mixer layers: the intra-patch mixing MLP, capturing local temporal patterns in the short-term period, and the inter-patch mixing MLP, capturing global temporal patterns in the long-term period. Notably, to address the varying importance of features in RUL prediction, we introduce a weighted loss function in the MLP-Mixer-based architecture, marking the first time such an approach has been employed. Our experiments demonstrate that IIP-Mixer achieves competitive performance in battery RUL prediction, outperforming other popular time series frameworks, such as Informer and DLinear, with relative reductions in mean absolute error (MAE) of 24% and 10%, respectively.

1. Introduction

Lithium-ion batteries are widely used in electric vehicles, unmanned aerial vehicles, and grid energy storage systems. Accurate prediction of the remaining useful life (RUL) of a battery is crucial for effective battery management and health estimation [1,2]. This task is challenging due to nonlinear degradation mechanisms caused by cycling and varied operational conditions. Typically, the battery RUL is defined as the number of cycles when a battery reaches 80% of its initial capacity. Previous works on battery RUL prediction can be categorized into two main approaches: physics-based approaches and data-driven approaches [3].

Physics-based approaches such as the single-particle model [4] and the pseudo-two-dimensional model [5] based on the electrochemical principles underlying lithium-ion batteries (LIBs) can simulate the current and voltage characteristics of a battery from kinetics and transport equations. Although these models are usually precise and understandable, they come with significant computational demands and difficulties when extending their applicability to different battery cell types [6].

In contrast, data-driven approaches do not make a priori assumptions about battery degradation mechanisms but instead leverage historical cycling data of batteries. Data-driven approaches, such as Dynamic Programming Methods (DPMs) [7], are efficient and adept at handling nonlinearities, but can be complex to implement and resource intensive for large-scale problems. Gaussian processes [8,9] offer flexible, probabilistic modeling and uncertainty quantification; however, they face challenges with scalability and are computationally expensive for large datasets. Other methods incorporate intricate structures such as gating mechanisms [6,10,11] and attention mechanisms [12,13,14]. Although these approaches are effective, they often require substantial training time to thoroughly explore temporal and intercorrelation patterns, which demands extensive computing resources [15]. In addition, the permutation-invariant self-attention mechanism leads to a loss of temporal information to some extent [16].

To tackle these challenges, we propose a novel MLP-Mixer-based architecture named “Intra–Inter-Patch Mixer” (IIP-Mixer). Unlike traditional neural network-based models [17], IIP-Mixer uses multilayer perceptrons (MLPs) exclusively, which simplifies the architecture while maintaining high performance, relying solely on basic matrix multiplication routines, changes to data layout, and scalar nonlinearities [18]. The key contributions of our work are as follows:

• Novel architecture: The IIP-Mixer architecture introduces parallel dual-head mixer layers: the intra-patch mixing MLP, which captures local temporal patterns in the short-term period, and the inter-patch mixing MLP, which captures global temporal patterns in the long-term period. This parallel dual-head approach allows the model to effectively learn from both short-term and long-term data patterns.
• Weighted loss function: To address the varying importance of features in RUL prediction, we introduce a weighted loss function in the MLP-Mixer-based architecture. This innovation marks the first time such an approach has been employed in this context, enhancing the model’s ability to prioritize more critical features during training.
• Performance and efficiency: Our experiments demonstrate that the IIP-Mixer achieves competitive performance in battery RUL prediction, outperforming other popular time series frameworks. Furthermore, the simplicity of the MLP-based design results in faster training times compared to attention-based models like Transformers, making it a more efficient solution for real-time applications.

2. Related Work

Remaining useful life (RUL) estimation forecasts when a battery will become ineffective, thereby minimizing risks by evaluating cell condition. This is essential for planning proper maintenance and reducing mishap risks. However, RUL estimation accuracy is often hindered by insufficient data, model complexity, and system constraints. To improve prediction accuracy, various approaches are utilized [19].

2.1. Physics-Based Approaches

Physics-based approaches examine the chemical and physical phenomena affecting battery performance, providing detailed evaluations [20]. The type of electrode material significantly influences the accuracy and effectiveness of RUL prediction methods. Graphite and lithium cobalt oxide (LCO) electrodes [21], known for their relatively predictable degradation patterns, are well suited for physics-based models. On the other hand, tin-based anodes [22] and composite materials exhibit more complex degradation behaviors and require more sophisticated modeling approaches to capture the intricacies of their performance. The intricate performance of these advanced materials over time highlights their suitability for more sophisticated predictive approaches. Physics-based approaches can be categorized into semi-empirical and empirical methods.

• Semi-empirical methods map degradation parameters to key battery metrics like static capacity and impedance under different loads and environmental conditions through offline testing. Sankararaman et al. [23] extended the first-order second-moment method (FOSM) and first-order reliability method (FORM) for use with state space models, creating a computational framework to determine the entire probability distribution of RUL predictions. Coulombic efficiency, a critical battery parameter, is closely linked to lithium inventory loss, the main aging factor for lithium-ion batteries. Leveraging this relationship, Yang et al. [24] developed a semi-empirical model to capture capacity degradation.
• Empirical methods continuously update their model parameters through online measurement and estimation of battery states. Given the nonlinear and non-Gaussian capacity degradation characteristics of lithium-ion batteries (LIBs), Zhang et al. [25] proposed an RUL prediction method using an exponential model and particle filter. In real environments, the discharge current of LIBs changes randomly within a charge–discharge cycle, significantly affecting battery life. Shen et al. [26] developed a two-stage Wiener process model, incorporating the unscented particle filtering algorithm, to predict the RUL considering variable discharge currents.

While physics-based methods have made significant strides in mechanism analysis, constructing a perfect battery simulation model remains challenging due to the complex interactions between components and the inherently active and nonlinear nature of battery degradation. Real-time observation of a battery’s internal conditions is difficult, complicating the development of an exact physical model [27].

2.2. Data-Driven Approaches

A data-driven approach uses operational data from advanced sensor technology to construct a degradation model for predicting the RUL. Instead of relying on accurate descriptions of the system mechanism, it utilizes data to capture the inherent degradation relationship and trend [27]. Various methods have been proposed for this purpose [28,29].

• MLP-based architecture: A multilayer perceptron (MLP) is a type of artificial neural network with a forward structure, consisting of an input layer, an output layer, and several hidden layers. Kim et al. [30] proposed a practical State of Health (SOH) classification scheme based on an MLP, performing classification using only data from discrete life spans. Das et al. [31] introduced a multilayer perceptron-based encoder–decoder model, the Time Series Dense Encoder (TiDE), for long-term time series forecasting. This model combined the simplicity and speed of linear models with the ability to handle covariates and nonlinear dependencies.
• RNN-based architecture: Recurrent neural networks (RNNs) are designed for processing sequential data. In [32], differential thermal voltammetry (DTV) signal analysis methods are combined with RNN-based data-driven methods to track battery degradation. De Brouwer et al. [33] introduced a continuous-time version of the Gated Recurrent Unit (GRU), built upon Ordinary Differential Equations, along with a Bayesian update network for processing sporadically observed time series.
• Transformer-based architecture: The Transformer employs an attention network to precisely capture sequential information, thereby enhancing neural network training performance. Chen et al. [34] introduced a model based on the Transformer to tackle challenges in capturing long-term dependencies and complex degradation patterns. This model integrated a position-sensitive self-attention (PSA) unit, which improved the model’s ability to incorporate local context by focusing on positional relationships in input data at each time step. To handle noisy battery capacity data, particularly during charge/discharge cycles, Chen et al. [13] developed a Transformer-based neural network using a Denoising Auto-Encoder (DAE) to learn from corrupted input and reconstruct data, effectively processing noisy battery capacity data.

Data-driven models can adjust to diverse battery types and operational conditions without requiring extensive knowledge of underlying physical processes. However, their adaptability and robustness remain persistent challenges in practical applications. Additionally, concerns often arise regarding the sensitivity of parameter settings, which can affect model performance. The accuracy of data-driven models is heavily contingent on the quality, quantity, and representativeness of the training data; inadequate data quality may result in less reliable predictions.

3. Methods

Building on the insights from previous works, we propose a novel architecture for RUL prediction that leverages the strengths of MLPs while addressing their typical shortcomings.

3.1. Problem Formulation

For systems like lithium-ion batteries, the degradation process often spans numerous cycles, with various sensor data collected during each cycle. We address the battery RUL prediction problem as a multivariate time series prediction task. Specifically, given the historical observations $\mathit{X}\in {\mathbb{R}}^{C\times L}$, where L is the length of the look-back window, C is the number of variables, we consider the task of predicting $\mathit{Y}\in {\mathbb{R}}^{C\times N}$, where N is the number of subsequent time steps [35]. In this work, we focus on the case when the values of the target time series are equal to the historical observation.

Problem definition:

• Input data: the input data consist of multiple time series variables $\mathit{X}$ collected over a series of cycles, representing various sensor measurements, such as voltage, current, and capacity.
• Output data: the output is the predicted future values $\mathit{Y}$, from which the remaining useful life of the battery can be determined.
• Objective: the goal is to develop a model that can accurately predict the future values $\mathit{Y}$ based on the historical data $\mathit{X}$.

3.2. The Framework

Inspired by the recent MLP architecture MLP-Mixer [18], which demonstrates that simple univariate linear models can outperform deep learning models like CNN and Transformer on several commonly used academic benchmarks, we extend this approach. In this paper, we explore the capabilities of MLP-based models for time series forecasting, specifically tailored to the context of long-term time series forecasting for battery RUL prediction. We present IIP-Mixer, an architecture based exclusively on MLPs. The idea behind the IIP-Mixer architecture is to separate the intra-patch mixing and inter-patch mixing to capture local temporal patterns and global temporal patterns simultaneously. Both operations are implemented with MLPs. The central architecture of the IIP-Mixer is an MLP-based design, which aggregates information from both intra-patch and inter-patch within the input series. It is worth noting that different features have different patch mixers. For univariate time series, as illustrated in Figure 1, this framework contains five major components.

3.2.1. Major Components of the Framework

Input series transformation: From the perspective of channel independence, the multivariate time series $\mathit{X}\in {\mathbb{R}}^{C\times L}$ are divided into C univariate series ${\mathit{X}}^{\left(i\right)}\in {\mathbb{R}}^L,$$i=1,2,\cdots C$. These univariate series are independently fed into the IIP-Mixer model. This approach breaks down these input univariate series into smaller and structured patches. It transforms each original univariate time series from a 1D series to 2D patches while preserving their original relative positions, expressed as follows:

$${\mathit{X}}^{\left(i\right)}\in {\mathbb{R}}^L\to {\mathit{X}}^{\left(i\right)}\in {\mathbb{R}}^{H\times W}$$

where W represents the length of the patch, and H denotes the number of patches for each univariate series, which is a tuning hyperparameter.

Intra-patch mixing MLP: The rows in the 2D input univariate series ${\mathit{X}}^{\left(i\right)}$ represent distinct patches, while the columns denote time steps. A trainable intra-patch mixing MLP shared across all patches is employed to map each patch to a hidden space. We utilize a multilayer perceptron with a single hidden layer to capture local temporal patterns within patches. The size of output ${\mathit{O}}_{\mathrm{intra}}^{\left(i\right)}$ is the same as that of input ${\mathit{X}}^{\left(i\right)}$, and the process can be summarized as the following equations [18]:

$${\mathit{O}}_{\mathrm{intra}}^{\left(i\right)}={\mathit{W}}_2\sigma \left({\mathit{W}}_1{\mathit{X}}^{\left(i\right)}\right)$$

(1)

It is essential to highlight that the parameters of the intra-patch mixing MLP shared across all patches prevent the architecture from growing too fast when increasing the length L of the look-back window and lead to significant memory savings.

Inter-patch mixing MLP: The 2D patch ${\mathit{X}}^{{\left(i\right)}^T}$ is a transposition of ${\mathit{X}}^{\left(i\right)}$, the rows of which represent time steps, while the columns denote distinct patches. A trainable inter-patch mixing MLP shared across all of the time steps is employed to map each time step to a hidden space. Similar to the intra-patch mixing MLP, the inter-patch mixing MLP utilizes a multilayer perceptron with a single hidden layer to capture global temporal patterns across patches. The size of output ${\mathit{O}}_{\mathrm{inter}}^{\left(i\right)}$ is the same as that of input ${\mathit{X}}^{{\left(i\right)}^T}$, and the process can be summarized as the following equations [18]:

$${\mathit{O}}_{\mathrm{inter}}^{\left(i\right)}={\mathit{W}}_4\sigma \left({\mathit{W}}_3{\mathit{X}}^{{\left(i\right)}^T}\right)$$

(2)

Linear projection: A linear projection is a trainable linear neural network that projects aggregated output information from the intra-patch mixing MLP and inter-patch mixing MLP to predict future time steps. Additionally, the input of the linear projection incorporates the skip connections of ${\mathit{X}}^{\left(i\right)}$. This residual design ensures that IIP-Mixer retains the capacity of temporal linear models while still being able to exploit intra–inter-patch information. The linear projection takes an input flattened from 2D patches to 1D time series and predicts long time series sequences in a single forward operation [36], significantly enhancing inference speed. The process can be summarized by the following equations:

$${\widehat{\mathit{Y}}}^{\left(i\right)}=W_5\mathrm{flatten}\left({\mathit{O}}_{\mathrm{intra}}^{\left(i\right)}+{\mathit{O}}_{\mathrm{inter}}^{\left(i\right)}{ }^T+{\mathit{X}}^{\left(i\right)}\right)$$

(3)

Weighted loss function: Considering the varying importance of different variables in battery RUL prediction, to further enhance prediction performance, different from the loss function of traditional multivariate time series prediction [36], we propose to use the weighted mean square error as our loss function, which can be rewritten as:

$$\mathrm{WMSELoss}=\sum _{i=1}^C{\alpha }_i\left({\displaystyle \frac{1}{N}}\sum _{j=1}^N{\left({\widehat{\mathit{Y}}}_j^{\left(i\right)}-{\mathit{Y}}_j^{\left(i\right)}\right)}^2\right)$$

(4)

where ${\alpha }_i$ represents the weight assigned to the ith variable, which is derived from the random forest feature importance measure. This measure computes the importance of a feature as the normalized total reduction in the criterion (e.g., Gini impurity for classification and variance for regression) caused by that feature. It indicates the significance of each variable in predicting the remaining useful life (RUL) of the battery.

In addition, the criterion is a function that is used to measure the quality of a split. In regression tasks, feature importance in a random forest is typically measured using the reduction in variance that each feature contributes to the splits in the trees. Each time a feature is used to split the data, the reduction in variance resulting from the split is calculated. The variance reduction can be calculated as follows:

$$\mathrm{Variance}\mathrm{Reduction}=\mathrm{Var}\left(T\right)-\left({\displaystyle \frac{N_L}{N}}\mathrm{Var}\left(T_L\right)+{\displaystyle \frac{N_R}{N}}\mathrm{Var}\left(T_R\right)\right)$$

(5)

where $\mathrm{Var}\left(T\right)$ is the variance of the target feature in the parent node. $\mathrm{Var}\left(T_L\right)$ and $\mathrm{Var}\left(T_R\right)$, are the variances of the target feature in the left and right child nodes, respectively, N is the number of samples in the parent node, and $N_L$ and $N_R$ are the numbers of samples in the left and right child nodes, respectively.

3.2.2. Data Processing Procedure

As shown in Figure 2, the time series within the look-back window, which contains historical observation data, is first divided into multiple patches. For example, a time series with 16 time steps can be divided into four patches, each containing data from 4 sequential time steps. As shown in Figure 1, the 4 different colored rectangles represent these 4 patches.

Next, we transform the 1D time series patches into 2D patches, where each row represents a distinct patch. Simultaneously, we create a transposed version of these 2D patches, with rows now representing time steps.

We then use the intra-patch MLP and inter-patch MLP to capture local temporal patterns and global temporal patterns from the 2D patches and their transposed counterparts, respectively. Following this, we aggregate the local and global temporal pattern information through an addition operation on the 2D patches.

Finally, we flatten the aggregated 2D patches back into a 1D time series and generate the forecasting output, such as the 4 sequential time steps after the look-back window, using a linear projection.

3.3. The Multivariate Time Series

In predicting the remaining useful life of rechargeable lithium-ion batteries, a lot of prior research uses only the capacity feature and barely considers more features of the battery in the charge–discharge cycles [6,13]. However, the remaining useful life (RUL) of batteries can vary significantly even among the same types of battery when subjected to different charge–discharge conditions, due to variations in temperature, voltage, and current. Consequently, predicting the RUL based solely on univariate time series data may diminish the generalization ability of the prediction model. To enhance the model’s generalization capability, it is essential to consider various load conditions by incorporating load-specific variables, such as charge voltage and charge current, into the prediction model.

To address these issues mentioned above, we propose a multivariate input representation that includes the features of charge and discharge cycles of rechargeable batteries. Scilicet, we predict the RUL of rechargeable batteries from a multivariate time series sequence. It is worth noting that feature generation and selection are crucial to the prediction performance of our proposed approach. To better capture the evolving trends in time series data, we generate a feature by calculating the mean of the accumulated capacity for each discharge cycle. Moreover, the presence of noise and redundant information in the raw measurements can impede model convergence. To address this, we introduce the random forest feature importance measure [37] to identify and incorporate the most important features, thereby improving both the convergence speed and accuracy of the model.

3.4. Dual-Head MLP

Motivated by PatchMixer [36], we employ the dual forecasting heads design in our IIP-Mixer model, including one intra-patch mixing MLP head and one inter-patch mixing MLP head.

As shown in Figure 1, the intra-patch mixing MLP head comprises two fully connected layers and a GeLU nonlinearity, facilitating communication between time steps within a patch and capturing short-term temporal dependencies. The parameters within the intra-patch mixing MLP act as the short-term memory of IIP-Mixer, emphasizing the learning of information among local time steps without considering the entire input sequence.

The structure of the inter-patch mixing MLP head is the same as that of the intra-patch mixing MLP head, allows communication between different patches, and captures temporal dependencies across the whole input sequence. It consistently processes the same time step of each patch independently, extracting long-term temporal dependencies within the entire input sequence.

It is essential to note that unlike PatchMixer, where serial dual heads are employed, our approach features parallel dual forecasting heads. As shown in Figure 2, the forecasting procedure simultaneously incorporates output information from the dual-head MLP and residual connections from past sequences to model future sequences using a linear projection. The training procedure of IIP-Mixer is summarized in Algorithm 1.

Algorithm 1: IIP-Mixer: PyTorch-like pseudocode.

$\#\mathrm{x}$: segments of the univariate time series $\#{\mathrm{x}}^{\top }:$ the transposition of input x $\#$params: parameters of the network: mixer${ }_{1:N}$ + fc for x in loader:                                     # load a minibatch       $\mathrm{x}=\mathrm{tf}\left(\mathrm{x}\right)$                                        # transform $\mathrm{x}$ from 1D to 2D       for i in range (N)                           # loop number of mixer layer            x = mixer ${ }_i\left(\mathrm{x}\right)$                         # the i-th mixer layer       $\mathrm{x}=$ ft(x)                                         # flatten $\mathrm{x}$ from 2D to 1D       pred $=\mathrm{fc}\left(\mathrm{x}\right)$                                 # full connection layer       loss = WMSELoss ( pred, true )  # weighted MSE loss function       loss.backward()                            # back-propagate       update(params)                           # SGD update # mixer layers def mixer (x):       ${\mathrm{O}}_{\mathrm{intra}}={\mathrm{mlp}}_{\mathrm{intra}}\left(\mathrm{x}\right)$                # intra-patch mixing MLP       ${\mathrm{O}}_{\mathrm{inter}}={\mathrm{mlp}}_{\mathrm{inter}}\left({\mathrm{x}}^{\top }\right)$             # inter-patch mixing MLP       $\mathrm{x}={\mathrm{O}}_{\mathrm{intra}}+{\mathrm{O}}_{\mathrm{inter}}^{\top }+\mathrm{x}$            # aggregate output information       return x

3.5. Differences between IIP-Mixer and MLP-Mixer

Different from MLP-Mixer, we made the following three modifications to enhance the model’s capability in predicting the remaining useful life of batteries:

• First, in IIP-Mixer, unlike MLP-Mixer’s patches and features, the two dimensions represent patches and time steps, allowing us to capture both local and global temporal patterns simultaneously. Particularly, the global temporal patterns captured from time steps are crucial for battery RUL prediction, as demonstrated in Table 5.
• Second, unlike MLP-Mixer’s serial structure of dual-head MLP, in IIP-Mixer, we introduce a parallel structure for inter-patch mixing and intra-patch mixing to capture temporal patterns simultaneously. As shown in Table 6, the parallel heads’ structure clearly outperforms the serial structure in battery RUL prediction.
• Consequently, unlike MLP-Mixer’s traditional loss function, we propose a weighted mean square error loss function that accounts for the varying importance of features in battery RUL prediction. This approach results in a 5% relative improvement in MAE.

4. Experiments

To evaluate the effectiveness of our proposed method, we conducted experiments using a well-established dataset and compared our model’s performance against several baselines.

4.1. Datasets

We conducted our experiments using PyTorch on the public NASA PCoE battery dataset, available from the NASA Ames Research Center website. This dataset includes four Li-ion batteries (B0005, B0006, B0007, and B0018) that were subjected to three different operational profiles (charge, discharge, and impedance) at room temperature. Charging was performed in a constant current (CC) mode at 1.5 A until the battery voltage reached 4.2 V, followed by a constant voltage (CV) mode until the charge current dropped to 20 mA. Discharging was conducted at a constant current (CC) level of 2 A until the battery voltage fell to 2.7 V, 2.5 V, 2.2 V, and 2.5 V for batteries B0005, B0006, B0007, and B0018, respectively. Repeated charge and discharge cycles accelerated the aging of the batteries. The experiments were halted when the batteries met the end-of-life (EOL) criteria, defined as a 30% reduction in rated capacity (from 2 Ahr to 1.4 Ahr). We uniformly collected data only before their discharge capacity decayed to 0.55 Ah, at which point they had cycled 167, 167, 167, and 131 times, respectively.

This dataset can be used to predict the remaining useful life (RUL) of batteries. To ensure the robustness of the model, the dataset was divided into training, validation, and test sets. The training set, comprising data from B0005 and B0006, was used to fit the model. The validation set, consisting of data from B0007, was used to tune the hyperparameters. Finally, the test set, containing data from B0018, was used to evaluate the model’s performance.

4.2. Baselines

We benchmarked our IIP-Mixer models against commonly used basic networks, including MLP, Transformer models, and their newly proposed variants such as DLinear [16] and Informer [38].

• MLP: A multilayer perceptron is like a mathematical function that maps input and output values. Multiple layers are used to learn the battery’s dynamic and nonlinear degradation trend.
• Transformer: Transformer is a model that uses an attention mechanism for model training; it mainly consists of two components: an encoder and a decoder, with which we can predict the capacity degradation trend of the battery.
• Informer: A variant of the Transformer architecture that efficiently handles extremely long input sequences by highlighting the dominating attention by halving the cascading layer input. It predicts the long time series sequences in one forward operation rather than in a step-by-step way, which drastically improves the inference speed.
• DLinear: In consideration of permutation-invariant and “anti-ordering” to some extent of the Transformer-based architecture [16], DLinear decomposes the time series into a trend and a remainder series and employs two one-layer linear networks to extract the temporal relations among an ordering set of continuous points.

4.3. Implementation

4.3.1. Parameter Selection and Optimization

• MLP: The MLP model was configured with one hidden layer. The optimal values for the learning rate and hidden dimension were determined through a grid search, resulting in 0.0005 and 32, respectively.
• Transformer and Informer models utilized two encoder layers and one decoder layer, each with 8 attention heads. The optimal values for the learning rate, hidden dimension, and dropout rate, identified through a grid search, were 0.0001, 512, and 0.05, respectively.
• For the DLinear model, we adopted the architecture parameters proposed in the original paper. The best performance was achieved with a decomposition kernel size of 25 and a learning rate of 0.0015, as determined through a grid search.
• IIP-Mixer: Our model includes six key parameters: patch size, learning rate, dropout rate, patch length, number of mixer blocks, and number of principal features of the time series. The optimal values, determined through a grid search, were 4, 0.001, 0.05, 4, 1, and 6, respectively. Detailed information about the training process is provided in

  Table 1. Hyper-parameter summary.

  Hyper-Parameter	Range of Values
  Patch size	{2, 4, 8}
  Learning rate	{0.0001, 0.0005, 0.001}
  Dropout	{0.05, 0.1, 0.2}
  Length of patch	{2, 4, 8}
  # of mixer blocks	{1, 2, 3, 4}
  # of principal features	{1,2,4,6,8,10,12,14,16}

  .

In short, we adopted the default architecture of Transformer, Informer, and DLinear proposed in [16]. For simplicity, a single hidden layer was used in both the MLP and IIP-Mixer models. To ensure a fair comparison, we standardized the input time step length to 16 and the output time step length to 4 across all models, following the methodology outlined in [13]. Additionally, to ensure the robustness of the results, each model’s hyperparameters were fine-tuned through a comprehensive grid search, and the models were evaluated using the validation set to mitigate overfitting and provide a reliable performance estimate.

4.3.2. Evaluation Metrics

In our experiments, four evaluation metrics were used to compare the performance of these models mentioned above. In addition to three common metrics, the Mean Absolute Error (MAE), Root-Mean-Square Error (RMSE), and Mean Absolute Percentage Error(MAPE) [39], we chose the Absolute Relative Error (ARE) [13] to evaluate the prediction performance of the battery RUL, which is defined as follow:

$$ARE=\frac{\left|RU{L}^{pre}-RU{L}^{ture}\right|}{RU{L}^{ture}}\times 100\%$$

(6)

where $RU{L}^{ture}$ denotes the true RUL of the battery, and $RU{L}^{pre}$ denotes the prediction of the RUL from models. Additionally, we conducted each experiment three times using three consecutive seeds and reported the mean of the evaluation metrics.

5. Results

5.1. Performance of IIP-Mixer

Despite its simplicity, IIP-Mixer achieved competitive results. To verify the performance of our methods, we conducted our experiments on four baseline methods. The MAE, RMSE, MAPE, and ARE scores obtained for all methods are shown in

Table 2. Performance of methods on the NASA PCoE battery dataset.

Methods	MAE (Ah)	RMSE (Ah)	MAPE (%)	ARE (%)
Transformer	0.055	0.073	3.697	9.589
Informer	0.049	0.063	3.281	4.110
MLP	0.050	0.066	3.402	6.393
DLinear	0.041	0.052	2.732	3.196
IIP-Mixer	0.037	0.048	2.480	1.370

, the best results are shown in bold. All experiments in this subsection were conducted on the same machine, utilizing a single GPU (NVIDIA Tesla T4) for consistent and reliable findings. From the result, we conclude the following:

• The Transformer and its variant, the Informer model, excel in modeling both long-term and short-term dependencies, showcasing superior performance on long-time series data. However, it is worth noting that they can easily lead to overfitting for small datasets, such as the NASA PCoE battery dataset.
• The MPL model is adept at capturing global temporal patterns but may struggle to capture local temporal patterns from time series data. Consequently, its performance remained average across all evaluation metrics compared to other methods.
• The DLinear model decomposes the time series into a trend and a remainder series. It utilizes two one-layer linear networks to extract temporal relations among an ordered set of continuous points, making it adept at capturing both the trend and season of a time series.
• Among all the baseline methods, our proposed model IIP-Mixer achieved the best experimental results. As shown in Figure 3, IIP-Mixer could capture the local and global temporal patterns in time series data; this is a great help for battery RUL prediction, especially in the overall degeneration trend of discharge capacity.

Table 3. Evaluation results on the NASA PCoE battery dataset.

Methods	MAE(Ah)	RMSE(Ah)	ARE
LSTM [42]	0.083	0.091	0.265
Dual-LSTM [41]	0.082	0.088	0.256
GRU [40]	0.081	0.092	0.304
DeTransformer [13]	0.071	0.080	0.225
IIP-Mixer (ours)	0.037	0.048	0.014

shows our model outperformed all baseline methods significantly; we achieved the best performance across all the metrics on the NASA PCoE battery dataset. IIP-Mixer demonstrated an overall relative reduction of 40% in the RMSE, 48% in the MAE, and 94% in the ARE in comparison to the state-of-the-art DeTransformer. From the result, we conclude the following:

• LSTM and its variants, the GRU [40] and Dual-LSTM [41] models, are types of neural networks designed for processing sequential data. These models can handle examples of varying lengths by sharing parameters across different parts of the network. However, recurrent neural networks, including LSTM networks, can experience performance degradation due to long-term dependencies, which impacts their effectiveness in battery RUL prediction.
• DeTransformer [13], a variant of Transformer, leverages the strength of the self-attention mechanism, excelling at extracting semantic correlations between paired elements in a long sequence, regardless of their order, thanks to its permutation-invariant nature. However, in time series analysis, the primary focus is on modeling the temporal dynamics among a continuous set of points, where the order of the data points is crucial [16]. This affects the effectiveness of Transformers in predicting battery RUL.
• IIP-Mixer, a straightforward MLP-Mixer-based architecture, is well suited for learning temporal patterns due to its time-step-dependent characteristics, as demonstrated by recent work [35]. Our results also highlight the importance of efficiently utilizing MLPs to capture both local and global temporal information, thereby improving the performance of time series forecasting. In summary, this approach achieved superior performance compared to RNN-based and Transformer-based architectures.

5.2. Computation Efficiency

We investigated the computational efficiency of all neural networks mentioned above based on the length L of the input time series. Comparisons of theoretical time complexity and memory usage [38] are summarized in

Table 4. L-related computation statistics of each layer.

Methods	Training		Testing
	Time	Memory	Step
Transformer	$\mathcal{O}\left(L^2\right)$	$\mathcal{O}\left(L^2\right)$	L
Informer	$\mathcal{O}\left(L\log L\right)$	$\mathcal{O}\left(L\log L\right)$	1
MLP	$\mathcal{O}\left(L^2\right)$	$\mathcal{O}\left(L^2\right)$	1
DLinear	$\mathcal{O}\left(L\right)$	$\mathcal{O}\left(L\right)$	1
IIP-Mixer	$\mathcal{O}\left(L\right)$	$\mathcal{O}\left(L\right)$	1

. It is important to note that, in our model, the patch size W can equal the patch number H, so we have $W\times H=W^2=H^2=L$. It is evident that our model, like DLinear, had the lowest cost across all computational metrics.

5.3. Ablation Study

5.3.1. Effect of Dual-Head MLP

To evaluate the effectiveness of the dual-head MLP, we individually removed the intra-patch mixing MLP head or inter-patch mixing MLP head.

Table 5. Ablation study of dual-head MLP.

Methods	MAE (Ah)	RMSE (Ah)	MAPE (%)	ARE (%)
w/o inter	0.081	0.095	5.443	34.703
w/o intra	0.044	0.055	2.940	5.023
IIP-Mixer	0.037	0.048	2.480	1.370

demonstrates that the dual-head MLP mechanism outperformed all other configurations. This result highlights the effectiveness of the dual-head mechanism in comparison to a single output head.

It is worth noting that as shown in Figure 4, the performance of the architecture without an intra-patch mixing MLP head was better than that without an inter-patch mixing MLP head. Specifically, the intra-patch mixing MLP conducts per-location (intra-patch) operations, capturing local temporal patterns from a single patch and identifying the short-term seasonal component of the time series. Conversely, the inter-patch mixing MLP performs cross-location (inter-patch) operations, capturing global temporal patterns from the same time step across several patches, thereby identifying the long-term trend component of the time series. In the context of battery RUL prediction, the long-term trend component contributes significantly more to predicting battery life than the seasonal component. As shown in Table 5, the inter-patch mixing MLP achieved a lower ARE of 5.02% compared to the intra-patch mixing MLP, which had an ARE of 34.70%. Naturally, the best performance was achieved with the dual-head MLP with an ARE of 1.37%.

5.3.2. Serial vs. Parallel Heads

Unlike the serial structure of the dual-head MLP in MLP-Mixer and PatchMixer, we introduced a parallel structure for dual heads. To evaluate the effectiveness of the parallel heads structure, we compared the performance of serial heads (intra-first), serial heads (inter-first), and parallel heads. As shown in

Table 6. Comparison of serial and parallel heads.

Methods	MAE (Ah)	RMSE (Ah)	MAPE (%)	ARE (%)
Serial heads (inter-first)	0.063	0.076	4.258	15.068
Serial heads (intra-first)	0.045	0.056	3.032	4.110
Parallel heads	0.037	0.048	2.480	1.370

, it is evident that the parallel heads’ structure outperformed the serial structure in battery RUL prediction.

5.3.3. Effect of Weighted Loss Function

In the majority of recent research on MLP-based models, the loss functions did not take into account the varying importance of different variables. As shown in

Table 7. Comparison of loss functions with weighting vs. w/o weighting.

Methods	MAE (Ah)	RMSE (Ah)	MAPE (%)	ARE (%)
w/o weighting	0.039	0.049	2.562	1.370
With weighting	0.037	0.048	2.480	1.370

, it is evident that the weighted loss function outperformed the loss function without weighting, resulting in relative improvements of 5%, 2%, and 3% in Mean Absolute Error (MAE), Root-Mean-Square Error (RMSE), and Mean Absolute Percentage Error (MAPE), respectively.

5.3.4. Effect of Multivariate Time Series

We investigated the impact of multivariate time series in

Table 8. Comparison of time series: univariate vs. multivariate.

Methods	MAE (Ah)	RMSE (Ah)	MAPE (%)	ARE (%)
Univariate	0.042	0.054	2.822	3.653
Multivariate (full)	0.052	0.064	3.510	7.306
Multivariate (principal)	0.037	0.048	2.480	1.370

. The learning performance with principal-feature multivariate time series surpassed that of univariate time series. Specifically, predicting the remaining useful life of batteries using a multivariate time series of principal features significantly enhanced the generalization ability of the prediction model.

It is important to highlight that training with multivariate time series containing all features may lead to a reduction in model performance. Therefore, the feature selection pipeline plays a critical role in determining prediction performance.

6. Conclusions

In this paper, we presented IIP-Mixer, an innovative MLP-Mixer-based architecture designed to predict the remaining useful life in batteries. IIP-Mixer incorporates a parallel dual-head MLP: the intra-patch mixing MLP and inter-patch mixing MLP. The intra-patch mixing MLP independently applies an MLP to each patch, capturing local temporal patterns in the short-term period. On the other hand, the inter-patch mixing MLP applies an MLP across all patches from the input sequence, capturing global temporal patterns in the long-term period. Moreover, recognizing the varying importance of features in RUL prediction, we proposed a weighted mean square error loss function to enhance prediction accuracy, which resulted in a 5% relative improvement in MAE. Our experiments demonstrated that IIP-Mixer achieved competitive performance in battery RUL prediction, outperforming other popular time series frameworks, such as Informer and DLinear, with relative improvements in MAE of 24% and 10%, respectively.