Extending the BESS Lifetime: A Cooperative Multi-Agent Deep Q Network Framework for a Parallel-Series Connected Battery Pack

Abstract

In this paper, we propose a battery management algorithm to maximize the lifetime of a parallel-series connected battery pack with heterogeneous states of health in a battery energy storage system. The growth of retired lithium-ion batteries from electric vehicles increases the applications for battery energy storage systems, which typically group multiple individual batteries with heterogeneous states of health in parallel and series to achieve the required voltage and capacity. However, previous work has primarily focused on either parallel or series connections of batteries due to the complexity of managing diverse battery states, such as state of charge and state of health. To address the scheduling in parallel-series connections, we propose a cooperative multi-agent deep Q network framework that leverages multi-agent deep reinforcement learning to observe multiple states within the battery energy storage system and optimize the scheduling of cells and modules in a parallel-series connected battery pack. Our approach not only balances the states of health across the cells and modules but also enhances the overall lifetime of the battery pack. Through simulation, we demonstrate that our algorithm extends the battery pack’s lifetime by up to 16.27% compared to previous work and exhibits robustness in adapting to various power demand conditions.

1. Introduction

Lithium-ion batteries used in electric vehicles (EVs) have a finite lifetime, and for safety reasons must be replaced when their capacity drops to 80% or below [1]. However, these batteries can still be repurposed for other applications, where the remaining capacity is sufficient for less demanding tasks. Figure 1 illustrates the life cycle of lithium-ion batteries, where a battery energy storage system (BESS) can effectively utilize retired batteries when their state of health (SOH) is between 80% and 60% [2,3]. The BESS comprises one or more battery packs, each of which uses a group of battery cells connected in parallel and in series [4], to store electrical energy as backup power for households, data centers, charging stations, etc. [5]. Battery cells in a battery pack can be connected in one of two architectures shown in Figure 2: (a) a module groups the batteries in series, and then the modules are connected in parallel (denoted S-P), which is useful for high voltage applications; and (b) a module groups batteries in parallel, and then the modules are connected in series (denoted P-S) for applications requiring high capacity. In the past few years, many projects around the world have implemented a BESS by repurposing EV batteries. In the Netherlands, a 2.8 MWh BESS was installed for Johan Cruijff Arena in 2018 by reusing Nissan LEAF battery packs, each consisting of 192 cells, in an S-P connection [6]. In Finland, a 2.6 MWh BESS was built in 2021 as a backup power resource for the power grid by repurposing Tesla Model S battery packs, each consisting of 7104 cells, in a P-S connection [7]. Table 1 summarizes recent projects that repurposed EV batteries for a BESS. By leveraging a BESS, the demand for new batteries can be significantly reduced, thereby lessening the environmental impact associated with battery production. However, a BESS that reuses retired lithium-ion batteries from EVs still has some limitations when implemented.

One significant challenge in implementing a BESS is the varying capacity levels of the repurposed batteries connected in parallel and in series, indicating heterogeneous SOHs. The cells within a retired battery pack that exhibits heterogeneous SOHs can result in capacity and voltage imbalances, leading to inefficient energy storage and distribution [8]. This discrepancy in capacity can cause weaker cells to discharge faster or to overheat, potentially shortening the lifetime of the entire battery pack and posing safety risks. Furthermore, the varying SOHs among the cells complicates the task of maintaining balanced charging or discharging across the battery pack, making it difficult to achieve an optimal performance and lifetime [9]. As a result, the cells and modules need to be appropriately connected or bypassed (known as scheduling) to mitigate these issues and ensure the reliable operation of a BESS that reuses EV batteries.

Table 1. Projects around the world that reused batteries from EVs.

Name’s Project	Applications	Capacity	EV Model	Battery Pack Configuration
Johan Cruijff Arena (Netherlands) [6]	PV power supply, emergency supply	2.8 MWh	590 Nissan LEAF battery packs	96S-2P 1 (192 cells)
Former coal-fired power plant in Elverlingsen (Germany) [7]	Energy storage system for households	3.0 MWh	72 Renault Zoe battery packs	96S-2P (192 cells)
Cactos One Energy Storages (Finland) [10]	Energy storage system for households	2.6 MWh	Tesla Model S battery packs	74P-96S 2 (1704 cells)
EUREF Campus (Germany) [11]	Multi-use storage unit compensates for fluctuations in the grid	1.9 MWh	Audi battery packs	4P-108S (432 cells)
TGN Energy battery energy storage (Norway) [12]	Increased self-consumption	216 kWh	Mercedes-Benz battery packs	NA
Landafors hydropower plant (Sweden) [13]	Offers fast frequency reserve regulation to the power markets	250 kWh	48 Volvo plug-in hybrid battery packs	NA

¹ Ninety-six series-connected batteries in a module, then two modules are connected in parallel. ² Seventy-four parallel-connected batteries in a module, then 96 modules are connected in series.

Table 1. Projects around the world that reused batteries from EVs.

Name’s Project	Applications	Capacity	EV Model	Battery Pack Configuration
Johan Cruijff Arena (Netherlands) [6]	PV power supply, emergency supply	2.8 MWh	590 Nissan LEAF battery packs	96S-2P 1 (192 cells)
Former coal-fired power plant in Elverlingsen (Germany) [7]	Energy storage system for households	3.0 MWh	72 Renault Zoe battery packs	96S-2P (192 cells)
Cactos One Energy Storages (Finland) [10]	Energy storage system for households	2.6 MWh	Tesla Model S battery packs	74P-96S 2 (1704 cells)
EUREF Campus (Germany) [11]	Multi-use storage unit compensates for fluctuations in the grid	1.9 MWh	Audi battery packs	4P-108S (432 cells)
TGN Energy battery energy storage (Norway) [12]	Increased self-consumption	216 kWh	Mercedes-Benz battery packs	NA
Landafors hydropower plant (Sweden) [13]	Offers fast frequency reserve regulation to the power markets	250 kWh	48 Volvo plug-in hybrid battery packs	NA

¹ Ninety-six series-connected batteries in a module, then two modules are connected in parallel. ² Seventy-four parallel-connected batteries in a module, then 96 modules are connected in series.

In the BESS, switches are integrated to schedule cells and modules by connecting or bypassing them [14]. These switches enable scheduling to selectively isolate degraded cells or modules, thereby extending the battery pack’s useful lifetime [15]. By dynamically adjusting the connections between cells and modules, it is possible to balance the load effectively and mitigate the impact of lower capacity cells and modules. Scheduling not only improves the reliability and efficiency of the BESS, but also reduces the need for new batteries by maximizing the utilization of existing battery resources. To achieve this, scheduling requires all the states in the BESS including the state of charge (SOC) and the SOH of the cells and modules, the terminal voltage and output current of the battery pack, the power demand (e.g., from households), and the available power supply (e.g., from solar energy) [16,17,18]. The SOC of a battery indicates the current charge level as a percentage of its maximum capacity, whereas the SOH represents the ratio of the battery’s current maximum capacity to its original rated capacity. The required power demand and available power supply, collectively known as external systems information, refer to the amount of power the BESS needs to discharge and recharge. Incorporating BESS states into scheduling policy protects the battery pack by preventing excessive charging or discharging, and helps connect or bypass cells and modules appropriately. Furthermore, scheduling balances the SOC and SOH across cells and modules, ensuring that no single cell or module is overloaded [19]. In particular, SOH balancing through scheduling reduces the difference between SOHs among cells by utilizing the cells with higher SOHs and bypassing the cells with the lowest SOHs. In this way, the rate of degradation of cells or modules with lower SOHs is minimized. This balance helps to distribute the load more evenly, further protecting the battery pack and extending its useful lifetime.

Scheduling for battery packs in a BESS has been explored in the literature. In [20], an adaptive control algorithm was proposed to balance the SOCs for a series-connected battery pack. However, this approach overlooked the SOH and did not address parallel connections, limiting its effectiveness in comprehensive battery management. In [21], a controller was focused on balancing the SOCs in parallel-connected batteries to prevent overcharge or overdischarge, but failed to consider SOHs and series connections. An approach was presented in [15], where SOC balancing in parallel-series connections was proposed for automatic configuration according to the dynamic load, the storage demand, and the condition of each cell (i.e., SOC and current), yet it ignored SOH, which impacts a BESS lifetime. Other methods proposed in [18,22] were aimed at balancing SOHs by adjusting the charge and discharge durations for cells with weaker SOHs, but they ignored SOC and external systems information. Traditional methods like those presented in [15,18,20,21,22] are limited when they do not consider both SOC and SOH, because ignoring one can lead to suboptimal performance and reduced battery lifetime [19].

Deep reinforcement learning (DRL) has become a promising direction for battery pack scheduling with its ability to observe multiple states in a BESS and develop appropriate scheduling policies to optimize problem formulation. The combination of neural networks with reinforcement learning in DRL has proven to be a significant breakthrough, enabling the development of more scalable and efficient battery management strategies [23]. Unlike traditional reinforcement learning, which struggles with high-dimensional state spaces, DRL can leverage neural networks to approximate value functions and policies more effectively [24], allowing it to manage a larger number of cells and modules while responding in real time to critical factors such as SOC, SOH, and external systems information. In [17], an SOH balancing framework was proposed based on DRL in a series-connected battery pack to minimize the SOH imbalance among battery cells by observing the cell SOCs and SOHs. However, this approach lacked the observation of factors like power demand, terminal voltage, and output current, which are essential for effective switch scheduling and battery pack longevity. In [16], the authors proposed a DRL-based battery management algorithm to maximize the lifetime of retired batteries with varying SOHs in a parallel-connected battery pack, but they ignored scheduling for series-connected modules, so the approach was limited to applications requiring higher voltages. Moreover, the computational complexity in [16,17] was relatively high due to the extensive state space considered, and the proposals were limited to the use of a single agent, which can lead to a struggle with scalability when there is a large number of cells and modules. In [25], a multi-agent DRL-based method was proposed to reduce the SOC and SOH imbalance among battery cells, but overlooked external systems information, which directly affects battery pack lifetime by preventing overcharging or discharging, especially for cells or modules with lower SOHs.

Table 2. Classification of battery scheduling algorithms.

References	Battery Connection			SOC Balancing	SOH Balancing	Lifetime Optimal	Dynamic Power Demand
	Parallel	Series	Parallel-Series
[20]	-	✓	-	✓	-	-	-
[21]	✓	-	-	✓	-	-	-
[15]	✓	✓	✓	✓	-	-	✓
[18,22]	-	✓	-	✓	✓	-	-
[17]	✓	✓	-	✓	✓	-	-
[16]	✓	-	-	✓	✓	✓	✓
[25]	✓	-	-	✓	✓	-	-
Proposed	✓	✓	✓	✓	✓	✓	✓

shows the classification among battery scheduling algorithms.

In this study, we propose a battery management algorithm to maximize the BESS lifetime in a parallel-series connected battery pack with heterogeneous SOHs. To carry this out, the proposed algorithm first estimates the SOC and SOH of all cells jointly online. Then, based on the SOCs and SOHs of the cells and modules, a cooperative multi-agent deep Q network framework is implemented to schedule switches in the parallel-series connected battery pack by connecting or bypassing battery cells and modules. The proposed algorithm maximizes the battery pack lifetime and reduces the SOH imbalance among cells and modules. The algorithm also adapts to changes in external systems (i.e., power demand and available power supply). We demonstrate the effectiveness of our proposed algorithm via simulation using real, measured data compared to previous work.

The rest of this paper is organized as follows: Section 2 explores the proposed parallel-series connected battery pack and the associated scheduling challenges. Section 3 formulates the optimization problem by minimizing the reduction in SOH in the battery pack. Section 4 presents the framework of the proposed algorithm. Section 5 details the simulation setup, results, and the algorithm’s impact. Finally, Section 6 provides the conclusion of this work.

2. System Model

In this paper, we consider a parallel-series connected BESS [15] with a power supply (e.g., a solar energy generator) and a load (e.g., a household), as shown in Figure 3. The BESS comprises a parallel-series connected battery pack and a battery management system (BMS) that controls charging and discharging. We consider a discrete-time model with time slot t ($t=0,1,2,\dots$) and durations of $\Delta t$.

2.1. Parallel-Series Connected Battery Pack Model

The battery pack consists of $m\times n$ battery cells, where m battery modules are connected in series to increase voltage at the battery pack terminals, and n battery cells are connected in parallel to form a battery module to provide a higher current (or capacity), as shown in Figure 3. To schedule modules and cells in the battery pack, we consider $m\times (n+1)$ controllable switches. Specifically, at time slot t, switch ${\mathcal{X}}_{ij}\left(t\right)$($i=\{1,2,\dots ,m\}$, $j=\{1,2,\dots ,n\}$) expresses connecting or disconnecting battery cell j of module i, denoted as cell $(i,j)$, and switch ${\mathcal{X}}_i\left(t\right)$ is used to express connecting or bypassing module i in the battery pack. Switch ${\mathcal{X}}_{ij}\left(t\right)$ for cell $(i,j)$ at time slot t is defined as

$$\begin{array}{c}\hfill {\mathcal{X}}_{ij}\left(t\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{cell}(i,j)\mathrm{is}\mathrm{connected}\hfill \\ 0,\hfill & \mathrm{if}\mathrm{cell}(i,j)\mathrm{is}\mathrm{bypassed}.\hfill \end{array}\right.\end{array}$$

(1)

Switch ${\mathcal{X}}_i\left(t\right)$ for module i at time slot t is defined as

$$\begin{array}{c}\hfill {\mathcal{X}}_i\left(t\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathcal{X}}_{ij}\left(t\right)=0\mathrm{for}\forall j\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(2)

Switch ${\mathcal{X}}_i\left(t\right)$ is turned ON (1) if all n cells in module i are disconnected (0) to ensure that the battery pack charging or discharging process is not interrupted, which means module i is bypassed from the power supply or load at time slot t. Otherwise, switch ${\mathcal{X}}_i\left(t\right)$ is turned OFF (0) if any cell in module i is connected (1), which means that module i can be charged or discharged at time slot t.

Each battery cell is modeled according to a second-order Thévenin equivalent model [26] based on the material structure of the lithium-ion battery [25], as shown in Figure 4. Battery cell $(i,j)$, which represents the jth cell in module i, has electrochemical impedance spectroscopy (EIS) parameters including open-circuit voltage $V_{ij}^O$, internal resistance $R_{ij}^S$, and two polarization RC pairs connected in series; each RC pair includes a resistor and a capacitor connected in parallel, i.e., ($R_{ij}^{P1}$,$C_{ij}^{P1}$) and ($R_{ij}^{P2}$,$C_{ij}^{P2}$). The terminal voltage of cell $(i,j)$ at time t, $V_{ij}\left(t\right)$, is calculated by

$$V_{ij}\left(t\right)=V_{ij}^O\left(t\right)-V_{ij}^{P1}\left(t\right)-V_{ij}^{P2}\left(t\right)-R_{ij}^S\left(t\right)I_{ij}\left(t\right),$$

(3)

where $I_{ij}\left(t\right)$ is the measured current of cell $(i,j)$ at time slot t; $V_{ij}^{P1}\left(t\right)$ and $V_{ij}^{P2}\left(t\right)$, respectively, are the polarization voltages of RC pairs ($R_{ij}^{P1}$,$C_{ij}^{P1}$) and ($R_{ij}^{P2}$,$C_{ij}^{P2}$) in cell $(i,j)$ at time slot t, and are calculated by using the EIS parameters at the previous time slot $t-1$ as [26]

$$V_{ij}^{P1}\left(t\right)=e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P1}(t-1)C_{ij}^{P1}(t-1)}}}{V}_{ij}^{P1}(t-1)+R_{ij}^{P1}(t-1)\left(1-e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P1}(t-1)C_{ij}^{P1}(t-1)}}}\right)I_{ij}(t-1)$$

(4)

and

$$V_{ij}^{P2}\left(t\right)=e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P2}(t-1)C_{ij}^{P2}(t-1)}}}{V}_{ij}^{P2}(t-1)+R_{ij}^{P2}(t-1)\left(1-e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P2}(t-1)C_{ij}^{P2}(t-1)}}}\right)I_{ij}(t-1).$$

(5)

Based on measurement data (terminal voltage, current) and EIS parameters, the BMS estimates the SOC and SOH of each cell in order to schedule switches in the battery pack. Details are explained in the following subsection.

2.2. Battery Management System

The BMS schedules the switches in the battery pack and external systems (i.e., the power supply or load demand) according to SOC and SOH estimation, as shown in Figure 3. To estimate SOC and SOH, the BMS monitors the voltage, current, and temperature of each battery cell. We define the SOC of cell $(i,j)$ at time t as the level of charge at time t relative to the maximum battery capacity by [18]

$$\begin{array}{c}\hfill SO{C}_{ij}\left(t\right)=SO{C}_{ij}(t-1)-{\displaystyle \frac{\eta \Delta t{I}_{ij}(t-1)}{M_{ij}\left(t\right)}},\end{array}$$

(6)

where $M_{ij}\left(t\right)$ is the estimated capacity level of cell $(i,j)$ at time slot t, and $\eta$ is the Coulombic efficiencies of the discharging and charging processes. We set the measured current $I_{ij}(t-1)$ to positive when discharged and negative when charged, to simplify Equation (6).

Module i, in which cells are connected in parallel, ensures that all cells share the same voltage while their individual currents add up, resulting in a cumulative increase in the total capacity of the module [27]. Therefore, capacity $M_i\left(t\right)$ of module i in parallel connection is the sum of the capacities of the individual cells based on Kirchhoff’s law as [28]

$$\begin{array}{c}\hfill M_i\left(t\right)=\sum _{j=1}^n{M}_{ij}\left(t\right).\end{array}$$

(7)

We define the SOC of module i at time slot t as the ratio of the total charge level of all cells in module i to the total capacity $M_i\left(t\right)$ of module i by [18]

$$\begin{array}{c}\hfill SO{C}_i\left(t\right)=\sum _{j=1}^{n}SO{C}_{ij}\left(t\right){\displaystyle \frac{M_{ij}\left(t\right)}{M_i\left(t\right)}}.\end{array}$$

(8)

The SOH of cell $(i,j)$ at time t, which is the ratio of the maximum battery capacity at time t to its rated capacity [29], is defined as

$$\begin{array}{c}\hfill SO{H}_{ij}\left(t\right)={\displaystyle \frac{M_{ij}\left(t\right)}{M_{ij}^{new}}}={\displaystyle \frac{M_{ij}\left(t\right)}{M^{new}}},\end{array}$$

(9)

where $M_{ij}^{new}$ is the initial capacity of new cell $(i,j)$; in this paper, we consider all battery cells to be the same type and have the same initial capacity as $M_{ij}^{new}=M^{new}$ [30,31]. The BESS exhibits heterogeneous SOHs between individual cells. From Equation (7), capacity $M_i\left(t\right)$ of module i at time slot t is the capacity summation of parallel-connected cells in the given module. Therefore, combining Equation (9), the SOH of module i at time slot t, $SO{H}_i\left(t\right)$, is defined as the SOH average of all parallel-connected cells in module i by

$$\begin{array}{c}\hfill SO{H}_i\left(t\right)={\displaystyle \frac{M_i\left(t\right)}{n{M}^{new}}}={\displaystyle \frac{1}{n}}\sum _{j=1}^{n}SO{H}_{ij}\left(t\right),\end{array}$$

(10)

where $SO{H}_{ij}\left(t\right)$ is the SOH of cell j in module i at time slot t.

The SOH of individual aging cells is not uniform, leading to SOH inconsistencies between modules. The battery pack, in which modules are connected in series, decreases the terminal voltage, limiting the fulfillment of demand with lower SOH modules [32]. Therefore, we define the SOH of the battery pack, $SO{H}_{\mathcal{P}}\left(t\right)$, as the lowest SOH of series-connected modules by [30,32]

$$\begin{array}{c}\hfill SO{H}_{\mathcal{P}}\left(t\right)=\underset{i}{min}SO{H}_i\left(t\right),\end{array}$$

(11)

where $SO{H}_i\left(t\right)$ is the SOH of module i at time slot t. The SOH of the battery pack $SO{H}_{\mathcal{P}}\left(t\right)$, which represents battery pack aging, is a non-increasing function until the end of its life cycle. In this paper, we define the lifetime of the battery pack, $T_{EoL}$, as the first time slot after the battery pack $SO{H}_{\mathcal{P}}\left(t\right)$ reaches a threshold, denoted as $T{h}_{EoL}$, which is assumed to be 60% of SOH as seen in Figure 1, by 

$$\begin{array}{c}\hfill T_{EoL}=arg\underset{t}{min}\{SO{H}_{\mathcal{P}}\left(t\right)\le T{h}_{EoL}\}.\end{array}$$

(12)

The BMS protects battery cells from overcharge, overdischarge, overheating, and excess current by controlling the BESS to fulfill demand when discharging to the load, then recharging from the power supply to recover power based on the SOC and SOH information. Note that the load and power supply change continuously under actual conditions, where $d_D\left(t\right)$ and $d_C\left(t\right)$ represent the power demand and available power supply, respectively, at time slot t and are constrained as [16]

$$\begin{array}{c}\hfill d_D\left(t\right)d_C\left(t\right)=0\mathrm{with}{d}_D\left(t\right),d_C\left(t\right)\ge 0,\end{array}$$

(13)

which means the battery pack only charges, discharges, or is idle at the given time slot. We define the first time slot of a discharge process $t_D$ if $d_D\left(t_D\right)>0$ and $d_D(t_D-1)=0$. Similarly, the first time slot of a charge process, $t_C$, is defined if $d_C\left(t_C\right)>0$ and $d_C(t_C-1)=0$. When discharging, the BMS considers the amount of discharged power, $l_D\left(t\right)$, and the maximum dischargeable power, $E_D\left(t\right)$, of the battery pack at time slot t as [33]

$$\begin{array}{c}\hfill l_D\left(t\right)=\sum _{i=1}^m\left(SO{C}_i\left(t_D\right)-SO{C}_i\left(t\right)\right)M_i\left(t\right)\end{array}$$

(14)

and

$$\begin{array}{c}\hfill E_D\left(t\right)=\sum _{i=1}^m\left(SO{C}_i\left(t_D\right)-SO{C}_{min}\right)M_i\left(t\right),\end{array}$$

(15)

where $SO{C}_{min}$ indicates the lower bound of the SOC to prevent overdischarge, and $t_D$ is the first time slot of the discharge process containing t. When charging, the BMS considers the amount of charged power, $l_C\left(t\right)$, and the maximum chargeable power, $E_C\left(t\right)$, of the battery pack at time slot t as [33]

$$\begin{array}{c}\hfill l_C\left(t\right)=\sum _{i=1}^m\left(SO{C}_i\left(t\right)-SO{C}_i\left(t_C\right)\right)M_i\left(t\right)\end{array}$$

(16)

and

$$\begin{array}{c}\hfill E_C\left(t\right)=\sum _{i=1}^m\left(SO{C}_{max}-SO{C}_i\left(t_C\right)\right)M_i\left(t\right),\end{array}$$

(17)

where $SO{C}_{max}$ indicates the upper bound of the SOC to prevent overcharging, and $t_C$ is the first time slot of the charge process containing t.

The BMS controls the battery pack discharge if $d_D\left(t\right)>0$ and $l_D\left(t\right)<E_D\left(t\right)$, or charge if $d_C\left(t\right)>0$ and $l_C\left(t\right)<E_C\left(t\right)$. Otherwise, the battery pack returns to idle for the remainder of the discharge or charge process. To that end, after estimating SOCs and SOHs, and controlling the charging or discharging process of the battery pack, the BMS schedules all switches in the battery pack in order to extend the lifetime and to reduce the SOH imbalance among the cells and modules.

3. Problem Formulation

The aim of this paper is to extend the lifetime of the battery pack in the BESS. To maximize the battery pack lifetime, we formulate the lifetime maximization of the battery pack as

$$\begin{array}{c}\max \hspace{1em}{T}_{\mathit{EoL}}\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.\hspace{1em}& I_{min}^{-}\le {\mathit{I}}_{\mathit{ij}}\left(t\right)\le I_{max}^{+},\hfill \\ & {\mathit{SOC}}_{\mathit{min}}\le {\mathit{SOC}}_{\mathit{ij}}\left(t\right)\le {\mathit{SOC}}_{\mathit{max}},\hfill \\ & l_D\left(t\right)\le E_D\left(t\right),\hfill \\ & l_C\left(t\right)\le E_C\left(t\right),\hfill \end{array}$$

(18)

where $I_{max}^{+}$ and $I_{min}^{-}$ represent the discharge current and charge current thresholds to limit the discharging and charging rates, respectively; $SO{C}_{min}$ and $SO{C}_{max}$ represent the lower and upper limits of the SOC, respectively, which are necessary to prevent overdischarging and overcharging; $l_D\left(t\right)$ and $l_C\left(t\right)$ represent the discharged and charged power, respectively, until time slot t; $E_D\left(t\right)$ and $E_C\left(t\right)$, respectively, indicate the maximum power load of the battery pack when discharging and charging. Maximizing battery pack lifetime $T_{EoL}$ means maximizing the number of time slots in this second life cycle (Stage 2 in Figure 1). To carry this out, we minimize the rate of battery pack aging (i.e., the SOH reduction in the battery pack). We first define the SOH reduction in the battery pack at time slot t as

$$\begin{array}{c}\Delta SO{H}_{\mathcal{P}}\left(t\right)=SO{H}_{\mathcal{P}}(t-1)-SO{H}_{\mathcal{P}}\left(t\right),\end{array}$$

(19)

where $SO{H}_{\mathcal{P}}(t-1)$ and $SO{H}_{\mathcal{P}}\left(t\right)$ denote the SOH of the battery pack, respectively, at time slots $t-1$ and t. $SO{H}_{\mathcal{P}}\left(t\right)$ is a non-increasing function, so we constrain it with

$$\begin{array}{c}\Delta SO{H}_{\mathcal{P}}\left(t\right)\ge 0.\end{array}$$

(20)

To achieve this, the problem is framed to minimize the reduction in SOH of the battery pack, which is mathematically expressed as

$$\begin{array}{c}\min \hspace{1em}{\displaystyle \sum _t}\Delta SO{H}_{\mathcal{P}}\left(t\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.\hspace{1em}& \Delta SO{H}_{\mathcal{P}}\left(t\right)\ge 0,\hfill \\ & I_{min}^{-}\le {\mathit{I}}_{\mathit{ij}}\left(t\right)\le I_{max}^{+},\hfill \\ & {\mathit{SOC}}_{\mathit{min}}\le {\mathit{SOC}}_{\mathit{ij}}\left(t\right)\le {\mathit{SOC}}_{\mathit{max}},\hfill \\ & l_D\left(t\right)\le E_D\left(t\right),\hfill \\ & l_C\left(t\right)\le E_C\left(t\right),\hfill \end{array}$$

(21)

4. The Proposed Algorithm

To enhance the lifetime of the battery pack in the BESS, we propose a battery management algorithm to minimize SOH reduction in the battery pack. The overall flow of the proposed algorithm is shown in Algorithm 1. At each time slot, the algorithm first gathers measurement data, including the terminal voltage, current, and temperature of each cell. It then estimates the SOC and SOH (Algorithm 2) and manages the BESS charging or discharging process based on the dynamic power demand (Algorithm 3). After updating the states in the BESS, the proposed algorithm schedules all switches in the battery pack to prolong its second lifetime based on the cooperative multi-agent deep Q network (Algorithm 4). The following subsections provide a detailed discussion of each component of the proposed algorithm.

Algorithm 1 BESS Scheduling Algorithm

1: Gather measurement data V, I, T 2: Estimate SOCs and SOHs (Algorithm 2) 3: Control discharge, charge, and idle processes of the BESS (Algorithm 3) 4: Update the BESS states 5: Schedule switches in the battery pack (Algorithm 4)

Algorithm 2 EKF-based SOC and SOH estimation

1: Input: Measurement data V, I, T; Data tables  2: Output: $SO{C}_{ij}\left(t\right)$, $SO{H}_{ij}\left(t\right)$  3: for each cell $(i,j)$ do  4:       Estimate state vector $\widehat{x_{ij}}\left(t\right)$ and error covariance $\widehat{P_{ij}}\left(t\right)$ using (23) and (24)  5:       Estimate terminal voltage $\widehat{V_{ij}}\left(t\right)$ using (28)  6:       Compute Kalman gain $G_{ij}\left(t\right)$ using (32)  7:       Correct state vector $x_{ij}\left(t\right)$ and error covariance $P_{ij}\left(t\right)$ using (33) and (34)  8:       Update $SO{C}_{ij}\left(t\right)$ and $M_{ij}\left(t\right)$  9:       Update $SO{H}_{ij}\left(t\right)$ using (35) 10: end for 11: Update SOC and SOH of each module using (8) and (10) 12: Update SOH of the battery pack using (11)

Algorithm 3 Charge, Discharge, and Idle Controlling

1: Input: $l_D\left(t\right)$, $l_C\left(t\right)$, $d_D\left(t\right)$, $d_C\left(t\right)$, $E_D\left(t\right)$, $E_C\left(t\right)$  2: Output: Discharge, charge, or idle  3: Determine the current process (discharge, charge, or idle)  4: if $d_D\left(t\right)>0$ then                  ▹ Discharging  5:     if $l_D\left(t\right)\ge d_D\left(t\right)$ then  6:           Convert discharge to charge  7:     else  8:           if $l_D\left(t\right)\ge E_D\left(t\right)$ then  9:               Convert discharge to idle 10:          else 11:              Continue to discharge 12:          end if 13:    end if 14: else if $d_C\left(t\right)>0$ then                  ▹ Charging 15:    if $l_C\left(t\right)\ge d_C\left(t\right)$ then 16:          Convert charge to discharge 17:    else 18:          if $l_C\left(t\right)\ge E_C\left(t\right)$ then 19:              Convert charge to idle 20:          else 21:              Continue to charge 22:          end if 23:    end if 24: end if

Algorithm 4 Switch Scheduling

1: Input: state vector $s\left(t\right)$  2: Output: Optimal joint action $a\left(t\right)$  3: Construct main network ${\mathcal{Q}}_i$ and target network ${\overline{\mathcal{Q}}}_i$ for each agent  4: Initialize acquired knowledge $\mathcal{K}$  5: Select module switch action $a_{m+1}\left(t\right)$ by $ϵ$-greedy policy   ▹ Switch ON/OFF modules  6: for each agent $i=1\mathrm{to}m$ do  7:       if module i is OFF then  8:            Limit action $a_i\left(t\right)$              ▹ Switch OFF all cells in module i  9:     else 10:            Select action $a_i\left(t\right)$ by $ϵ$-greedy policy       ▹ Switch ON/OFF cells in module i 11:       end if 12: end for 13: Execute joint action $a\left(t\right)$ 14: for each agent $i=1\mathrm{to}m$ do 15:       Compute local reward $r_i\left(t\right)$ 16: end for 17: Compute global reward $r_{m+1}\left(t\right)$ 18: Update new state $s(t+1)$ 19: Add $\langle s_i\left(t\right),a_i\left(t\right),r_i\left(t\right),s_i(t+1)\rangle$ into $\mathcal{K}$ for each agent 20: Compute target action value ${\overline{Q}}_i\left(t\right)$ for each agent 21: Perform a gradient descent to minimize loss function $\mathcal{L}\left({\varphi }_i\left(t\right)\right)$ for each agent

4.1. SOC and SOH Estimation

To observe the states of the battery pack, the proposed algorithm estimates the SOC and SOH of each battery cell by gathering its terminal voltage, current, and temperature. To carry this out, the algorithm first uses a fourth-order extended Kalman filter (EKF) to estimate the SOC and SOH of each battery cell, and then updates the SOC and SOH information of each battery module and the battery pack.

For each cell $(i,j)$, we define the corrected state vector of cell $(i,j)$ at time $t_k$, denoted as $x_{ij}\left(t_k\right)$, by

$$\begin{array}{c}\hfill x_{ij}\left(t_k\right)={[SO{C}_{ij}\left(t_k\right),V_{ij}^{P1}\left(t_k\right),V_{ij}^{P2}\left(t_k\right),1/M_{ij}\left(t_k\right)]}^T.\end{array}$$

(22)

Based on measured current $I_{ij}(t-1)$ and the modeled EIS parameters at the previous time slot, the algorithm estimates state vector $\widehat{x_{ij}}\left(t\right)$ and error covariance $\widehat{P_{ij}}\left(t\right)$ as

$$\begin{array}{c}\hfill \widehat{x_{ij}}\left(t\right)=A_{ij}(t-1)x_{ij}(t-1)+B_{ij}(t-1)I_{ij}(t-1)\end{array}$$

(23)

and

$$\begin{array}{c}\hfill \widehat{P_{ij}}\left(t\right)=A_{ij}(t-1)P_{ij}(t-1)A_{ij}{(t-1)}^T,\end{array}$$

(24)

where $x_{ij}(t-1)$ is the correct state vector of cell $(i,j)$ at time $t-1$; $A_{ij}(t-1)$ and $B_{ij}(t-1)$ denote the transition matrix and the input matrix, respectively. Matrices $A_{ij}(t-1)$ and $B_{ij}(t-1)$ are defined as

$$A_{ij}(t-1)=\left[\begin{array}{cccc}1& 0& 0& -\eta \Delta t{I}_{ij}(t-1)\\ 0& e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P1}(t-1)C_{ij}^{P1}(t-1)}}}& 0& 0\\ 0& 0& e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P1}(t-1)C_{ij}^{P1}(t-1)}}}& 0\\ 0& 0& 0& 1\end{array}\right]$$

(25)

and

$$B_{ij}(t-1)=\left[\begin{array}{c}0\\ R_{ij}^{P1}(t-1)(1-e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P1}(t-1)C_{ij}^{P1}(t-1)}}})\\ R_{ij}^{P2}(t-1)(1-e^{-{\displaystyle \frac{\Delta t}{R_{ij}^{P2}(t-1)C_{ij}^{P2}(t-1)}}})\\ 0\end{array}\right],$$

(26)

where $I_{ij}(t-1)$ is the measured current of cell $(i,j)$ at $t-1$. EIS parameters $R_{ij}^S$, $R_{ij}^{P1}$, $C_{ij}^{P1}$, $R_{ij}^{P2}$, and $C_{ij}^{P2}$ are functions of $SO{C}_{ij}$ and $SO{H}_{ij}$. Specifically, the EIS parameters are exponential functions of $SO{C}_{ij}(t-1)$, such as

$$\begin{array}{c}\hfill x_{1}exp\left(x_{2}SO{C}_{ij}(t-1)\right)+x_3,\end{array}$$

(27)

where $x_1$, $x_2$, and $x_3$ are real numbers, depending on the SOH of each cell $SO{H}_{ij}$. A dataset [34] is used to construct look-up tables of each EIS parameter $(R_{ij}^S$, $R_{ij}^{P1}$, $C_{ij}^{P1}$, $R_{ij}^{P2}$, or $C_{ij}^{P2})$ based on the above exponential functions. The algorithm estimates terminal voltage $\widehat{V_{ij}}\left(t\right)$ using state vector $\widehat{x_{ij}}\left(t\right)$ and Jacobian matrices $C_{ij}\left(t\right)$ and $D_{ij}\left(t\right)$ as

$$\begin{array}{c}\hfill \widehat{V_{ij}}\left(t\right)=C_{ij}\left(t\right)\widehat{x_{ij}}\left(t\right)+D_{ij}\left(t\right)I_{ij}\left(t\right),\end{array}$$

(28)

where $I_{ij}\left(t\right)$ is the measured current of cell $(i,j)$ at time slot t. Matrices $C_{ij}\left(t\right)$ and $D_{ij}\left(t\right)$ are, respectively, defined as

$$C_{ij}\left(t\right)=\left[\begin{array}{cccc}{\displaystyle \frac{\delta V_{ij}^O\left(t\right)}{\delta SO{C}_{ij}\left(t\right)}}& -1& -1& 0\end{array}\right]$$

(29)

and

$$\begin{array}{c}\hfill D_{ij}\left(t\right)=-R_{ij}^S\left(t\right).\end{array}$$

(30)

Open-circuit voltage $V_{ij}^O\left(t\right)$ is defined as the ath-order polynomial function of $SO{C}_{ij}\left(t\right)$ by

$$\begin{array}{c}\hfill V_{ij}^O\left(t\right)=\sum _{b=0}^a{y}_b{\left(SO{C}_{ij}\left(t\right)\right)}^b,\end{array}$$

(31)

where $y_b$ is a real number depending on $SO{H}_{ij}$. The algorithm calculates Kalman gain $G_{ij}\left(t\right)$ to consider the error of the estimated value as

$$\begin{array}{c}\hfill G_{ij}\left(t\right)=\widehat{P_{ij}}\left(t\right)C_{ij}{\left(t\right)}^T{\left(C_i\left(t\right)\widehat{P_{ij}}\left(t\right)C_{ij}{\left(t\right)}^T\right)}^{-1}.\end{array}$$

(32)

Using the measured terminal voltage of cell $(i,j)$, $V_{ij}\left(t\right)$, the algorithm corrects state vector $x_{ij}\left(t\right)$ and error covariance $P_{ij}\left(t\right)$ as

$$\begin{array}{c}\hfill x_{ij}\left(t\right)=\widehat{x_{ij}}\left(t\right)+G_{ij}\left(t\right)\left(V_{ij}\left(t\right)-\widehat{V_{ij}}\left(t\right)\right)\end{array}$$

(33)

and

$$\begin{array}{c}\hfill P_{ij}\left(t\right)=\left(1-G_{ij}\left(t\right)C_{ij}\left(t\right)\right)\widehat{P_{ij}}\left(t\right).\end{array}$$

(34)

From corrected state vector $x_{ij}\left(t\right)$, $SO{C}_{ij}\left(t\right)$ and maximum capacity $M_{ij}\left(t\right)$ of cell $(i,j)$ at time slot t are updated. The estimation algorithm updates the SOH of cell $(i,j)$ at time slot t by averaging it after completing a charge or a discharge process in the battery pack, as the SOH does not degrade immediately after single or multiple time slots [35], as

$$\begin{array}{c}\hfill SO{H}_{ij}\left(t\right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{t-t_D+1}}\sum _{\tau =t_D}^{t}SO{H}_{ij}\left(\tau \right)\hfill & \mathrm{if}\mathrm{completing}\mathrm{discharge}\mathrm{at}\mathrm{time}\mathrm{slot}t;\hfill \\ {\displaystyle \frac{1}{t-t_C+1}}\sum _{\tau =t_C}^{t}SO{H}_{ij}\left(\tau \right)\hfill & \mathrm{if}\mathrm{completing}\mathrm{charge}\mathrm{at}\mathrm{time}\mathrm{slot}t;\hfill \\ SO{H}_{ij}(t-1)\hfill & \mathrm{otherwise}\hfill \end{array}\right.,\end{array}$$

(35)

where $t_D$ and $t_C$ are the first time slots of the discharging and charging processes containing t. To that end, the estimation algorithm updates the SOC and SOH of modules ($SO{C}_i\left(t\right),SO{H}_i\left(t\right)\mid \forall i=\{1,2,\dots ,m\}$) using Equations (8) and (10). $SO{H}_{\mathcal{P}}\left(t\right)$ of the battery pack is also updated using Equation (11). Algorithm 2 summarizes the EKF-based combined SOC and SOH estimation.

4.2. Charge, Discharge, and Idle Controlling

The proposed algorithm controls the charging and discharging processes of the battery pack to prevent overcharging and overdischarging in the BESS. By comparing loaded power ($l_D\left(t\right)$, and $l_C\left(t\right)$), maximum power load ($E_D\left(t\right)$, and $E_C\left(t\right)$), power demand ($d_D\left(t\right)$), and available power supply ($d_C\left(t\right)$), the algorithm determines whether the status of the BESS is discharging, charging, or idle in order to control it accordingly.

On the one hand, if the battery pack is discharging, i.e., $d_D\left(t\right)>0$, we calculate the amount of power discharged $l_D\left(t\right)$ and maximum dischargeable power $E_D\left(t\right)$ using Equations (14) and (15). If $l_D\left(t\right)$ reaches power demand $d_D\left(t\right)$, the algorithm converts the BESS status from discharging to charging. When $l_D\left(t\right)$ reaches the maximum power load of the battery pack when discharging $E_D\left(t\right)$, the algorithm lets the battery pack go idle. Otherwise, the discharge process continues.

On the other hand, if the battery pack is charging, i.e., $d_C\left(t\right)>0$, we calculate the amount of power charged $l_C\left(t\right)$ and maximum chargeable power $E_C\left(t\right)$ by using Equations (16) and (17). If $l_C\left(t\right)$ reaches $d_C\left(t\right)$, the algorithm converts the BESS status from charging to discharging. When $l_C\left(t\right)$ reaches the maximum power load of the battery pack at time slot t when charging $E_C\left(t\right)$, the algorithm lets the battery pack go idle. Otherwise, it continues charging. The process of the battery pack charging and discharging controlling is summarized in Algorithm 3.

4.3. Cooperative Multi-Agent Deep Reinforcement Learning-Based Battery Switch Scheduling

A cooperative multi-agent deep Q network (CM-DQN) scheduling algorithm is proposed to control switches in order to minimize the SOH reduction in the battery pack. The algorithm uses $m+1$ agents in which agent $m+1$ collectively evaluates the battery pack and comes up with module switch scheduling for serial connections, while m agents (1 to m) correspond to m modules for scheduling cell switches in parallel networks. While agents 1 to m optimize the local reward (i.e., minimize the SOH reduction in each module) by sharing policies, agent $m+1$ obtains global states (from modules and the BESS) to minimize the SOH reduction in the battery pack in the second life cycle. The proposed algorithm perceives environment state $s\left(t\right)=\{s_i\left(t\right)\mid i=1,\dots ,m+1\}$ based on the estimation algorithm (Algorithm 2) and the charge and discharge control algorithm (Algorithm 3), and chooses a joint action $a\left(t\right)=\{a_i\left(t\right)\mid i=1,\dots ,m+1\}$ by achieving a cumulative reward $r\left(t\right)=\{r_i\left(t\right)\mid i=1,\dots ,m+1\}$. The proposed switch scheduling algorithm first monitors the current environmental state of the battery pack and then derives state vectors for each agent, respectively, as

$$\begin{array}{c}\hfill s_i\left(t\right)=\left\{\begin{array}{c}\left[{\mathcal{C}}_i\left(t\right),{\mathcal{H}}_i\left(t\right),I_{\mathcal{P}}\left(t\right),d_D\left(t\right),d_C\left(t\right)\right],\mathrm{with}i=1,2,\dots ,m\hfill \\ \left[\mathcal{C}\left(t\right),\mathcal{H}\left(t\right),V_{\mathcal{P}}\left(t\right),d_D\left(t\right),d_C\left(t\right)\right],\mathrm{with}i=m+1,\hfill \end{array}\right.\end{array}$$

(36)

where $\mathcal{C}\left(t\right)$ and $\mathcal{H}\left(t\right)$ represent sets of the SOCs and SOHs of m series-connected modules, respectively; ${\mathcal{C}}_i\left(t\right)$ and ${\mathcal{H}}_i\left(t\right)$ are value sets of the SOCs and SOHs of n parallel-connected cells in module i, respectively; $V_{\mathcal{P}}\left(t\right)$ and $I_{\mathcal{P}}\left(t\right)$ represent the measured terminal voltage and the measured current of the battery pack at time slot t; and $d_D\left(t\right)$ and $d_C\left(t\right)$ are the load demand and available power supply at time slot t, respectively. Action $a_i\left(t\right)$ with $i=1,\dots ,m+1$ controls switches in the battery pack and is defined as

$$\begin{array}{c}\hfill a_i\left(t\right)=\left\{\begin{array}{c}\left[{\mathcal{X}}_{i1}\left(t\right),\dots ,{\mathcal{X}}_{in}\left(t\right)\right],\mathrm{with}i=1,2,\dots ,m\mathrm{for}\mathrm{cell}\mathrm{switches}\hfill \\ \left[{\mathcal{X}}_1\left(t\right),\dots ,{\mathcal{X}}_m\left(t\right)\right],\mathrm{with}i=m+1\mathrm{for}\mathrm{module}\mathrm{switches}.\hfill \end{array}\right.\end{array}$$

(37)

All agents continuously interact with each other by sharing states and policies so the training is synchronous and accurate.

To minimize SOH reduction in the battery pack in accordance with the obtained state, the algorithm utilizes acquired knowledge $\mathcal{K}$ that includes a replay buffer to store in the form of $\langle s_i\left(t\right),$ $a_i\left(t\right),r_i\left(t\right),s_i(t+1)\rangle$ with an experience in each time slot. For each agent $i=\{1,\dots ,m+1\}$, the algorithm constructs a main network ${\mathcal{Q}}_i$, and a target network ${\overline{\mathcal{Q}}}_i$ with the same structure and random weights ${\varphi }_i={\overline{\varphi }}_i$, where ${\mathcal{Q}}_i$ approximates the action-value function $Q_i\left(s_i\left(t\right), a_i\left(t\right)\mid {\varphi }_i\right)$ at time slot t, and ${\overline{\mathcal{Q}}}_i$ computes the target action value ${\overline{Q}}_i\left(t\right)$.

To utilize the past experiences, the proposed algorithm looks at acquired knowledge $\mathcal{K}$ to determine whether state $s_{m+1}\left(t\right)$ is in $\mathcal{K}$ or not. If state $s_{m+1}\left(t\right)$ belongs to $\mathcal{K}$, the algorithm chooses action $a_{m+1}\left(t\right)$ using an $ϵ$-greedy policy [36]. Specifically, it either chooses a random action with probability $p=ϵ$ or opts for the action with the highest $Q_{m+1}\left(t_k\right)$ with probability $p=1-ϵ$, by

$$\begin{array}{c}\hfill a_{m+1}\left(t\right)=\left\{\begin{array}{c}\mathrm{random}\mathrm{action},\mathrm{with}p=ϵ\hfill \\ arg\underset{a_{m+1}\left(t\right)}{max}{Q}_{m+1}\left(t_k\right),\mathrm{with}p=1-ϵ\hfill \end{array}\right..\end{array}$$

(38)

In the case in which state $s_{m+1}\left(t\right)$ is not in $\mathcal{K}$, scheduling action $a_{m+1}\left(t\right)$ is performed at random. We determine which modules are bypassed at time slot t, and then limit scheduling action in each agent of cell switch scheduling (agent 1 to m). If module i is bypassed, we limit action $a_i\left(t\right)$ by turning OFF all cells in module i. Otherwise, the algorithm continues to assess the acquired knowledge $\mathcal{K}$ to determine whether state $s_i\left(t\right)$ is included in $\mathcal{K}$. If state $s_i\left(t\right)$ is in $\mathcal{K}$, the algorithm selects action $a_i\left(t\right)$ according to an $ϵ$-greedy policy as specified in Equation (38). In the case in which state $s_i\left(t\right)$ is not in $\mathcal{K}$, scheduling action $a_i\left(t\right)$ is chosen randomly. After all actions are chosen, a joint action $a\left(t\right)=\{a_i\left(t\right)\mid i=1,\dots ,m+1\}$ is executed.

For agent i of module i corresponding to cell switch scheduling, the algorithm evaluates the local immediate reward as

$$\begin{array}{c}\hfill R_i\left(t\right)=\mathbb{E}\left[-\Delta SO{H}_i\left(t\right)\right],\end{array}$$

(39)

where $\Delta SO{H}_i\left(t\right)$ is the SOH reduction in module i at time slot t, and is defined as a non-increasing function by

$$\begin{array}{c}\hfill \Delta SO{H}_i\left(t\right)=SO{H}_i(t-1)-SO{H}_i\left(t\right)\ge 0.\end{array}$$

(40)

The algorithm evaluates the local cumulative reward through interactions with the environment and seeks an optimal policy to maximize it as

$$\begin{array}{c}\hfill r_i\left(t\right)=\sum _{h=t}^{\infty }{\gamma }^{h-t}{R}_i\left(h\right).\end{array}$$

(41)

To that end, the algorithm computes the global cumulative reward to minimize SOH reduction in the battery pack as

$$\begin{array}{c}\hfill r_{m+1}\left(t\right)=\sum _{h=t}^{\infty }{\gamma }^{h-t}{R}_{m+1}\left(h\right),\end{array}$$

(42)

where $R_{m+1}\left(t_h\right)$ is the global immediate reward and is computed by

$$\begin{array}{c}\hfill R_{m+1}\left(t\right)=\mathbb{E}\left[-\Delta \sum _{i=1}^{m}SO{H}_i\left(t\right)\right].\end{array}$$

(43)

By executing joint action $a\left(t\right)$, the algorithm updates new state $s(t+1)$, then stores sample $\langle s_i\left(t\right),$ $a_i\left(t\right),r_i\left(t\right),s_i(t+1)\rangle$ into accquired knowledge $\mathcal{K}$. Target action value ${\overline{Q}}_i\left(t\right)$ is computed by

$$\begin{array}{c}\hfill {\overline{Q}}_i\left(t\right)=r_i\left(t\right)+\gamma \underset{a_i\left(t\right)}{max}{Q}_i\left(s_i(t+1),a_i(t+1)\mid {\overline{\varphi }}_i\right),\end{array}$$

(44)

where $\gamma \in (0,1]$ is the discount factor that determines the emphasis on future rewards. The CM-DQN updates the acquired knowledge $\mathcal{K}$ by minimizing the loss function $\mathcal{L}\left({\varphi }_i\left(t\right)\right)$ through gradient descent. The loss function is defined as

$$\begin{array}{c}\hfill \mathcal{L}\left({\varphi }_i\left(t\right)\right)\leftarrow \mathbb{E}\left[{\left({\overline{Q}}_i\left(t\right)-Q_i\left(s_i\left(t\right),a_i\left(t\right)\mid {\varphi }_i\right)\right)}^2\right].\end{array}$$

(45)

Weight ${\varphi }_i\left(t\right)$ is updated by the loss function as

$$\begin{array}{c}\hfill {\varphi }_i\left(t\right)={\varphi }_i(t-1)+\alpha \nabla \mathcal{L}\left({\varphi }_i\left(t\right)\right)\end{array}$$

(46)

where $\alpha \in (0,1]$ is the learning factor. After computing the loss for an action, the target network ${\overline{\mathcal{Q}}}_i$ updates its weights to match those of the main network ${\mathcal{Q}}_i$, i.e., ${\overline{\varphi }}_i={\varphi }_i$ after P time slots to ensure algorithm stability [36]. Loss function $\mathcal{L}\left({\varphi }_i\left(t\right)\right)$ is minimized so that action value $Q_i\left(s_i\left(t\right),a_i\left(t\right)\mid {\varphi }_i\right)$ has the same value as target action value ${\overline{Q}}_i\left(t\right)$, which also means that the SOH reduction in the battery pack is minimized. The proposed switch scheduling algorithm is summarized in Algorithm 4. The training process in the CM-DQN is shown in Figure 5.

5. Performance Evaluation

5.1. Simulation Environment

The BESS was performed via simulation using a lithium-ion battery pack model and was implemented in MATLAB and Simulink 2023b. To assess the performance of the proposed algorithm, we designed a $6\times 4$ parallel-series connected battery pack [15], listed in

Table 3. Initial SOHs of the cells and modules.

	Module 1	Module 2	Module 3	Module 4	Module 5	Module 6
$SO{H}_{i1}\left(0\right)$	90.00%	82.17%	79.23%	89.96%	86.72%	82.84%
$SO{H}_{i2}\left(0\right)$	86.77%	84.72%	88.16%	84.23%	83.46%	86.11%
$SO{H}_{i3}\left(0\right)$	84.13%	80.02%	87.35%	85.76%	79.98%	78.72%
$SO{H}_{i4}\left(0\right)$	78.15%	79.28%	81.19%	87.54%	81.75%	80.14%
$SO{H}_i\left(0\right)$	84.77%	81.55%	83.98%	86.87%	82.98%	81.95%
$SO{H}_{\mathcal{P}}\left(0\right)$	81.55%

, including 24 lithium-ion batteries with heterogeneous SOHs. The battery cell was modeled as a second-order Thévenin equivalent battery model with reductions in SOHs by utilizing a NASA dataset [34]. That dataset includes 28 lithium-ion cobalt oxide 18,650 cells with a nominal voltage of 3.7 V and nominal capacity of 2.2 Ah with a maximum current of 4 A, encompassing real-time measurements of terminal voltage, current, cell temperature, discharging capacity, and EIS impedance readings. We identified the EIS parameters, including $V_{ij}^O$, $R_{ij}^S$, $R_{ij}^{P1}$, $C_{ij}^{P1}$, $R_{ij}^{P2}$, and $C_{ij}^{P2}$, in the 90% to 60% SOH range by using the dataset. A dynamic power demand condition was used to evaluate the effectiveness of the algorithm. The power demand was obtained by generating values from a uniform distribution within 60 Wh to 100 Wh, while the terminal voltage was obtained by generating values from a uniform distribution ranging from 13.75 V to 16.25 V (at least four series-connected modules ON at a given time because the nominal voltage of each cell is 3.7 V). We set the load current for both discharging and charging the battery pack to 8 A (at least two parallel-connected batteries ON at a given time because the maximum output current of one battery is 4 A).

Two deep Q network architectures were constructed for cell scheduling and module scheduling, each including one input layer, two hidden layers, and one output layer. The number of hidden layers and the number of neurons in each hidden layer can be selected by trial and error [37]. To determine the optimal network size, including the number of hidden layers and the number of neurons in each layer, two approaches are commonly used: the constructive approach and the destructive approach [38]. We used a constructive approach to network sizing [39]. We started with a small network and gradually added neurons or hidden layers to improve the performance of the network, where the number of hidden layer neurons varied among 64, 128, and 256, and the number of hidden layers started from 1. Both deep Q network architectures with two 128-dimension hidden layers had the maximum cumulative reward with minimum episodes. For cell scheduling, the input layer included 11 neurons corresponding to the number of dimensions in state $s_i$ of module i with four parallel-connected cells. The output layer consisted of 12 neurons corresponding to possible actions, $a_i$, for cell switch scheduling. For module scheduling, the input layer included 15 neurons corresponding to the number of dimensions in state $s_{m+1}$ of the battery pack with six series-connected modules. The output layer consisted of 15 neurons corresponding to possible actions, $a_{m+1}$, for module switch scheduling. During learning, we set the learning rate $\alpha$ to 0.001, which helped reduce the loss function $\mathcal{L}\left({\varphi }_i\left(t\right)\right)$ episodes and balance the speed of learning with the stability of the training process. We set the discount factor $\gamma$ to 0.99 to effectively balance prioritizing long-term cumulative rewards, i.e., minimizing SOH reduction in the battery pack, while avoiding slowing down the convergence process. Other simulation parameters are summarized in

Table 4. Simulation parameters.

Parameter	Value
Number of battery cells	$6\times 4$
Battery type	Lithium-ion 3.7 V/2.2 Ah
Total capacity (new)	195.36 Wh
$I_{discharge}$	8 A
$I_{charge}$	−8 A
$I_{min}^{-},I_{max}^{+}$	−4 A, 4 A
$SO{C}_{min},SO{C}_{max}$	10%, 90%
$\eta$	1 (discharge)/0.98 (charge)
Total working time	1600 h
$\Delta t$	10 min
Hidden layers in each network	2
Neurons in each hidden layer	128
Neurons in input layer (cell scheduling)	11
Neurons in output layer (cell scheduling)	12
Neurons in input layer (module scheduling)	15
Neurons in output layer (module scheduling)	15
Learning rate ($\alpha$)	0.001
$ϵ$-greedy value	$0.9$
Discount factor ($\gamma$)	$0.99$
Period of target network update P	100 time slots

.

For the performance evaluation, we first investigated the effect of the proposed algorithm on SOH balancing to enhance the lifetime of the battery pack. Then, we evaluated the impact of the proposed algorithm on dynamic power demand. To validate the performance of the proposed cooperative multi-agent deep Q network (denoted as CM-DQN) algorithm, we compare it with previous works, including a self-X multicell battery algorithm [15] denoted as Self-X, a multilayer SOH equalizer [22] denoted as M-SOH, a DOD-SOH balancing algorithm [17] denoted as DOD-SOH, and a multi-actor–critic scheduling algorithm [25] denoted as M-A2C.

5.2. Capacity Balancing and Battery Pack Lifetime

We evaluated the performance of the proposed algorithm on balancing the SOH of modules, as shown in Figure 6. The SOH balancing of modules in the battery pack in all the algorithms is shown in Figure 6a–e. The proposed algorithm (CM-DQN) achieved more SOH balancing than previous work. During learning, the CM-DQN framework determines the optimal actions known as switches to connect or bypass cells and modules by observing BESS states such as the SOCs and SOHs of cells and modules, power demand $d_D\left(t\right)$, or available power supply $d_C\left(t\right)$. CM-DQN ensures that no single module degrades significantly faster than the others by fully observing the states in the BESS (including SOC, SOH of modules and cells, charging and discharging states, and the power demand). Figure 6f compares the standard deviation in SOHs among the modules over time for each algorithm. CM-DQN exhibited the lowest standard deviation (close to zero), compared to previous work, further affirming its ability to maintain SOH balance among modules. Self-X showed an SOH imbalance that was almost unchanged without considering SOHs. Other methods (M-SOH, DOD-SOH, and M-A2C), which considered SOHs, reduced the SOH imbalance but, without observations of the states in the BESS, the standard deviation was still higher than CM-DQN, indicating that SOHs between modules were not balanced until the end of the lifetime. With heterogeneous SOHs in the battery pack, CM-DQN offered better performance than other algorithms by balancing SOHs among the modules.

The performance of the proposed algorithm on battery pack lifetime, measured by minimizing SOH reduction, was evaluated and is shown in Figure 7. By balancing the SOHs among the modules in the battery pack, the proposed algorithm (CM-DQN) achieved the lowest SOH reduction in the battery pack compared to the other algorithms by minimizing the SOH reduction for each time slot, as shown in Figure 7a. The reduction in SOH under Self-X and M-SOH increased rapidly after 1200 h, compared to the other algorithms, due to the lack of consideration of both SOC and SOH. M-A2C had the closest performance to CM-DQN, but ignoring external systems information caused the SOH reduction, $\Delta SO{H}_{\mathcal{P}}\left(t\right)$, to increase faster as the battery lifetime weakened after 1250 h. The SOH reduction in the battery pack was minimized with CM-DQN, resulting in an extended battery pack lifetime, as shown in Figure 7b. The proposed algorithm CM-DQN achieved a longer battery pack lifetime compared to previous work. The SOH of the battery pack under CM-DQN reached 60% (the end of its second life) after a working time of 1558 h. By not observing the states in the BESS, all other work finished the second lifetimes significantly faster than CM-DQN.

Table 5. Battery pack lifetime improvement of the proposed algorithm compared with previous work.

Method	Self-X	M-SOH	DOD-SOH	M-A2C
Extended lifetime by CM-DQN	16.27%	14.14%	11.93%	4.49%

shows that the battery pack lifetime improvement of the proposed algorithm (CM-DQN) compared to previous work was up to 16.27%. Hence, the proposed algorithm can efficiently schedule switches in the battery pack to maximize the BESS lifetime and balance the SOHs among the modules and cells.

To show the impact of SOC balancing on the lifetime of the battery pack, the standard deviations of the SOCs in a battery pack under different algorithms are shown in Figure 8. Self-X, which only focused on SOCs, had better SOC balancing than other algorithms but stopped decreasing after some time and had a worse battery pack lifetime, as shown in Figure 7b. CM-DQN had a continuously decreasing standard deviation over time and it was lower than that of Self-X after 1130 h. Although the proposed algorithm (CM-DQN) focused on only balancing the SOHs of cells by observing SOCs, it also decreased the SOC imbalance among the cells in a battery pack and achieved a longer lifetime. As a result, scheduling cells and modules in a battery pack for SOH balancing is more important to extend the lifetime of the battery pack, as shown in Figure 6f, Figure 7 and Figure 8.

5.3. Performance of the Proposed Algorithm on Varied Demands

We studied the performance of the proposed algorithm in scenarios with different power demands, as shown in

Table 6. Different energy demand conditions.

Scenario	Energy Demand	Mean $\mathit{\mu }$	Variance ${\mathit{\sigma }}^2$
	U(75 Wh, 85 Wh) *		8.33
	U(70 Wh, 90 Wh)		33.33
1	U(60 Wh, 100 Wh)	80 Wh	133.33
	U(50 Wh, 110 Wh)		300
	U(40 Wh, 120 Wh)		533.33
	U(40 Wh, 80 Wh)	60 Wh
	U(50 Wh, 90 Wh)	70 Wh
2	U(60 Wh, 100 Wh)	80 Wh	133.33
	U(70 Wh, 110 Wh)	90 Wh
	U(80 Wh, 120 Wh)	100 Wh

* U(75 Wh, 85 Wh) was obtained by generating values from a uniform distribution across 75 Wh to 85 Wh.

. We considered various conditions in Scenario 1 with the same mean (80 Wh) and incremental variance (from 8.33 to 533.33), as shown in Figure 9a, and in Scenario 2 with the same variance (133.33) and incremental mean (from 60 Wh to 100 Wh), as shown in Figure 9b. Understanding the impact of demand variance is essential for evaluating the robustness and adaptability of battery management algorithms in real-world applications.

Figure 10 illustrates the impact of different load demand conditions, characterized by varying demand variance (${\sigma }^2$), on the lifetime of algorithms under the same mean demand. In Figure 10a, it is observed that, as the demand variance increased from 8.33 to 533.33, the standard deviation for all algorithms also increased. The proposed algorithm minimizes the SOH reduction in the battery pack at each time slot by balancing the SOHs of the battery modules. The charge and discharge control algorithm (Algorithm 3) and observation of dynamic demands (shown in state vectors in (36)) helped CM-DQN consistently achieve the lowest standard deviation, indicating robustness against fluctuations in demand. In contrast, the previous algorithms showed an increasing standard deviation proportional to the variation in power demand, indicating a lack of adaptation to demand dynamics. Figure 10b examines the lifetime obtained by the algorithms. The proposed algorithm (CM-DQN) demonstrated a significantly longer lifetime than the others, maintaining a robust performance in SOH balancing and extending lifetime even as demand variance increased. This suggests the proposed algorithm is more adaptive and effective in extending a battery’s operational lifetime under varying demand conditions. In summary, the proposed CM-DQN algorithm outperformed the others in robustness (lower standard deviation and longer lifetime) when subjected to increasing demand variance.

Figure 11 presents the performance of the algorithms under different mean demand conditions ($\mu$) while keeping the variance constant. Figure 11a shows the standard deviation of the algorithms’ performance as the demand mean increased from 60 Wh to 100 Wh. The CM-DQN algorithm consistently achieved the lowest standard deviation, indicating its robustness against variations in the demand mean. Figure 11b illustrates the lifetimes of the algorithms as the demand mean increased. The proposed CM-DQN algorithm exhibited the longest lifetime across all mean demand levels, demonstrating its efficiency in extending the battery lifetime under varying power demands. Overall, CM-DQN minimized the SOH reduction in the battery pack in handling different demand mean conditions, providing an extended battery lifetime and an SOH balancing among the modules.

6. Conclusions

In this paper, we proposed a battery management algorithm using a cooperative multi-agent deep Q network to maximize battery lifetimes for the battery pack in a BESS. The BESS consisted of a battery pack in which retired batteries with heterogeneous SOHs are connected in parallel and series. The proposed algorithm scheduled switches in the battery pack by connecting or bypassing battery cells and modules. The algorithm maximized the battery pack lifetime by reducing the SOH imbalance among cells and adapting to varied power demands. Via simulation, we showed that the proposed algorithm outperformed the other algorithms by attaining a more extended battery pack lifetime and reduced SOH imbalance among the cells and modules. The proposed algorithm extended the lifetime of the battery pack by up to 16.27% compared to the other algorithms.