Online Charging Strategy for Electric Vehicle Clusters Based on Multi-Agent Reinforcement Learning and Long–Short Memory Networks

Abstract

The electric vehicle (EV) cluster charging strategy is a key factor affecting the grid load shifting in vehicle-to-grid (V2G) mode. The conflict between variable tariffs and electric-powered energy demand at different times of the day directly affects the charging cost, and in the worst case, can even lead to the collapse of the whole grid. In this paper, we propose a multi-agent reinforcement learning and long-short memory network (LSTM)-based online charging strategy for community home EV clusters to solve the grid load problem and minimize the charging cost while ensuring benign EV cluster charging loads. In this paper, the accurate prediction of grid prices is achieved through LSTM networks, and the optimal charging strategy is derived from the MADDPG multi-agent reinforcement learning algorithm. The simulation results show that, compared with the DNQ algorithm, the EV cluster online charging strategy algorithm can effectively reduce the overall charging cost by about 5.8% by dynamically adjusting the charging power at each time period while maintaining the grid load balance.

1. Introduction

In today’s tight global energy supply and demand, countries around the world are striving to achieve carbon neutrality, and electric vehicles (EV) are rapidly expanding their market with their low pollution and noiseless features, but the huge user base overwhelms community home charging stations with limited community power supply capacity and insufficient power load. The traditional EV cluster charging strategy favors charging during low peak periods, but the electricity demand explodes at night when electric vehicles become popular, grid load peaks will be shifted, and electricity costs will increase exponentially. Traditional methods encourage users to use electricity through prices [1], but when large quantities of electricity are consumed, there will be an avalanche in the wholesale power market, and its flexibility is low. Many real time incentive schemes gradually emerged with the development of reinforcement learning (RL) [2,3], which hedges the avalanche risk in the grid market by RL and dynamically adjusts the incentive scheme in real time to induce the most favorable choice for the users. Vehicle-to-grid (V2G) [4] technology is proposed to form a certain buffer by feeding back to the grid through the EV’s power storage when the grid load is high, reaching a win–win situation for both users and the grid.

This paper proposes an online charging strategy that combines long and short-term memory networks (LSTM) and multi-agent reinforcement learning to solve the problem of smart charging of community home EV clusters under uncertain electricity prices. This framework mainly uses LSTM to predict the next electricity price to guide the charging piles to make the best charging and discharging decisions to minimize the charging cost and maintain a good load balance in the community grid. Nowadays, there are a variety of EV charging scheduling strategies; literature [5] proposes a nonlinear programming algorithm that includes energy price models to optimize EV charging for fleets, and literature [6] takes Monte Carlo sampling and proposes a solution which is infinitely close to the optimal one following the approximate dynamic programming idea of an infinite level dynamic program. However, such nonlinear programming and dynamic programming are too computationally large and too slow in the improvement in relative progress to satisfy only the charging scheduling in a specific environment. The literature [7] has developed a stochastic optimized dispatch model that combines energy retailers with local operators with a high degree of flexibility and has been tested in several scenarios with uncertain parameters, but with the popularity of the V2G model, the instability of grid-connected tariffs should be taken into account and should be combined with the V2G model to further reduce charging costs by selling excess electricity to local operators. In recent years, with the increasing maturity of annual machine learning techniques, we find that RL in the EV cluster charging strategy has started to become popular, and the literature [8] seeks the best charging strategy by controlling the charging of EVs throughout the charging station through a batch reinforcement learning approach, where the value network takes a fully connected neural network for fitting; conversely, the ability to handle high-dimensional data is poor when the smart body state parameters increase, and the charging characteristics of individual EVs are retained in the process of binning the total demand and the approximation error is introduced in the discretization process, which affects the learning speed of the whole algorithm. In the literature [9], a RL-based intelligent charging coordination system for EV fleets is proposed from the grid perspective, which keeps the grid facilities operating at low load by creating charging schedules for EVs 24 h in advance. At the same time, this system only provides charging strategies for individual EVs and does not propose a charging strategy that adapts to the global situation.

Smart charging strategies for electric vehicle clusters need to meet the charging demand of users while taking into account the grid load as well as the charging cost, where the changeable electricity price and uncertain usage time of users become a challenge. In the literature [10], a long–short memory network is used to improve the prediction accuracy of unstable tariffs, overcoming the challenges posed by realistic battery models of limited types and non-negligible uncertainties for parameters, but not taking into account the uncertainty of user usage. The literature [11] uses a feedforward neural network (NN) based on an extreme learning machine (ELM) for predicting uncertainty in electricity prices and electric vehicle commuting behavior by fitting the value function of the best action through a Q-network, while the ELM network has a slow learning rate, which is difficult to determine and prone to fall into local minima.

The main contributions of this paper are mainly as follows:

• We proposed a prediction strategy based on LSTM. This strategy can provide accurate prediction values for floating tariffs in the charging strategy algorithm and improves the accuracy of the algorithm greatly.
• We proposed a multi-agent RL-based online charging strategy algorithm for EV clusters, which takes into account uncertainties such as variable electricity prices and user usage.
• We designed an approach of centralized training and distributed execution by coordinating the charging strategies of each charging pile within a community home charging station to maintain a cooperative and competitive relationship between each charging post while globally controlling and coordinating the whole, minimizing user usage costs and maintaining load balancing on the community grid.

The remainder of this paper is as follows. Section 2 presents the specific design process of a multi-agent RL-based model for EV cluster charging strategy. The simulation results of the algorithm are discussed in Section 3. Finally, Section 4 provides the conclusion and follow-up work in the future.

2. Real-Time Grid Price Forecasting Model Based on Long and Short Memory Networks

2.1. RNN-Based LSTM Network

Recurrent neural networks (RNNs) [12] are neural networks that process data with certain sequences, which can allow the manipulation of sequences of input and output vectors to better process data with sequences relative to the traditional neural networks. RNNs are equipped with a memory module, through which information about the current state and the old state is continuously passed to the next stage; thus, enabling the prediction of the next state.

The RNN structure is shown in Figure 1, where $x$ is the input of sequence data in the current state, $y$ is the output of the current state, and $h$ is the memory module, which stores the output of the previous state and passes on the output of the current state.

LSTM [12] is a special form of RNN, which aims at solving the gradient disappearance as well as gradient explosion problems that occur during the training process of long sequence data. For the purpose of this paper, LSTM networks are well suited for long-term real-time price prediction.

The LSTM electric car price prediction model mainly includes an input layer, convolutional layer, fully connected layer, and output layer. It is calculated as follows:

$$c_t=\sigma (W_f[x_t,h_{t-1}])\odot c_{t-1}+\sigma (W_i[x_t,h_{t-1}])\odot \tanh (W[x_t,h_{t-1}])$$

(1)

$$h_t=\sigma (W_0[x_t,h_{t-1}])\odot \tanh (c_t)$$

(2)

$$y_t=\sigma (W^{\prime }{h}_t)$$

(3)

where $x_t$ is the feature vector of the input layer, $y_t$ is the output state vector, $h_{t-1}$ and $h_t$ are the states at the time of t−1 and t. $W_f, W_i, W_o$ are the weights of the forgetting gate, the selection gate, and the output gate, respectively, $W, W^{\prime }$ are the input and output weights, $\sigma$ and $\tan \mathrm{h}$ are the $sigmoid$ function and $\tan \mathrm{h}$ function, respectively.

The convolutional layer of the LSTM model consists of several memory gates, and its internal structure is shown in Figure 2, where $c_{t-1}$ is the memory state which has very small transition amplitude in the transfer process, $h_{t-1}$ is the hidden layer at time t−1, and $x_t$ is the input feature vector at time t. $h_{t-1}$ and $x_t$ pass together as inputs to the first forgetting gate, which has a selective memory function that chooses to remember the part of $c_{t-1}$ from the previous state that needs to be remembered; the second gate is a memory gate, where the input $x_t$ is memorized and subsequently added to the result calculated in the previous step to obtain the next memory state $c_t$; finally, after the output gate, the previous state is deflated through the tanh function to output $h_t$. $h_t$ is finally multiplied with the output weight $W^{\prime }$ to output $y_t$ through the $sigmoid$ function.

2.2. LSTM-Based Real-Time Grid Price Prediction Model

The LSTM-based real-time grid price prediction model proposed in this paper focuses on defining an LSTM model with 300 single hidden layers as a decoder that reads an input sequence and outputs a 300-element vector that captures the features in the input sequence. By taking as input the real-time charging demand quantity of EVs, then, a fully connected layer is used to interpret each time step in the output sequence, and the output layer predicts a step in the output sequence.

3. A Multi-Agent Reinforcement Learning-Based Model for Electric Vehicle Cluster Charging Strategy

3.1. Introduction to Multi-Intelligence Reinforcement Learning

Multi-agent reinforcement learning [13] is a sub-field of reinforcement learning, which is still in its infancy in various areas of artificial intelligence development today. Multi-agent reinforcement learning is the study of the interaction of multiple intelligent bodies in a common environment, which usually collaborate towards a final goal.

There are cooperative and competitive relationships when this intelligence interacts with the environment and each other, and their complexity increases exponentially as the number of intelligent bodies increases. The training framework of intelligent bodies can be divided into centralized training and distributed training, where a centralized training is coordinated by a central controller, and each intelligent body communicates with each other to update its strategy during the training period, but its strategy update speed is too slow and is mostly used in traffic signal control [14,15], multi-robot coordination [16], production scheduling [17], etc. In contrast, in distributed training, each intelligent body only focuses on its information for policy update during training, which is fast but less scalable and mostly used for distributed sensors [18]. The MADDPG algorithm used in this paper takes a centralized training distributed execution approach to speed up its policy updates while maintaining the independence and scalability of community charging stake policies.

3.2. Electric Vehicle Cluster Charging Strategy Model

3.2.1. Cluster Charging Behavior Analysis

In this paper, we focus on the charging strategy of community home electric vehicle clustering by dividing a day into 288 time periods with a 24 h cycle and a 5 min time window length and treating each charging pile as a single intelligent body to optimize the output power of the charging post for decision making when the charging pile is in working condition. The cluster charging cost consists of the user charging cost and the overall charging load cost of the whole charging station. Among them, the user charging cost includes the cost of power sold in V2G mode and the cost of the conventional use of power, expressed as follows:

$$C={\displaystyle \sum _{k=0}^N\mathrm{c}(i,k)Ep(i,k)P(i,k)}\Delta t$$

(4)

In Equation (4), $Ep\left(i,k\right)$ is the real-time tariff, $P\left(i,k\right)$ is the charging power of the charging post, $\Delta t$ is the time window, i.e., 5 min, $N$ is the total time period, and $c\left(i,k\right)$ is the charging status of the current time period, as follows:

$$\mathrm{c}(i,t)=\left\{\begin{array}{ccc}1,\hfill & & output\\ 0,\hfill & & done\\ -1,\hfill & & ch\arg ing\end{array}\right.$$

(5)

In Equation (5), $c\left(i,t\right)$ is equal to 1 if it is in charging state, −1 if it is selling power, and 0 if it is in standby. The overall charging load cost of a charging station $Q$ can be expressed as follows:

$$Q(t)={\displaystyle \sum _{k=0}^N\alpha Q_{burden}}(t)$$

(6)

In Equation (6), $\alpha$ is the load factor and is the grid load factor, in which $Q_{burden}$ is calculated as follows:

$$Q_{burden}(i,t)={\displaystyle \sum _t^{t+\Delta t}\frac{P(i,t)}{Sni\times \mathrm{Cos}\varphi 2\times 100\%}}$$

(7)

In Equation (7), $Sni$ is the rated power of the charging post, $\mathrm{Cos}\varnothing 2$ is the power factor of the secondary, and $P\left(i,t\right)$ is the output power.

3.2.2. EV Cluster Charging Strategy: Markov Decision Process

The biggest challenge of EV cluster charging [19] is to make charging choices for each charging stake while maintaining load balancing and minimizing user usage costs. Since there are different cooperative or competitive relationships between each charging stake, it is difficult for ordinary reinforcement learning to satisfy its conditions, while multi-agent reinforcement learning coordinates the strategies between each intelligence through a central processor, ensuring the independence of the intelligent body’s decisions while maintaining its cooperative communication capabilities.

As described above, a Markov decision model for EV cluster charging is as follows:

• State-space: The states of EV cluster charging can be divided into two groups: (1) single-intelligent charging state, which is primarily accountable for ensuring that individual EV charging is satisfied; (2) multi-agent charging state, which is a collective state and is mainly responsible for maintaining the overall state stability.
• As shown in Table 1, the EV battery power $SOC\left(i,t\right)$ is directly obtained when the charging post starts charging tasks, and the charging status $c\left(i,t\right)$ is updated at the same time. In order to further reduce the user’s usage cost and extend the battery life, our idea is to set the charging threshold $u\left(i,t\right)$ for EV charging during the daytime charging according to the past usage $SO{C}_{his}\left(i,t\right)$ of the community users and charge the remaining power during the low power consumption period. At the same time, there is a discharge threshold $v\left(i,t\right)$ in the V2G mode to ensure that users have electricity available when they use the car temporarily.

In addition to the four single-intelligent state features, there are two multi-agent charging states, tariff $ep\left(t\right)$ and area load $W_{burden}\left(t\right)$, where the area load is a real-time value that is a weighted average of the load factor of all charging piles, and the tariff is stored in historical data to continuously improve the neural network in Section 2.

Table 1. EV cluster charging state characteristics.

	Single Smart Body Charging Status	Multi-Agent Body Charging Status
Real-time data	$\mathrm{Amount}\mathrm{of}\mathrm{electricity}SOC\left(i,t\right)$	$\mathrm{Area}\mathrm{load}{W}_{burden}\left(t\right)$
	$\mathrm{Charging}\mathrm{status}c\left(i,t\right)$
	$\mathrm{Charging}\mathrm{threshold}\mathrm{status}{y}_u\left(i,t\right)$$\mathrm{Discharge}\mathrm{threshold}\mathrm{state}{y}_v\left(i,t\right)$
Historical data	$\mathrm{Historical}\mathrm{electricity}SO{C}_{his}\left(i,t\right)$	$\mathrm{Electricity}\mathrm{price}ep\left(t\right)$

Table 1. EV cluster charging state characteristics.

	Single Smart Body Charging Status	Multi-Agent Body Charging Status
Real-time data	$\mathrm{Amount}\mathrm{of}\mathrm{electricity}SOC\left(i,t\right)$	$\mathrm{Area}\mathrm{load}{W}_{burden}\left(t\right)$
	$\mathrm{Charging}\mathrm{status}c\left(i,t\right)$
	$\mathrm{Charging}\mathrm{threshold}\mathrm{status}{y}_u\left(i,t\right)$$\mathrm{Discharge}\mathrm{threshold}\mathrm{state}{y}_v\left(i,t\right)$
Historical data	$\mathrm{Historical}\mathrm{electricity}SO{C}_{his}\left(i,t\right)$	$\mathrm{Electricity}\mathrm{price}ep\left(t\right)$

• Action space: EV cluster charging action space $a\left(i,t\right)$ is the output power of each charging post.
• Reward: Reward provides delayed feedback to intelligence for its action decisions, enabling it constantly to optimize its strategies. Therefore, the reward function has a great influence on the optimization of intelligent decision-making. In the multi-agent reinforcement learning algorithm proposed in this paper, the reward function is set to a negative penalty function, which helps the rapid iterative convergence of the neural network.

The reward function can be divided into the charging cost function, charging power penalty, and charging power penalty function, where the charging cost reward function is expressed as follows:

$$r_{charge}(i,t)=-\lambda c(i,t){\displaystyle \underset{t}{\overset{t+\Delta t}{\int }}ep(t)P(i,t)dt}$$

(8)

In Equation (8), $\lambda$ is the discount parameter, $c\left(i,t\right)$ is the state of charge function, $ep\left(t\right)$ is the real-time tariff, and $P\left(i,t\right)$ is the charging power. The charging power reward function is as follows:

$$w_p(t)=\left\{\begin{array}{cc}\delta W_{burden}(t)\hfill & 75\%\le W_{burden}(t)\le 80\%\\ -\delta W_{burden}(t)\hfill & W_{burden}<75\%,W_{burden}(t)>80\%\end{array}\right.$$

(9)

In Equation (9), $\delta$ is the discount parameter, $W_{burden}\left(t\right)$ is the regional grid load factor, and is a weighted average of the load factors of all charging posts, calculated as follows:

$$W_{burden}(t)={\displaystyle \sum _{i=0}^N{Q}_{burden}(i,t)}$$

(10)

In Equation (10), $Q_{burden}\left(i,t\right)$ is the load factor of a single charging post. Finally, the charging power reward function:

$$w_s(i,t)=-\sigma {(1-SOC(i,t))}^2$$

(11)

In Equation (11), $\sigma$ is the power discount parameter, and the function mainly judges whether the charging is completed. Considering the uncertainty of user use scenarios, set the charging threshold $g$, the threshold limit function is as follows:

$$r_g(i,t)=\left\{\begin{array}{ll}\epsilon & c(i,t)>0,SOC(i,t)>80\%\\ -\epsilon & c(i,t)>0,SOC(i,t)<80\%\end{array}\right.$$

(12)

In Equation (12), $\epsilon$ is the threshold discount parameter, when the power is less than this threshold then the charging state is maintained and discharging is prohibited, when greater than this threshold then the discharging operation is allowed once the threshold is reached then the discharging is stopped. The final total reward function is shown as Equation (13):

$$R(i,t)=r_{charge}(i,t)+w_P(t)+w_s(i,t)+r_g(i,t)$$

(13)

• Observation value: To give the central processor a better grasp of the global information, an observation value is set for each intelligent body. The settings are as Equation (14):

$$o(i)=\{SOC(i,t),ep(t+1),c(i,t)\}$$

(14)

3.2.3. Multi-Intelligence Reinforcement Learning-Based Charging Strategy Process for Electric Vehicle Clusters

The pseudo-code of the multi-agent RL-based EV cluster charging strategy is shown in Algorithm 1, where a random process $N$ is initialized for the first time at the beginning of each training round for the intelligent body actions, and subsequently the initialized states are fed into the model. At the time of exploration, each smart body chooses the best action $a_{i,t}$ based on the current policy and the latest tariff $e{p}_t$ obtained from the LSTM-based tariff prediction network, and observes the reward $r_{i,t}$, the new state $s_{t+1}$, and the latest real-time tariff $E{p}_t$ after performing the action, and stores $\left(x_t,a_t,r_t,x_{t+1}\right)$ in the buffer pool $D$. The latest real-time tariff is returned to the LSTM network for storage and training. Next, the state $s_{t+1}$ is updated, and random samples are taken from the cache pool $D$ to update the critic network and the performer network. Finally, the target network parameters are updated for each intelligent body.

Algorithm 1: MADDPG Electric Vehicle Cluster Charging Strategy Process

1: for episode = 1 to $M$ do

2:          Initialize the random process $N$

3:          Accept initial state $s$

4:          for $t=1$ to maximum time period $n$ do

5:           Forecasting electricity prices $e{p}_t$ based on LSTM networks

6:           Select action $a_{i,t}$ according to current strategy

7:           Perform actions $a_{i,t}$, observe rewards $r_{i,t}$ and new states $s_{i,t+1}$

8:           Obtain real-time electricity prices $E{p}_t$ to return to LSTM

9:           network optimization

10:           Put $\left(s_{i,t}, a_{i,t}, r_{i,t},s_{i,t+1}\right)$ into the experience pool $D$

11:           $s_{i,t}=s_{i,t+1}$

12:           for agent i = 1 to $K$ do

13:                Draw samples $\left(s_{i,t}, a_{i,t}, r_{i,t},s_{i,t+1}\right)$ from experience pool $D$

14:                Follow the New Critics Network and the Performers Network

15:            end for

16:            Update the target network parameters of each charging post smartbody

17:        end for

18: end for

4. Algorithm Analysis

4.1. Analysis of LSTM-Based Electricity Price Prediction Algorithm

4.1.1. Data Description

In this paper, we use electricity price data from the Australian Energy Market (AEMO) [20], with data starting on 1 January 2022 and ending on 31 January 2022, with a data interval of 5 min, i.e., 288 data items for one day, and monthly electricity price data as shown in Figure 3.

4.1.2. Parameter Setting

The algorithm takes 7 days a week as a cycle, and a total of 2016 time period samples are included in a cycle, and the electricity price of the next time period is predicted by learning from the data of the first 7 days. As shown in Figure 4, a total of 8928 data is included, the first 6696 are training data and the last 2232 are test data set. A sliding window of size 2017 with step size 1 is adopted, in each window the first 2016 data are the training set and the last one is the validation set. The learning rate in LSTM algorithm is set to 0.001 and the number of training rounds is 300.

4.1.3. Analysis of Results

The results of the LSTM-based electricity price prediction algorithm are shown in Figure 5 below, from which it can be seen that the LSTM network has a very small error in predicting the electricity price with an accuracy of 90.12%, while the traditional ARIMA prediction algorithm predicts the results as shown in Figure 6 with an accuracy of 89.57%. From the error comparison graph of the two algorithms in Figure 7, we can easily see that the blue dashed LSTM-based forecasting algorithm has a smoother error compared to the orange dashed ARIMA algorithm. Additionally, as shown in Algorithm 1, the charging strategy algorithm obtains the predicted electricity price at the current time by LSTM every time period and returns it to the LSTM network store for continuous training after obtaining the current actual electricity price in order to improve the accuracy of the algorithm.

4.2. Analysis of a Cluster Charging Strategy Algorithm for Community Home Electric Vehicles Based on Multi-Agent Reinforcement Learning

4.2.1. Experiment Description

This experiment mainly simulates the charging scenario of a community home charging station, setting 12 charging piles of rated power of 37.5 KW and battery capacity of 60 Kwh for electric vehicles, and in order to reduce the number of training rounds, the charging piles are taken to work continuously and uninterruptedly for 24 h. Specifically, the initial charging battery power of each charging stake is initialized at the beginning of each round of training, and the uncertainties of the noise simulation real charging scenario are added in each charging time step, and the charging battery power is initialized when one of the charging stakes completes the charging task, and the charging task is continued until the end of 288 time steps of the day.

4.2.2. Parameter Setting

To verify the algorithm of this paper the following experiments are conducted and the simulation environment is shown in

Table 2. Simulation environment configuration.

Experimental Configuration	Specific Parameters
CPU	Intel Core i7-11700K@ 5.0 GHz
GPU	NVIDIA GeForce GTX 1660 Super
Memory	16 GB
Operating System	Windows10
Programming Environment	python3.7, pytorch 1.10.2 + cu102

. In this algorithm, the target policy network learning rate is ${10}^{-4}$, and the value network learning rate is ${10}^{-4}$. Combining the data in Figure 3 and Figure 5, we can see that the average electricity price in January is AUD 75, so we set the charging cost discount parameter $\lambda$ to 0.1, so that the average charging cost $r_{charge}$ is 0.625; from Equation (9), we can see that the grid load rate $W_{burden}$ is maintained between 0 and 1, so the solid charging power discount parameter $\delta$ is 1; from the charging power reward function $w_s\left(i,t\right)$, we can see that $w_s\left(i,t\right)\approx {10}^{-2}$, so the charging power discount parameter $\sigma$ is 10; finally, the threshold discount parameter $\epsilon$ is 0.5.

4.2.3. Analysis of Results

To demonstrate the performance of the proposed EV cluster charging strategy algorithm, we next discuss the simulation results for 1 February 2022, in detail. The average execution time of the algorithm proposed in this paper is 1.884 milliseconds, which has a strong real-time performance and meets the requirements of industrial applications. As shown in Figure 8, the blue line shows the electricity price for each period on 1 February and the green bar shows the output power of the charging post during the period. It is obvious that the electricity price fluctuates a lot around 13:00 and 18:00, which are the peak hours of electricity consumption, and the charging strategy successfully avoids the high charging cost during this period and reverses the output voltage to sell to the grid for a lower charging cost.

In terms of load, the dynamic power load versus the rated power load is shown in Figure 9, where the dynamic power load is lower than the rated power load for most of the period, with an average load factor of 68% throughout the day, which is 32% lower than the rated power. Compared with the single strategy of the DQN algorithm, this algorithm adopts a centralized training and distributed execution method, and each charging stake maintains both cooperative communication and policy independence, which can better adjust its strategy to achieve the overall load balancing of the community home charging stations. As shown in Figure 10, Figure 11 and Figure 12, MADDPG greatly reduces the load pressure on the community grid and helps the community grid to operate efficiently and smoothly.

The cost of charging is another key indicator to verify the performance of the charging strategy. From Figure 13, Figure 14 and Figure 15, we can see that the cost of electricity of the proposed algorithm in each period is lower than that of a single power charging strategy. It is easy to see that the reinforcement learning combined with V2G technology significantly reduces the user’s electricity cost, and the MADDPG algorithm is more flexible than the DQN algorithm for each charging station charging strategy, which, to a large extent, can effectively reduce the use cost.

5. Conclusions

In this paper, a cluster online charging strategy algorithm for a community household electric vehicle was proposed. The algorithm, combined with multi-agent reinforcement learning and short and long memory network, aims to reduce the load of regional power grid, expand the operation scope of local operators to achieve better energy scheduling, and reduce the charging cost of community users to achieve a win–win situation among three parties. In particular, in the context of today’s volatile energy prices, the algorithm predicts the next stage of energy prices through a long–short memory network and combines a multi-agent reinforcement learning algorithm with a centralized training and distributed execution model to maintain communication between intelligence while ensuring their decision independence. Simulation results show that this algorithm, combined with the V2G model, can effectively avoid peak consumption periods and reverse power sales to the grid, which greatly reduces the cost of use for users. In the future, the recommendation algorithm for a user-oriented charging strategy will be researched to guide users to avoid peak usage periods through incentive strategies to further reduce charging costs.