Smart Carbon Emission Scheduling for Electric Vehicles via Reinforcement Learning under Carbon Peak Target

Abstract

Electric vehicles (EVs) have become popular in daily life, which influences carbon dioxide emissions and reshapes the curves of community loads. It is crucial to study efficient carbon emission scheduling algorithms to lessen the influence of EVs’ charging demand on carbon dioxide emissions and reduce the carbon emission cost for EVs coming to the community. We study an electric vehicle (EV) carbon emission scheduling problem to shave the peak community load and reduce the carbon emission cost when we do not know future EV data. First, we investigate an offline carbon emission scheduling problem to minimize the carbon emission cost of the community by predicting future data with regard to incoming EVs. Then, we study the online carbon emission problem and propose an online carbon emission algorithm based on a heuristic rolling algorithm. Furthermore, we propose a reinforcement learning smart carbon emission algorithm (RLSCA) to achieve the dispatching plan of the charging carbon emission of EVs. Last but not least, simulation results show that our proposed algorithm can reduce the carbon emission cost by $21.26\%$, $16.60\%$, and $8.72\%$ compared with three benchmark algorithms.

1. Introduction

With the increasing awareness of the need for environmental protection and the support of national policy, people are more willing to use electric vehicles (EVs). Chinese national policy has also contributed to the rapid adoption of EVs. Different from the community load, EVs have intermittent charging demands, which is a remarkable feature [1]. For the intermittent characteristic of EVs and unknown future charging demand, it is crucial and challenging to design online electric vehicle (EV) carbon emission scheduling strategies to shave the peak load [2]. According to the IPCC report on climate change, it will become carbon neutral around 2050 if the temperature rise is less than 1.5 degrees Celsius. To achieve carbon peak and neutrality, we need optimal scheduling methods and green technologies to control carbon emissions. In this work, we study an online carbon emission scheduling algorithm under the constraint of carbon peak.

There has been a large body of literature on the EV charging scheduling and the design of EV charging algorithms. Mou et al. [3] proposed a dynamic wireless charging system powered by wind energy with traffic flow-based charging demand prediction to cut down carbon emissions. Hanahi et al. [4] investigated optimal charging problems for EVs supported by on-route public charging stations to reduce carbon emissions. For the dynamic charging demand, some online EV charging problems with carbon emission are studied in the following works. A linear function approximator is proposed in [5] for the charging scheduling state-value function to optimize pricing strategies for a public EV charging station. Ren et al. [6] studied a low-carbon schedule model with the constraints of carbon emission costs and rights to reduce the total system carbon emissions. The references [3,4,5,6] discussed the EV charging scheduling problem with the target of reducing carbon emissions, but did not emphasize the EV carbon emission scheduling problem. We formulate our EV carbon emission model from these works.

The log-mean divisia index method is utilized in [7] to analyze the influence of influencing factors to reduce road transport carbon emission in eight policy scenarios. Vehicle carbon emissions were calculated in [8] based on monthly average energy consumption and carbon emissions of electricity generation. The carbon emission monetary value method is utilized in [9] to convert the carbon emission reduction problem into a cost optimization problem, to better evaluate energy transition. An online EV energy schedule model with an evolutionary method was proposed in [10] to manage the energy storage. Yang et al. [11] proposed a layered transactive energy model to coordinate EV charging demand and energy-resources generation in distributed power systems. Masoum et al. [12] proposed an online EV charging fuzzy coordination algorithm to reduce the total energy cost and power losses. Jian et al. [13] proposed a valley-filling policy for large-scale EVs charging problem. Dong et al. [14] developed a charging price strategy based on a model of EV mobility for EV fast charging stations. Wang et al. [15] proposed a dynamic traffic assignment model to optimize the spatiotemporal distribution of EV flows on roads to enhance renewable energy source absorption and reduce carbon emissions. Bilh et al. [16] proposed an EV charging algorithm and aimed to utilize the flexibility of the load of EVs to flatten the power fluctuation caused by wind energy. The studies [14,15,16] are dependent on specific models and scenarios, which are not completely applicable to the carbon emission scenario. We formulate our EV online carbon emission model from these works. We aim at developing a less model-dependent online carbon emission algorithm and we utilize the model-free method to design the optimal EV carbon emission strategy. To our knowledge, this paper is the first work to propose an EV online carbon emission model with the constraint of dynamic charging demand.

There are some works using reinforcement learning (RL) methods to solve the scheduling problems about EVs. A deep q-network-based energy and emission management strategy and two distributed deep RL algorithms for for hybrid electric vehicles were proposed to enhance training efficiency [17]. A double-Q learning-based arbitrage strategy is proposed in [18] to reduce the operation costs. Standard Q-learning methods have discrete charging action values, which may not achieve the optimal plan over practical continuous action space. Therefore, in our work, we don’t discretize the decisions and aim at achieving the optimal EV charging carbon emission dispatching plan with continuous scheduling action. We utilize the actor–critic (AC) method [19] with continuous scheduling action to solve the practical scheduling problem with continuous states and actions. Some works on the EV charging scheduling problem are studied with the AC method. An online incentive RL-based algorithm was proposed in [20] to schedule electricity to shave the peak and enhance the reliability of power system. Bahrami et al. [21] developed an online scheduling algorithm based on the AC method, which can lower the peak-to-average ratio and users’ cost. A policy-gradient method with continuous actions is proposed in [22] to solve the resource allocation problem. which is not completely applicable to the carbon emission scenario. Reinforcement learning algorithms in the studies [17,18,19,20,21] are not completely applicable to the continuous EV charging schedule in the carbon emission scenario. Then, we design an improved AC algorithm to achieve a continuous EV charging carbon emission value. A deep reinforcement learning method was utilized in [23] to solve the optimal dispatch problem of electricity–gas systems to improve the scheduling efficiency. This work is the research closest in nature to our own. To our knowledge, this work is the first one to design an EV carbon emission problem based on an improved AC algorithm to achieve a continuous charging emission under carbon peak.

We aim to study a carbon emission scheduling algorithm for EVs arriving in a community; by this algorithm, we can dispatch the EVs in response to day-ahead price. The scheduling of EV-charging carbon emissions can result in the reduction of the system cost and the fluctuation of total load. We design the carbon emission schedule as a Markov decision process, considering uncertainty about the price and loads in the community. We first propose an offline EV carbon emission scheduling problem to minimize the carbon emission cost by knowing the future data. Then, we work on the online carbon emission problem and apply a heuristic rolling algorithm to solve this problem. Furthermore, we propose a smart charging carbon emission scheduling algorithm via the improved AC method to achieve an optimal EV charging action, which has a good convergence [24]. Finally, we compare the average cost per EV of the RLSCA algorithm with three benchmark algorithms. The contributions of our paper are summarized by the following three points.

• A carbon emission scheduling model that considers the random arrival of EVs with power constraints is formulated to minimize the carbon emission cost in accord with the community’s known EV profiles. Our model is practical and easily adaptable to different scenarios.
• We propose an online carbon emission scheduling algorithm based on a heuristic rolling algorithm by making carbon emission decisions according to the current EV profiles, such as arrival time and state of charge. This serves as a baseline by which to evaluate our proposed learning-based algorithm.
• This work proposes a reinforcement learning-based smart charging carbon emission scheduling algorithm (RLSCA) via the asynchronous advantage AC method. The simulation results show that the RLSCA algorithm can reduce the expected carbon emission cost of EVs by $21.26\%$, $16.60\%$, and $8.72\%$, compared with three benchmark algorithms, respectively.

This work contains the following five parts. Section 2 studies the offline carbon emission scheduling problem. Section 3 reformulates the offline carbon emission schedule to an online carbon emission scheduling problem. Section 4 studies and analyzes the reinforcement learning algorithm RLSCA. Section 5 shows simulation performance and analysis. Section 6 shows the conclusion and shows the future work.

2. Offline Carbon Emission Scheduling Problem

In this part, we study the optimal offline carbon emission scheduling problem without knowing the future data ahead.

2.1. System Model

From Figure 1, we study a community carbon emission model with enough charging piles. We need to minimize the carbon emission cost of this community. In particular, we study the charge and discharge scheduling of electric vehicle battery in a time horizon of T with the same intervals—that is, $t=1,2,\cdots ,T$. Without loss of generality, we denote that EVs come to the community in the order from 1 to N. We denote the state of charge (SoC) for EV k as $So{C}_{k,t}$ at time slot t. The initial SoC of EV k is $So{C}_{k,0}$. The expected SoC at the departure time for EV k is $So{C}_{k,end}$. The arrival time of EV k is denoted as $t_k^{arr}$. The departure time of EV k is set as $t_k^{dep}$. We set the charging demand of EV k as $D_k$. The profiles $D_k,t_k^{arr}$ and $t_k^{dep}$ will be known to the community when EV k arrives in an offline scheduling problem. The expected SoC value should be reached before the EV leaves.

According to the practical constraints of battery, the carbon emission of EV k according to the charging energy is set as $b_k\left(t\right)$ and the carbon emission per time slot has a range limit

$$\begin{array}{c}\hfill b_k\left(t\right)\in [0,b_{k,max}],\end{array}$$

(1)

where $b_{k,max}$ is the upper limit of EV k carbon emission in each time slot. We have the battery capacity of EV k as $B_{k,max}$ and denote $\mathcal{G}\left(t\right)$ as EVs set in the community at time slot t. $\mathcal{U}\left(t\right)$ is the time set from t to $\overline{t}$, where $\overline{t}$ is the maximum EV leaving time in $\mathcal{G}\left(t\right)$,

$$\begin{array}{c}\hfill \mathcal{U}\left(t\right)=\left\{\overline{t}\right|\overline{t}\ge t\&\overline{t}\le max\left\{t_k\right|k\in \mathcal{G}\left(t\right)\}\}.\end{array}$$

(2)

We explain the concept of $\mathcal{G}\left(t\right)$ and $\mathcal{U}\left(t\right)$ by the example shown in Figure 2. There are five EVs in the community in this example. At time slot 4, there are four EVs in the community (that is, ${\mathcal{G}}_4=\{2,3,4,5\}$) and in this time slot, the maximum service time of EVs parking in the community is from 4 to 9 (that is, ${\mathcal{U}}_4=\{4,5,6,7,8,9\}$) and EV 5 is the last one that leaves the community in time slot 9. Our community can control the charging rate $b_k\left(t\right)$. We denote the charging load of the station $w_{ce}\left(t\right)$ at time slot t,

$$\begin{array}{c}\hfill w_{ce}\left(t\right)=\sum _{k\in \mathcal{G}\left(t\right)}{b}_k\left(t\right).\end{array}$$

(3)

We denote the inelastic base load of the community as $w_b\left(t\right)$, such as washing machine or lighting, which fits a certain distribution. We have the total community load $W\left(t\right)$ at time slot t,

$$\begin{array}{c}\hfill W\left(t\right)=w_{ce}\left(t\right)+w_b\left(t\right).\end{array}$$

(4)

The total community load $W\left(t\right)$ has an upper bound $L_{max}$, and the unit price can be formulated as a linear function of the load (that is, ${\xi }_0+2{\xi }_{1}z$), where ${\xi }_0$ and ${\xi }_1$ are nonnegative constants. We can get the following carbon emission cost $c\left(t\right)$ of the community,

$$\begin{array}{cc}\hfill c\left(t\right)& ={\int }_{w_b\left(t\right)}^{W\left(t\right)}({\xi }_0+2{\xi }_{1}z)dz\hfill \\ \hfill & ={\xi }_0(W\left(t\right)-w_b\left(t\right))+{\xi }_1\left[{\left(W\left(t\right)\right)}^2-{\left(w_b\left(t\right)\right)}^2\right]\hfill \\ \hfill & ={\xi }_0{w}_{ce}\left(t\right)+{\xi }_1\left[w_{ce}{\left(t\right)}^2+2w_{ce}\left(t\right)w_b\left(t\right)\right]\hfill \\ \hfill & ={\xi }_0\sum _{k\in \mathcal{G}\left(t\right)}{b}_k\left(t\right)+{\xi }_1{\left[\sum _{k\in \mathcal{G}\left(t\right)}{b}_k\left(t\right)\right]}^2+2{\xi }_1{w}_b\left(t\right)\sum _{k\in \mathcal{G}\left(t\right)}{b}_k\left(t\right),\hfill \end{array}$$

(5)

where the carbon emission cost $c\left(t\right)$ is the integral of the price ${\xi }_0+2{\xi }_{1}z$. Let $\mathbb{J}$ denote the index set of the time intervals from 1 to T, and let $e_j(j\in \mathbb{J})$ denote the length of jth interval. Then we define the total electricity bill paid by the community as $C\left(t\right)$ during the period T in the following equation:

$$\begin{array}{c}\hfill C\left(t\right)=\sum _{t=1}^{T}c\left(t\right).\end{array}$$

(6)

2.2. Problem Formulation

We then formulate the carbon emission scheduling problem to minimize the total carbon emission cost in the following expression,

$$\begin{array}{cc}\hfill & \underset{b_k\left(t\right)}{min}\sum _{t\in \mathbb{J}}\sum _{k\in \mathcal{G}\left(t\right)}\left[{\xi }_0{b}_k\left(t\right)+2{\xi }_1{b}_k\left(t\right)w_b\left(t\right)\right]+{\xi }_1{\left[\sum _{k\in \mathcal{G}\left(t\right)}{b}_k\left(t\right)\right]}^2{e}_j\hfill \end{array}$$

(7a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\sum _{t_k^{arr}}^{t_k^{dep}}{b}_k\left(t\right)e_j=D_k/\eta ,k=1,2,\dots ,N\hfill \end{array}$$

(7b)

$$\begin{array}{cc}\hfill & 0\le b_k\left(t\right)\le b_{k,max}\hfill \end{array}$$

(7c)

$$\begin{array}{cc}\hfill & 0\le W\left(t\right)\le L_{max},\hfill \end{array}$$

(7d)

where $\eta$ is the conversion coefficient between electricity consumption and carbon emissions. If $w_b\left(t\right),t_k^{arr},t_k^{dep},b_{k,max}$, and $D_k$ are known in advance, we can achieve the optimal carbon emission schedule $b_k\left(t\right)$ by solving the problem (7). Then, we need to reformulate the offline problem without knowing the future profiles of an online carbon emission problem.

3. Online Carbon Emission Scheduling Algorithm

In this section, we design a heuristic online algorithm (OA) in a rolling fashion. We study the carbon emission scheduling without considering any demand from arriving EVs. We assume that the inelastic load in the community will not change in each time slot and in practice we can set the time interval to be smaller.

Figure 3 shows an illustrative example. An EV comes to the community in the time slot $t_s$. We set $\mathcal{U}\left(t_s\right)$ as the set of interval indices at time slot $t_s$ and $e_j\left(t_s\right)$ is the interval from $t_s$ to $j\in \mathcal{U}\left(t_s\right)$, which is the length of the jth interval. For instance, in Figure 3, $\mathcal{U}\left(t_s\right)=\{t_3^{dep},t_1^{dep},t_2^{dep}\}$ and $e_1\left(t_s\right)=t_3^{dep}-t_s$, $e_2\left(t_s\right)=t_1^{dep}-t_3^{dep}$, $e_3\left(t_s\right)=t_2^{dep}-t_1^{dep}$. We define $\mathcal{G}(j,t_s)$ as the set of EVs that arrive at time slot $t_s$ and stay in the community at the interval j, $j\in \mathcal{U}\left(t_s\right)$. We set the set of time interval indices that EV k stay at the community as $\mathcal{Q}(k,t_s)$. The charging rate of EV k is set as $b_k\left(t\right)$, $t\in \mathcal{U}\left(t_s\right)$, which is constant in each time interval. Then, an online carbon emission optimization problem is formulated as follows:

$$\begin{array}{cc}\hfill & \underset{b_k\left(t\right)}{min}\sum _{t\in \mathcal{U}\left(t_s\right)}\left[\sum _{k\in {\mathcal{G}}_{j,t_s}}\left({\xi }_0{b}_k\left(t\right)+2{\xi }_1{b}_k\left(t\right)w_b\left(t_s\right)\right)+{\xi }_1{\left(\sum _{k\in {\mathcal{G}}_{j,t_s}}{b}_k\left(t\right)\right)}^2\right]e_j\left(t_s\right)\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\sum _{\mathit{j}\in \mathcal{Q}(\mathit{k},{\mathit{t}}_{\mathit{s}})}{\mathit{b}}_{\mathit{k}}\left(\mathit{j}\right){\mathit{e}}_{\mathit{j}}\left({\mathit{t}}_{\mathit{s}}\right)={\mathit{D}}_{\mathit{i}}\left({\mathit{t}}_{\mathit{s}}\right),\mathit{k}\in {\mathcal{G}}_{{\mathit{t}}_{\mathit{s}}}\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill & 0\le b_k\left(t\right)\le b_{k,max},k\in {\mathcal{G}}_{t_s},t\in \mathcal{Q}(k,t_s)\hfill \end{array}$$

(8c)

$$\begin{array}{cc}\hfill & 0\le W\left(t\right)\le L_{max}.\hfill \end{array}$$

(8d)

The community will stick with the optimal carbon emission schedule to Problem (8) if there are no arriving EVs. When the base load in the community changes, the system will update $w_b\left(t_s\right),{\mathcal{G}}_{j,t_s},\mathcal{Q}(k,t_s),D_i\left(t_s\right)$ and resolve Problem (8).

4. Smart Charging Carbon Emission Scheduling Algorithm

The OA algorithm tackles the problem with the heuristic rolling algorithm without considering any demand from EVs that may arrive in the future time slots. This algorithm executes the online scheduling in a rolling manner according to the current profile of the community. This results in poor performance because it neglects the incoming EVs that bring stochastic charging demand. Therefore, we seek a more effective charging algorithm. Reinforcement learning aims to find an optimal policy to make the agent obtain the maximum reward by taking the action according to the current state. Sarsa [25] and Q-learning [26] estimate the future reward of a policy by utilizing temporal difference iteration and then choose the discrete action drawn from the highest reward. However, it is difficult to set the convincing discrete action. Therefore, we utilize the AC approach to solve the online carbon emission schedule to choose the continuous charging action. The AC method will learn the optimal policy and the value function, which includes two parts: the actor part and the critic part. The actor part is designed to define a parameterized policy and achieve the corresponding carbon emission action for EVs. The critic is to evaluate and criticize the current policy by the rewards according to the action of the state. Unlike the reward function in the policy gradient method, the convergence of the AC algorithm is not ideal [24] because it tries to evaluate the policy similar to that used to generate the data, and the approximate value of state is self-generated. Therefore, an asynchronous advantage AC approach [24] is applied to solve the convergence of AC algorithm in our problem.

4.1. RL Framework Formulation

We dispatch the carbon emission in a time window of T and denote the state space as $\mathsf{\Phi }$ where the state is $\phi \left(t\right)$. The carbon emission state $\phi \left(t\right)\in \mathsf{\Phi }$ contains the unit price and the percentage of the battery capacity $So{C}_i\left(t\right)$ of EV k, which is shown as follows,

$$\begin{array}{c}\hfill \phi \left(t\right)=(So{C}_1\left(t\right),\cdots ,So{C}_N\left(t\right),w_b\left(t\right)),\end{array}$$

(9)

where $w_b\left(t\right)$ is defined as community energy demand in Section 2. In practice, the charging process is a continuous state, and we know that the probability of staying in a specific state approaches zero. We utilize the EV carbon emission cost to design the following reward function $r\left(t\right)$,

$$\begin{array}{c}\hfill r\left(t\right)=-[{\xi }_0+2{\xi }_{1}a\left(t\right)+2{\xi }_1{w}_b\left(t\right)]a\left(t\right).\end{array}$$

(10)

Then, we set the following state-action function $Q^{{\mathsf{\Theta }}_{\epsilon }}(\phi \left(t\right),a\left(t\right))$ to estimate the expected accumulated rewards of current community carbon emission state,

$$\begin{array}{c}\hfill Q^{{\mathsf{\Theta }}_{\epsilon }}(\phi \left(t\right),a\left(t\right))={\mathbb{E}}_{{\mathsf{\Theta }}_{\epsilon }}\left\{\sum _{k=0}^{\infty }{ϵ}^{k}r(t+k+1)\right|\phi \left(t\right),a\left(t\right),{\mathsf{\Theta }}_{\epsilon }\},\end{array}$$

(11)

where the discount factor is $ϵ\in (0,1)$.

4.2. The Actor Part

The state-action function is the expected rewards according to current state. We give the following parameterized policy to generate the carbon emission action

$$Q^{\mathsf{\Theta }}(\phi \left(t\right),a\left(t\right))=\mathbb{E}\{r\left(t\right)+\gamma Q^{\mathsf{\Theta }}(\phi (t+1),a(t+1))\},$$

(12)

where $\mathbb{E}\{·\}$ is the anticipation function, $\gamma$ is the discount factor to estimate the action decisions, and $\mathsf{\Theta }$ can be estimated by ${\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))$. The gradient policy ${\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))$ is differentiable in $\epsilon$, and we give the gradient of $\epsilon$ in the following expression,

$$\begin{array}{c}\hfill \Delta \epsilon ={\beta }_a{\nabla }_{\epsilon }\mathcal{J}\left({\mathsf{\Theta }}_{\epsilon }\right)={\beta }_a\frac{\partial \mathcal{J}\left({\mathsf{\Theta }}_{\epsilon }\right)}{\partial {\mathsf{\Theta }}_{\epsilon }}\frac{\partial {\mathsf{\Theta }}_{\epsilon }}{\partial \epsilon },\end{array}$$

(13)

where ${\beta }_a$ is the actor learning rate. We need to minimize ${\beta }_a$ to avoid the policy oscillation [24]. The Gaussian distribution is widely used in the actor part based on the maximum entropy principle [27]. Then, we design the following parameterized policy to generate continuous carbon emission action,

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))=\frac{1}{\sqrt{2\mathsf{\Theta }}\sigma }{e}^{[-{(a\left(t\right)-\kappa \left(\phi \left(t\right)\right))}^2/2{\sigma }^2]},\end{array}$$

(14)

where $\kappa \left(\phi \right(t\left)\right)$ is the action of the carbon emission state, and $\sigma$ is the standard deviation of all the probable carbon emission actions. The parameter $\kappa \left(\phi \right(t\left)\right)$ is defined as follows,

$$\begin{array}{c}\hfill \kappa \left(\phi \left(t\right)\right)=\sum _{j=1}^n{\epsilon }_j{x}_j\left(\phi \left(t\right)\right),\end{array}$$

(15)

where $x_j\left(\phi \left(t\right)\right)$ is the feature vector in state $\phi \left(t\right)$. ${\mathsf{\Theta }}_{\epsilon }\left(\phi \left(t\right)\right)$ shows the probability of choosing action $a\left(t\right)$ in state $\phi \left(t\right)$. We optimize the policy ${\mathsf{\Theta }}_{\epsilon }$ to make carbon emission decisions.

We use the AC approach to solve the EV carbon emission problem of continuous states and actions. It is impossible to enumerate and store the whole value functions in a lookup table for each state-action couple. Therefore, the target of the AC approach is to find an optimal strategy ${\mathsf{\Theta }}_{\epsilon }$ by which to maximize the following function,

$$\begin{array}{cc}\hfill \mathcal{J}\left({\mathsf{\Theta }}_{\epsilon }\right)& =\mathbb{E}\left\{Q^{{\mathsf{\Theta }}_{\epsilon }}(\phi \left(t\right),a\left(t\right))\right\}\hfill \\ \hfill & ={\int }_{\mathsf{\Phi }}{d}^{{\mathsf{\Theta }}_{\epsilon }}\left(\phi \left(t\right)\right){\int }_{a\left(t\right)}{\mathsf{\Theta }}_{\epsilon }\left(a\left(t\right)\right|\phi \left(t\right))Q^{{\mathsf{\Theta }}_{\epsilon }}(\phi \left(t\right),a\left(t\right))da\left(t\right)d\phi \left(t\right),\hfill \end{array}$$

(16)

where $d^{{\mathsf{\Theta }}_{\epsilon }}\left(\phi \right)$ is the function of strategy ${\mathsf{\Theta }}_{\epsilon }$. We need to improve the parameters of policy ${\mathsf{\Theta }}_{\epsilon }$ iteratively.

4.3. The Critic Part

We estimate the value function by the function approximation, where the parameter vector $w={(w_1,w_2,\cdots ,w_n)}^T$,

$$Q_w(\phi \left(t\right),a\left(t\right))\approx Q_{\epsilon }^{\mathsf{\Theta }}(\phi \left(t\right),a\left(t\right)).$$

(17)

We apply the linear combination method to approximate the value function because it has a good function approximation and an easy-to-show gradient. Here, we have

$$\begin{array}{cc}\hfill & Q_w(\phi \left(t\right),a\left(t\right))=w^T\Psi (\phi \left(t\right),a\left(t\right))=\sum _{k=1}^n{w}_i{\psi }_i(\phi \left(t\right),a\left(t\right)),\hfill \end{array}$$

(18)

where we define the basis function vector $\Psi \left(\phi \right(t),a(t\left)\right)$ for EV carbon emission action $a\left(t\right)$ at state $\phi \left(t\right)$ in time slot t, as follows,

$$\begin{array}{cc}\hfill & \Psi (\phi \left(t\right),a\left(t\right))={({\psi }_1(\phi \left(t\right),a\left(t\right)),{\psi }_2(\phi \left(t\right),a\left(t\right)),\cdots ,{\psi }_n(\phi \left(t\right),a\left(t\right)))}^T,\hfill \end{array}$$

(19)

where $\Psi \left(\phi \right(t),a(t\left)\right)$ is the vector of features visible in state $\phi \left(t\right)$ taking carbon emission action $a\left(t\right)$, which is created by tile coding [19] and combined linearly with the parameter vector ${\psi }_i(\phi \left(t\right),a\left(t\right))$. Any of the methods in the linear model can guarantee that the algorithm can converge to the local optimum or the global optimum [19]. We apply the temporal difference error to update the critic value function, which shows the error between true value and approximated value. We set temporal difference (TD) error as follows,

$$\begin{array}{cc}\hfill e_t& =r(t+1)+ϵQ_w[\phi (t+1),a(t+1)]-Q_w(\phi \left(t\right),a\left(t\right)),\hfill \end{array}$$

(20)

where $r(t+1)+ϵQ_w[\phi (t+1),a(t+1)]$ is the actual return value of slot t. The critic parameters are updated in the following expression,

$$\begin{array}{c}\hfill \Delta w={\beta }_c{e}_t{\nabla }_w{Q}_w(\phi \left(t\right),a\left(t\right)),\end{array}$$

(21)

where ${\beta }_c$ is the critic learning rate and ${\beta }_c$ will cause the oscillation if ${\beta }_c$ is too big, but it will take a long time to achieve the convergence if ${\beta }_c$ is too small.

4.4. Smart Charging Carbon Emission Scheduling Algorithm

The critic part updates the value function and its parameters. Then, the actor part utilizes the critic’s output to update its policy parameters. The TD error can indicate whether the learning process is becoming better or worse than we expect because the learning process is positively correlated with the critic output. We aim to reduce the TD error, which can adjust the actor part and the critic part in the gradient direction. The TD error is generated according to the current state, the next state, and the current reward in the critic part, and then this TD error will help update the policy parameter with the current EV carbon emission action and state. The critic part estimates the strategy from the state-action function $Q_w(\phi \left(t\right),a\left(t\right))$. The actor part utilizes the policy gradient in the following expression, where we utilize a as an abbreviation for charging action $a\left(t\right)$,

$$\begin{array}{cc}\hfill & {\nabla }_{\epsilon }\mathcal{J}\left({\mathsf{\Theta }}_{\epsilon }\right)\approx {\int }_{\mathsf{\Phi }}{d}^{{\mathsf{\Theta }}_{\epsilon }}\left(\phi \left(t\right)\right){\int }_a{Q}^{{\mathsf{\Theta }}_{\epsilon }}(\phi \left(t\right),a){\nabla }_{\epsilon }{\mathsf{\Theta }}_{\epsilon }\left(a\right|\phi \left(t\right),\epsilon )dad\phi \left(t\right).\hfill \end{array}$$

(22)

We use the Gaussian policy to achieve continuous charging action values. Because ${\nabla }_{\epsilon }{\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))={\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right)){\nabla }_{\epsilon }$ln${\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))$, we have

$$\begin{array}{c}\hfill {\nabla }_{\epsilon }{\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))={\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))\frac{\left(a\right(t)-\varrho (\phi \left(t\right)\left)\right)\Gamma \left(\phi \right(t\left)\right)}{{\sigma }^2}.\end{array}$$

(23)

We update the actor parameter $\epsilon$ and critic parameter w at the same time. We update the actor parameter $\epsilon$ in the direction determined by the critic output. However, unlike the reward function in the policy gradient method, the convergence of the AC algorithm is not ideal because of its on-policy search method [19], which tries to evaluate the same policy as that used to generate the data. However, the approximate value of state in the AC algorithm is self-generated. To enhance the convergence of the AC algorithm, we utilize the asynchronous advantage AC algorithm to dispatch EV carbon emission actions, and we give the complete RLSCA algorithm in Algorithm 1.

Algorithm 1 Reinforcement learning-based smart charging carbon emission scheduling algorithm (RLSCA)

Input: Discount factor $ϵ$, critic learning rate ${\beta }_c$, actor learning rate ${\beta }_a$, strategy ${\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))$, $a\sim N(\varrho \left(\phi \left(t\right)\right),{\sigma }^2)$  1: Init: Thread step operator $t=1$, global shared operator $k=0$. Set parameter $\epsilon ={\epsilon }_0$.  2: For each thread do  3: For each step do  4: Reset gradients: $d{\epsilon }^{\prime }=0$, $d{\omega }^{\prime }=0$.  5: Update thread parameters $\epsilon ={\epsilon }^{\prime }$, $\omega ={\omega }^{\prime }$  6: Set $t_{start}=t$, get state $\phi \left(t\right)$  7: Repeat  8: Generate carbon emission action $a(t+1)\sim {\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))$, move to the state $\phi (t+1)\sim P\left(\phi \right(t),a(t),\phi (t+1\left)\right)$, achieve the reward $r(t+1)$.  9: Update $t\leftarrow t+1,k\leftarrow k+1$ 10: Until$t-t_{start}=t_{max}$ 11: Critic: 12: Update: $\Psi (\phi \left(t\right),a\left(t\right))={\nabla }_{\epsilon }$ ln ${\mathsf{\Theta }}_{\epsilon }(\phi \left(t\right),a\left(t\right))$ 13: Update: $Q_w(\phi (t+1),a(t+1))=w^T{\nabla }_{\epsilon }$ ln ${\mathsf{\Theta }}_{\epsilon }(\phi (t+1),a(t+1))$ 14: Update critic parameters: $w_{t+1}=w_t+{\beta }_c{e}_{t}I$, where $e_t$ is updated by (20) 15: Actor: 16: Update the policy parameter: ${\epsilon }_{t+1}={\epsilon }_t+{\beta }_a{e}_t{\nabla }_{\epsilon }J\left({\mathsf{\Theta }}_{\epsilon }\right)$ 17: Update: $\phi \left(t\right),a\left(t\right),z\left(t\right),Q_w(\phi \left(t\right),a\left(t\right))\leftarrow Q_w(\phi (t+1),a(t+1))$ 18: end 19: Update ${\epsilon }^{\prime }$ using $d{\epsilon }^{\prime }$ and of ${\omega }^{\prime }$ using $d{\omega }^{\prime }$ according to (24) and (25). 20: Until$k>{\xi }_{max}$ 21: End

RLSCA has the policy $\mathsf{\Theta }\left(a_t\right|{\phi }_t;\epsilon )$ and the approximate state function $Q({\phi }_t;\omega )$ and adopts the multi-step returns [19] to update the strategy and the value function after every $t_{max}$ actions. We update the critic parameters in the following expression,

$$\begin{array}{c}\hfill d{\omega }^{\prime }=d{\omega }^{\prime }+\partial {\left(R-Q({\phi }_t;\omega )\right)}^2/\partial \omega ,\end{array}$$

(24)

and we update the actor parameters in the following expression,

$$\begin{array}{c}\hfill d{\epsilon }^{\prime }=d{\epsilon }^{\prime }+{\nabla }_{\epsilon }log\mathsf{\Theta }\left(a_t\right|{\phi }_t;\epsilon )(R-Q({\phi }_t;\omega )).\end{array}$$

(25)

The cumulative reward is $R(t+1)=r\left(t\right)+ϵR\left(t\right)$ and the immediate reward is $r\left(t\right)=-[{\xi }_0+2{\xi }_{1}a\left(t\right)+2{\xi }_1{w}_b\left(t\right)]a\left(t\right)$. Each thread has an agent that works in a replicated environment. Each step generates a gradient of parameters.

5. Simulation

In this part, we estimate the performance of three benchmark algorithms and RLSCA algorithm by utilizing the community load profiles. For our simulation, the base load profile and the statistical EV data, which includes EV arrival distribution and the energy level at their arrival, are given and known ahead of time.

5.1. Simulation Parameter

We utilize the base load profile for two days from [28]. We obtain the statistical EV data in [29], where Figure 4 shows the EV arrival distribution. The initial SoC of an EV’s battery influences the load profile of the community. The initial SoC is estimated based on urban dynamometer driving schedule [30]. According to measured statistics from [31], the probability density function of the EVs’ battery SoC is shown in Figure 5. We consider two types of EVs—Type 1, the maximum charging rate of which is $3.2$ kW with battery capacity 36 kWh, and Type 2, the maximum charging rate of which is $1.4$ kW with battery capacity 16 kWh.

5.2. Simulation Performance

The simulation performance of the RLSCA algorithm is compared with the online optimal charging algorithm, the eagerly charging algorithm, and the reinforcement learning adaptive energy management algorithm. Here are the three algorithms’ benchmarks.

(1)

OA: The EV k selects the optimal carbon emission $b_k^{*}\left(t\right)$ from the community, which is the optimal solution of Problem 8. We set the carbon emission cost by OA as ${\Theta }_{OA}$.

(2)

EC [32]: The EV k selects the maximum carbon emission $b_{max}$ for every time slot in the community. We set the carbon emission cost by EC as ${\Theta }_{EC}$.

(3)

RLAEM [33]: The EV k selects the carbon emission $b_k^{*}\left(t\right)$ from the community solved by the adaptive Q-learning algorithm, which is an adaptive reinforcement learning algorithm. We set the carbon emission cost by RLAEM as ${\Theta }_{RL}$.

The AEM algorithm is based on the Q-learning method, which has discrete charging action values which may not achieve the optimal plan over practical continuous action space. It is impossible to enumerate and store the whole value functions in a lookup table for each state-action couple. In our work, we don’t discretize the decisions and aim at achieving the optimal EV charging carbon emission dispatching plan with continuous scheduling action. The RLSCA algorithm is based on the AC approach to solve the EV carbon emission problem of continuous states and actions. The RLSCA algorithm is applicable to other places and scenarios if the base load profile and statistical EV data are given. The OA algorithm is simulated by CVXPY [34] which is a Python toolbox for convex optimization problems. We compare four algorithms in a case study with 40 Type-1 EVs over 48 h, and show the optimal charging results of four algorithms in a case study with 40 Type-1 EVs over 48 h in Figure 6. The EV axis shows the index of EV, the time slot axis shows the index of time slot, and the z axis shows the total load of the community. The fluctuation of total community load of RLSCA algorithm is less than the EC, OA, and RLAEM algorithms. From Figure 6, we see that RLAEM and RLSCA charge EVs more smoothly than EC and OA and shave the total peak load together with the base load. Among all tested algorithms, the RLSCA algorithm achieves the best performance in terms of minimizing the total peak load. We show the carbon emissions from EV charging in Figure 7. We present four simulation results of 40 EVs and show the total load including the EV load and the inelastic load of the community in Figure 8. Our RLSCA algorithm has the lowest total load peak. The carbon emission peaks of EC, OA, RLAEM, and RLSCA algorithms are $67.51$ kg, $60.34$ kg, $60.37$ kg, and $56.94$ kg. The carbon emission peak of RLSCA algorithm is 15.66%, 5.63%, and 5.68% less than the carbon emission peak of EC, OA, and RLAEM algorithms. The fluctuation of carbon emission peak of RLSCA algorithm is less than the EC, OA, and RLAEM algorithms. We also calculate the carbon emission costs for four simulated algorithms and show the results in Figure 9. The total carbon emission costs of EC, OA, RLAEM, and RLSCA algorithms in this system are $\$20.6$, $\$19.45$, $\$17.77$, and $\$16.22$, and the carbon emission costs of the EC, OA, and RLAEM algorithms in this system are $21.26\%$, $16.61\%$, and $8.72\%$ higher than that of our RLSCA algorithm. Similarly, for another type of EV in which the maximal charging rate is $1.4$ kW and cell capacity is $36,16$ kWh, the total carbon emission costs of the EC, OA, RLAEM, and RLSCA algorithms are $\$14.98$, $\$13.50$, $\$12.43$, and $\$11.32$, and the carbon emission costs of the EC, OA, and RLAEM algorithms are $24.43\%$, $16.15\%$, and $8.93\%$ higher than that of our RLSCA algorithm, respectively.

In addition, RLSCA algorithm can reduce EVs’ expected cost by $16.61\%$, compared with the OA algorithm. In Figure 9, we compare the electricity bill of EVs with two types of EVs in the EC, OA, RLAEM, and RLSCA algorithms. The discount factor $ϵ$ is a crucial operator that can cut down on the TD error in the critic part. For Type-1 EV, the total EV carbon emission costs of EC, OA, RLAEM, and RLSCA algorithms are $\$21.32,\$18.95,\$18.01,$ and $\$17.32$, and the EV carbon emission costs of the EC, OA, and RLAEM algorithms are $23.09\%,9.41\%,$ and $3.98\%$ higher than the cost of our RLSCA algorithm. For Type-2 EV, the total EV carbon emission costs of EC, OA, RLAEM, and RLSCA algorithms are $\$15.58,\$13.95,\$11.95,$ and $\$11.31$, and the EV carbon emission costs of the EC, OA, and RLAEM algorithms are $37.75\%,23.34\%,$ and $5.66\%$ higher than the cost of our RLSCA algorithm, respectively.

From Figure 10, we can conclude that the discount factor has an important impact on the convergence of our algorithm, and we need to find a low discount factor. We simulate our algorithm with discount factors $0.005,0.01,$ and $0.05$, and we can find that a lower discount factor will get a better performance. Considering the computational efficiency of the algorithm, we can select $ϵ=0.01$ to get a trade-off between the reward and convergence, achieving both reasonably fast convergence and high reward. The actor learning rate ${\beta }_a$ is a crucial parameter in the actor part. From Figure 11, the actor learning rate will have an important impact on the convergence of our algorithm and the update of policy. A larger actor learning rate results in large overshoots of the rewards and the state reward depends on the actor learning rates. A large actor learning rate (e.g., 0.0005 and 0.001) leads to big overshoots in the evolution of rewards, and the steady-state reward depends on the configuration of the actor learning rates as well. From the simulation results in Figure 11, we can see that ${\beta }_a=0.0001$ has the better convergence performance and the higher reward.

Last but not least, we give the results of average carbon emission costs for all the simulated algorithms in Figure 12. From Figure 12, for 40 EVs, carbon emission costs per EV of EC, OA, RLAEM, and RLSCA are $\$0.5447,\$0.5129,\$0.4756,$ and $\$0.4372$. The carbon emission costs per EV of RLSCA are $24.59\%,17.31\%,$ and $8.78\%$ less than those of EC, OA, and RLAEM. For 50 EVs, carbon emission costs per EV of EC, OA, RLAEM, and RLSCA are $\$0.5946,\$0.5573,\$0.5168,$ and $\$0.4607$. The carbon emission costs per EV of RLSCA are $29.06\%,20.97\%,$ and $12.18\%$ less than those of EC, OA, and RLAEM. The average costs of four algorithms steadily rise with the increase of the number of EVs. Our RLSCA algorithm has the lowest average cost among the tested algorithms and the effectiveness of the RLSCA algorithm is verified in the tested scenarios with a different number of electric vehicles.

6. Conclusions

In this work, we study an offline carbon emission scheduling problem, which minimizes the carbon emission cost of a community without future data. Then, we modify the offline EV carbon emission problem as an online problem, and we propose the OA algorithm based on a heuristic rolling algorithm. Furthermore, we propose the RLSCA algorithm based on the AC method to update the continuous EV carbon emission action. In our algorithm, the actor part has parameterized policy, and we can update the policy parameter with a policy gradient. The critic part uses the TD error about the state and action to update the critic parameter, which helps the actor part to optimize the policy. Simulation results showed that our RLSCA algorithm is lower by $21.26\%$$16.60\%$, and $8.72\%$ than the EC, OA and RLAEM algorithms, respectively, while having a good convergence. The average cost per EV of the RLSCA algorithm is the lowest among all the simulated algorithms. In future work, we will formulate the carbon emission of renewable energy as a negative cost function or a reward function and study the transactive carbon emission schedule problem with renewable energy according to the transactive energy framework under carbon peak.