Blockchain-Assisted Secure Energy Trading in Electricity Markets: A Tiny Deep Reinforcement Learning-Based Stackelberg Game Approach

Abstract

Electricity markets are intricate systems that facilitate efficient energy exchange within interconnected grids. With the rise of low-carbon transportation driven by environmental policies and tech advancements, energy trading has become crucial. This trend towards Electric Vehicles (EVs) is bolstered by the pivotal role played by EV charging operators in providing essential charging infrastructure and services for widespread EV adoption. This paper introduces a blockchain-assisted secure electricity trading framework between EV charging operators and the electricity market with renewable energy sources. We propose a single-leader, multi-follower Stackelberg game between the electricity market and EV charging operators. In the two-stage Stackelberg game, the electricity market acts as the leader, deciding the price of electric energy. The EV charging aggregator leverages blockchain technology to record and verify energy trading transactions securely. The EV charging operators, acting as followers, then decide their demand for electric energy based on the set price. To find the Stackelberg equilibrium, we employ a Deep Reinforcement Learning (DRL) algorithm that tackles non-stationary challenges through policy, action space, and reward function formulation. To optimize efficiency, we propose the integration of pruning techniques into DRL, referred to as Tiny DRL. Numerical results demonstrate that our proposed schemes outperform traditional approaches.

1. Introduction

The electricity market is a structured marketplace where electricity is traded, aiming to ensure the efficient allocation and use of electrical resources to maintain a balance between supply and demand. Key participants in this market include generators, transmission companies, distribution companies, Load-Serving Entities (LSEs), and end users [1]. Price signals within the electricity market are essential, as they incentivize the optimal use and dispatch of electricity resources. The market operates through various segments, notably the day-ahead market, where electricity is traded a day in advance based on forecasts, and the real-time market, which addresses immediate imbalances in electricity demand and supply [2]. Moreover, intraday markets offer additional flexibility by allowing market participants to electricity closer to the time of delivery, further enhancing the market’s ability to respond to unforeseen changes in demand or supply [3]. These mechanisms collectively contribute to the reliable and cost-effective delivery of electricity to consumers, facilitating the integration of renewable energy sources and supporting the overall stability of the power grid.

Electricity trading is a key strategy for achieving low-carbon transportation and offers additional benefits such as improving urban air quality and reducing environmental pressures [4]. This dual benefit motivates both national and local governments to take more decisive actions. In recent years, stronger environmental protection policies and significant reductions in technology costs have solidified commitments from governments and automakers toward the development of Electric Vehicles (EVs). These developments indicate that EVs are poised to become the mainstream choice for future transportation [5]. EV charging operators play a crucial role in the growing EV ecosystem, offering charging facilities and services that are vital for widespread EV adoption. As EV usage increases, these operators become key players in the electricity market. Beyond providing charging services, they actively participate in energy trading, mainly electricity trading, through the use of smart grid technologies [6]. EVs can function as mobile energy storage units, charging during periods of low electricity demand and prices and discharging back to the grid when demand and prices are high. This bidirectional energy flow helps optimize electricity distribution, enhance grid stability, and maximize economic benefits. The integration of vehicle-to-grid technology further enhances this capability by enabling more efficient energy management and supporting the overall stability and efficiency of smart grids [7].

Currently, power trading faces several key challenges that hinder its efficiency and reliability. First, there are inadequate incentive mechanisms in environments with incomplete information. Without proper incentives, market participants may be reluctant to provide reliable and accurate electricity resources, leading to inefficiencies. This issue is compounded by the lack of transparency and trust between stakeholders, which can further discourage active and honest participation in the market [8]. Secondly, the dynamic nature of the trading environment presents significant difficulties. Traditional methods often fail to achieve optimal trading strategies in real time due to rapid price fluctuations and the complex requirements of demand response. These methods are generally not equipped to handle the high volatility and the swift changes in supply and demand, which are characteristic of modern electricity markets. As a result, there is a pressing need for the development of more flexible and efficient technological solutions. Advanced approaches such as machine learning and Deep Reinforcement Learning (DRL), have shown promise in this regard. These technologies can adapt to changing market conditions and optimize trading strategies in real time, thereby ensuring more effective market operations [9,10].

The electricity market is essential for the optimal allocation of electrical resources, ensuring a balance between supply and demand. EV charging operators enhance this process by integrating smart grid technologies and actively participating in the electricity market, thereby promoting efficient electricity utilization. To overcome the challenges posed by incomplete information and dynamic trading environments, it is crucial to continuously innovate and improve market mechanisms and technological solutions [11]. These advancements are vital for maintaining the efficient operation of the electricity market and supporting the green transition of energy systems. In the context of emerging technologies and market dynamics, continuous innovation and the development of advanced market mechanisms are crucial. For instance, integrating Demand Response (DR) strategies and employing advanced algorithms, such as those based on game theory and DRL, can significantly enhance market efficiency and reliability. By leveraging these technologies, the market can better accommodate the variability of renewable energy sources and ensure a more resilient and adaptive power system.

Therefore, to address the challenge of ensuring that electricity markets provide real and reliable resources, we propose a Stackelberg game. This game-theoretic approach effectively structures interactions between market participants, promoting optimal decision making and efficient resource allocation. Furthermore, we integrate DRL with pruning techniques to solve the model efficiently. This combination enables dynamic adaptation to changing market conditions, significantly enhancing the security and efficiency of electricity resource provision. The main contributions of this paper are summarized as follows:

• We introduce a blockchain-assisted secure electricity trading framework that facilitates transactions between EV charging operators and the electricity market. Central to this framework is an aggregator that leverages blockchain technology to securely record and manage these transactions. By employing blockchain, we ensure the integrity and security of electricity trading operations.
• To address the pricing challenges within the electricity market, we propose a single-leader, multi-follower Stackelberg model involving the electricity market and EV charging operators. Here, the electricity market assumes the role of the leader, establishing the selling price of electric energy units. EV charging operators, as followers, adjust their resource demand strategies based on the pricing set by the market leader. This model aims to optimize resource allocation and pricing decisions within the system.
• Recognizing the computational complexity associated with training traditional DRL models, we present a Tiny DRL algorithm that integrates pruning techniques with DRL methodologies. This novel approach enhances computational efficiency while aiming to achieve Stackelberg equilibrium. By combining pruning techniques with DRL, our algorithm efficiently navigates complex and dynamic environments, ultimately improving performance in reaching the desired equilibrium state.

The rest of this paper is organized as follows: Section 2 reviews the related work and introduces the combination of DRL with pruning techniques. In Section 3, we introduce the system model considering electricity trading between EV charging operators and the electricity market. In Section 4, we introduce the single-leader, multi-follower Stackelberg game model between EV charging operators and the electricity market in detail. In Section 5, we propose a Tiny DRL algorithm to find the Stackelberg equilibrium. The numerical results of the proposed scheme are shown in Section 6. Section 7 concludes the paper.

2. Related Work

In this section, we review several related works, with a focus on reliable energy trading in electricity markets. Ensuring reliable energy trading is crucial for maintaining grid stability and optimizing resource allocation. Therefore, compared to traditional schemes, blockchain technology is employed to enhance the security of transactions in this paper, safeguarding the integrity and transparency of the process. Furthermore, advanced DRL methods incorporating pruning techniques are utilized to optimize strategic bidding, energy trading, and load management. By removing less significant neurons or parameters from the network, these techniques enable more efficient decision making, leading to faster convergence and more robust learning outcomes.

2.1. Reliable Energy Trading in Electricity Markets

EVs possess dual attributes as both electrical loads and power sources. They play a crucial role in creating a safe, economical, and environmentally friendly intelligent power system. EVs significantly contribute to the solving of transportation, energy, and environmental challenges by reducing greenhouse gas emissions, enhancing energy efficiency, and supporting grid stability through the use of smart charging and vehicle-to-grid technologies [12]. Integrating EVs into smart grids and urban infrastructure not only mitigates pollution but also fosters the development of sustainable and resilient energy systems. Therefore, numerous scholars have undertaken extensive and in-depth research on the integration of EVs into the electricity market [13,14,15,16]. The authors of [13] proposed a joint demand response and energy trading model for electric vehicles in off-grid microgrid systems, optimizing transaction prices through a broker-led Stackelberg game approach. The results demonstrated that this model achieves up to 25.8% lower transaction prices compared to existing markets while maintaining high power reliability, showcasing its suitability for isolated microgrid environments. The authors of [14] presented a Peer-to-Peer (P2P) local electricity market model that integrates both energy and uncertainty trading to enhance the reliability of energy trading in electricity markets, particularly with the incorporation of EVs. The model significantly improves the local balancing of photovoltaic forecast errors by matching forecast power with time-flexible demand and uncertain power with power-flexible demand. The authors of [17] presented a data-driven probabilistic evaluation method for determining the hosting capacity of hydrogen fuel cell vehicles, incorporating a directional mapping approach, a probabilistic model considering high-dimensional uncertainties, and a cross-term decoupled polynomial chaos expansion for efficient computation. The authors of [15] introduced a decentralized Quality of Service (QoS)-based system for P2P energy trading among EVs, leveraging smart contracts to ensure reliable and resilient transactions without a third party. By employing QoS attributes and a fuzzy-based approach, the system effectively matches energy providers and consumers while implementing penalties to maintain contract integrity, thereby enhancing reliability in electricity markets. The authors of [18] presented a comprehensive analysis of the application and evolution of cooperative, non-cooperative, and evolutionary game theory within the electricity market. They examined the effects of these game theory models on the power generation, power sale, and power consumption sectors, with a particular focus on energy trading. Additionally, the study assessed the current status and scale of electricity markets, both domestically and internationally, providing insights and prospects for future research and the application of game theory in this domain.

2.2. Blockchain-Based Energy Trading in the Electricity Market

With the exponential increase in data volume and the inherent value of these data, transactions within the electricity market are encountering a critical demand for enhanced security measures [19,20,21]. For example, the authors of [22] presented FedPT-V2G, a federated transformer learning approach for real-time vehicle-to-grid dispatch that addresses non-IID data issues and data privacy concerns through the use of proximal algorithms and transformer models, achieving performance comparable to that of centralized learning in both balanced and imbalanced datasets. The adoption of blockchain technology also represents a viable solution for the establishment of trustworthiness and ensuring the continuity of secure transactions within the electricity market. By leveraging blockchain for secure storage and management, a decentralized system can be established that guarantees data integrity through encryption protocols, ensuring transparency and robust security throughout the entire process [23]. The authors of [19] reviewed the role of blockchain technology, combined with smart contracts, in facilitating peer-to-peer energy trading among prosumers, highlighting its potential to reshape the energy sector, the challenges it faces, emerging start-ups, and its application in EV charging. The authors of [20] proposed a novel blockchain-based distributed community energy trading mechanism designed to optimize energy trading efficiency and security in the context of shifting from consumers to producers. The authors of [21] pointed out that the security characteristics of blockchain technology can improve the efficiency of energy transactions and establish the basic stability and robustness of the energy market, e.g., the electricity market, and also reviewed the basic characteristics of blockchain and energy markets. In conclusion, the pivotal role of blockchain technology in energy trading, particularly electricity trading, is increasingly recognized by scholars, as evidenced by the growing body of research in this area. Therefore, in this paper, blockchain technology is utilized to enhance the security of the transaction process, underscoring its significance in ensuring the integrity and trustworthiness of energy transactions compared to current electricity trading methods.

2.3. Deep Reinforcement Learning with Pruning Techniques

DRL combines the advantages of deep learning and reinforcement learning, enabling the creation of algorithms that dynamically interact with and adapt to their environment. By employing privacy-preserving techniques, DRL algorithms iteratively learn and optimize decision making while safeguarding sensitive information. In the context of Stackelberg games, which involve leader–follower dynamics and strategic decision making, participants might be hesitant to disclose too much information due to competition or security concerns. DRL is essential for effectively reaching equilibrium solutions in such settings, as it allows agents to learn optimal strategies through interaction without requiring full disclosure of private information [24]. This capability is particularly beneficial in applications like security games, energy trading, and multi-agent systems, where balancing strategic advantage and information privacy is crucial [25].

However, training DRL models is resource-intensive in terms of computing power and storage. To address the need for more efficient DRL models in specific scenarios, researchers have increasingly adopted pruning techniques to optimize and enhance DRL performance [24,26,27,28,29]. For example, the authors of [29] introduced a novel multi-agent deep reinforcement learning method for urban distribution network reconfiguration, incorporating a “switch contribution” concept to reduce the action space, an improved QMIX algorithm for policy enhancement, and a two-stage learning structure with reward sharing to improve learning efficiency, which was validated through numerical results on a 297-node system. Pruning techniques are mainly divided into structured pruning and unstructured pruning [30]. Structured pruning involves the removal of entire components of a neural network, e.g., layers, neurons, or channels [24]. With the pruning of these larger structures, the shape of the model changes, leading to a more streamlined and often faster-to-execute network. Unstructured pruning, also known as magnitude pruning, targets individual parameters or weights within the neural network [30]. It removes weights that have the smallest magnitude, resulting in a sparse network. Pruning techniques have emerged as a promising approach to compress DRL models and improve algorithm efficiency, with an increasing amount of research focused on integrating pruning techniques with DRL. The authors of [26] proposed a novel model compression framework for DRL models using a sparse regularized pruning method and policy-shrinking technology, achieving a balance between high sparsity and compression rate. The authors of [27] proposed a compact DRL algorithm that leverages adaptive pruning and knowledge distillation to achieve high long-term transaction efficiency and lightweight routing for payment channel networks in resource-limited Internet of Things (IoT) devices. Simulation results show that the algorithm significantly outperforms baseline methods. The authors of [24] proposed a Tiny Multi-Agent DRL (Tiny MADRL) algorithm to facilitate the efficient migration of Unmanned Aerial Vehicle Twins (UTs) in Unmanned Aerial Vehicle (UAV) metaverses. By using pruning techniques, the algorithm reduces the network parameters and computational demands, optimizing Roadside Unit (RSU) selection and bandwidth allocation for seamless UT migration.

3. System Model

Decarbonizing transportation is crucial for climate change mitigation. With the increasing supply of renewable energy, governments are actively promoting the electrification of vehicle fleets [31]. Figure 1 shows the proposed blockchain-assisted secure electricity trading between EV charging operators and the electricity market with renewable energy sources. We provide more details of the system model as follows:

• EV Charging Operator: EV charging operators are responsible for managing and operating charging stations where EV owners can recharge their vehicles [32]. They ensure the availability, functionality, and efficiency of charging infrastructure. These charging operators purchase electricity from different kinds of electricity markets to supply their charging stations, maintaining a reliable energy source for EVs.
• Aggregator: Traditionally, the aggregator purchases time-varying electricity from the power grid and sells it to traditional users [33]. In this paper, we consider the aggregator responsible for managing electric energy trading between the EV charging operator and electricity markets. Specifically, the aggregator utilizes blockchain technology to securely record and verify energy trading transactions [34], which ensures the transparency, traceability, and efficiency of electric energy trading between the EV charging operator and electricity markets [34]. Note that the Practical Byzantine Fault Tolerance (PBFT) consensus algorithm is used in the blockchain system to achieve lightweight consensus. The incorporation of blockchain technology into energy trading enhances security, transparency, and traceability, surpassing the capabilities of traditional electricity market trading mechanisms [21]. This advancement empowers EV charging operators to make well-informed and optimized operational decisions, thereby ensuring the efficiency and reliability of electric energy trading processes.
• Electricity Markets: Electricity market operators facilitate the buying and selling of electrical energy, with EV charging operators participating by purchasing the electricity needed to supply their stations. These markets—especially those incorporating renewable energy resources—regulate prices by continuously adjusting them based on the supply-and-demand dynamics of EV charging operators.

4. Stackelberg Model for Electric Energy Trading

In this section, we consider that one electricity market and a set ($\mathcal{M}=\{1,\dots ,m,\dots ,M\}$) of M EV charging operators participate in electric energy trading.

During electric energy trading, the electricity market is the sole electricity resource, and EV charging operators rely on electricity resources provided by the electricity market to supply energy for EVs. Since electric energy trading between the electricity market and EV charging operators is an incomplete process [35], a monopoly market is formed [36]. Specifically, the electricity market operates as a monopoly with the authority to regulate electricity, while market supply and demand drive price adjustments. Electric vehicle charging operators must decide how much electricity to purchase based on the prevailing prices. If prices are low, EV charging operators may buy more energy to ensure a reliable supply for EVs. Conversely, high prices may discourage purchases. Therefore, balancing energy trading is crucial to maximizing the utility of the electricity market while maintaining its monopoly power.

The Stackelberg game, acting as an effective game-theoretical model, has been widely used to strategically regulate the price of oligopolies, which can be described as an oligopoly model [23,36]. The Stackelberg game has two stages, where the leader sets its strategy first, followed by the followers, who respond accordingly. We model this as a single-leader, multi-follower Stackelberg game between the electricity market and EV charging operators. In the first stage, the electricity market, as the leader, sets the selling price to maximize its utility. In the second stage, each EV charging operator, as a follower, determines its energy demand to maximize its utility. The Stackelberg game model is described in detail as follows.

4.1. Electric Energy Demands of EV Charging Operators in Stage II

We formulate the utility function of EV charging operators, which is the difference between the profit corresponding to the purchased electric energy and the cost of purchasing electric energy. Specifically, for EV charging operator m, we define $E_m$ as the electric energy provided by the electricity market. The more electric energy obtained from the electricity market, the more profits that EV charging operators can obtain. Thus, motivated by [36], the profit of EV charging operator m can be defined as

$${\mathcal{G}}_m\left(E_m\right)={\alpha }_{m}log(1+E_m),$$

(1)

where ${\alpha }_m$ is the unit profit for the purchased electric energy of EV charging operator m. Thus, the utility function of EV charging operator m is given by

$$U_m\left(E_m\right)={\mathcal{G}}_m\left(E_m\right)-P·E_m,$$

(2)

where $P>0$ is the unit selling price of electric energy. In Stage II, each EV charging operator (m) aims to maximize its utility $U_m\left(E_m\right)$ by deciding the optimal electric energy demand to purchase. Therefore, the optimization problem that maximizes the utility of EV charging operator m is formulated as

$$\begin{array}{cc}\hfill \mathbf{P}\mathbf{1}:& \underset{E_m}{max}{U}_m\left(E_m\right)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.E_m>0.\hfill \end{array}$$

(3)

4.2. Selling Price of the Electricity Market in Stage I

The electricity market, as the energy provider, ensures that its energy allocation meets the demands of EV charging operators while maximizing its utility [23]. To achieve this, it formulates a dynamic pricing strategy, adjusting based on the energy demands of the EV charging operators. The utility of the electricity market is the difference between the total charges paid by EV charging operators and the cost of energy harvesting and transmission. Thus, the utility of the electricity market is expressed as

$$U_e\left(p\right)=\sum _{m=1}^M(P·E_m-C·E_m),$$

(4)

where $C>0$ is the unit cost of supplying electric energy to EV charging operators. From (4), we know that the electricity market can obtain profits by providing electric energy to EV charging operators but needs to pay the costs of supplying electric energy. Considering that the renewable energy harvested by the electricity market is not unlimited, the energy sold by the electricity market has an upper limit of $E_{max}$, and the energy price also has an upper limit of $P_{max}$. The electricity market aims to maximize its utility by deciding a selling price under the constraints that the total electric energy sales do not exceed $E_{max}$ and the energy price does not exceed $P_{max}$. Hence, the optimization problem of maximizing the utility of the electricity market is given by

$$\begin{array}{cc}\hfill \mathbf{P}\mathbf{2}:& \underset{P}{max}{U}_e\left(p\right)=\sum _{m=1}^M(P·E_m-C·E_m)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.0<{\sum }_{m=1}^M{E}_m\le E_{max},\hfill \\ \hfill & E_m>0,\forall m\in \{1,\dots ,M\},\hfill \\ \hfill & 0<C\le P\le p_{max}.\hfill \end{array}$$

(5)

Note that no EV charging operator would buy electric energy from the electricity market if the selling price of unit electric energy were to exceed $P_{max}$. Finally, we formulate the Stackelberg game based on (3) and (5).

4.3. Stackelberg Equilibrium Analysis

In this part, we seek the Stackelberg equilibrium to find the optimal solution for the game. This equilibrium ensures that the electricity market maximizes its utility, while EV charging operators can design energy request policies based on their best response. Both parties maximize their utility by adjusting strategies until they reach equilibrium [23]. The Stackelberg equilibrium is defined as follows:

Definition 1 

(Stackelberg Equilibrium). We denote ${\mathit{E}}^{*}=\left\{E_m^{*}\right\},m\in \mathcal{M}$ and $P^{*}$ as the optimal electric energy demands of EV charging operators and the optimal energy pricing of the electricity market, respectively. The strategy $({\mathit{E}}^{*},P^{*})$ can be the Stackelberg equilibrium if and only if the following set of inequalities is strictly satisfied [23,36]:

$$\left\{\begin{array}{c}{U}_e(P^{*},{\mathit{E}}^{*})\ge U_e(P,{\mathit{E}}^{*}),\hfill \\ U_m(E_m^{*},{\mathit{E}}_{-m}^{*},P^{*})\ge U_m(E_m,{\mathit{E}}_{-m}^{*},P^{*}),\forall m\in \mathcal{M}.\hfill \end{array}\right.$$

(6)

In the following, we utilize the backward induction method to analyze the Stackelberg equilibrium [23,36].

4.3.1. EV Charging Operators’ Optimal Strategies as Equilibrium in Stage II

In the Stackelberg game, EV charging operators act as followers, which determine the optimal strategies of electric energy demands based on the selling price of a unit of electric energy (P), thereby maximizing their profits.

Theorem 1. 

The perfect equilibrium in the EV charging operators’ subgame is unique.

Proof. 

We derive the first-order derivative and the second-order derivative of $U_m\left(E_m\right)$ with respect to $E_m$ as follows:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial U_m\left(E_m\right)}{\partial E_m}}& ={\displaystyle \frac{{\alpha }_m}{1+E_m}}-P,\hfill \\ \hfill {\displaystyle \frac{{\partial }^2{U}_n\left(b_n\right)}{\partial b_n^2}}& =-{\displaystyle \frac{{\alpha }_m}{{(1+E_m)}^2}}<0.\hfill \end{array}$$

(7)

Since the first-order derivative of $U_m\left(E_m\right)$ has a unique zero point and the second-order derivative of $U_m\left(E_m\right)$ is negative, the utility function ($U_m\left(E_m\right)$) of EV charging operators is strictly concave with respect to the electric energy demand strategy ($E_m$) of EV charging operators. Based on the first-order optimality condition, i.e., ${\displaystyle \frac{\partial U_m\left(E_m\right)}{\partial E_m}}=0$, we can obtain the best response function ($E_m^{*}$) of EV charging operator m, which is given by

$$E_m^{*}={\displaystyle \frac{{\alpha }_m}{P}}-1.$$

(8)

Therefore, perfect equilibrium in the subgame of EV charging operators is unique.    □

4.3.2. The Electricity Market’s Optimal Strategy as Equilibrium in Stage I

In this part, we focus on studying the concavity of the utility function of the electricity market, proving the existence and uniqueness of the Stackelberg equilibrium. In Stage I, the electricity market acts as the leader in maximizing its utility by predicting the strategies of EV charging operators.

Theorem 2. 

The uniqueness of the Stackelberg equilibrium $({\mathit{E}}^{*},P^{*})$ can be guaranteed in the formulated Stackelberg game.

Proof. 

According to Theorem 1, there exists a unique Nash equilibrium among EV charging operators under any given value of P. Thus, the electricity market can maximize its utility by choosing the optimal value of P. Based on the optimal electric energy demand strategies of EV charging operators, the utility function of the electricity is given by

$$U_e\left(P\right)=\sum _{m=1}^M(P-C)\left({\displaystyle \frac{{\alpha }_m}{P}}-1\right).$$

(9)

By taking the first-order derivative and the second-order derivative of $U_e\left(P\right)$ with respect to P, we can obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial U_e\left(P\right)}{\partial P}}& =\sum _{m=1}^M\left({\displaystyle \frac{{\alpha }_{m}C}{P^2}}-1\right),\hfill \\ \hfill {\displaystyle \frac{{\partial }^2{U}_e\left(P\right)}{{\partial }^{2}P}}& =\sum _{m=1}^M-{\displaystyle \frac{2{\alpha }_{m}C}{P^3}}<0.\hfill \end{array}$$

(10)

Since the first-order derivative of $U_e\left(P\right)$ has a unique zero point, we can obtain $P^{*}=\sqrt{{\displaystyle \frac{C{\sum }_{m=1}^M{\alpha }_m}{M}}}$, and the second-order derivative of $U_e\left(P\right)$ is negative, so $U_e\left(P\right)$ is also strictly concave, which indicates that the electricity market has a unique optimal solution to the formulated game [36]. Based on the optimal strategy of the electricity market, the optimal strategies of EV charging operators can be obtained [37]. Therefore, the uniqueness of the Stackelberg game’s equilibrium is proven.    □

Due to the dynamic nature of the environment of energy trading between the electricity market and EV charging operators [38], traditional methods may be difficult to adapt to the dynamics of energy trading and not be able to efficiently find the Stackelberg equilibrium. Since DRL agents can learn to adapt their behavior based on environmental dynamics [39], we utilize a DRL algorithm to find the Stackelberg equilibrium. Furthermore, we innovatively add dynamic structured pruning techniques to the DRL algorithm for efficient implementation in energy trading.

5. Tiny Deep Reinforcement Learning for an Optimal Pricing Strategy

In intricate decision-making contexts, sophisticated AI methodologies such as DRL [40,41] represent promising approaches for the development of incentive mechanisms while addressing privacy concerns [42,43]. In this section, we model the formulated Stackelberg game between the electricity market and EV charging operators as a Partially Observable Markov Decision Process (POMDP) [36,44]. To address the challenge posed by incomplete information and enhance the efficiency of finding the Stackelberg equilibrium, we propose a Tiny DRL algorithm. The Tiny DRL algorithm is designed to find the Stackelberg equilibrium by identifying the optimal solutions of the Stackelberg game, enabling the electricity market to quickly converge to near-optimal decisions. Unlike traditional DRL approaches that focus on estimating fixed policies or single-step models, the proposed method leverages Markov properties to effectively decompose the problem.

5.1. POMDP for the Stackelberg Game between the Electricity Market and EV Charging Operators

Because of the effect of the competition between the electricity market and EV charging operators, each EV charging operator has local incomplete information in the Stackelberg game and determines electric energy strategies in a completely non-cooperative manner. The energy trading environment following a POMDP is needed to train the DRL agent, which is formulated by conceptualizing the dynamic relationship between the electricity market and EV charging operators as a Stackelberg game. Let $\mathcal{F}=\{\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{R},\gamma \}$ represent a POMDP [45], where $\mathcal{S}$, $\mathcal{O}$, $\mathcal{A}$, $\mathcal{R}$, and $\gamma$ represent the state space, partially observable policy, action space, reward function, and discounted factor for the electricity market, respectively [36,45].

In each time step (t, where $t\in \mathcal{T}=\{0,\dots ,t,\dots ,T\}$), the electricity market interacts with the environment to determine its current state, which is denoted as $\mathit{S}\left(t\right)$. During the training process, the electricity market, acting as the DRL agent, engages in interactions with the environment. At each time step, when the electricity market executes an action ($P\left(t\right)$) according to the current state ($\mathit{S}\left(t\right)$), the environment provides an immediate reward ($R\left(t\right)$) [46]. In the realm of electric energy trading, the electricity market functions as a game leader, responsible for selecting the action, i.e., the pricing policy ($P\left(t\right)$). After that, EV charging operators, acting as followers, identify an optimal strategic decision based on (8). Following this, the environment provides a reward ($R\left(t\right)$) to the electricity market by considering the strategies decided by all EV charging operators. The system contains a finite relay buffer, denoted as $\mathcal{D}$, which can store historical operation data, and the capacity of the finite relay buffer is defined as D. Relevant data of the electricity market can be extracted from the relay buffer to create new states, triggering the subsequent time step [24].

5.1.1. State Space

In each time step ($t\in \mathcal{T}=\{0,\dots ,t,\dots ,T\}$), the state space is defined as a union of the current pricing strategy of the electricity market and the electric energy demand strategies of EV charging operators, which is denoted as

$$\mathit{S}\left(t\right)\triangleq \left\{P\left(t\right),\mathit{E}\left(t\right)\right\},$$

(11)

where $P\left(t\right)$ and $\mathit{E}\left(t\right)$ are the price of the electricity market and the electric energy demand vector of EV charging operators at time step t, respectively.

5.1.2. Partially Observable Policy

We formulate the partially observable space for energy trading between the electricity market and EV charging operators, tackling the non-stationary problem in the DRL system. Throughout the POMDP, the electricity market agent can solely base decisions on local environmental observations. We define the observation space of the electricity market in at time step t as $O\left(t\right)$, which is a union of its historical pricing strategies and the electric energy demand strategies of EV charging operators for the previous L games involving the electricity market and all EV charging operators. Consequently, the observation space ($O\left(t\right)$) of the electricity market at time step t is represented as

$$\mathit{O}\left(t\right)\triangleq \left\{P(t-L),\mathit{E}(t-L),P(t-L+1),\mathit{E}(t-L+1),\dots ,P(t-1),\mathit{E}(t-1)\right\},$$

(12)

where $P(t-L)$ and $\mathit{E}(t-L)$ can be generated randomly during the initial stage when $t<L$. By considering historical information, the electricity market agent can learn how changes in its strategy impact the game result in the current time slot [36]. When receiving an observation ($\mathit{O}\left(t\right)$) from the environment, the electricity market agent needs to design the selling price ($P\left(t\right)$) of electric energy to maximize its utility.

5.1.3. Action Space

$\mathcal{A}\triangleq \left\{P\right\}$ denotes the action space of the electricity market. Given the lower-bound cost (C) and the upper-bound price ($P_{max}$) for the pricing action, the electricity market decides its action ($P\left(t\right)$) at each time step (t), where $P\left(t\right)\in [C,P_{max}]$. This decision-making process relies on the information encapsulated in the observation space ($\mathit{O}\left(t\right)$).

5.1.4. Reward Function

$\mathcal{R}\triangleq \left\{R\right\}$ denotes the reward function of the electricity market. Following the state transition, the electricity market can acquire an immediate reward based on the current state ($\mathit{S}\left(t\right)$) and the corresponding action ($P\left(t\right)$) [36]. The reward function is defined as the utility function of the electricity market that we construct in the Stackelberg game. At time step t, the reward function for the electricity market is represented as $R\left(t\right)=U_e\left(t\right)$.

In the actor–critic network framework, the system consists of two crucial elements, i.e., the actor network and the critic network [24]. Proximal Policy Optimization (PPO) is a DRL algorithm based on policy gradients [47]. By employing proximal optimization techniques on the policy, the stability and convergence of agent learning can be enhanced, ensuring more reliable and efficient learning processes. In the proposed tiny DRL framework, we denote the actor–critic network as $(\mathit{\theta },\mathit{\omega })$. Note that the actor and critic networks are all neural networks. The actor network essentially functions as a policy function (${\pi }_{\mathit{\theta }}\left(P\right|\mathit{S})$) with parameters ($\mathit{\theta }$), which helps to generate the action of the electricity market, namely the pricing strategy (P), and facilitate interactions with the environment. Conversely, the critic network, characterized by the value function ($V_{\mathit{\omega }}\left(\mathit{S}\right)$) parameterized by $\mathit{\omega }$, evaluates the performance of the electricity market agent and guides the actions of the agent in subsequent phases, which is defined as

$$V_{\mathit{\omega }}\left(S\right)\triangleq {\widehat{\mathbb{E}}}_{{\pi }_{\mathit{\theta }}}\left[\sum _{t=0}^T{\gamma }^{t}R\left(S\left(t\right),P\left(t\right)\right)\mid S_0=S\right],$$

(13)

where ${\widehat{\mathbb{E}}}_{{\pi }_{\mathit{\theta }}}(·)$ is the expected value of a random variable, given that the electricity market agent follows the policy (${\pi }_{\mathit{\theta }}$).

The primary objective of the critic network is to minimize the Temporal Difference (TD) error, which is expressed as

$$d=R\left(t\right)+\gamma V_{\mathit{\omega }}\left(\mathit{S}(t+1)\right)-V_{\mathit{\omega }}\left(\mathit{S}\left(t\right)\right),$$

(14)

where $V_{\mathit{\omega }}\left(\mathit{S}\left(t\right)\right)$ and $V_{\mathit{\omega }}\left(\mathit{S}(t+1)\right)$ represent the value functions associated with the current state ($\mathit{S}\left(t\right)$) and the subsequent state ($\mathit{S}(t+1)$), respectively. Therefore, the loss function of the critic network is derived by minimizing the expected value of the squared temporal difference (TD) value, which is given by [27]

$$\begin{array}{c}\hfill \underset{\mathit{\omega }}{min}{L}_c\left(\mathit{\omega }\right)=\underset{\mathit{\omega }}{min}\mathbb{E}\left[\right(R\left(t\right)+\gamma V_{\mathit{\omega }}\left(\mathit{S}(t+1)\right)\\ \hfill -V_{\mathit{\omega }}\left(\mathit{S}\left(t\right)\right){)}^2].\end{array}$$

(15)

Furthermore, the objective of the actor network is specifically defined as

$$\begin{array}{c}\hfill \underset{\mathit{\theta }}{max}{J}_a\left(\mathit{\theta }\right)=\underset{\mathit{\theta }}{max}\mathbb{E}[min(\zeta \left(\mathit{\theta }\right){\widehat{A}}_{{\pi }_{\mathit{\theta }}}(\mathit{S},P),\\ \hfill I\left(\iota ,\zeta \left(\mathit{\theta }\right)\right){\widehat{A}}_{{\pi }_{\mathit{\theta }}}(\mathit{S},P)\left)\right],\end{array}$$

(16)

where $\zeta \left(\mathit{\theta }\right)={\displaystyle \frac{{\pi }_{\mathit{\theta }}\left(P\right|\mathit{S})}{{\pi }_{\widehat{\mathit{\theta }}}\left(P\right|\mathit{S})}}$ represents the important ratio between the old policy and the new policy, $\widehat{\mathit{\theta }}$ represents the parameters of the strategy used for sampling (P), and ${\pi }_{\widehat{\mathit{\theta }}}\left(P\right|\mathit{S})$ denotes the policy employed for importance sampling [48]. $I\left(\iota ,\zeta \left(\mathit{\theta }\right)\right)$ is a piece-wise function with intervals, which is given by [48]

$$I\left(\iota ,\zeta \left(\mathit{\theta }\right)\right)=\left\{\begin{array}{cc}1+\iota ,\hfill & \zeta \left(\mathit{\theta }\right)>1+\iota ,\hfill \\ \zeta \left(\mathit{\theta }\right),\hfill & 1-\iota \le \zeta \left(\mathit{\theta }\right)\le 1+\iota ,\hfill \\ 1-\iota ,\hfill & \zeta \left(\mathit{\theta }\right)<1-\iota ,\hfill \end{array}\right.$$

(17)

where $\iota$ represents an adjustable hyper-parameter. ${\widehat{A}}_{{\pi }_{\mathit{\theta }}}(\mathit{S},P)$ denotes the estimator for the advantage function that utilizes $V_{\mathit{\omega }}\left(S\right)$, which is expressed as

$$\begin{array}{cc}\hfill {\widehat{A}}_{{\pi }_{\mathit{\theta }}}\left(\mathit{S}\left(t\right),P\left(t\right)\right)=& {\gamma }^{T-t}{V}_{\mathit{\omega }}\left(\mathit{S}\left(T\right)\right)-V_{\mathit{\omega }}\left(\mathit{S}\left(t\right)\right)\hfill \\ \hfill & +\sum _{x=t}^{T-1}{\gamma }^{x-t}R(\mathit{S}\left(x\right),P\left(x\right)),\hfill \end{array}$$

(18)

5.2. Dynamic Structured Pruning

In the DRL algorithm, the actor and critic networks are essentially deep neural networks [49], which typically consist of an input layer, multiple hidden layers, and an output layer [24]. These layers have numerous parameters, like neurons and weights. Without loss of generality, we consider an actor network with K layers and denote the weights in the k-th fully connected layer as ${\mathit{\theta }}^{\left(k\right)}$, where $k\in \{1,\dots ,K\}$. By inputting the state ($\mathit{S}\left(t\right)$) at time step t into the first layer, the output of the first layer is calculated as

$${\mathit{h}}^{\left(1\right)}={\sigma }^{\left(1\right)}\left({\mathit{\theta }}^{\left(1\right)}\mathit{S}\left(t\right)+b^{\left(1\right)}\right),$$

(19)

where ${\sigma }^{j\left(1\right)}$ represents the nonlinear response of the first layer, which is typically set to the ReLU function, and $b^{\left(1\right)}$ is the deviation at the h-th layer. The output of each layer in the network is fed to the subsequent laye ras the input. Therefore, the output of the k-th layer is expressed as

$${\mathit{h}}^{\left(k\right)}={\sigma }^{\left(k\right)}\left({\mathit{\theta }}^{\left(k\right)}{\mathit{h}}^{(k-1)}+b^{\left(k\right)}\right).$$

(20)

Finally, at time step t, the actor network outputs the action, i.e., the price strategy ($P\left(t\right)$), which is expressed as

$$P\left(t\right)={\sigma }^{\left(K\right)}\left({\mathit{\theta }}^{\left(K\right)}{\mathit{h}}^{(K-1)}\right).$$

(21)

To achieve the Tiny DRL algorithm, we incorporate dynamic structured pruning techniques into the actor network. This helps eliminate neurons and weights that do not significantly contribute to the performance of the actor network [49]. Unlike unstructured pruning techniques for the acceleration of DRL training, which often results in irregular network structures [24], structured pruning is a technique used to reduce model complexity by strategically eliminating redundant neurons or connections [50].

To indicate the pruning status of neurons, a binary mask (${\mathit{m}}^{\left(k\right)}$) is employed. Specifically, we denote ${\mathit{m}}_i^{\left(k\right)}=1$ as a non-pruning neuron ($o_i^{\left(k\right)}$) and ${\mathit{m}}_i^{\left(k\right)}=0$ as a pruning neuron ($o_i^{\left(k\right)}$) [24]. Thus, the action output from the actor network is expressed as

$$P\left(t\right)={\sigma }^{\left(H\right)}\left({\mathit{\theta }}^{\left(H\right)}{\mathit{h}}^{(H-1)}\odot {\mathit{m}}^{\left(H\right)}\right),$$

(22)

where ⊙ denotes the element-wise multiplication of two matrices. Based on the above analysis, the loss function of the actor network is rewritten as [24,50]

$$\begin{array}{c}\hfill J_a(\mathit{\theta },\mathit{m})=\mathbb{E}[min(\zeta (\mathit{\theta },\mathit{m}){\widehat{A}}_{{\pi }_{\mathit{\theta }}}(\mathit{S},P),\\ \hfill I\left(\iota ,\zeta (\mathit{\theta },\mathit{m})\right){\widehat{A}}_{{\pi }_{\mathit{\theta }}}(\mathit{S},P)\left)\right].\end{array}$$

(23)

As D records accumulate in the replay buffer, the actor and critic networks are updated. Specifically, the electricity market updates the parameters of the actor network by using the gradient ascent method, which is given by

$${{\mathit{\theta }}^{\left(k\right)}}^{\prime }={\mathit{\theta }}^{\left(k\right)}-\epsilon {\displaystyle \frac{\partial J_a(\mathit{\theta },\mathit{m})}{\partial \left({\mathit{h}}^{\left(k\right)}\odot {\mathit{m}}^{\left(k\right)}\right)}}·{\displaystyle \frac{\partial \left({\mathit{h}}^{\left(k\right)}\odot {\mathit{m}}^{\left(k\right)}\right)}{\partial {\mathit{\theta }}^{\left(k\right)}}},$$

(24)

where $\epsilon$ represents the learning rate employed in the training process of the actor network and ${{\mathit{\theta }}^{\left(k\right)}}^{\prime }$ represents the updated parameters of the actor network. The parameters of the critic network are updated through the gradient descent method as follows [24,50]:

$${{\mathit{\omega }}^{\left(k\right)}}^{\prime }={\mathit{\omega }}^{\left(k\right)}-ϵ{\displaystyle \frac{\partial L_c\left(\mathit{\omega }\right)}{\partial {\mathit{\omega }}^{\left(k\right)}}},$$

(25)

where $ϵ$ represents the learning rate employed in the training process of the critic network and ${{\mathit{\omega }}^{\left(h\right)}}^{\prime }$ represents the updated parameters of the critic network.

The dynamic structured pruning of non-essential neurons consists of two key steps, namely, determining the pruning threshold and updating the binary mask used for pruning [24,50]. The pruning threshold plays a crucial role in identifying and eliminating unnecessary parameters or connections during the pruning process. Motivated by [24,50], we formulate a dynamic pruning threshold, which is given by

$$\chi \left(t\right)=\sum _{n=1}^N\sum _{k=1}^K{\rho }_n^{\left(k\right)}·\tau \left(t\right),$$

(26)

$$\tau \left(t\right)=\check{\tau }+(\widehat{\tau }-\check{\tau }){\left(1-{\displaystyle \frac{t}{Y▵t}}\right)}^3,$$

(27)

where ${\rho }_n^{\left(k\right)}$ and K represent the neuronal importance of the n-th neuron of layer k and the total number of pruning steps, respectively. ${\displaystyle ▵}$ represents the pruning frequency. $\tau \left(t\right)$, $\widehat{\tau }$, and $\check{\tau }$ represent the current sparsity in epoch t, the initial sparsity, and the target sparsity, respectively. This dynamic pruning method can adaptively enhance the sparsity of the model as the iteration goes on, providing a more refined and effective method for structured pruning. Neurons are ranked according to their importance, from least to most important. Neurons whose ranks are below a set threshold are then pruned to improve the overall sparsity of the model. The mask of the n-th neuron of layer k is updated as

$$m_n^{\left(k\right)}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{abs}\left[m_n^{\left(k\right)},{\mathit{\theta }}_n^{\left(k\right)}\right]\ge \psi ,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(28)

The above process of dynamic structured pruning is shown in Algorithm 1. In the Tiny DRL model, we adopt a fully connected deep neural network architecture for the actor network, which consists of K layers. Algorithm 1 consists of a two-step process, namely, initially training the DRL model, then using a dynamic pruning threshold to remove unimportant neurons. Note that the complexity of Algorithm 1 over T episodes is $\mathcal{O}\left(T\left|\mathit{S}\right|\right)+\mathcal{O}\left(T{\sum }_{k=1}^{K-1}{u}^{\left(h\right)}\right)$, where $u^{\left(h\right)}$ is the number of neurons in each hidden layer (k) up to the penultimate layer [24,50].

Algorithm 1: Tiny DRL algorithm with dynamic structured pruning for Stackelberg equilibrium.

Input: State $\mathit{S}$.  Output: The optimal strategy $({\mathit{E}}^{*},P^{*})$.

6. Numerical Results

In this section, we present numerical results to demonstrate the effectiveness of the proposed tiny DRL algorithm and analyze the proposed Stackelberg game model.

Figure 2 presents a performance comparison between the proposed Tiny PPO algorithm and the PPO algorithm. We set the pruning rate, the learning rate of the actor and critic networks, the discount factor, the training epoch, and the batch size as $0.05$, $1\times {10}^{-4}$, $0.95$, 400, and 512, respectively. From Figure 2, we can observe that the proposed Tiny PPO algorithm is more stable than the PPO algorithm and can obtain more test rewards. The Tiny PPO algorithm can also promote higher utility of the electricity market and sum utilities of all EV charging operators, demonstrating the superior performance of the proposed Tiny PPO algorithm.

Figure 3 shows the utilities and optimal strategies of the electricity market and EV charging operators under different costs (C), with $M=5$ corresponding to the number of EV charging operators and a unit profit of $\alpha =50$. From Figure 3, we can observe that as the unit cost (C) increases, the selling price of a unit of electric energy (P) set by the electricity market also rises. Concurrently, the electric energy demands ($E_m$) determined by the EV charging operators decrease. The underlying reason for this trend is that an increase in the unit cost (C) compels the electricity market to raise prices to maintain stable and increasing profits. Then, this price hike discourages EV charging operators from purchasing large amounts of electricity, leading to a reduction in electric energy demands. Moreover, the utilities of the electricity market and EV charging operators decrease as the unit cost (C) increases. That is because the electric energy demands of EV charging operators decrease, while the selling price of a unit of electric energy (P) increases. Specifically, the reduction in electric energy demand has a more substantial negative impact on the utility of the electricity market than the positive impact of the increased selling price, resulting in a net decrease in the utility of the electricity market. Similarly, for EV charging operators, the adverse effect of a higher selling price per unit of electric energy outweighs the effect of reduced electric energy demands, leading to a decrease in their utility as well.

Figure 4 illustrates the utilities and strategies of the electricity market and EV charging operators under different numbers of EV charging operators, with a unit cost of $C=5$ and unit profit of $\alpha =50$. From Figure 4, it is evident that the electric energy demands increase as the number of EV charging operators (M) rises, while the selling price of a unit of electric energy (P) remains stable, regardless of changes in the number of EV charging operators. According to the the equation $P^{*}=\sqrt{{\displaystyle \frac{C{\sum }_{m=1}^M{\alpha }_m}{M}}}$, since C and $\alpha$ are constant, P does not change. Specifically, the stability of the selling price (P) amidst increasing demand can be attributed to the constancy of the unit cost (C) and unit profit ($\alpha$), ensuring that the price equilibrium is maintained. Additionally, we can observe that the utilities of both the electricity market and the EV charging operators increase as the number of EV charging operators grows. This is because the increased energy demands of EV charging operators positively impact the utilities of both the electricity market and the EV charging operators.

Figure 5 shows the utilities and strategies of the electricity market and EV charging operators under different unit profits ($\alpha$), with $M=5$ corresponding to the number of EV charging operators and a cost of $C=5$. It is observed that as the unit profit ($\alpha$) increases, both the electric energy demands and the selling price per unit of electric energy rise. This is because a higher unit profit ($\alpha$) incentivizes EV charging operators to purchase more electricity resources. The increased demand for electric energy enables the electricity market to set higher prices to maximize its profit. Furthermore, we can observe that the utilities of the electricity market and EV charging operators increase as the unit profit ($\alpha$) increases. It is obvious that the simultaneous growth in selling price and electric energy demands boosts the utility of the electricity market. For EV charging operators, the increase in utility may be attributed to the fact that the positive impact of higher electric energy demands outweighs the negative impact of rising selling prices.

Figure 6 shows the security performance of the PBFT consensus algorithm in the proposed blockchain system for electricity trading. From Figure 6, we can see that regardless of the probability of a delegate being malicious, $p_m$ exists, and the security probability increases as the number of miners increases. The PBFT algorithm relies on a majority consensus, requiring more than half of the nodes to agree. Therefore, as the number of miners increases, the proportion of honest nodes involved in the consensus process also grows, which enhances the overall robustness of the system, making it increasingly difficult for malicious attackers to compromise its integrity [51]. Therefore, the proposed blockchain system utilizing the PBFT consensus algorithm ensures reliable and secure electricity trading by guaranteeing trustworthy block verification.

7. Conclusions

In this paper, we proposed a blockchain-assisted secure energy trading framework. Specifically, we utilized blockchain technology to securely manage energy trading between the electricity market and EV charging operators. Then, we proposed a single-leader, multi-follower Stackelberg game model to address the electricity trading problem between the electricity market and EV charging operators. In this model, the electricity market acts as the leader, setting the price of a unit of electric energy. The EV charging operators, as followers, determine their electricity demand based on the price set by the electricity market. During the trading process, blockchain technology is utilized by EV charging aggregators to securely record and verify energy transactions. To find the Stackelberg equilibrium, we employed a DRL algorithm. Given the resource-intensive nature of training DRL models, we introduced pruning techniques into the DRL framework, referred to as Tiny DRL, to enhance the efficiency of the algorithm in terms of computing power and storage requirements. In future work, we will consider formulating a multi-leader, multi-follower Stackelberg game between electricity markets and EV charging operators. Our focus will be on enhancing the verification of our model through rigorous testing and validation procedures. Furthermore, we will aim to enhance consensus mechanisms, optimize smart contract functionalities, and explore interoperability with other blockchain networks to improve security, scalability, and efficiency within the energy trading ecosystem.