Game-Theory Modeling for Social Welfare Maximization in Smart Grids

Abstract

In this paper, we study the Stackelberg game-based evolutionary game with two players, generators and energy users (EUs), for monetary profit maximization in real-time price (RTP) demand response (DR) systems. We propose two energy strategies, generator’s best-pricing and power-generation strategy and demand’s best electricity-usage strategy, which maximize the profit of generators and EUs, respectively, rather than maximizing the conventional unified profit of the generator and EUs. As a win–win strategy to reach the social-welfare maximization, the generators acquire the optimal power consumption calculated by the EUs, and the EUs obtain the optimal electricity price calculated by the generators to update their own energy parameters to achieve profit maximization over time, whenever the generators and the EUs execute their energy strategy in the proposed Stackelberg game structure. In the problem formulation, we newly formulate a generator profit function containing the additional parameter of the electricity usage of EUs to reflect the influence by the parameter. The simulation results show that the proposed energy strategies can effectively improve the profit of the generators to 45% compared to the beseline scheme, and reduce the electricity charge of the EUs by 15.6% on average. Furthermore, we confirmed the proposed algorithm can contribute to stabilization of power generation and peak-to-average ratio (PAR) reduction, which is one of the goals of DR.

1. Introduction

With advanced communication networks and intelligent controllable electrical devices or energy users (EUs), the smart grid makes it possible to achieve the distributed control and the distributed energy management (DEM) [1]. DEM plays a key role in the distributed monitoring, controlling, scheduling, and optimization of the profit of both generators and demands for the implementation of demand response (DR) programs [2]. DR, defined as the energy-usage changes of users in response to varying electricity prices or to incentive payments [3], induces EUs to consume less energy during periods of high wholesale market prices or at the peak power consumption. This price-based DR program can be optimally implemented through a continuous interaction between the users and the service provider. The user needs to adjust the electricity usage in consideration of the varying electricity price over time, while the service provider also needs to properly adjust the electricity price with the amount of power generation to motivate users to evenly use electricity over time [4]. The most efficient use of the smart grid, for example, is the increasing of the energy efficiency through measures such as the decrease of the peak-to-average ratio (PAR) of the energy demand; however, in reality, this will determine the extent of the monetary profits in the generator and demand sides.

To maximize the monetary profit (or social welfare), various studies on smart-home scheduling have been completed, as follows: the formulation of a linear programming problem for smart-home scheduling in consideration of the uncertainty of energy consumption [5], a Markov-chain model of the scheduling problem and the development of the backtrack algorithm based on a decision threshold [6], a dynamic-programming algorithm to schedule home appliances in consideration of multiple power levels [7], the consideration of the distributed-load-management problem as a congestion game with a dynamic pricing strategy to discourage the energy consumption at peak hours [8], and the deployment of pricing strategies by local aggregators to control the energy load [9,10]. In [11], an optimal management system of battery energy storage to enhance the resilience of the microgrid is proposed while maintaining its operational cost at a minimum level. In [12], a two stage energy management strategy for the contribution of plug-in electrical vehicles (PEVs) in demand response programs of commercial building microgrids is addressed for energy management optimization. In [13], by forming coalitions for gaining competitiveness in the energy market, a smart transactive energy (TE) framework in which home microgrids (H-MGs) can collaborate with each other in a multiple H-MG system is presented and analyzed. In [14], an optimization-based algorithm in which an objective function premised on economic strategies, distribution limitations and the overall demand in the market structure is proposed with emphasizing optimum use of electrical/thermal energy distribution resources, while maximizing profit for the owners of the H-MGs.

Over the past few years, related studies that are based on game-theory modeling for the energy management in a real-time price (RTP)-based DR have also been conducted. In [15], a cake-cutting game (CCG) for the selection of discriminate prices for different users was investigated. In [4], an RTP-based DR algorithm for the achievement of an optimal load control regarding the devices in a facility that is obtained through the forming of a virtual electricity-trading process is proposed. In [16], an aggregate game is adopted for the modeling and analysis of the energy-consumption control in the smart grid and Nash seeking strategies are developed. In [17,18], the Stackelberg game was leveraged to model the interaction between the demand-response aggregators and generators. In [19], the authors propose a light-weight DR scheme for managing energy consumption based on a non-iterative Stackelberg model and historical real-time pricing without iterations for the massive smart manufacturing systems. In [20], a multiagent-based energy market for multi-microgrid systems using game-theoretic and hierarchical optimization approaches is proposed to achieve the optimal operation of smart microgrids in distribution systems. In [21], an advanced retail electricity market based on game theory for the optimal operation of H-MGs and their interoperability within active distribution networks is proposed and the optimal solution is achieved using the Nikaido-Isoda Relaxation Algorithm (NIRA) in a non-cooperative gaming structure. In [22] and [23], a consensus-based distributed-energy-management algorithm for both sides of an indirectly connected network is proposed. However, the following shortcomings relative to the game structure and formulas for profit maximization can be identified in the previous research, which have been addressed in this paper:

• It is necessary to study an optimization methodology that maximizes the monetary profit of the generators and the demands, respectively, rather than the optimization methodology based on the unified profit of the generators and the demands [24,25].
• As an revolutionary game based on the Stackelberg game structure, it is necessary to study that the players (demands and generators) obtain the each other’s energy parameters, such as the time-varying electricity price and an amount of electricity consumption with spying on each other’s energy strategy to effectively and adaptively maximize their profit.

In view of these needs for research, in this paper, the Stackelberg game model where the generators and the demands that are the players of the stackelberg game alternatively maximize their respective profits using their own energy strategy over time while watching each other’s energy strategies is studied. The contributions of this paper are as follows:

• We propose two energy strategies for profit maximization for both the generators and EUs (Generator’s Best-Pricing and Power-Generation Strategy in Section 3.1, Demand’s Best Electricity-Usage Strategy in Section 3.2) based on the Stackelberg game as an evolutionary game where the players (demands and generators) alternately perform their energy strategy with spying on each other’s energy strategy to update their own energy parameters to achieve profit maximization over time in smart grid demand response. Whenever the generators and the EUs execute their energy strategy in the proposed Stackelberg game structure, the generators acquire the optimal power consumption calculated by the EU, and the EUs obtain the optimal electricity price calculated by the generator. To the best knowledge of the authors, this game structure based on the aforementioned parameter exchange (optimal power consumption and optimal electricity price) for profit maximization in the smart grid demand response is first studied.
• We newly formulate a generator profit function including the additional parameter, the electricity-consumption of EUs, compared with that of the conventional profit function [26] since the profit of generators can be influenced by the electricity-consumption of EUs.
• We greatly improve the monetary profit of the generators and EUs using the proposed two energy strategies by optimizing the amount of the power generation and the electricity price in the generator side, and electricity consumption in the EU side.
• As one of the simple and powerful electricity-usage control strategy of the demand that is applicable to the time-varying electricity market, we newly propose a market-adaptive electricity-usage scheduling algorithm which maximizes the demand’s profit by calculating the amount of power that should be consumed according to the time-varying electricity price.
• The proposed profit maximization algorithm can solve the existing PAR reduction problem because the energy usage immediately increases as the electricity price goes down, and the energy usage goes down as the price goes up according to the proposed game structure. It is also possible to alleviate the problem that the price greatly fluctuates because if the price changes greatly, the power plant will lose its benefits and will not be able to withstand it. The problem of making power generation stable is consistent with reducing PAR, which is one of DR’s ultimate goals. Stable power generation can reduce useless power generation, which makes our energy resources the most efficient to use, and can contribute to addressing issues such as global warming in the energy industry.

The Stackelberg game model is used to realistically model the profit maximization scenarios wherein two players, the generators and the demands, repeatedly modify their energy strategies by checking on each other’s energy strategy in the order of time. It is confirmed here that the monetary profit of the both is optimized when the two players compete strategically over time. Further, a modeling of the demand profit and the optimization of the EU energy usage is performed based on a time-varying electricity price. Since the electricity price at different times can be significantly different even in a single day, it is necessary to optimally schedule the electricity usage for all of the demands, while the electricity-price changes are monitored. It is assumed here that the comprehensive judgment and control of the EU electricity usage are optimally performed by the smart-home scheduler. In this paper, we provide the profit maximization methodology based on the profit of the generators and EUs, respectively, not the unified profit of the generators and EUs.

The rest of this paper is organized as follows: in Section 2, we formulate the new profit function of the generators and the profit maximization problem for the generators and EUs, respectively, based on the Stackelberg game. In Section 3, we propose two profit maximization algorithms as a solution of the problem, and present the schematic overview and application of the proposed algorithms. In Section 4, simulation results are presented and the practical benefits and advantages of the proposed algorithm are described. Finally, we conclude the paper in Section 5.

2. System Model and Problem Formulation

For the problem formulation, a smart grid with a network of $K$-distributed EUs and $I$ generators was considered. It was assumed that each EU $k\in \mathcal{K}=\left\{1,\dots ,K\right\}$ and each generator $i\in \mathcal{I}=\left\{1,\dots ,I\right\}$ are directly connected as the communication topology of the network (i.e., any two nodes are connected by a directed path). In this paper, it should be noted that the assumption of the strongly connected communication network renders the distributed-energy-management problem to be more general and relaxes the undirected-connection assumption [22]. A block-processing model was adopted to schedule the load demands according to periodic time blocks. We divide the total scheduling time into t time periods $t\in \mathcal{T}=\left\{1,\dots ,T\right\}$ with a constant length $t_l$. We consider that scheduling horizon for an EU is 24 h (a single day), that is $t_l$ and $T$ are set to 1 h and 24, respectively, in this paper. All parameters and functions used in this paper are listed in

Table 1. Notation list.

Notations	Meanings of Notations
$\mathcal{K}=\left\{1,\dots ,K\right\}$, $k\in \mathcal{K}$	The set of energy users (EUs)
$\mathcal{I}=\left\{1,\dots ,I\right\}$, $i\in \mathcal{I}$	The set of generators
$\mathcal{T}=\left\{1,\dots ,T\right\}$, $t\in \mathcal{T}$	Total scheduling time set
$t_l$	The constant time length
$p_t$ (cents/kWh)	The electricity price per unit energy
$P_{i,t}$ (kWh)	The amount of the actual providable power
$P_{i,t}^L$ (kWh)	The transmission losses
$P_{i,t}^T$ (kWh)	The total amount of generated power
$C_{i,t}\left(P_{i,t}^T\right)$ (cents)	The cost to generate power for generator i
$a_i\left(\mathrm{cents}/{\mathrm{kWh}}^2\right)$, $b_i\left(\mathrm{cents}/\mathrm{kWh}\right)$, and $c_i\left(\mathrm{cents}\right)$	The fitting parameters of the cost function $C_{i,t}\left(P_{i,t}^T\right)$
$d_i$	The coefficient for power loss
$P_i^m$ and $P_i^M$	The minimum and maximum bound of $P_{i,t}^T$
${\mathcal{U}}_{i,t}^G$	The profit function for generator i
${\mathcal{U}}_t^G$	The total profit for all generators
$R_t$	Real profit of generators
$M_{i,t}$	Maximum achievable profit of the generator i
$P_{k,t}$	The amount of the electricity usage of UE k
${\widehat{P}}_{k,t}$	The amount of electricity to be used in the future divided by the remaining time
$∆P_{k,t}$	The additional or abandoned electricity usage
${\mathcal{U}}_k^{EU}$	The electricity charge for UE k
${\mathcal{U}}^{EU}$	The total electricity charge for all UEs
$\mathcal{G}=\left(\mathcal{N}, \mathcal{A}, \mathcal{U}\right)$	The Stackelberg game where $\mathcal{N}$ is a player set that is composed of generators and EUs, $\mathcal{A}$ is the constraint set, and $\mathcal{U}$ is the profit set
$p^m$ and $p^M$	The minimum and maximum prices of the electricity
$P_k^m$ and $P_k^M$	The minimum and maximum electricity-usage
${\phi }_k$	The minimum required electricity consumption
${\eta }^{\star }$	The optimum solution of P1
$P_{i,t}^{\star },p_t^{\star }$ and $P_{k,t}^{\star }$	The optimized value of $P_{i,t}$, $p_t$ and $P_{k,t}$
$F$	A set of feasible solutions
${\mathcal{L}}_{\mathrm{G}}$	The Lagrange dual function of P3
${\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t$, and ${\theta }_t$	The Lagrange multipliers
$N_M^G$ and $N_S^G$	The number of iterations for the master-loop and slave-loop in Algorithm 1
${\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4$, and ${\omega }_5$	The iteration steps for optimization
${\overline{p}}_t$	The base price as the historical average of $p_t$
$t_w$	The window size of the time slots
${\overline{\mathcal{U}}}^{EU}$	The total electricity charge accumulated for $T$

. Furthermore, the systemic model of the electricity buying and selling is shown in Figure 1. The notion used is listed in Table 1. The users are equipped with an advanced metering infrastructure (AMI) and an energy-management controller (EMC) [22]. Load information for each EU is exchanged between these two modules. The AMI is used to schedule, control and optimize the electricity usage for each EU and enables bidirectional communication between the EUs and the DEM which is connected to the generators. DEM plays a role in optimizing the benefits of both the EU network (demand) and the generator network (energy supply) based on the DR programs and real-time pricing to improve the energy-usage efficiency. Further, the following three key optimization variables (the electricity price, the generation power, and the demand power) are employed to realize the coordination of the generators and the EUs.

In the proposed evolutionary game, we assume that the players have perfect rationality. They always act in a way that maximizes their profit, and are capable of obtaining the energy information to calculate the best response to other players’ energy strategies. The tractability of the perfect rationality game can be realized by using the infrastructure, such as smart meter, EMC, DEM and wired/wireless communications to obtain maximum profit of the cooperative and win-win players.

2.1. Generator Profit

As the player of the Stackelberg game for each time t, each generator i aims to maximize its own profit by adjusting and optimizing the electricity price per unit energy $p_t$ (cents/kWh) and the amount of the actual providable power $P_{i,t}$ (kWh), excluding the transmission losses $P_{i,t}^L$ (kWh), in the total amount of generated power $P_{i,t}^T=P_{i,t}+P_{i,t}^L$ (kWh). To design a practical electric-power-transmission system, the parameters of the modeling of the transmission-loss factors were considered since these are inevitable in a power grid [22]. According to the micro-incremental transmission losses of each generator [24], the transmission-loss amount $P_{i,t}^L$ that is induced by the i can be represented using the following simple quadratic function:

$$P_{i,t}^L=d_i{P}_{i,t}^T{}^2$$

(1)

where $d_i$ is the loss coefficient. The total amount of generated power $P_{i,t}^T$ is derived by jointly solving the Equation (1) and $P_{i,t}^T=P_{i,t}+P_{i,t}^L$ as follows:

$$P_{i,t}^T=P_{i,t}+P_{i,t}^L=P_{i,t}+\frac{1-2P_{i,t}{d}_i-\sqrt{1-4P_{i,t}{d}_i}}{2d_i}$$

(2)

In reference to [25,26,27], the cost $C_{i,t}\left(P_{i,t}^T\right)$ can be represented by the quadratic function of $P_{i,t}^T$, as follows:

$$C_{i,t}\left(P_{i,t}^T\right)=a_i{P}_{i,t}^T{}^2+b_i{P}_{i,t}^T+c_i$$

(3)

where $a_i\left(\mathrm{cents}/{\mathrm{kWh}}^2\right)$, $b_i\left(\mathrm{cents}/\mathrm{kWh}\right)$, and $c_i\left(\mathrm{cents}\right)$ are the fitting parameters of the cost function, and the minimum (maximum) bound of $P_{i,t}^T$ is denoted by $P_i^m$ $\left(P_i^M\right)$. Therefore, a new profit function ${\mathcal{U}}_{i,t}^G\left(P_{i,t},p_t|P_{k,t}\right)$ and the real-profit-to-maximum-achievable-profit ratio (RMR), with the latter composed of the two profit functions for the i for each time t, was defined, as follows:

$$\begin{array}{c}{\mathcal{U}}_{i,t}^G\left(P_{i,t},p_t|P_{k,t}\right)=\frac{\mathrm{Real}\mathrm{profit}}{\mathrm{Maximum}\mathrm{achievable}\mathrm{profit}}=\frac{R_t\left(p_t|P_{k,t}\right)}{M_{i,t}\left(P_{i,t},p_t\right)}\\ =\frac{p_t{\sum }_{k=1}^K{P}_{k,t}}{p_t{P}_{i,t}-C_{i,t}\left(P_{i,t}+P_{i,t}^L\right)}\end{array}$$

(4)

where $P_{k,t}$ is the amount of the electricity usage of each EU k for each t. It becomes evident that the newly defined profit function of (4) is more reasonable and practical compared with that of [26], since the additional parameter $P_{k,t}$ is considered in (4) to reflect the relationship between the real profit and the maximum achievable profit. The generator total profit for each t is represented as the sum of the profits of the i, as follows:

$${\mathcal{U}}_t^G\left(P_{i,t},p_t|P_{k,t}\right)=\frac{R_t\left(p_t|P_{k,t}\right)}{M_t\left(P_{i,t},p_t\right)}=\frac{R_t\left(p_t|P_{k,t}\right)}{{\sum }_{i=1}^I{M}_{i,t}\left(P_{i,t},p_t\right)}$$

(5)

2.2. EU Profit

It was supposed that EUs schedule their energy consumption in consideration of the time-varying electricity price to minimize their electricity charge without reducing the total amount of electricity that should be used. The electricity charge accumulated until the $t$ for the $P_{k,t}$ of the k is written as ${\mathcal{U}}_k^{EU}\left(P_{k,t}|p_t\right)$, as follows:

$${\mathcal{U}}_k^{EU}\left(P_{k,t}|p_t\right)={\displaystyle \sum }_{t=1}^t{\mathcal{U}}_{k,t}^{EU}\left(P_{k,t}|p_t\right)={\displaystyle \sum }_{t=1}^t{p}_t{P}_{k,t}={\displaystyle \sum }_{t=1}^t{p}_t\left({\widehat{P}}_{k,t}+∆P_{k,t}\right)$$

(6)

where ${\widehat{P}}_{k,t}$ is the amount of electricity to be used in the future divided by the remaining time, and $∆P_{k,t}$ is the additional or abandoned electricity usage for the current t in consideration of the current electricity price $p_t$.

Remark 1.

Reducing the total electricity usage itself will obviously reduce the electricity charge, but its usage is not intended. In this paper, $∆P_{k,t}$ was modeled to reduce the electricity charge through the adjusting of the $∆P_{k,t}$ while the total electricity consumption was retained. For example, the $∆P_{k,t}$ increased at the low $p_t$ and decreased at the high $p_t$ in consideration of the setting of different prices for different periods by the generators. Then, the total electricity charge accumulated until the $t$, and then ${\mathcal{U}}^{EU}$ was introduced as the sum of each k electricity charge, as follows:

$${\mathcal{U}}^{EU}\left(P_{k,t}|p_t\right)={\displaystyle \sum }_{k=1}^K{\mathcal{U}}_k^{EU}\left(P_{k,t}|p_t\right)$$

(7)

2.3. Optimization of the Problem Formulation Based on the Stackelberg Game Model

In the game model of the present study, the generator acts as a follower to observe the electricity usage of the EUs for each t and the maximization of its profit; then, the EU acts as a leader to observe the generator electricity price of each t and the minimization of its electricity charge. Based on the game model, the Stackelberg game was formulated as $\mathcal{G}=\left(\mathcal{N}, \mathcal{A}, \mathcal{U}\right),$ where $\mathcal{N}$ is a player set that is composed of generators and EUs, $\mathcal{A}$ is the constraint set, and $\mathcal{U}$ is the profit set. Regarding P1, the generator profit, ${\mathcal{U}}_t^G$ is defined as the extent to which the generator can increase the profit given the EU electricity usage. In this paper, the constraints of the power generation unit were not considered. In other words, we assume that the power generators are able to flexibly adjust the amount of the power generation to maximize their profit by the proposed dynamic algorithm in Section 3. To solve P1, it was assumed that the generator can observe the EU electricity usage $P_k$. In P2, ${\mathcal{U}}_t^{EU}$ is the EU electricity charge that is for the maximization of its profit in consideration of the current electricity price. The optimization problem for the two players is formulated as follows:

$$\left(\mathrm{P}1\right)\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\eta ={\mathcal{U}}_t^G\left(P_{i,t},p_t|P_{k,t}\right)$$

(8)

subject to:

$${\mathcal{A}}_G=\left\{\left(P_{i,t},p_t|P_k\right)|\begin{array}{c}{P}_i^m\le P_{i,t}\le P_i^M\\ p^m\le p_t\le p^M\\ \sum _{k=1}^K{P}_{k,t}\le \sum _{i=1}^I{P}_{i,t}\end{array}\right\}$$

(9)

$$\left(\mathrm{P}2\right)\underset{\left\{P_{k,t}\in {\mathcal{A}}_{EU}\right\}}{\min }{\mathcal{U}}^{EU}\left(P_{k,t}|p_t\right)$$

(10)

subject to:

$${\mathcal{A}}_{EU}=\left\{\left(P_{k,t}|p_t\right)|\begin{array}{c}{P}_k^m\le P_{k,t}\le P_k^M\\ \sum _{t=1}^T{P}_{k,t}\ge {\phi }_k\end{array}\right\}$$

(11)

Here, $p^m$ and $p^M$ are the minimum and maximum prices of the electricity, and $P_k^m$ and $P_k^M$ are the minimum and maximum electricity-usage values of each k, respectively. For each responsive demand, a certain adjustable range of the electricity usage maximizes its own profit through the adjusting of the $P_{k,t}$ for the t. Furthermore, the minimum required electricity consumption ${\phi }_k$ of each EU is available for the entire time and an electricity amount should be efficiently used. This enables the EUs to shift heavy consumption loads from the peak-price time slots to the nonpeak-price time slots [28].

3. Profit Maximization

In this section, the two profit-optimization algorithms which can maximize the social welfare, which are for the generators and the EUs, are proposed based on the time-hierarchy structure of the Stackelberg game.

3.1. Generator’s Best-Pricing and Power-Generation Strategy

In this subsection, to maximally increase the generator profit with the knowledge of the UE energy consumption, the generator profit is maximized as a part of the Stackelberg game. Firstly, to successfully acquire the P1 maximum profit, the nonconvex function of P1 was transformed into the convex function using nonlinear fractional programming [29], since the solving of the nonconvex function of P1 is extremely complex, and the optimum values can only be found using a brute-force approach. Then, the Lagrangian dual decomposition was applied as a greedy-type iterative solution to the transformed convex function to estimate the optimum argument set $\left\{P_{i,t},p_t\right\}$, where the constraint set ${\mathcal{A}}_G$ is guaranteed. By exploiting the properties of nonlinear fractional programming [29], P1 is equivalent to P3, as follows:

$$\left(\mathrm{P}3\right)\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\left\{R\left(p_t|P_{k,t}\right)-\eta M\left(P_{i,t},p_t\right)\right\}$$

(12)

subject to ${\mathcal{A}}_G$, where ${\eta }^{\star }$ is the P1 optimum solution when the $\left\{P_{i,t},p_t\right\}$ is equal to the optimal argument set $\left\{P_{i,t}^{\star },p_t^{\star }\right\}.$ To mathematically prove that P3 is the convex and equivalent function, the theorem for the transformation was given with the defining $F$ as a set of feasible solutions and the maximum profit as ${\eta }^{\star }$ in the maximization problem P1, as follows [29]:

Theorem 1.

${\eta }^{\star }=\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\left\{{\mathcal{U}}_t^G\left(P_{i,t}, p_t|P_{k,t}\right)|\forall \left\{P_{i,t}, p_t\right\}\in F\right\}=\frac{R\left(p_t^{\star }|P_{k,t}\right)}{M\left(P_{i,t}^{\star },p_t^{\star }\right)}$ if and only if, $\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{ \max }\left\{R\left(p_t|P_{k,t}\right)-{\eta }^{\star }M\left(P_{i,t}, p_t\right)\right\}=R\left(p_t^{\star }|P_{k,t}\right)-{\eta }^{\star }M\left(P_{i,t}^{\star },p_t^{\star }\right)=0,$ for $R\left(p_t|P_{k,t}\right)\ge 0$ and $M\left(P_{i,t}, p_t\right)>0$.

Theorem 1 represents P1 in the fractional form that can equivalently be transformed into the subtractive form of P3. To prove Theorem 1, the transformed function $F\left(\eta \right)$ was defined as follows:

$$F\left(\eta \right)=\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\left\{R\left(p_t|P_{k,t}\right)-\eta M\left(P_{i,t},p_t\right)\right\}$$

(13)

By following the approaches of [29], it was possible to prove Theorem 1.

Proof of Theorem 1.

Convexity and equivalence. □

Lemma 1.

$F\left(\eta \right)$ is exactly monotonically decreasing for $\eta$ (i.e., $F\left({\eta }^{\prime }\right)>F\left({\eta }^{″}\right)$ if ${\eta }^{\prime }<{\eta }^{″}$.

Proof.

Let ${\eta }^{″}$ maximize $F({\eta }^{″})$, then $F({\eta }^{″})=\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\left\{R(p_t|P_{k,t})-{\eta }^{″}M(P_{i,t},p_t)\right\}=R(p_t^{″}|P_{k,t})-{\eta }^{″}M(P_{i,t}^{″},p_t^{″})<R(p_t^{″}|P_{k,t})-{\eta }^{\prime }M(P_{i,t}^{″},p_t^{″})\le R(p_t^{\prime }|P_{k,t})-{\eta }^{\prime }M(P_{i,t}^{\prime },p_t^{\prime })=\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\left\{R(p_t|P_{k,t})-{\eta }^{\prime }M(P_{i,t},p_t)\right\}=F({\eta }^{\prime })$, where $R(p_t^{″}|P_{k,t})-{\eta }^{\prime }M(P_{i,t}^{″},p_t^{″})\le R_{i,t}(p_t^{\prime }|P_{k,t})-{\eta }^{\prime }M(P_{i,t}^{\prime },p_t^{\prime })$ is reasonable, because $R(p_t^{\prime }|P_{k,t})-{\eta }^{\prime }M(P_{i,t}^{\prime },p_t^{\prime })$ is the maximized value when the input value ${\eta }^{\prime }$ is given in the function $F(·)$. □

Lemma 2.

Let any set be $\{P_{i,t}^{\prime }, p_t^{\prime }\}$ and the set satisfies ${\eta }^{\prime }=\frac{R(p_t^{\prime }|P_{k,t})}{M(P_{i,t}^{\prime },p_t^{\prime })},$ then $F({\eta }^{\prime })\ge 0$.

Proof.

$F({\eta }^{\prime })=\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }\left\{R(p_t|P_{k,t})-{\eta }^{\prime }M(P_{i,t},p_t)\right\}\ge R(p_t^{\prime }|P_{k,t})-{\eta }^{\prime }M(P_{i,t}^{\prime },p_t^{\prime })=0$. □

As shown in Lemma 1 and Lemma 2, it is natural that the transformed function $F\left(\eta \right)$ is convex, since $F\left(\eta \right)$ is monotonically decreasing and converges to zero. Further, the convergence of $F\left(\eta \right)$ to zero is representative of the generator profit, $\eta$, reaching the maximum value. Thus, it became possible to see P3 as equivalent to P1, and P3 is used as an equivalent objective function of P1 in the rest of this paper. Furthermore, to illustrate the solving of P1 through P3 as pseudocode, we construct the iterative algorithm, Algorithm 1: Generator’s Profit Maximization, in page 10. In Algorithm 1, the lines 3–8 are performed by the generators with the operation of the master-loop algorithm that is based on the slave-loop algorithm.

Remark 2.

When the number of master-loop iterations, $N_M^G$, continues, the $\eta$ increases and converges to ${\eta }^{\star }$ if $F\left(\eta \right)<\tau$, as shown in the lines 3–8 in Algorithm 1. Note that, for the convergence of the function $F\left(\eta \right)$, the threshold parameter 𝜏 is set to approximately positive zero, as represented by $\tau \approx + 0$. Furthermore, we have introduced nonlinear fractional programming [29] to ensure that our proposed solution has stability in reaching equilibrium. From Proof of Theorem 1, we confirmed that the P3 is not only equivalent to the P1, but also a function which is monotonically decreasing and converges to zero (Lemma 1 and Lemma 2). The P3 is monotonically decreasing converges to zero when the iteration number $N_M^G$ of Master-loop algorithm in Algorithm 1 increases, which means the equilibrium of P1 can be obtained with stability.

The slave-loop algorithm of Algorithm 1 line 5, can be considered as the solution of P3. By using the Lagrange dual method [30], it was then possible to solve the convex optimization problem of P3 to propose the slave-loop algorithm. To estimate the optimal arguments of $P_{i,t}^{\star }$ and $p_t^{\star }$, the Lagrange dual function of P3 was derived as follows:

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{G}}\left(P_{i,t},p_t,{\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t,{\theta }_t\right)=R_t\left(p_t|P_{k,t}\right)-\eta M_t\left(P_{i,t},p_t\right)-{\beta }_{i,t}{\sum }_{i=1}^I\left(P_i^m-P_{i,t}\right)-\\ {\gamma }_{i,t}{\sum }_{i=1}^I\left(P_{i,t}-P_i^M\right)-{\delta }_t\left(p^m-p_t\right)-{\epsilon }_t\left(p_t-p^M\right)-{\theta }_t\left({\sum }_{k=1}^K{P}_{k,t}-{\sum }_{i=1}^I{P}_{i,t}\right)\end{array}$$

(14)

where ${\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t$, and ${\theta }_t$ are the Lagrange multipliers. The Lagrange dual problem of P3 can be formulated as follows:

$$\underset{\left\{{\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t,{\theta }_t\right\}}{\min }\underset{\left\{P_{i,t},p_t\in {\mathcal{A}}_G\right\}}{\max }{\mathcal{L}}_{\mathrm{G}}\left(P_{i,t},p_t,{\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t,{\theta }_t\right)$$

(15)

By solving the following formulas of (16) and (17), which were derived using the Karush–Kuhn–Tucker (KKT) conditions [30], it is possible to simply derive the optimal values, $P_{i,t}^{\star }$ and $p_t^{\star }$, as follows:

$$\frac{\partial {\mathcal{L}}_G}{\partial P_{i,t}}=\frac{\eta \left(a_i+b_i{d}_i\right)}{d_i\sqrt{1-4d_i{P}_{i,t}}}-\eta +{\beta }_{i,t}-{\gamma }_{i,t}+{\theta }_t-\eta p_t\{\begin{array}{c}=0, P_{i,t}>0. \\ <0, \mathrm{otherwise}.\end{array}$$

(16)

$$\frac{\partial {\mathcal{L}}_G}{\partial p_t}=-\eta P_{i,t}+\frac{{\delta }_t-{\epsilon }_t+{\sum }_{k=1}^K{P}_{k,t}}{I}\{\begin{array}{c}=0, p_t>0. \\ <0, \mathrm{otherwise}.\end{array}$$

(17)

$$P_{i,t}^{\star }={\left[\frac{\left(A-\eta b_i\right)\ast \left(2\eta a_i+\eta b_i{d}_i+d_{i}A\right)}{{\left(2\eta a_i+2d_{i}A\right)}^2}\right]}^{+}$$

(18)

$$p_t^{\star }={\left[\frac{a_i+a_i{B}^{-\frac{1}{2}}+b_i{d}_i{B}^{-\frac{1}{2}}}{d_i}+\frac{{\beta }_{i,t}-{\gamma }_{i,t}+{\theta }_t}{\eta }\right]}^{+}$$

(19)

where $A=\eta p_t-{\beta }_{i,t}+{\gamma }_{i,t}-\theta$, and $B=1-4d_i({\delta }_{i,t}-{\epsilon }_{i,t}+\sum _{k=1}^K{P}_{k,t})/I\eta$, ${\left[X\right]}^{+}=\max \left\{X,0\right\}$. Moreover, the Lagrange multiplier ${\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t$ and ${\theta }_t$ can be updated by using gradient methods in a distributed manner, as follows:

$${\beta }_{i,t}\left(N_S^G+1\right)={\left[{\beta }_{i,t}\left(N_S^G\right)-{\omega }_1{\displaystyle \sum }_{i=1}^I\left(P_i^m-P_{i,t}^{\star }\right)\right]}^{+}$$

(20)

$${\gamma }_{i,t}\left(N_S^G+1\right)={\left[{\gamma }_{i,t}\left(N_S^G\right)-{\omega }_2{\displaystyle \sum }_{i=1}^I\left(P_{i,t}^{\star }-P_i^M\right)\right]}^{+}$$

(21)

$${\delta }_t\left(N_S^G+1\right)={\left[{\delta }_t\left(N_S^G\right)-{\omega }_3{\displaystyle \sum }_{i=1}^I\left(p^m-p_t^{\star }\right)\right]}^{+}$$

(22)

$${\epsilon }_t\left(N_S^G+1\right)={\left[{\epsilon }_t\left(N_S^G\right)-{\omega }_4{\displaystyle \sum }_{i=1}^I\left(p_t^{\star }-p^M\right)\right]}^{+}$$

(23)

$${\theta }_t\left(N_S^G+1\right)={\left[{\theta }_t\left(N_S^G\right)-{\omega }_5\left({\displaystyle \sum }_{k=1}^K{P}_{k,t}-{\displaystyle \sum }_{i=1}^I{P}_{i,t}^{\star }\right)\right]}^{+}$$

(24)

where the iteration steps ${\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4$, and ${\omega }_5$ are positive values, which are like a learning rate, for a more rapid convergence of the algorithm, and the parameter $N_S^G$ is the number of iterations for the slave loop in the line 5 of Algorithm 1: Generator’s Profit Maximization. The pseudocode of the iterative slave-loop algorithm is proposed in Algorithm 3. This is the operating structure of the master- and slave-loop algorithms, where the result of the slave-loop algorithm is the input of the master-loop algorithm.

In this paper, we assumed that there are multiple power generators and multiple EUs as game players in the proposed game algorithm. Each profit of the power generators and EUs, respectively, was defined in Section 2.1 and Section 2.2. However, we maximize the sum of the profits of the power generators and the EUs, so there exist two profit sums for the power generators and the EUs, given by Equations (5) and (7), respectively. If the profit sums are successfully maximized by the proposed algorithm, the profit of each player can be distributed according to the pre-defined profit function given by Equations (4) and (6) in Section 2.

The detailed description of the operation of Algorithm of Social-Welfare Maximization is as follows. The Algorithm of Social-Welfare Maximization is the hierarchical bi-level iterative algorithm with the Stackelberg-loop iteration number $t\in \mathcal{T}=\left\{1,\dots ,T\right\}$, and is composed of the two pseudocode tables, Algorithms 1 and 2. We define the generators as the “leader”, and the energy users as the “follower” as a game player in the proposed game structure. Originally $t$ denotes time set, but it is recognized as an iteration number. If the $T$ is set to be 24 and the time interval is 1 h, the Algorithm of Social-Welfare Maximization repeats 24 times until the Stackelberg-loop iteration number $t$ becomes 24. Whenever the iteration for $t$ is performed, the proposed Algorithms 1 and 2 are performed in succession. They play their algorithm in the Stackelberg game and do not play the game at the same time but play alternately over time to “interact” with each other and maximize their profit. Algorithm 1 is for generator’s profit maximization, and iterates with the master-loop iteration number $N_M^G$ until the line 6, $R\left(p_t^{\star }|P_{k,t}\right)-{\eta }^{\star }M(P_{i,t}^{\star },p_t^{\star })<\tau$, is satisfied. Note that the threshold parameter 𝜏 for the convergence of the function $R\left(p_t^{\star }|P_{k,t}\right)-{\eta }^{\star }M(P_{i,t}^{\star },p_t^{\star })$ should be set to around positive zero, $\tau \approx +0$, by the proof of Lemmas 1 and 2. Furthermore, before the line 6 is performed, the optimal value $P_{i,t}^{\star }$ and $p_t^{\star }$ should be calculated in the line 5 as an outcome of the Slave-loop Algorithm of Algorithm 1, Algorithm 3, with the slave-loop iteration number $N_S^G$. The Slave-loop Algorithm is performed to solve the Lagrange dual problem P3 based on Equations (18)–(24). When the line 6 is satisfied, the maximized profit for the generators is obtained with the optimal electricity price $p_t^{\star }$ and the optimal power generation $P_{i,t}^{\star }$. To interact with each other and effectively maximize the profit, if Algorithm 1 ends, the optimal price $p_t^{\star }$ calculated from Algorithm 1 is passed to the input of Algorithm 2, and Algorithm 2 is performed. Algorithm 2 is for energy user’s profit maximization, and the specific methodology is described in next Section 3.2. If Algorithm 2 is successfully performed and the maximum EU’s profit is calculated with the optimal electricity usage of the EUs $P_{k,t}^{\star }$, the optimal electricity usage $P_{k,t}^{\star }$ is also passed to the input of Algorithm 1 (which means the “interact”) and $t$ is incremented by one. We have described this passing of values as “spy on” in Introduction. By running the algorithm repeatedly and alternatively over time and spying on each other’s energy parameters, they can effectively maximize their own profit. The Algorithm of Social-Welfare Maximization (combination of Algorithms 1 and 2, lines 1–18) is as follows:

Algorithm 1. Generator’s Profit Maximization (lines 3–8, performed by the generator, master-loop algorithm).

1:  Input: time set $t\in \mathcal{T}=\left\{1,\dots ,T\right\}$. 2:  While $t\le T$, do 3:   Input: active generator set $i\in \mathcal{I}=\left\{1,\dots ,I\right\},$ $P_{i,t},$ $p_t,$ $P_{k,t}={\widehat{P}}_{k,t},$ $\eta =0,$ $\tau \approx +0$ (positive zero), $N_M^G=1$. 4:   Initialize $\eta ={\mathcal{U}}_t^G\left(P_{i,t},p_t|P_{k,t}\right)=\frac{R\left(p_t|P_{k,t}\right)}{M\left(P_{i,t},p_t\right)}=0$. 5:   Update $P_{i,t}^{\star }$ and $p_t^{\star }$ from Slave-loop algorithm (“Algorithm 3”). 6:    If $R\left(p_t^{\star }|P_{k,t}\right)-{\eta }^{\star }M(P_{i,t}^{\star },p_t^{\star })<\tau$ 7:     Return optimal parameters $\{P_{i,t}^{\star },p_t^{\star }\}$ and optimal profit of generators, ${\eta }^{\star }=\frac{R\left(p_t^{\star }|P_{k,t}\right)}{M\left(P_{i,t}^{\star },p_t^{\star }\right)}$. 8:    else Update $\eta =\frac{R\left(p_t^{\star }|P_{k,t}\right)}{M\left(P_{i,t}^{\star },p_t^{\star }\right)}$ and $N_M^G=N_M^G+1$, then Go to line 5.

Algorithm 2. Energy-User’s Profit Maximization (Market-adaptive Electricity-Usage Scheduling Algorithm, lines 9–17, performed by the EU).

9:   Input: $p_t^{\star }$, active EU set $k\in \mathcal{K}=\left\{1,\dots ,K\right\},$ $P_{k,t},$ ${\phi }_k,$ $t,$ $T,$ $t_w,$ ${\overline{P}}_{k,t}.$ 10:   For $k\in \mathcal{K}$ do 11:    Calculate ${\overline{p}}_t$ with $p_t^{\star }$ according to (26). 12:    Calculate $∆P_{k,t}$ with ${\overline{p}}_t$ according to (25). 13:    Calculate $P_{k,t}^{\star }$ with $∆P_{k,t}$ according to (27). 14:    Update ${\widehat{P}}_{k,t+1}$ with ${\widehat{P}}_{k,t}$ and $∆P_{k,t}$ according to (28). 15:   end for 16:   Calculate ${\mathcal{U}}_{k,t}^{EU}$ with (7). 17:   Let $t=t+1$. 18:  end while

Algorithm 3. Slave-loop Algorithm of the Generator’s Profit Maximization.

1:  Input: active generator set $i\in \left\{1,\dots ,I\right\},$ $P_{i,t},$ $p_t,$ $P_{k,t},$ $a_i,$ $b_i,$ $c_i,$ $d_i,$ $P_i^m,$ $P_i^M,$ $p^m,$ $p^M,$ ${\beta }_{i,t},$ ${\gamma }_{i,t},$. ${\delta }_t,$ ${\epsilon }_t,$ ${\theta }_t,$. ${\omega }_1,$ ${\omega }_2,$ ${\omega }_3,$ ${\omega }_4,$ ${\omega }_5,$ $N_S^G=1$. 2:  While ${\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t$ and ${\theta }_t$ are not converged do 3:   for $i\in \mathcal{I}$ do 4:    Update $P_{i,t}^{\star }=P_{i,t}^{\star }\left(N_S^G+1\right)$ with (18), $p_t^{\star }=p_t^{\star }\left(N_S^G+1\right)$ with (19), ${\beta }_{i,t}\left(N_S^G+1\right)$ with (20), ${\gamma }_{i,t}\left(N_S^G+1\right)$ with (21), ${\delta }_t\left(N_S^G+1\right)$ with (22), ${\epsilon }_t\left(N_S^G+1\right)$ with (23), and ${\theta }_t\left(N_S^G+1\right)$ with (24). 5:   end for 6:   Let $N_S^G=N_S^G+1$. 7:  end while

3.2. Demand’s Best Electricity-Usage Strategy

While the total electricity consumption was retained, the electricity charge of the EU was minimized with the knowledge of the current electricity price $p_t$, and this is another part of the Stackelberg game along with Algorithm 2: Energy User’s Profit Maximization (Market-adaptive electricity-usage scheduling algorithm) which is newly proposed in this subsection to solve P2. Algorithm 2 is one of the electricity-usage controlling strategies of the demand that is applicable to the time-varying electricity market. Firstly, it was assumed that the $P_k^m$ and $P_k^M$ are the minimum and maximum electricity-usage values for each t of each EU, respectively, and the total sum of the electricity usage of the entire time for each EU should be greater than or equal to the minimum requisite electricity consumption ${\phi }_k$. Furthermore, the entire time of the scheduling of the electricity usage of the EU was set to $T\in \mathcal{T}=\left\{1,\dots ,T\right\}$.

To initialize the electricity usage of each scheduling time before the applying of the proposed algorithm, it was assumed that each EU is supposed to consume ${\phi }_k/T$ during every scheduling time, and the electricity of ${\phi }_k$ is used for the entire time $T$. Further, ${\widehat{P}}_{k,t}$ was set as the amount of electricity that the k is expected to use during the current t, and the initial value of ${\overline{P}}_{k,t}$ is allocated to ${\phi }_k/T$. Then, in consideration of the electricity price of every scheduling time, the ${\widehat{P}}_{k,t}$ can be increased or decreased by the fluctuation of the electricity price. The amount of change in the electricity usage that is due to changes in the electricity price, $∆P_{k,t}$, is as follows:

$$∆P_{k,t}={\widehat{P}}_{k,t}\ast \left(1-\frac{p_t}{{\overline{p}}_t}\right)$$

(25)

$${\overline{p}}_t=\left(1-\frac{1}{t_w}\right)\ast {\overline{p}}_{t-1}+\frac{1}{t_w}\ast p_t$$

(26)

$$P_{k,t}^{\star }={\widehat{P}}_{k,t}+∆P_{k,t}$$

(27)

where ${\overline{p}}_t$ is the base price as the historical average of $p_t$ based on the electricity prices of the previous time slots, $t_w$ is the window size of the time slots, and $P_{k,t}^{\star }$ is the adjusted electricity usage according to the proposed algorithm. In Equation (25), a calculation of the amount of the change in the electricity usage is performed using the ratio of the base price to the current price, thereby acknowledging that the price that breaks past the average is a substantially increased price and the electricity consumption is reduced by the rate of increase, and vice versa. Then, in Equation (27), the electricity usage is updated for the current scheduling time.

To obtain insight regarding ${\overline{p}}_t$ according to $t_w$ in (26), it was assumed that the $t_w$ values are 1, 3, and 5. When $t_w$ is 1, $∆P_{k,t}$ is not generated in its structure, thereby meaning that the electricity consumption is calculated regardless of the electricity-price fluctuation, and this can be used for the control of the proposed algorithm result. Alternatively, the increasing $\mathrm{of}{t}_w$ to 3 or 5 means the determination of how sensitively $∆P_{k,t}$ is able to react to the current market price, since it was assumed that ${\overline{p}}_t$ is the average of the prices in the previous $t_w$ occasions. This provides an opportunity to adaptively reflect the market characteristics to the rapidly fluctuating or the gentle market. To retain the total electricity consumption in the proposed algorithm, the $∆P_{k,t}$ is uniformly collected or distributed for each remaining time block. If the $∆P_{k,t}$ is negative, the $∆P_{k,t}$ x amount is divided by the total remaining time $T-t$, and it is then distributed among each of the scheduling time blocks, and ${\widehat{P}}_{k,t+1}$ is updated as follows:

$${\widehat{P}}_{k,t+1}={\widehat{P}}_{k,t}-∆P_{k,t}/\left(T-t\right)$$

(28)

By performing (28), it is possible to constantly retain the electricity that is for consumption, ${\phi }_k$. We propose Algorithm 2: Energy User’s Profit Maximization reflecting the whole description and formulas in Section 3.2 as a profit maximization algorithm performed by the EU.

Remark 3.

Algorithm 2 was proposed to solve the problem P2. The equilibrium of P2 is involved in the iteration t in Algorithm of Social-Welfare Maximization, which gradually reaches the equilibrium point whenever t increases. According to the maximization structure of the proposed algorithm, if the pre-defined total energy consumption ${\phi }_k$ are all distributed according to the algorithm, the equilibrium can be stably reached.

3.3. Complexity Analysis

The computational complexity of the Algorithm of Social-Welfare Maximization based on the optimization technique used in this paper can be evaluated as follows. First, the complexity of the gradient method updating dual variables to obtain the optimal price $p_t^{\star }$ and the optimal power generation $P_{i,t}^{\star }$ in Slave-loop Algorithm (Algorithm 3) of Algorithm 1 linearly increases with the number of generators I and the number of iterations $N_S^G$, i.e., $\mathcal{O}\left\{N_S^{G}I\right\}$ where $\mathcal{O}\left\{·\right\}$ is Big O notation. Second, as provided in Section 3.1, the dual function P3 is always convex by proof of Theorem 1, and the gradient method was employed to update $\left\{{\beta }_{i,t},{\gamma }_{i,t},{\delta }_t,{\epsilon }_t,{\theta }_t\right\}$ toward the optimal solution with guaranteed convergence [29]. Thus, in Master-loop Algorithm of Algorithm 1, the complexity of the Dinkelbach method [29] to update $\eta$ is independent of I and linearly increases with the number of iterations $T_2$, i.e., $\mathcal{O}\left\{N_M^G{N}_S^{G}I\right\}$. Third, when we consider the number of EUs $K$ in Algorithm 2 and the number of iteration $t$, the complexity becomes $\mathcal{O}\left\{t{N}_M^G{N}_S^{G}IK\right\}$. Therefore, the total complexity of the proposed algorithm is $\mathcal{O}\left\{t{N}_M^G{N}_S^{G}IK\right\}$. For comparisons, the complexity of the exhaustive search [31] is roughly $\mathcal{O}\{{\left(K+I\right)}^{\left(K+I\right)}\}$, where K and I are the number of generators and EUs, respectively.

3.4. Schematic Overview and Application of Proposed Algorithms

In this paper, the two game players (generators and EUs) are supposed to participate in the Stackelberg game, and play the game in order to maximize their monetary profit in the smart grid demand response. We proposed two algorithms (Algorithm 1: Generator’s Profit Maximization and Algorithm 2: Energy User’s Profit Maximization) in Section 3.1 and Section 3.2, respectively, as an energy strategy to achieve profit maximization, and the generators and the EUs play Algorithms 1 and 2 alternately in time, respectively, as shown in Figure 2. The proposed game structure is “dynamic game” from the following two reasons:

• The game players, generators and EUs, interact with observing the each other’s energy strategy for profit maximization. The generators observe the electricity consumption of the EUs, and the EUs observe the electricity price of the generators in the proposed game operation.
• The proposed profit maximization game was constructed as an iterative algorithm where the energy strategies (Algorithms 1 and 2) are repeatedly performed up to the specified number of times, for example 24 times as 24 h a day.

They play their algorithm in the Stackelberg game and do not play the game at the same time but play alternately over time since the EU needs to know how much the generator has set the current electricity price in order to optimally control his energy consumption $P_{k,t}^{\star }$, and the generator needs to know how much power the EU is currently consuming in order to determine the electricity price $p_t^{\star }$ and the amount of power generation in each game play (the game play means the algorithm operation). By running the algorithm repeatedly and alternatively over time and spying on each other’s energy parameters, they can effectively maximize their own profit. The Algorithm for Social-Welfare Maximization is detailed in Algorithms 1 and 2 as the Stackelberg-loop algorithm with the iteration number $T$. The total architecture of the proposed social welfare maximization including Algorithms 1 and 2 is described in Figure 2 as an overview of the algorithms proposed in this paper.

The proposed methodology to achieve the social-welfare maximization can provide the advantage to maximize the monetary profit of the generators and EUs, but we also reveal the following considering points expected in real-world implementation; 1. Algorithm 1 proposed to maximize the profit of the generator in this paper derives a sub-optimal solution that can reduce the computational complexity compared to the brute-force approach to be suitable for real-time optimization in smart grid demand response. The performance of the algorithm can vary slightly depending on the initial point and the values of the variables that make up the Algorithm; 2. Algorithms 1 and 2 proposed in this manuscript can be implemented in a smart meter or the EMC as a form of software to automatically control the power consumption of energy user’s appliances and facilities. It is a system that can operate only in limited areas equipped with the AMI; 3. Generators are required to have the ability to flexibly control their power generation to participate in the proposed algorithm. We need to consider practical implementations to enable real-time operation of the proposed cooperative and simultaneous usage of coalitional game theory methods.

• In the EU side, the implementation method of the proposed profit maximization algorithm is as follows: The proposed profit maximization algorithm and formulas is applicable to general demand response applications between generators and EUs such as, residential households, electrical appliances, new smart appliances and internet of things (IoT) devices as a real-world scenario. In the EU side which can be used at high priced hours, we can effectively reduce electricity charges by adjusting the energy usage according to the electricity price with the proposed Market-Adaptive Electricity-Usage Scheduling Algorithm (Algorithm 2 in the manuscript). To implement the Market-Adaptive Electricity-Usage Scheduling Algorithm, automatic electricity usage controller can be needed to be implemented and connected to EU applications (the residential households, IoT devices and etc.). The automatic electricity usage controller can be developed by porting a functional software which quickly and dynamically performs the automatic electricity usage control to an AMI or an EMC. In the case of AMI and EMC, it is possible to transmit and receive electricity price information in real time through power line communications in smart grid. Based on this, the proposed algorithm can be fully implemented and operate to achieve energy usage optimization and electricity charge savings. To summarize, EUs should choose their appliances or facilities to automatically control their energy usage to maximize monetary profit, and if they are connected to a smart meter or EMC equipped with our proposed algorithms, the proposed profit maximization system will simply be able to operate. We think that the proposed system can be implemented in the direction of utilizing existing infrastructure such as the smart meter and EMC.
• On the generator side, the implementation method of the proposed profit maximization algorithm is as follows: In order to realize the optimal power generation and optimal pricing based on the proposed algorithm in the generators side, the generators should be able to integrate and manage the total power generation and the electricity price by forming a coalitional for profit maximization themselves. Or as a top authority for power generators that are already integrated and managed by the government can implement the proposed coalitional game theory methods. Or a third party such as a power retailer that runs various demand response programs can implement the proposed game theory. If the proposed algorithm is implemented and operated, it should be able to interact with another game player, energy user, with through power line communication or wireless local area network (WLAN) on the smart grid for information communication, such as real-time electricity price and electricity consumption exchange required by this algorithm.

In view of appropriate time intervals for this real-time operation and implementation, we consider the time interval of one hour is reasonable, and we can think about a smaller or larger interval based on this one hour. For example, a 10-min interval that is smaller than 1 h is expected to cause confusion because too much dynamic power generation and power consumption changes for both the generators and the energy users can be caused. On the other hand, if we set a time interval greater than one hour to a time interval, the proposed profit maximization system can be somewhat inefficient if we run the system once a day at a specific time because we have a fairly wide variation in power consumption trends during a day. Therefore, it is reasonable to set the time interval appropriately between minimum 1 h and maximum 24 h in consideration of country, region and environment in which the proposed profit maximization system operates.

4. Simulation Results

In this section, numerical results are provided to demonstrate the effectiveness of the proposed algorithm. The system setup is as follows: six generators and 12 EUs are considered based on the IEEE 39-BUS system [32]. The graph of the communication network of these generators and EUs shows that they are strongly connected. The parameters of the EUs and the generators are given in

Table 2. Parameters of generators for IEEE 39-BUS [32].

Generator Parameters
Node	${\mathit{a}}_{\mathit{i}}$	${\mathit{b}}_{\mathit{i}}$	${\mathit{c}}_{\mathit{i}}$	${\mathit{d}}_{\mathit{i}}$	${\mathit{P}}_{\mathit{i}}^{\mathit{m}}$	${\mathit{P}}_{\mathit{i}}^{\mathit{M}}$
1	0.0024	5.56	30	0.00021	60	339.69
2	0.0056	4.32	25	0.00031	25	479.10
3	0.0072	6.60	25	0.00011	28	290.4
4	0.0047	3.14	16	0.00022	40	306.34
5	0.0091	7.54	6	0.00041	35	593.80
6	0.0046	4.76	12	0.000121	30	443.41

and

Table 3. Parameters of EUs for IEEE 39-BUS [32].

Demand Parameters
Node	${\mathit{P}}_{\mathit{k}}^{\mathit{m}}$	${\mathit{P}}_{\mathit{k}}^{\mathit{M}}$	Node	${\mathit{P}}_{\mathit{k}}^{\mathit{m}}$	${\mathit{P}}_{\mathit{k}}^{\mathit{M}}$
1	50	100.34	7	80	137.93
2	100	159.13	8	50	84.19
3	40	80.56	9	50	104.06
4	30	123.98	10	78	119.36
5	80	109.55	11	103	176.19
6	40	76.34	12	67	147.26

[32]. The constant time length $t_l$ was set to 1 h, and the total scheduling time was set to $T=24$ (one day) with the minimum requisite electricity consumption ${\phi }_k=T\ast \left(P_k^m+P_k^M\right)/2$. Further, the initial value of the ${\widehat{P}}_{k,t}$ was set to ${\widehat{P}}_{k,t}={\phi }_k/T$, and it was assumed that the lower bound of the electricity price per unit of energy is nonzero, whereas the upper bound is 50 (cents/kWh). The window size $t_w$ was set to be from 1 to 10. In the following results, the maximum number of slave-loop iterations was set to 6.

Figure 3 shows the evolution of the generation power (kWh) $P_{i,t}$ and the electricity price (cents/kWh) $p_t$ of each generator optimized by the proposed iterative Algorithm 1. “Iterations” in Figure 3 is the number of master-loop iterations $N_M^G$ of Algorithm 1, and the result was averaged over 1000 independent simulations, each of which involved different scheduling time slots. From Figure 3, it is evident that the convergence of the generation power and the price can be achieved within seven iterations on average to maximize the ${\mathcal{U}}_t^G$. All of the values in each iteration change with satisfying the ${\mathcal{A}}_G$ in (9). To maximize the generator profit, the generation power of each generator converges to a different optimal value, which is affected by the constant parameters such as $a_i$, $b_i$, $c_i$, $d_i$, $P_i^m$, and $P_i^M$, and the price converges to its optimal value in conjunction with the generation power.

Figure 4 demonstrates the evolution of the generator profit calculated with the employment of the optimal values of the generation power and the electricity price from Figure 3. The optimization process is performed using the proposed master- and slave-loop algorithms in Algorithm 1. Even though the generator power curves in Figure 3 show fluctuations with the iterations, it is evident that all of the curves continually increase as the iterations continue. Furthermore, a baseline scheme was set as a profit maximization strategy which optimizes only one of the two optimization parameters $\left(P_{i,t},p_t\right)$ compared to the proposed Algorithm 1, where the other optimization parameter was fixed. The “fixing” means that the fixed parameter was not optimized but its value is constant, and the fixed value of $P_{i,t}$ was set as just one of the possible values to $\left(P_i^m+P_i^M\right)/2$. In Figure 4, it is confirmed that the profit of the proposed algorithm is greater than that of the baseline scheme, thereby confirming the proper operability of the algorithm. Furthermore, the proposed optimization problem in this paper is a non-convex optimization problem (P1 in Section 2) and the global optimum for the non-convex optimization problem is usually only achieved by using a brute-force approach (or exhaustive search). In this paper, to solve P1, Lagrange dual method and non-linear fractional programming were applied to find a sub-optimal point close to the global optimum. From Figure 4, the global optimum point was further suggested, and it was confirmed that the difference from the sub-optimal point is within 1% when the iteration is converged to 7. Since the sub-optimal point we found may differ depending on the initial point in the proposed profit maximization algorithm. So we propose a method to calculate multiple sub-optimal points with multiple initial points to determine the more maximized point among the multiple sub-optimal points. In the implementation phase, this algorithm should be devised to generate multiple initial points and calculate the sub-optimal point corresponding to the multiple initial points.

Remark 4.

It is not possible to run the baseline scheme by fixing the $p_t$. When the $p_t$ is fixed, it is obvious that the $P_{i,t}$ will be set to the minimum value to maximize the profit according to the structure of Equation (4).

In Figure 5, Figure 6 and Figure 7, the electricity price, the electricity usage of one of the EUs according to the price, and the accumulated electricity charge are depicted over time. All of the results were simulated for $T=24$ h (one day) with 1-h intervals, and the window size was set to be $t_w=3$. Figure 5, Figure 6 and Figure 7 show the changes in the electricity price as Case #1: side-crawl trend, Case #2: rising-tide trend, and Case #3: falling-tide trend, respectively, and these are case studies where the possible trends were analyzed in the real market to determine the effectiveness of the proposed Algorithm 2. Please note that, regarding the totals of Algorithms 1 and 2, Algorithm of Social-Welfare Maximization, the electricity prices of all of these cases were calculated using Algorithm 1, while the electricity usage and the electricity charge were calculated using Algorithm 2. Also, these three cases are three of the results that were obtained by the performance of the entire Algorithm of Social-Welfare Maximization over 1000 times. From Figure 5, Figure 6 and Figure 7, the changes of the electricity usage of EU 1 show that the proposed Algorithm 2 operates in a market-adaptive manner (EU 1 is merely a representative of the EUs, and the rest of the EUs show the same tendency), as the expectation of this is described in Section 3.2. Based on this market-adaptive manner, the electricity usage is reduced when the electricity is expensive, whereas the electricity usage increases when the electricity is cheap, thereby meaning that the total electricity consumption is the same, but the electricity charge can be considerably reduced. Please note that this means that the proposed algorithm can alleviate the existing PAR reduction problem because the electricity usage immediately responds to the price according to the results of Figure 5, Figure 6 and Figure 7. In the real world implementation, it is also possible to alleviate the problem that the price greatly fluctuates because if the price changes greatly, the generators will lose its profits and will not be able to withstand it. We can contribute to stabilization of power generation and PAR reduction, which is one of the ultimate goals of DR through our proposed algorithm. Figure 5, Figure 6 and Figure 7 show that the average electricity usage is the same in any trend according to the ${\mathcal{A}}_{EU}$, and the electricity charge can be greatly reduced from 13–18%. The comparator, which consumes the same power at all times, is “unadjusted” in the legend. Furthermore, from Figure 5, Figure 6 and Figure 7, it can be seen that the greater the variability in the market, the greater the possibility of the adaptation to the market that facilitates the attainment of a greater benefit.

We can think of the proposed game methodology as a type of DR because the demand is responsive to price as the reviewer commented, but it is not exactly a DR program. In general DR should include the ability to convert excess power consumption to optimal power consumption (“load reduction”), as well as to provide price elasticity to eliminate inefficiencies due to fixed prices. In the proposed game methodology, we do not reduce the power consumption of the energy user, but optimize how efficiently the specified power consumption will be consumed in a given period of time. In Figure 5, Figure 6 and Figure 7, to maximize the profit of the EU, we estimated the adjusted electricity usage which should be consumed for each hour when we know an amount of energy we should use during the day. When the electricity price changes as shown in the top figure of Figure 5, Figure 6 and Figure 7, respectively, the electricity charge can be reduced by 13%, 18% and 16% when calculating the electricity usage as shown in the middle figure of Figure 5, Figure 6 and Figure 7 based on the proposed algorithm. We studied in this paper how to use the load efficiently within the reduced load when the energy user receives an instruction to reduce the load on the DR.

Remark 5.

In the proposed algorithm, the profit function of each generator and EU is defined mathematically in Section 2 so that the profit generation and distribution can be fair. Each generator and EU does not need special skills for fairness because it is an independent entity and can take its own profits by the defined profit function.

Figure 8 represents the way the market volatility (degree of change of the electricity price) provides the EU with benefits when the window size changes from 1 to 10. The benefit function is first given in Figure 8, as follows:

$$\mathrm{Benefit}=\left(1-\frac{{\mathcal{U}}^{EU}}{{\overline{\mathcal{U}}}^{EU}}\right)\ast 100$$

(29)

where ${\overline{\mathcal{U}}}^{EU}$ is the total electricity charge that is accumulated for the $T$ when the electricity usage is unadjusted and allocated in a totally flat manner. Note that the reduction percentages in Figure 5, Figure 6 and Figure 7 were calculated using Equation (29). Also note that, in Figure 8, the peak-to-average ratio (PAR) is an approximate indicator of the electricity usage as it is known, and it was assumed that the change in the PAR reflects the volatility of the electricity price, because the electricity usage fluctuates when the electricity price fluctuates according to Algorithm 2. It is evident that Algorithm 2 responds sensitively to the changes in the electrical price; that is, the smaller that $t_w$ is, the greater the benefit that is derived, with the exception of the case where $t_w$ is 1. Here, $t_w$ is 1, and this means that the electricity usage has not been adaptively adjusted to the market using Equation (26), while it is also accurate that the PAR is 1. In the meantime, it is possible to observe the trends of the increasing benefit as the PAR is increased, and this means that, as the market volatility is increased, Algorithm 2 can increase the benefit. The purpose of this simulation, however, is not the raising of the PAR to increase the benefit. The purpose is the demonstration of the benefit that is attained from the proposed algorithm when the EU has some PAR.

Summary of Simulation Results and Insights

We showed the results of maximizing the profit of the generators through Algorithm 1 in Figure 3 and Figure 4, and we demonstrated the results of maximizing the profit of the EUs based on Algorithm 2 in Figure 5, Figure 6 and Figure 7. In Figure 3 and Figure 4, we confirmed that the profit of the generators can be improved to about 45% compared to existing (baseline scheme) scheme, and the electricity charge of the EUs can be reduced by 15.6% on average compared to that of when algorithm was not applied. Please note that the amount of power consumption of the EUs is same when the algorithm was applied and not applied. From Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7, we confirmed that the proposed profit maximization algorithms effectively improves the monetary profit of generators and EUs. We summarize the gain of monetary profit from the proposed algorithms in

Table 4. The gain of monetary profit from the proposed algorithm.

Energy Strategy	Gain of Profit
Algorithm 1: Generator Profit Maximization	About 45% (compared to existing (beseline scheme) scheme)
Algorithm 2: Energy User Profit Maximization	15.6% on average (compared to that of when algorithm was not applied)

.

To provide and investigate the influence of the changes of the PAR on the monetary profit generated from the proposed algorithms, Figure 8 showed the change of the profit of the EUs. From Figure 8, we confirmed that the EUs gain more profit as the PAR increases. What this means that more EUs will participate in this game and algorithms when the PAR increases, and it can lead to PAR reduction because the monetary gain that can be obtained for the current PAR has no choice but to be limited, and it can be predicted that as many people share the profit, the PAR would decrease. This also suggest the proposed algorithm not only can contribute to maximization of the profit of the generators and the EUs, but also the desired goal of demand response, PAR reduction, at the same time.

5. Conclusions

In this paper, to maximize the monetary profit in real-time price DR systems, we formulated the Stackelberg game-based non-convex optimization problem, and proposed two energy strategies, Algorithm 1: Generator’s Profit Maximization and Algorithm 2: Energy User’s Profit Maximization, as an optimal solution. In the problem formulation, we newly formulated the generator profit function to reflect the influence of the electricity usage of EUs. To solve the non-convex optimization problem, nonlinear fractional programming and the Lagrange-multiplier method were adopted in proposing the energy strategy for the generators. Also, we newly proposed the energy strategy for the EUs based on the time-window-based market-adaptive manner. We greatly improve the monetary profit of the generators and EUs using the proposed two energy strategies by optimizing the amount of the power generation and the electricity price in the generator side, and electricity consumption in the EU side. In Figure 3 and Figure 4, we confirmed that the profit of the generators can be improved to about 45% compared to the existing (baseline) scheme, and the electricity charge of the EUs can be reduced by 15.6% on average compared to that of when algorithm was not applied. Furthermore, we confirmed that the simulation result from Figure 8 suggests the proposed algorithm can contribute to not only maximization of the profit of the generators and the EUs, but also the desired goal of demand response, PAR reduction, at the same time.