Modeling and Comparative Analysis of Multi-Agent Cost Allocation Strategies Using Cooperative Game Theory for the Modern Electricity Market

Abstract

The electrical market scenario has changed drastically in the last decade. In the presence of increased competition and less tolerant players, more sophisticated methods are required to balance the diversity and differential pricing while promoting cooperation among the agents. In the monopolistic environment, the central utility incurred the total cost of the transmission expansion. But as the current scenario demands, there are several public and private market players. The growth will benefit all the players, so the total cost in transmission expansion can be divided among players as per the benefit received by each player. In this paper, a transmission system expansion planning problem in the cooperative environment using cooperative game theory (CGT) is framed for the power sector, in which various players can cooperate in a coordinated manner to maximize their benefit but ultimately strengthen the power grid. In this paper, we have modeled, analyzed and compared various cost allocation methods of cooperative game theory specifically for the cost allocation in a transmission expansion planning problem. The present work focuses on forming coalitions to calculate the costs using the forward search and frog leap optimization approach. We have compared the SCRB, BSV, ENSC, and ACA methods for transmission expansion planning while attempting to satisfy the axioms. We have also observed that bilateral Shapely value efficiently allocated the costs due to its decentralized approach and the sequencing of coalition formations to achieve the best possible cost allocations.

1. Introduction

In the deregulated environment, the electricity supply industry consists of several players, and hence planning decisions should involve the decisions taken by these market players of the expansion scenario [1,2,3]. Cooperative game theory [4,5] is used for cost allocation of transmission system expansion planning while analyzing the correlations among the agents as per their preferences [6,7].

In cooperative games, the players can form groups and share the profits by cooperating using side payments and payoffs [8]. The outcome is divided among players at the game’s termination state (end-state) [9]. This outcome of players in quantitative terms is called payoff. These game players are also termed ‘agents’ in a multi-agent system (MAS). A MAS has several decision-makers independent of each other. These agents make the decisions depending on the market conditions, as they are autonomous bodies working to maximize their own profit.

Initially, in the vertically integrated environment, the transmission system expansion planning (TSEP) decisions were taken by the central utility alone, as the primary purpose of the electricity industry is to serve consumers reliably. But because of industrialization and globalization, the deregulation of the electricity industry also began. The restructuring and deregulation of the industry started a new era of electricity trading with several private players and joint ventures. So far, the significant privatization impacts are found in generation and distribution facilities, so transmission is also not left untouched. Considering all these aspects, transmission planning decisions in this new era cannot be centralized; they should consider various players’ decisions, as this process is intertwined [10,11]. Several players of the game will use the newly built transmission lines.

For this reason, the planning tools (e.g., optimal power flow) used by the central utility in the past need to be reworked to adapt to the present electricity scenario. Game theory (GT) is one solution that fits well to a new decentralized and competitive environment, thereby allowing the use of the tools of the traditional utility to address the problems without losing their power to solve. It is used in the non-cooperative environment to model sustaining cooperation.

The game can be modeled as follows [12]. Consider the N-player game on an infinite time horizon:

$$x=\mathrm{f}\left(x\right)+{\displaystyle \sum }_{j=1}^{N}gj\left(x\right)uj$$

(1)

where state x(t) ∈ Rⁿ, players or controls uj(t) ∈ R^m_j.

The associated cost with each player can be

$$\begin{gathered} Ji \left(x\right(0), u_1,u_2,\dots \dots . u_N) \\ ={\int }_0^{\infty }(Qi\left(x\right)+{\sum }_{j=1}^N{u}_j^T R_{ij}{u}_j) dt \\ ={\int }_0^{\infty }{r}_i\left(x\left(t\right), u_1,u_2,\dots \dots . u_N\right) dt;i\in N \end{gathered}$$

(2)

where $R_{ii}$ > 0, $R_{ij}$ ≥ 0 are symmetric matrices.

A cooperative game in game theory is defined as a competitive game where players attempt to form a coalition by interacting among themselves to maximize the profits (collective payoffs) via agreements.

The Nash equilibrium defines the optimal solution in non-cooperative games. The player doesn’t gain anything or book any profit in deviating from the initial strategy. The Nash equilibrium facilitates the participants to get the best possible outcome while predicting the decisions of other players [13].

A game is said to be in Nash equilibrium if the following conditions are met:

(a)

All players are required to maximize the gain within the limits and rules of the game;

(b)

All players should have flawless execution;

(c)

All players should have ample knowledge to analyze the problem domain to achieve the solution;

(d)

The equilibrium strategies should be communicated clearly and known to all;

(e)

Any deviation in an individual strategy should not cause any change in the strategy of other players. Also, the individual strategy should not be affected by any other player in the game.

In a pure Nash equilibrium, none of the players can get a higher payoff by deviating from the move, and all other players need to stick to their initial choices. As per Nash equilibria, the self-enforcing coalitions can take place where each player is allowed negotiated moves. In Nash equilibrium, each player behaves best in response to the other players [14].

Cooperative and non-cooperative game theories are types of game theory where the classification is based on the modeling concept employed [15,16]. Both the theories have the following general components:

• Individual decision-makers: Players;
• Decisions (choices) of the players (within constraints): Actions;
• Rules of the game.

The outcome is the end state of the game, and the payoff is the final cost with the player after the game. The payoff of the i-th player is represented by x_i. The payoff vector is the collection of the entire player’s payoff. The coalition is accepted in cooperative games where players can form groups to maximize their profits. It is generally based on certain agreements [17]. The participating agents will ally to book more profits.

This paper presents the application of a few cooperative game theory (CGT) concepts in a MAS environment to allocate the transmission expansion cost among all market players. The ultimate purpose of cooperation and coalition formation among the agents is to expand the existing transmission system, satisfying various constraints economically.

Highlights of this paper can be summarized as follows:

• We have applied and analyzed various approaches for allocating the expansion cost among the power market players;
• We have modeled the transmission expansion planning as a game in a cooperative environment with multiple rules;
• We have compared and analyzed various coalition formation methods to expand the system and to divide the expansion cost among all the market players fairly;
• We have reached the different techniques and showed that bilateral Shapley value (BSV) provides the best combination of agents, as the coalition-creating algorithm is also calculated using BSV.

2. Related Works

The application of cooperative game theory in power systems is not a new concept [9]. It was created in 1974 with the successful application of game theory to share the gains from regional cooperation [17]. Various researchers have used game theory in various fields of decentralized multitasking environments, such as the stock market [18], and databases [2]. Its application to power systems ranges from allocating the transaction cost [19] to the cost saved in the energy-brokerage system [20].

J. Contreras [8] has proposed a cooperative game theory planning model within a multi-agent system environment, where the agents cooperate to achieve the optimal common expansion goal. The cooperative framework encourages an independent, distributed decision-making process in the competitive environment. Various researchers have proposed various approaches related to cooperative game theory for generation and transmission system expansion planning. Still, the main emphasis of this paper is on transmission system expansion planning. Therefore, only the most relevant studies are included.

With a multiple objectives scenario within a global framework, a decentralized and simultaneous generation and transmission system expansion planning was proposed by N. Hariyanto et al. [21]. R. Serrano et al. [22] had used static expansion planning considering various hydrology scenarios to study the expansion of the Chilean Central Interconnected System based on cooperative game theory. L. Geerli et al. [23] have developed an operation rule for a market model based on the Independent Power Producers (IPPs) and an electric utility as the players of the non-cooperative game.

Many studies are based on game theory for transmission cost allocation in the deregulated market. For the same purpose, J. Contreras et al. [8] have developed a multi-agent system that is based on the kernel approach. The kernel-oriented approach provides higher rewards to stronger players than a bilateral Shapely value approach [8]. P. A. Ruiz et al. [24] proposed a game theory-based method for viable distributed costs. J. Contreras et al. [25] have used the bilateral Shapley value (BSV), which is well-accepted, to form the coalition in the system. J. Yen et al. [26] have also proposed a BSV-based multi-agent system for coalition formation.

Further, M. Ventosa et al. [27] have used the Nash–Cournot game concept and presented two different approaches to model expansion planning in the electricity market under imperfect competitive conditions. R. Serrano [28] studied transmission system expansion planning from the perspective of private players.

The expert system can take into account human reasoning and all the heuristic knowledge into account [29]. The steps in building an expert system are identification, knowledge acquisition, implementation, and testing. The concept was used by R. C. G. Teive et al. [30] to manage the transmission expansion planning process to select feasible transmission routes and the definition of the expansion plan itself. R. C. G. Teive et al. [30] have proposed a scheme for energy supply system planning in the market environment. They used Shapely values to distribute gains among the coalition members.

3. Multi-Agent Environments in Power System

A multi-agent system is key to coordination and communication among agents to accomplish a task. Specifically for power system studies, they can be generators, consumers, transmission line owners, or any other entity physically connected to the system [31,32]. In such a system, a coordinator (e.g., ISO) collaborates with agents by coordinating their plans and deriving a profitable coalition under fair play.

Intelligent behavior can coordinate by adapting to the new environment and communicating. In a MAS, the agents coordinate accordingly due to the actions of the other agents and communicate by sharing only the relevant information (intentions) [33]. These intentions are then negotiated among them. In such an environment, each agent is an independent problem-solver whose priority is to resolve its problems, i.e., they have the knowledge and capabilities with some goals and intentions. In a MAS, each agent can include the others’ actions as a planning part [34].

Figure 1 shows the coordination plan of agents in a MAS and is somewhat simplified because it neglects the dynamics of coordination and communication among the agents. Firstly, the agents plan individually, sharing their plans through some agents. These agents play a vital role in forming an alliance based on profits and are bound by the agreements. It results in global planning with many such alliances.

4. Simplified Network Expansion Model

The main objective of transmission expansion planning is to minimize the overall cost (investment and operational) associated with the system under expansion, where the associated constraints can be physical or economical. Further, we have assumed that there is congestion in the system. A simple 6-bus system as specified by Graver [35] is used to illustrate the planning process. Here, the expansion problem is formulated as a cost-minimizing problem, minimizing operational and constructional cost, subjected to technical and economic constraints, as mentioned in [8].

Figure 2 is the cost allocation problem described in detail for this proposed expansion plan.

In Figure 2, the dashes denote the candidate lines added, while the straight lines represent the existing transmission lines that connect the buses. Figure 2 shows the expansion result with generation rescheduling.

The network proposed by Garver in Figure 2 is the extension of a 5—Bus system with future load and generation requirements.

The main objective of the traditional transmission system expansion planning problem is cost minimization.

$$C_{total}=C_{new-line}$$

(3)

where the utility function can be termed as,

$$C_{new-line}={\displaystyle \sum _{i\in L_c}{C}_L(x_i)L_i}$$

(4)

where L_i is the length of transmission line of the candidate,

• L_c is the set of candidates,
• x_i is the transmission type of the candidate and
• C_L (x_i) is the investment cost per km for type x_i.

The DC load flows are used and are in the form:

$${\displaystyle \sum _{j=1}^N{B}_{ij}({\theta }_i-\hspace{0.17em}}{\theta }_j)=\hspace{0.17em}{P}_{Gi}-P_{Di}\hspace{0.17em}\forall \hspace{0.17em}i\hspace{0.17em}\subset n$$

(5)

$${\displaystyle \sum _{j=1}^N{B}^m{}_{ij}({\theta }^m{}_i-\hspace{0.17em}}{\theta }_j{}^m)=\hspace{0.17em}{P}^m{}_{Gi}-P_{Di}\hspace{0.17em}\forall \hspace{0.17em}i\hspace{0.17em}\subset n\hspace{0.17em}\cap \hspace{0.17em}m\hspace{0.17em}\subset \hspace{0.17em}C$$

(6)

where ${\theta }_i$ and ${\theta }_j$ are the voltage phase angles of buses i and j, respectively;

• B_ij is the imaginary part of the element ij of the admittance matrix;
• P_Gi is the power generation at bus i;
• P_Di is the power demand at bus i, the index m shows the contingency parameters and variables;
• C is the set of contingencies;
• N is the number of buses.

For the transmission lines, the power transfer should not violate its rating (limits) during both normal and contingency conditions, so

$$b_k=({\theta }_i\hspace{0.17em}-{\theta }_j)\le \hspace{0.17em}{P}_K-P_{DI}\hspace{0.17em}\forall \hspace{0.17em}i\hspace{0.17em}\subset n$$

(7)

5. Formation of Coalitions

For any game with N players, the coalition is the subset S of the set of the N players. Being in a coalition means no other agreement between a member in S and (N-S) subsets. The coalition that has all the game players is termed a grand coalition N. For an n player game, the maximum number of possible coalitions is 2ⁿ.

Playing a game in the transmission system expansion planning process provides a minimum cost plan and allocates the expansion cost among the market players. The player in the expansion game can be a load or a generator, or an independent party attached to a bus. In an expansion game, each bus is considered an agent, i.e., a generator or load attached to a bus acts as a single agent. So, there are six agents for the Garver 6-bus system.

The coalition is a set of agents and the associated transmission lines that connect them, but the coalitions must satisfy the following four conditions [8]:

• There should be a minimum of one generator, one load, and at least one transmission line to be included in the set of agents;
• All the loads, including transmission losses, are always satisfied by the generation outputs;
• The thermal limits of existing lines can never be exceeded;
• There should be some transmission lines to connect all agents.

The coalitions that satisfy these conditions can make their expansion plans without negotiating with other agents.

5.1. Coalition Formation Algorithm

5.1.1. Self-Calculation Phase

In this phase, each agent determines the investment cost incurred to the agent if he goes for the expansion plan alone. This monetary value is called ‘self-value.’ For the Garver 6-bus system, self-values are calculated for all six agents. The self-value of agents 1 and 3, i.e., v (1) and v (3), is 0, as these agents are self-sufficient, satisfying axiom two, and do not need any expansion plan. However, the self-value of agents 2, 4, 5, and 6, shown in

Table 1. Bus Data for 6-Bus system.

Bus No.	Power Generation (pu)	Load (pu)
		Active Power	Reactive Power
1	0.50	0.8	0.079
2	0	2.4	0.237
3	1.650	0.4	0.04
4	0	1.6	0.158
5	0	2.4	0.237
6	5.45	0	0

, shows that they need new lines added to the system to either satisfy their load or transmit generated power to load centers (

Table 2. Candidate Line Data for Garver 6-Bus system.

Terminals	Length (km)	R (pu)	X (pu)	Capacity (MW)	Guide No.
1–3	38	0.095	0.38	100	38
1–6	68	0	0.68	70	68
2–5	31	0.0775	0.31	100	31
2–6	30	0	0.30	100	30
3–4	59	0.1475	0.59	80	59
3–6	48	0	0.48	100	48
4–5	63	0.1575	0.63	80	63
4–6	30	0	0.30	100	30
5–6	61	0	0.61	78	61

).

5.1.2. Communication Phase

After the self-calculation phase, the communication phase begins, where these agents communicate with each other to calculate ‘joint-cost’ through the coordinator. The coordinator is an independent system operator who does not intend to profit from this expansion game. The current system is large enough for agents to understand it completely. Hence, these expansion requests are sent to the central coordinator, who calculates the adequate number of lines required for the coalition to form while checking network security and reliability, and communicates the joint cost of the coalition to the agents involved. The most important task of the central coordinator is not to allow coalitions that are detrimental to the system’s security and reliability.

5.1.3. Joint-Cost Calculation Phase

At this stage, the agent is aware of the self-cost and expansion cost when forming the coalition with other agents. Then, the agents check for individual rationality, i.e., the agent should at least get as good as he would get alone (self-value), then he determines individually rational lists of preferred agents. It is an arranged preference list of local agents’ values for only two entity coalitions in the first iteration. This arranged preference list will change in the next iteration, or when this step is called again, new multi-parties (two two-entity) players will act as a single agent.

5.1.4. Bilateral Negotiation Phase

In this phase, each agent (sender) sends an offer indicating the amount the receiver agent would attain on collaborating with the sender as per the preference list. If both agents find it beneficial, they may join hands and start acting as single agents. All agents in the system are informed about this agreement to erase the members from their preference list. If the receiver is not interested in forming a coalition, the sender looks for the second one in his preference list, and the process is repeated until he reaches the end of the list.

Figure 3 depicts the general process of coalition formation. The coordinator broadcasts the START signal to all the players to initiate the negotiations. The players who do not wish to participate send the COMPLETE signal back to the coordinator; otherwise, they send the REQUIRE signal to each other. If the terms and conditions match and the proposed coalition satisfies the requirements, they send a COMPLETE signal to the coordinator. If the coalition conditions don’t match, then the REFUSE signal is to be sent. After one cycle of coalition formation, the coordinator again broadcasts the START signal to initiate the next round of negotiations.

5.2. Simulation Results

These four phases are repeated until no more coalitions are possible for the system. Figure 3 shows the overall coalition formation framework. After running the coalition formation simulation for the Garver 6-bus system, we obtained the self and joint-cost as shown in

Table 3. Coalition Costs.

Coalition	Cost	Coalition	Cost	Coalition	Cost
{2}	90	{1,3,4}	118	{1,2,5,6}	232
{4}	60	{1,3,5}	20	{1,3,5,6}	116
{5}	40	{1,4,6}	60	{1,4,5,6}	202
{6}	60	{1,5,6}	183	{2,3,4,6}	90
{2,3}	40	{2,3,6}	88	{2,3,5,6}	100
{2,6}	90	{2,4,6}	150	{3,4,5,6}	161
{3,4}	118	{2,5,6}	334	{1,2,3,4,6}	90
{3,5}	40	{3,4,6}	246	{1,2,3,5,6}	70
{4,6}	60	{3,5,6}	101	{1,2,4,5,6}	272
{5,6}	244	{4,5,6}	304	{1,3,4,5,6}	118
{1,2,3}	20	{1,2,3,6}	30	{2,3,4,5,6}	191
{1,2,6}	60	{1,2,4,6}	120	{1,2,3,4,5,6}	130

.

The Intel Core i7, 1 Terabyte hard disk, and 16 GB ram are used to perform these simulations. The simulation starts with all the agents creating the list of the preferred partners to form a coalition. The monetary values obtained here are taken as negative to represent the expansion cost incurred by the agents, i.e., they have to pay the amount. An assumption is made that a team is preferred whenever a choice is to be made between being solo or a team. Here, to find the order of the coalition formation, we used bilateral Shapley value, as in [8]. It is necessary when calculating the cost allocation using a backward induction algorithm.

6. Cost Allocation Methods

Various methods that we have used and analyzed for cost allocation of transmission expansion are:

6.1. Shapely Value

Shapley value is the associated value of each participating player, which is a characteristic function concerning the coalition structure [17].

It is the weighted average of marginal contributions of each participating member towards all the possible coalitions.

$$Y_i={\displaystyle \sum }_{\begin{array}{c}S\subseteq N\\ i\in S\end{array}}\frac{\left(\left|s\right|-1\right)!\left(\left|N\right|-\left|S\right|\right)!}{\left|N\right|!} \left[C\left(S\right)-C\left(s-\left\{i\right\}\right)\right], \left(i\in N\right)$$

(8)

where,

• |N| is the total number of players in the game;
• |S| is the total number of players in the coalition s;
• C(S) is the function of cost for the building the coalition of players with s-person;
• C(S − {i}) is the function of cost for building the coalition of players with s-person if any one player i is removed.

6.2. Separable Cost (SC) & Non Separable Cost (NSC) Methods

Separable cost says that no participant should get less than their marginal contribution in the coalition. The participant i is assumed to enter into a coalition with N − 1 participants; then the cost of including the participant can be

C′(i) = C(N) − C(N − i)

(9)

Non-separable cost is defined as the incentive for each player to participate in a grand coalition and is

$$NSC=C\left(N\right)-{\displaystyle \sum }_{i=1}^N{C}^{\prime }\left(i\right)$$

(10)

Based on the division of NSC, there are three approaches in the cost allocation concept using GT. They are as follows:

6.2.1. Separable Cost Remaining Benefits (SCRB)

This principle states that the cost allocated to any player should not be less than the cost for his inclusion in the said coalition. In the SCRB method, the participant i’s remaining benefit shown by r(i) is the difference between the cost of the non-cooperative situation and the marginal cost [36].

r(i) = [C(i) − C′(i)]

(11)

The SCRB method for N number of players can be modeled as:

$$y\left(i\right)=C\left(i\right)+\left[\frac{r\left(i\right)}{{\sum }_N{r}_N}\right]\left[NSC\right]$$

(12)

where,

5.

C′ (i) is the marginal cost of player i;

6.

$r\left(i\right)$ is the remaining benefit of player i;

7.

[${\sum }_N{r}_N$] is the cumulative sum of remaining benefits of all players;

8.

NSC is the non-separable cost.

6.2.2. Egalitarian Non-Separable Cost (ENSC) Method

The proportional allocation based on this method is fulfilled after allotting each participant’s marginal cost and is then added to distribute non-separable cost among all the participants equally [37].

$$y\left(i\right)=C\left(i\right)+\frac{1}{N}\left[NSC\right]$$

(13)

This second approach may often fail even to meet the individual rationality.

$$y\left(i\right)\ge C\left(i\right)$$

(14)

6.2.3. Alternate Cost Avoided Method (ACA)

This approach considers the remaining alternate cost of other coalitions for the allocation of NSC, whereas SCRB considers the remaining alternate cost of one user only [38].

The cost gap of a coalition S is

$$g^c\left(S\right)=C\left(S\right)-{\displaystyle \sum }_{j\in S}SC\left(j\right)$$

(15)

The concession amount for player i is

$$\lambda \left(i\right)=\underset{T:iϵT}{\underbrace{min} }{g}^C\left(T\right)$$

(16)

T is a coalition to which player i belongs, cost allocation is

$$y\left(i\right)=C\left(i\right)+\frac{\lambda \left(i\right)}{{\sum }_{j=1}^N\lambda \left(j\right)}NSC$$

(17)

6.3. Bilateral Shapely Value

In BSV [39], coalition structure can formally be defined as CS ⊆ P(A) with the player (agents) A = {a₁, ..., a_m}, where C_i ∪ C_j ⊆ A is a bilateral coalition of disjoint n-agent coalitions C_i and C_j. The bilateral Shapley value can be formulated as follows,

BSV_{{Ci, Cj}} (C_i) = 1/2V(C_i) + ½(V(C) − V(C_j))

(18)

V(C) is the self-value while C_i and C_j are known as the founders of C.

7. Cost Allocation Results

The results for the cost allocation using all the methods described above are shown in

Table 4. Cost Allocation Results.

Player	Shapely Value	SCRB	ENSC	ACA	BSV
1	−22.76	−203.33	−45.833	−34.6919	−12.5
2	16.30	40	27.166	45.6398	49.375
3	−34.81	−473.33	−126.833	−80.7583	−10.625
4	29.88	200	75.1667	60	55
5	41.21	133	55.1667	40	29.375
6	100.18	433	145.1667	99.8104	19.375

below. Similarly, the cost allocation for various expansion plans can be formulated considering different operating conditions in the market.

From Figure 4, we can observe that BSV has more stable cost allocations, whereas SCRB has more deviations, which denotes the unrealistic allocations.

8. Observations and Discussion

While analyzing various game-theoretic approaches for cost allocation for proposed expansion plans in TSEP, the following contributions were made:

• We did a thorough analysis of various cost allocation methods of game theory and modeled a few of the most commonly used methods for the study of cost allocation in a TSEP problem;
• The coalition formation algorithm was implemented, and coalition costs were calculated using the Forward search and EMSFLA approach. After this phase, the centralized and decentralized approaches were used for cost allocation;
• As per the comparative results, it is observed that all these approaches can provide the acceptable result of cost allocation for TSEP, except SCRB;
• Bilateral Shapely value is best suited for cost allocation for TSEP, as it is a decentralized approach, and the sequence of coalition formation can give the best possible cost allocations.

9. Conclusions

This paper has applied and analyzed various approaches for allocating the expansion cost among the power market players. We have modeled the transmission expansion as a game in a cooperative environment with multiple rules. The result is to expand the system and divide the expansion cost among all the market players fairly. Comparing the results of all the test runs shows that BSV provides the best combination of agents, as the coalition creating algorithm is also calculated using BSV. Hence, the results follow the strategies of the players. Therefore, the agents’ collaboration choice is crucial to cost allocation, specifically for transmission expansion planning, whereas the other methods, such as SV, etc., that are most commonly used in water resource distribution cost allocation are found inappropriate, as they process the coalition formation algorithm terminates in polynomial time. Abandoning the players’ right to choose the right coalition can result in a situation where the value with the agent may not be the legitimate value of the game. Therefore, decentralized approaches, which reflect individual agents’ strengths, should be used. A similar approach can protect the sensor nodes in a clustered wireless sensor network by detecting malicious sensor nodes that drop the high priority packets to maximize the high priority data trustworthiness [40]. Also, cooperative game theory can be used to enhancing the security and data trustworthiness in IoT applications [41].