Multi-Agent Reinforcement Learning Optimization Framework for On-Grid Electric Vehicle Charging from Base Transceiver Stations Using Renewable Energy and Storage Systems

Abstract

Rapid growth in a number of developing nations’ mobile telecommunications sectors presents network operators with difficulties such as poor service quality and congestion, mostly because these locations lack a dependable and reasonably priced electrical source. In order to provide a sustainable and reasonably priced energy alternative for the developing world, this study provides a detailed examination of the core ideas behind renewable energy technology (RET). A multi-agent-based small-scaled smart base transceiver station (BTS) site reinforcement strategy is presented to manage energy resources by boosting resilience so to supply power to essential loads in peak demand periods by leveraging demand-side management (DSM). Diverse energy sources are combined to create interconnected BTS sites, which enable energy sharing to balance fluctuations by establishing a market that promotes economical energy. A MATLAB simulation model was developed to assess the effectiveness of the proposed system by using real load data and fast electric vehicle charging loads from five different base transceiver stations (BTSs) located throughout Pakistan’s southern area. In this proposed study, the base transceiver station (BTS) sites can share their energy through a multi-agent-based system. From the results, it is observed that, after optimization, the base transceiver station (BTS) sites trade their energy with the grid at rate of 0.08 USD/kWh and with other sites at a rate of 0.04 USD/kWh. Therefore, grid dependency is decreased by 44.3% and carbon emissions are reduced by 71.4% after the optimization of the base transceiver station (BTS) sites.

1. Introduction

With increased environmental and economic restrictions, countries are turning to renewable energy sources, including wind, solar, and hydropower, to preserve energy and advance the usage of distributed generators. As a result, more people are using electric vehicles (EVs). Energy resources are vital to almost every aspect of the economy, so their significance cannot be exaggerated. Electricity is the most widely used and well-liked energy source worldwide. It significantly affects the social and technological development of a country. Many mobile communications organizations are operating to serve a huge number of subscribers as a result of the growth of technology, notably mobile communications systems. In order to accommodate the increasing need for wide coverage and high-quality transmission, cellular network providers need to build more towers [1].

For telecom operators, the absence of dependable grid electricity in isolated or rural areas poses significant challenges. This increases the risk of global warming and CO₂ emissions, since telecom equipment employs diesel generators (DGs) or batteries when there is a large demand. This places a focus on using renewable energy sources to power telecom towers, including solar, wind, biomass, hydro-, and tidal power [2]. Pakistan is the nation best suited to use renewable energy sources, as there is about eight to ten hours of sunlight per day. Research was performed on substitutes for the powering of cellular BTSs [3].

In previous studies [4], it was stated that microgrids that use dispersed generation units to generate energy have numerous positive effects on the environment, politics, and the economy. It is critical to produce an ideal design with appropriate control over every microgrid component in order to reap these benefits and make the best use of the microgrids [3]. Microgrids employ a variety of control strategies known as distributed, decentralized, and centralized control strategies. More adaptable and efficient control features are offered by the distributed and decentralized control strategies. The agents are computer systems that are placed in a specific context and which have the ability to act independently to achieve design goals. The development of artificial intelligence and agent concepts involves importing human traits into the computer environment, such as experience-based learning and reasoned decision-making. Intelligent agents are defined as “anything that can notice and affect its environment through sensors,” and they are based on the idea of intelligence, which is the capacity to perceive and analyze a situation and its surroundings, make decisions, and govern behaviors [5].

Creating autonomous systems that respond automatically and appropriately to events they observe in their surroundings is the primary goal of employing agents. Agents use their autonomous, sociable, reactive, and proactive features to accomplish this purpose. Agents are able to attempt to affect outcomes without intervention and have some degree of control over their internal states and actions because of their autonomy. Their proactive tendency leads to target-oriented conduct and taking the initiative to attain goals, while their reactive nature produces quick responses to changes in their surroundings. Multi-agent systems are systems that have numerous agents interacting with each other and provide control strategies [6].

The BTS sites can be controlled and monitored by the autonomous agent. The agent can behave independently to accomplish objectives as an information processing system in its surroundings. The multi-agent-based approach makes the EMS in multi-BTS sites extremely successful. Each agent possesses social, reactive, and proactive skills [7]. Integrated multi-agent systems (MASs) provide a self-sustaining setup and increased overall efficiency for optimal control operation. Energy storage systems (ESSs) and load demand scheduling are the most suited to the dynamic pricing provided by the electricity market [8]. Improving dependability, protecting consumer privacy, and cutting operating expenses are the top priorities of BTS sites.

It is clear from previous studies that multi-agent systems can enhance energy-efficient distribution and utilization systems. However, no studies have been carried out on multi-agent systems for the telecom sector. Three types of agents are used in this proposed study, which are named as the generating agent, the load agent, and the storage agent. A total of five distinct BTS sites have been chosen for the integration of renewable energy (solar and wind) with battery storage systems in order to improve Pakistan’s telecom industry. Then, these grid extension and electric vehicle charging stations are connected with these five BTS locations. All of the simulations were designed with MATLAB Pro Simulink. After the integration of the multi-agent systems, the BTSs could coordinate with each other and could import and export energy directly at favorable prices rather than being directly taken from the grid.

Problem Statement and Key Objectives of Proposed Research

Base transceiver stations (BTSs) are essential to the operation of modern telecommunications infrastructure because they maintain connectivity. To reduce their environmental impact and operating costs, BTSs are increasingly being powered by renewable energy sources. Adding electric vehicle (EV) batteries as storage units further improves their energy management capabilities, enabling increased efficiency and reliability. However, due to the dynamic nature of renewable energy generation, fluctuating demand patterns, and the interaction of multiple stakeholders (e.g., grid operators, BTS operators, and EV owners), conventional optimization techniques frequently find it difficult to effectively handle these complexities in real time.

By allowing agents (representing BTSs, EVs, and energy providers) to learn optimal tactics through the interaction with their environment, Multi-Agent Reinforcement Learning (MARL) offers a promising solution to address these difficulties. Each agent can independently modify its behaviors depending on local observations and global objectives, such as lowering operating costs, increasing the exploitation of renewable energy sources, and guaranteeing grid stability, by modeling the problem as a MARL framework by linear integer programming with relay optimization.

The following are the main objectives of this study:

• Optimizing Energy Consumption: To reduce grid reliance and operating expenses, BTS and EV energy consumption should be dynamically managed.
• Optimizing the Integration of Renewable Energy: Skillfully incorporating intermittent renewable energy sources, such as wind and solar, into BTS operations.
• Increasing System Reliability: Using EV batteries as storage units to improve energy resilience and reduce power fluctuations.
• Maintaining Grid Stability: To keep the grid stable and avoid interruptions, energy supply and demand must be balanced.

2. Literature Review

Pakistan has a large and diverse renewable energy resource base. As a result, Pakistan’s northern and western areas have a lot of potential for generating energy from biomass, solar, wind, and hydropower. Like the northeast, the southwest has an abundance of solar radiation, making it an excellent location for large-scale PV system installations for homes and businesses. Numerous important studies are available in the literature on techno-economic inquiries that address the power imbalance. The research of [7] evaluated the scientific and financial feasibility of combining a biomass energy system with photovoltaics to suit the electrical and agricultural needs of a modest hamlet in Punjab province’s Layyah area.

The article by [8] evaluated the techno-economic feasibility of different hybrid system configurations at fourteen sites in Gilgit Baltistan (GB). The study also assessed how well forests can reduce greenhouse gas emissions. In the study by [9], a grid-free HRES was developed to suit the electrical needs of a small South Indian community. The investigation was finished and an economical HRES setup was built using the HOMER Pro program. The load demand characteristics and the RERs at the study site were used in the simulations. Simulations that illustrated component sizes, electrical energy generation, and greenhouse gas emissions were evaluated, with the conclusion that the HRES system formed of the PV/DG/battery was the most practicable. In the energy market system paradigm, a multi-agent control system has been presented in [10] for the purpose of facilitating the best possible information exchange across multi-microgrids. Prices, load demand, and renewable-energy-resource-related uncertainties were handled in the day-ahead stage, and optimal scheduling storage power was sent in real time to govern energy arbitrage services, which was recommended in [11].

The authors of [12] proposed a comparison analysis using JAYA, teacher learning-based optimization (TLBO), and Rao1 to validate the findings. In [13], it was recommended that the distribution network operator (DNO) use the MILP technique to find the equilibrium point in order to incentivize MGs to participate in an incentive-based pricing plan. To reduce electricity expenses, the author in [14] scheduled problems relating to energy usage using the MILP approach. Energy exchange between nearby MGs and efficient storage scheduling are shown in the literature [15] to reduce grid dependency.

Multi-microgrids (MMGs) were presented in [16], where energy was handled by networked microgrids cooperating in distributed systems. Assuring MMG dependability and minimizing operating expenses with the demand response strategy was presented in [17]. In order to produce a successful energy management plan, [18] took customer privacy preservation into consideration. In [19], it was hypothesized that a multi-agent system could efficiently manage dispersed resources in MMGs. The bottom-up method was recommended in [20] for centralized energy management based on the MAS paradigm in order to guarantee higher system reliability. In [21], a meta-heuristic-based strategy for scheduling resources within the framework of the energy market paradigm was developed.

The research in [22] presents a combined framework for scheduling controllable loads and electric vehicles (EVs) for a microgrid with the goal of minimizing emissions and operating costs. The microgrid was outfitted with solar photovoltaic panels and wind turbines for the generation of sustainable electricity. In this regard, regulated loads would be utilized to meet the system’s reserve requirements, primarily brought on by intermittent renewable power generation, and EVs would be used to flatten the load profile. In order to determine the anticipated operating costs and reserves, the problem was stated as a two-stage scheduling program.

Due to the inherent uncertainties, managing energy in microgrids with a high concentration of renewable energy sources, like photovoltaics, can be challenging. Energy production and consumption are random, which makes it difficult to predict and determine which trades are ideal. In the study by [23], the authors assessed how uncertainties affected the optimality of several resilient energy-exchange techniques. In response, they introduced AIROBE, a data-driven system that computes robust energy exchange plans using a multiband robust optimization approach to protect against deviations, utilizing machine-learning-based estimates of the energy supply and demand as inputs.

Optimal energy management has evolved into a difficult undertaking in the sophisticated energy systems of today [24]. Significant benefits to society, including a stronger economy and reduced environmental degradation, can be attained by energy management performed as efficiently as possible. The microgrids can be operated in both isolated and grid-connected modes. In order to clarify the best way to operate microgrids that are loaded with both sustainable and non-sustainable energy sources, the authors of the current study introduced a new optimization technique called the Oppositional Gradient-based Grey Wolf Optimizer (OGGWO).

The literature analysis makes it clear that a study of the multi-agent-based evaluation of integrating on-grid and freestanding hybrid renewable systems with BTS locations for electric car charging stations has not yet been conducted in Pakistan. Furthermore, few studies have examined whether the recommended strategy should focus on just two or three development-related parameters, mainly the NPC and LCOE. Prior studies have often focused on regions with similar geographies and climates. Preliminary national research on site variety has been lacking, which has restricted earlier studies to a small number of targeted geographic locations.

Table 1. Previous studies on the techno-economic–environmental analysis of HRES with contribution of proposed study.

Sr. No.	Location	Analysis Type		Technologies Integrated					OGIs	Simulation Tool	Multi-Agent System	Load Scenario	Ref.
		T	E	BA	WE	PV	EV	On-Grid
1	Africa	✔	✔	✔	✘	✔	✘	✘	No	HOMER	✘	DOM	[25]
2	Australia	✘	✔	✘	✔	✔	✘	✔	No	PSO	✘	DOM	[26]
3	Bangladesh	✔	✔	✔	✔	✔	✘	✔	No	HOMER	✘	RSD	[27]
4	China	✔	✔	✘	✔	✔	✘	✔	No	HOMER	✘	IL	[28]
5	Colombia	✔	✔	✔	✔	✔	✘	✘	Yes	HOMER	✘	DOM	[29]
6	Europe	✔	✔	✘	✔	✔	✘	✔	No	GA, PSO	✘	DOM	[30]
7	Egypt	✘	✔	✘	✔	✔	✘	✘	No	HOMER	✘	COM	[31]
8	Iran	✔	✔	✘	✔	✔	✘	✘	No	GA	✘	COM	[32]
9	India	✘	✔	✘	✔	✔	✘	✘	No	HOMER	✘	AGR	[33]
10	Iraq	✔	✔	✘	✘	✔	✘	✘	No	HOMER	✘	RSD	[34]
11	Iran	✘	✘	✔	✔	✔	✘	✔	No	HOMER	✘	IND	[35]
12	Malaysia	✘	✔	✔	✘	✔	✘	✔	No	HOMER	✘	DOM	[36]
13	Namibia	✔	✔	✔	✘	✔	✘	✘	No	HOMER	✘	RSD	[37]
14	Nigeria	✔	✔	✘	✔	✔	✘	✘	No	HOMER	✘	DOM	[38]
15	India	✘	✔	✔	✔	✔	✘	✔	No	HOMER	✘	DOM	[39]
16	Nigeria	✔	✔	✘	✔	✔	✘	✘	No	MATLAB	✘	RSD	[40]
17	Saudi Arabia	✔	✘	✘	✔	✔	✘	✔	Yes	HOMER	✘	COM	[41]
18	South Korea	✘	✔	✔	✘	✔	✘	✘	No	HOMER	✘	COM	[42]
19	Saudi Arabia	✔	✔	✘	✔	✔	✘	✔	No	PSO	✘	DOM	[43]
20	Turkey	✘	✔	✘	✔	✔	✘	✘	No	HOMER	✘	RSD	[44]
21	USA	✔	✔	✔	✔	✘	✘	✘	No	HOMER	✘	RSD	[45]
22	Yamen	✔	✔	✔	✔	✔	✘	✘	No	HOMER	✘	DOM	[46]
23	Pakistan	✔	✔	✔	✔	✔	✔	✔	Yes	MATLAB	✔	TELEC	[PS]

Note: T: technical, E: economical, OGIs: on-ground inputs PV: photovoltaic, WE: wind energy, DOM: domestic, COM: commercial, PSO: Project management software, GA: General availability, AGR: agricultural, RSD: residential, IND: industrial, TELEC: telecom BTS load, PS: proposed study.

presents a detailed comparison of the contributions and creativity of the suggested inquiries with previous research.

In past research, every site used the same energy sources for the duration of the experiment, and the sites were primarily selected by random analysis. The majority of the research, which has not yet proven to be commercially feasible, chose common and standard components for each place or employed the hit-or-trial method to obtain results that were almost perfect. Conversely, storage technology varies depending on the climate zone in each site under study. It is crucial to emphasize that the recommended investigation was evaluated in perfect circumstances by the previous studies. The innovation of this proposed study is that standalone BTS sites are integrated with the grid extension and electric vehicle charging loads are also added for complete economic optimization. However, in previous studies, HOMER was used for the optimization of renewable energy resources with the BTS sites, and individuals BTSs were optimized without interfacing each other. In this proposed study, a multi-agent-based system is used and five BTS sites in Pakistan’s southern region are designed in MATLAB Simulink, named as Karachi, Badin, Hyderabad, Rajan Pur, and Quetta.

Research Novelty and Major Contribution of Proposed Study

According to the geographical region and renewable resource potential of Pakistan, solar and wind energy are integrated with each site and battery storage devices are used. These sites are integrated with the grid for net metering. These five selected BTS sites can interface with each other using three types of agents that are designed in this proposed study, named the storage agent, load agent, and generating agent. However, the main novelty of the proposed study that the BTS sites can interface with each other and can import energy during peak hours from neighboring BTS sites through controlling agents at optimal minimum price rates rather than directly from the grid. Another distinctive aspect of this research is the utilization of real input datasets. The study makes use of discount rates and inflation from the fiscal year 2021–2022, taking into account the status of the country at the time. The suggested study focused on areas with different topographical and climatic characteristics. Local wind farms with commercial installations are used to calculate the losses. In summary, the suggested study is novel, since it takes into account all of the flaws reported in the literature.

The following is a list of contributions:

• Trading carbon emission offset credits via the energy market infrastructure.
• Reducing the mismatch between the predicted generation and load with the hybrid base energy management system by using demand response programs, the optimal scheduling of distributed energy resources, and multi-agent-based energy trading.
• Generating revenue with BTS sites as a seller by selling energy to the market at a lower cost, and, as a buyer, obtaining clean energy at a lower cost than the utility grid.
• Exporting energy to the main grid with BTS sites during periods of high demand and importing energy from the main grid during periods of low demand, which produces the highest amount of income.
• Using relay optimization with integer linear programming to obtain the best generation pattern in accordance with the load pattern in order to lower each BTS site’s operating costs.
• Contributing to increased grid resilience by reducing the non-essential load, charging during off-peak hours, and discharging the ESS during peak demand hours.

The paper is divided into the following seven sections. The Introduction and Literature Review are covered in Section 1 and Section 2. Section 3 presents the research methodology. The system architecture and modeling, which includes the solar PV system, wind power system, battery storage system, and evaluation criteria with objective functions, are covered in Section 4. In Section 5, the results and discussion are supported by corresponding figures and information. Section 6 and Section 7 contain optimized comparisons of the proposed BTS sites and conclusions, respectively.

3. Research Methodology

The BTS locations used for this investigation are located in Pakistan’s southern area. Figure 1 displays the complete graphical map depiction of the Pakistani-nominated places. The current inflation and discount rates for 2021–2022 provided by the State Bank of Pakistan (SBP) were utilized in this analysis because they accurately reflect the country’s high inflation and variable policy rates. Photovoltaic–wind–battery (PV-W-BA) hybrid RESs-based systems are integrated with these five BTS sites. In addition, hybrid BTS sites are connected to the grid for net metering, and fast charging stations for electric vehicles are hooked in. Within each microgrid, the relay optimization control agent oversees the work of the storage agent, load agent, and generating agent.

Table 2. BTS sites with power rating and multiple defined resources.

Sr. No	BTS Sites	PV (kW)	Wind (kW)	No. of Batteries	System Converter (kW)
BTS-01	Karachi	54.2	10	11	38.9
BTS-02	Badin	10.4	10	11	7.64
BTS-03	Hyderabad	15.7	10	11	11.0
BTS-04	Rajan Pur	66.8	10	11	46.8
BTA-05	Quetta	23.9	10	11	17.7

lists the resources with the power ratings. The five sites, named Karachi, Badin, Hyderabad, Rajan Pur, and Quetta, are from both provinces of Pakistan, named Balochistan and Sindh, and cover the southern region of Pakistan. The research methodology flow chart is shown in Figure 2.

3.1. Study Area with Load Assessment

Pakistan has a sizable landmass and abundant solar and wind energy resources. PV and wind electricity production are guaranteed by an abundance of solar and wind energy resources. Because of this, Pakistan’s telecom industry exclusively uses costly, ecologically damaging diesel generators and battery storage technologies. The BTS sites in Pakistan are spread out around the country. For this planned project, five different BTS locations from the south region were chosen in order to connect electric vehicle charging loads with on-grid hybrid renewable energy resources and additional trade energy with each other. The Pakistan TELCO Corporation dispatches the real-time loads for each of these facilities. A dedicated power system library was used to simulate both static and variable loads (EV loads).

The load impedance is regarded as a constant when the voltage at the load terminal is lower than the specified value. The load’s consumption of both active and reactive power changes as the voltage is varied. Equations (1), and (2) provide a mathematical representation, with the terminal voltage denoted by V_m, the reference voltage represented by V_ref, and the exponents denoted by (n_p, n_q), which estimate the power sensitivity variation with the voltage. However, P_ref and Q_ref are determined with the reference voltage. Therefore

Table 3. Related information regarding selected BTS sites in Pakistan with energy demand profiles.

Sr. No.	Under-Study BTS Site Names, South Region	Annual Average Solar Radiation (kWh/m2/day)	Annual Average Wind Speed (m/s)	Annual Average Temperature (°C)	Coordinates		BTS Load Profile
					Latitude (°N)	Longitude (°E)	ED(h) (kWh)	ED (kWh/day)
01	Karachi-I	5.45	5.96	26.30	24.9148	66.8888	8.6	206.4
02	Badin	5.24	6.14	27.96	24.6459	68.8467	2.6	62.4
03	Hyderabad	5.27	6.21	27.45	17.3850	78.4867	3.4	81.6
04	Rajan Pur	5.02	4.61	27.26	29.1044	70.3301	8.6	206.4
05	Quetta	5.57	3.86	16.47	30.1798	66.9750	3.1	74.4

and Figure 3 represent the entire load profile of all selected BTS sites.

$$P_{active}=P_{ref}\times \left[{\displaystyle \frac{1+T_{P_{activ{e}_{{\_}_1}}}s}{1+T_{P_{activ{e}_{{\_}_2}}}s}}\right]\times {\left({\displaystyle \frac{V_m}{V_{ref}}}\right)}^{n_p}$$

(1)

$$Q_{reactive}=Q_{ref}\times {\left({\displaystyle \frac{V_m}{V_{ref}}}\right)}^{n_q}\times \left[{\displaystyle \frac{1+T_{Q_{reactiv{e}_1}}s}{1+T_{Q_{reactiv{e}_2}}s}} \right]\times {\left({\displaystyle \frac{V_m}{V_{ref}}}\right)}^{n_q}$$

(2)

3.2. Design Framework of Proposed BTS Stations

In normal BTS installations, the only energy source utilized to maintain the continuous load of the facilities is the diesel generator. Previous studies have suggested that a diesel generator might be more economical and environmentally friendly. Consequently, battery storage systems are integrated with renewable energy sources such as solar, wind, and batteries at BTS locations. These hybrid renewable BTS sites are then linked to the grid for net metering because there is a significant amount of excess energy. Thus, the transportation load is moved from fuel engines to electric engines, helping Pakistan decarbonize, in order to assess hybrid renewable sites with grid development and, finally, combining the EV charging load into a comprehensive hybrid model. Finally, all five sites are engaged with each other for the import/export of energy at economical prices. Therefore, the multi-agent system is designed on the basis of control agents, including the storage control agent, load control agent, and generating control agent, which cumulatively work together. The main control unit is called the relay optimization control agent. This relay optimization agent collects information from all agents and decides the trading of energy within the sites. The proposed BTS site infrastructure is presented in Figure 4.

There are some important investigations that each site must follow while trading energy with each other are, which are as follows:

• Each site must meet their load demands (base load and EV load) from their respective renewable energy resources that are integrated within the sites.
• After meeting the load demand, the remaining power can be used for charging the batteries present in the respective sites. After charging the batteries, the energy can be exported to the grid or to other sites.
• When there is a requirement for excess energy for any site, the battery can be used to meet the load demand.
• If the renewable energy and battery storage energy cannot meet the load demand, then the site can import energy from the grid or the site. The import energy is determined by the relay optimization control agent, which decides on the economical energy imported from other sites or the grid.

4. System Architecture and Modeling

As described in Figure 4, there are five sites, named Karachi, Badin, Hyderabad, Rajan Pur, and Quetta, nominated for this proposed study. However, each site contains renewable energy resources, i.e., solar, wind, and battery power, with base loads as well as EV charging station loads. These five sites are integrated with the grid for the import–export of energy when required, and the sites can trade energy directly with each other with the relay optimization control system for the economic model.

4.1. Solar PV System

The solar PV system designed in MATLAB Simulink is presented by Equation (3), where solar irradiance is denoted by I_rr(T), measured in W/m², and the efficiency of the solar converter is denoted by η_MPPT. The average solar irradiance of the selected sites is shown in Figure 5.

$$P_{PV}\left(T\right)=P_{nom}\times \left({\displaystyle \frac{1}{{10}^3}}\right)\left(Irr\left(T\right)\right)\times {\eta }_{Mppt}$$

(3)

4.2. Wind Power System

The average wind speed of the selected BTS sites is shown in Figure 5. However, the wind source model is designed by Equation (4), where performance efficiency is denoted by C_n, nominal wind speed is denoted by u_nom, and u(T) denotes the wind speed measured in m/s. In addition, ρ denotes the air density, u_min denotes the cut in the wind speed, and A_s denotes the rotor-swept area of the wind turbine.

$$P_{wind}\left(T\right)=\left\{\begin{array}{c}\begin{array}{cc}0& u\left(T\right)<u_{min}\\ \left({\displaystyle \frac{1}{2}}\right)\left(C_n\times \rho \times A_s\times u{\left(T\right)}^3\right)& u_{min}<u\left(T\right)>u_{nom}\\ P_{nom}& u_{nom}\le u\left(T\right)\le u_{max}\\ 0& u\left(T\right)>u_{max}\end{array}\end{array}\right\}$$

(4)

4.3. Battery Storage System

Regarding the climate condition with the model design, a generic lithium-ion battery is used as the energy storage device for the optimal performance. In the energy storage system (ESS), the SOC is the state of charge, which describes the storage level with its respective capacity. The SOC is denoted in percentage and varies from 0 to 100%. The standard prediction of the SOC of the battery is considered as 30%, which can prevent battery damage or battery-related accidents. The mathematical formulation of the SOC is described in Equation (5), where W_o(T) represents the battery charging time at a specific period. The revised formulation for the SOC for this study is described in Equation (6).

To avoid battery aging and the enhancement of battery efficiency, the mathematical formulation for the limit of the SOC is explained in Equation (7).

$$SOC\left(T\right)=\left(100\%\right)\times \left({\displaystyle \frac{W_o\left(T\right)}{W_{nom}}}\right)$$

(5)

$$SOC\left(T\right)=\left(100\%\right)\times \left(1-{\displaystyle \frac{\eta \ast W_o\left(T\right)}{W_{nom}}}\right)$$

(6)

$$SO{C}_{\min \left(limit\right)}\le SOC\left(T\right)\le SO{C}_{\max \left(limit\right)}$$

(7)

4.4. Evaluation Criteria with Objective Functions

After site optimization, the sites can trade energy with each other after using their own resources (wind, solar, and battery) by the relay optimization. There is a limit to the battery charge levels at each site when the SOC is 30%; when the battery SOC is 80%, energy can be exported. However, instead of importing energy from the grid, the energy can be taken from the sites with economic rates. All of the simulations are performed in MATLAB Simulink and MATLAB function blocks are used for the relay operation. The optimization is performed in multiple periods of time.

The objective functions for the conventional (non-optimization) BTS sites are formulated in Equations (8)–(10), as follows:

$$OF-01\colon\colon P_{Demand}\left(base load \& EV charing load\right)=P_{p{v}_{source}}+P_{w_{source}}+P_{B_{dis}}-P_{B_{ch}}+(P_{Grid}\left(if required\right))$$

(8)

$$P_{Bat\_max}\ge P_{Bat} \ge P_{Bat\_min}$$

(9)

$$E_{Bat\_max}\ge E_{Bat} \ge E_{Bat\_min}$$

(10)

The objective functions for the proposed (relay optimized) BTS sites are formulated in Equations (11)–(13), as follows:

$$OF-02\colon\colon P_{Demand}\left(base load \& EV charing load\right)=P_{p{v}_{source}}+P_{w_{source}}+P_{B_{dis}}-P_{B_{ch}}++P_{Other sites}+(P_{Grid}\left(if required\right))$$

(11)

$$P_{Bat\_max}\ge P_{Bat} \ge P_{Bat\_min}$$

(12)

$$E_{Bat\_max}\ge E_{Bat} \ge E_{Bat\_min}$$

(13)

4.5. Respective Constraints of Proposed Study

A number of restrictions must be taken into account when creating an optimal framework for Multi-Agent Reinforcement Learning (MARL) for on-grid electric vehicle base transceiver stations (BTSs) with renewable energy integration and storage systems. These limitations have the potential to greatly affect judgment calls and the framework’s overall efficacy.

• Resource constraints:
  • Energy Availability: It can be difficult to provide a steady and dependable energy supply for BTS operations because of the unpredictability and erratic nature of renewable energy sources (such solar and wind).
  • Storage Capacity: When the renewable energy supply is minimal or during high-demand periods, the quantity of energy that can be stored and used is limited due to the limited storage capacity of EV batteries.
  • Infrastructure for Charging: The viability of employing EVs for energy storage might be impacted by the availability of charging stations and their ability to fully charge EV batteries within the allotted time.

2.

Operational constraints:

• Grid Guidelines: Operational flexibility may be restricted by adhering to grid laws and standards for energy usage, the integration of renewable energy, sources, and the power quality.
• Service-Level Agreements: BTS performance, dependability, and uptime agreements with telecom service providers may limit the operational choices pertaining to energy management.

3.

Technical constraints:

• Communication Latency: Real-time decision making and coordination may be impacted by delays in communication between agents (BTSs, EVs, and energy providers) brought on by network latency.
• System Compatibility: The choice of components and how they are integrated may be restricted by the compatibility and interoperability of various technologies (such as energy storage systems, renewable energy systems, and BTS equipment).

4.

User Behavior and Preferences:

• The viability and efficiency of employing EVs as energy storage devices can be impacted by the preferences and behaviors of EV owners with relation to charging schedules and willingness to engage in energy management measures.

5. Multi-Agent-System-Based Approach for Base Transceiver Stations

There are multiple agents that are involved in the multi-agent system, including the storage agent, load agent, and generating agent.

5.1. Cummulative Approach by Agents for Ideal Operation

The primary function of the BTS control agent (BTSCA) is to coordinate and regulate with other agents, including as the storage, generating, and load agents, in order to maximize BTS functioning. As a server agent, it gathers all of the data from every other agent in order to carry out necessary tasks, as shown in Figure 2. The BTSCA asks the GA to schedule power produced by diesel generators and to reduce excess generation following trade and storage. An integer linear programming technique is used in relay circuits to optimize the generating pattern in order to make this selection. In response to the RTP, the request to reduce non-critical load is likewise sent to the LA.

5.2. Role of Storage Agent

Figure 2 shows how the storage agent monitors the state of charge (SOC) of the battery in order to optimize the balance between supply and demand dynamics by discharging during off-peak hours and storing extra energy during high-demand charges.

5.3. Role of Load Agent

As seen in Figure 2, the primary goal of a load agent is to optimize energy consumption patterns in accordance with energy costs. This involves reducing the non-critical load during periods of peak demand in order to save money and maintain grid stability.

5.4. Role of Generating Agent

Figure 2 illustrates how excess generation from a hybrid energy source system is reduced by a generating agent that uses an integer linear programming optimization technique to prevent overload or voltage stability problems. After energy storage, and in order to feed the grid with enough power while taking the thermal stability limit into consideration, energy curtailment is the last task for the producing agent. Making the best use of RERs increases the stability and dependability of the system. In situations where renewable energy is not available and the storage source is not sufficiently charged, the generating agent schedules the energy from the grid in order to meet the load demand.

6. Results and Discussion

The results and analysis, with related discussion, are covered in the following five sections. In the first section, the conventional (non-optimization) BTS site results are discovered and analyzed. In second, third, fourth, and fifth sections, the optimizations of the proposed BTS sites are analyzed.

6.1. Integration of Renewable Energy Resources with On-Grid Under-Study BTS Sites

In this section, the techno-economic aspects of the selected on-grid BTS sites with renewable energy resources are discussed. The generation, demand, SOC, and battery storage power of each site are displayed in Figure 6.

It can be seen from Figure 6 that the BTS-04 Rajan Pur site has the maximum power generation compared to all other BTS sites, having a peak of 130 kW. However, the BTS-02 Badin site has the minimum power generation, with a peak of 23.5 kW. Therefore, in the BTS site demand, the BTS-01 Karachi site has the maximum load demand and the BTS-02 site has the minimum load demand. The SOC of all BTS sites starts from 80% and ends at an average of 79.5%. The BTS-04 Rajan Pur site has the maximum battery storage compared to all other BTS sites.

The Karachi BTS site has 54.2 kW of solar energy with 10 kW of wind energy. However, the system converter used 38.9 kW. The Badin BTS site has 10.4 kW of solar energy and 10 kW of wind energy, while the Hyderabad and Rajan Pur BTS sites each have 10 kW of wind energy, and 15.7 kW and 66.8 kW of solar energy, respectively. Finally, the Quetta BTS site has 23.9 kW of solar energy with 10 kW of wind energy. The solar and wind energy output of each respective BTS site is shown in Figure 7.

In the conventional BTS sites, the renewable energy met the base loads and EV charging loads; afterward, the batteries were charged. However, in the case of excess energy, the maximum energy could be exported directly to grid. The cost of energy imported by the grid was 0.08 USD/kWh (PKR 20 per one unit), while the energy exported to grid was 0.04 USD/kWh (PKR 10 per one unit).

From the results, it is observed that all of the sites imported and exported energy to the grid at the specific period of time when the energy minimum was in excess form. The total cost of the energy at each period of time for all BTS sites is shown in Figure 8. Thus, it can be seen that the energy cost from Karachi varies from USD −200 million to USD +200 million. The positive value of the cost shows the imported energy from the grid and the negative values show the exported energy to the grid. Therefore, the Badin and Hyderabad energy costs vary from USD −40 million to USD 80 million and USD −20 million to USD 400 million. However, the energy costs at Rajan Pur and Quetta both varied from USD −200 million to USD +200 million.

6.2. BTS-01 Karachi Optimization

In this study, the optimization was performed by the relay operation. Artificial intelligence was used with the help of the controlling agent at each site, and the relay decided to import/export energy based on its on/off status. The relay was set to closed at a value of one and the relay opened at a value of zero. When relay is at one, it means that energy is traded, and if relay is at zero, then the energy cannot be traded by the relay. Here, USD 1 is set as equal to PKR 250. The cost of energy that is traded with the grid is set at 0.08 USD/kWh (PKR 20), while the cost of energy that is traded (imported/exported) with other sites is set at 0.04 USD/kWh (PKR 10). In the BTS-01 Karachi site, the model was run for 20 s. The relay used for grid trading is named the grid relay state, and the relay used for trading with the other sites is called the other site relay state. The grid relay state and the other site relay state for the BTS-01 Karachi site are shown in Figure 9.

It can be seen from the results that, from 9 s to 14 s, and at 18 s, the grid relay state was at one. The energy was traded with the grid in this time slot because the renewable energy and battery could not meet the load demand. At the times from 7.5 s to 9 s, 14 s to 18 s, and 19 s to 19.5 s, the other site relay state was at one. When energy was required, energy was taken from the other sites due to the reduced cost, but this depended upon the excess energy availability in the other sites. The cumulative power imported/exported from the grid and the other sites is shown in Figure 10.

From the results, it can be observed that, when the grid relay operates, power is taken from the grid, and when the other site relay is in the on state, energy is taken from the other sites. Therefore, dependency on the grid decreases by taking energy from the other sites. The cumulative cost of the energy imported/exported by Karachi is shown in Figure 11. Therefore, it can be seen from the results that the cost of Karachi decreased compared to the conventional site. The conventional non-optimized Karachi site cost between USD −220 million and USD 200 million, while, after optimization, most costs were taken from the other sites, so the cost decreased and varied from USD −5 million to USD 25 million. The per unit cost of energy (USD/kWh) that was traded at import or export by the Karachi site is shown in Figure 12. The cost of energy (USD/kWh) varied from 0 USD/kWh to 0.08 USD/kWh. It can be observed that, most of the time, the per unit cost of energy was less than 0.05 USD/kWh because the maximum energy was traded with the other sites at a lower cost than the grid. The overall solar and wind output and power consumption with the battery SOC are shown in Figure 13.

6.3. BTS-02 Badin Optimization

After the optimization of the Badin site, the grid relay state and the other site relay state are shown in Figure 14. It can be seen that, when the grid relay state is in the open position, the other site relay state is in the closed position. Therefore, the grid relay state was in the closed position from 12.5 s to 14 s and traded energy with the grid. On the other hand, at the initial stage, the solar and wind energy were not enough, so the site was able to trade energy with the other sites, and so closed positions of the other site relay were seen from 0.1 s to 1.5 s, 6.5 s to 10.5 s, and 14 s to 19.5 s. Hence, the maximum energy was traded with the other sites instead of the grid due to the reduced cost. Therefore, due to the maximum import/export of energy with the other sites, the energy cost of the Badin site is less than the conventional site. The cumulative power traded from the grid and the other sites is shown in Figure 15.

The results show that the import and export of power from the grid depends upon the grid relay state, while trading with another site depends upon the other site relay. From the results, it is observed that the grid relay state is in the closed position from 12.5 s to 14 s; hence, the power traded in this time is from 20 kW to 100 kW, dependent upon the demand. The cumulative cost of the import/export of energy in the Badin site from the grid and other sites is shown in Figure 16. It is noted from the results that the Badin conventional site shows a cumulative energy cost varying between USD −40 million and USD 80 million, while, after optimization, the maximum energy is traded with the other sites instead of the grid, so the cumulative cost decreases and varies from USD −5 million to USD 16 million. The per unit cost (USD/kWh) of energy import/export by the Badin site is shown in Figure 17. The cost of one unit of energy that is imported or exported by the site varied from 0.02 USD/kWh to 0.08 USD/kWh. Mostly, the cost of one unit of energy varied from 0.02 USD/kWh to 0.04 USD/kWh because the maximum energy was traded with the other sites rather than with the grid.

The overall solar and wind output and the power consumption with the battery SOC are shown in Figure 18. It can be observed that the maximum active power from wind is 10 kW and the solar power output is 22 kW. However, the maximum power consumption is 100 kW, while the SOC varies from 80% to 79.55%, which determines the state of charge of the battery at the BTS-02 Badin site.

6.4. BTS-03 Hyderabad Optimization

In the BTS-03 Hyderabad site optimization, the grid relay state and the other site relay state is shown in Figure 19. The grid relay state is in the closed position between 12.5 s and 14 s. However, the maximum energy is traded with the other sites instead of the grid, and the other site relay state is in the closed position in multiple intervals of time, i.e., between 0.01 s and 1.5 s, 7 s and 12.5 s, and 14 s and 19.9 s. The other site relay state is in the closed position in the initial state due to the low amount of renewable energy resources. The power imported and exported from the grid and the other sites is shown in Figure 20. The cumulative cost of the energy exported and imported is shown in Figure 21.

The conventional BTS-03 Hyderabad site had a higher cumulative energy cost which varied from USD −50 million to USD 400 million, while the optimized Hyderabad site varied from USD −20 million to USD 160 million because energy was traded at a reduced cost, with a variation of USD −5 million to USD 12 million. The cost of the energy (USD/kWh) at the BTS-03 Hyderabad site is shown in Figure 22. The cost of the energy (USD/kWh) varied from 0.01 USD/kWh to 0.08 USD/kWh. The major portion of the energy had an average value of 0.04 USD/kWh, which was due to the maximum energy trading with the other sites at cheaper rates. The active solar and wind output with the power consumption and SOC of the battery are displayed in Figure 23. The maximum active wind output is 10 kW, with a solar output of 20 kW.

6.5. BTS-04 Rajan Pur Optimization

The renewable energy resources covered 80% of the energy; therefore, less energy was traded with the grid and the other sites. The grid relay state was in the closed position from 8.9 s to 14.5 s, and the remaining time the grid relay remained open, as shown in Figure 24. Hence, the other site relay remained closed during the time intervals from 8 s to 8.2 s, 9 s to 9.5 s, 14.5 s to 14.8 s, and 19 s to 19.2 s. The cumulative power consumption by trading with the grid and the other sites is shown in Figure 25. The cumulative cost of the energy traded from outside sites is shown in Figure 26. The conventional BTS-04 Rajan Pur site had a higher cost of energy, which varied from USD −200 million to USD 200 million. The optimized BTS-04 Rajan Pur site had an energy cost variation from USD −2 million to USD 12 million.

The per unit cost of energy (USD/kWh) of the energy that was imported and exported is shown Figure 27. The average cost of energy (USD/kWh) was 0.04 USD/kWh. After optimization, the BTS-04 Rajan Pur site traded energy with the other sites at economical rates. The active wind and solar outputs with the overall power consumption with the SOC are shown in Figure 28.

6.6. BTS-05 Quetta Optimization

The grid relay state and the other site relay state for BTS-05 Quetta is shown in Figure 29. The grid relay is in the closed position from 9.5 s to 14.5 s, while the other site relay remained closed between 7 s and 9.5 s and 14.5 s and 19.5 s.

A larger share of the other site relay remained closed because the energy was traded with the other sites at economical prices. The cumulative power imported/exported with the grid and the other sites is displayed in Figure 30. From the results, it is noted that the maximum power was negative, as the negative power shows that the site exported power to the other sites and the grid. The BTS-05 Quetta site mostly exported power to the grid and the other sites due to the excess amount of renewable energy.

The cumulative cost of the energy that was traded is shown in Figure 31. The conventional BTS-05 Quetta site had a cumulative cost of energy that varied from USD −200 million to USD 200 million, while the optimized Quetta site had an energy cost variation from USD −21 million to USD 21 million. The negative energy cost shows the exported energy while the positive energy shows the imported energy. After optimization of the BTS-05 Quetta site, the results show that the energy cost was reduced due to the trade with the other sites and, when required, the grid was used to import/export energy. The per unit cost of energy (USD/kWh) that was imported or exported is shown in Figure 32. It can be seen that the average energy cost of one unit was 0.04 USD/kWh, while the maximum energy cost was 0.08 USD/kWh and the minimum energy cost was 0.01 USD/kWh.

The overall solar and wind active power output with the power consumption and SOC are displayed in Figure 33. The solar maximum active output power is 50 kW, while the wind active power is 10 kW. Therefore, the power consumption is 250 kW. The SOC of the battery varies from 80% to 79.4%.

7. Optimized Comparison of Proposed BTS Sites

In this section, all of the proposed sites are studied with respect to their optimization, as shown in Figure 34. From the results, it is observed that, at the BTS-01 Karachi site, renewable energy resources covered the maximum demand of the site. The solar and wind energy at their peak gave up to 110 kW of output power, while the remaining power was managed by the discharging battery at specific periods of time. At the initial period of time, the battery took power from the renewable energy resources and imported power from the other sites at economical rates. At the BTS-02 Badin site, at the peak time, renewable energy resources were not enough to meet the demand of the site; therefore, battery discharge was used to meet 70% of the load of the site, while 30% of the load was covered by the grid and the other sites. Therefore, during peak hours, the energy was imported from the grid and the battery was charged. However, in the final period of time, no energy was traded from the grid and the site used the charged battery to meet the demand.

At the BTS-03 Hyderabad site, at the initial period of time, both the base and EV charging loads were managed by the renewable energy resources, and in the middle of the peak time, the power consumption was higher, up to 175 kW; therefore, the other site relay operated and energy was imported from the other sites to meet the demand. When energy from the other sites was unavailable, energy was traded with the grid to meet the demand. During peak time, the battery was charged up to 75 kW and discharged later when there was the requirement of energy. At the BTS-04 Rajan Pur site, solar energy was bolstered by the maximum solar irradiance, and the active solar output was 130 kW, while the wind energy was set to a generic output of 10 kW. Therefore, at this site, 80% of the energy was met by their renewable energy resources, with the remaining energy covered by the battery. Some part of the energy was exported to the other sites when there was an excess amount of energy. At the start point, the solar active output power was used for the total power consumption; therefore, the excess energy was used to charge the battery. Finally, the BTS-05 Quetta site showed a maximum power consumption of up to 275 kW, and peak demand was covered by the solar and wind active output power, with the importing of energy from the other sites and the grid, as shown in Figure 34.

8. Conclusions

This study proposed agent-based multi-BTS sites with hybrid energy resources to trade energy between BTS sites and the grid in order to achieve incentive-based energy. Artificial intelligence-based relay optimization was used in the optimization process to determine the best generation pattern solution while taking the expected load pattern into account. Relays were used at each BTS site, and each relay had a further relay system to coordinate within the sites. The relays were updated with respect to artificial intelligence. The relays were designed so that each BTS site’s relay configured the energy requirements and compared the cost of energy, and then traded the most economical energy by operating circuit breakers. The flow of energy was completely monetarized in real time, and action was taken according to the energy requirements. In the optimized BTS sites, the energy was traded with the grid and with the other sites at rate of 0.08 USD/kWh and 0.04 USD/kWh, respectively. Therefore, it is observed that the maximum energy traded with the other sites had an average rate of 0.04 USD/kWh, and if energy was not available, then energy could be imported from the grid. In the non-optimized conventional BTS-01 Karachi site, the cumulative cost of energy varied from USD −220 million to USD 200 million, while, after the optimization, the cumulative cost of the energy reduced and varied from USD −8 million to USD 25 million. At the BTS-02 Badin site, the average cost of one unit of energy that was traded was 0.03 USD/kWh, with a reduced cumulative cost of energy for the whole period of time, varying from USD −2 million to USD 16 million. At the BTS-03 Hyderabad site, minimum energy was traded with the grid at peak times at the rate of 0.08 USD/kWh, while, 80% of time, the energy was covered by the renewable energy resources and the battery, with the remaining energy being covered by imported energy from the other sites at a rate of 0.04 USD/kWh. Finally, at the BTS-04 Rajan Pur and BTS-05 Hyderabad sites, the cumulative cost of energy for the whole period of time varied from USD −3 million to USD 12 million and USD −21 million to USD 21 million. Grid reliance was lowered to 44.3%, and the savings on consumption outweighed the costs incurred by the main grid following the trade. By purchasing more clean energy from the other locations rather than the grid, BTS-1, BTS-2, and BTS-3 received a sizable carbon offset credit compared to MG4. As a result, the total amount of carbon emissions from the interconnected BTS decreased by 71.4%.

9. Limitations of Current Study and Future Work

On-grid electric vehicle base transceiver stations (BTSs) can be optimized with renewable energy integration and storage systems by taking advantage of Multi-Agent Reinforcement Learning (MARL); however, there are a number of limitations and difficulties that must be taken into account by researchers and developers.

• Complexity and Scalability:
  BTSs, EVs, and energy providers are only a few of the agents that must be coordinated while maintaining their own decision-making processes in MARS. Scalability is a major difficulty because the complexity of the learning process increases exponentially with the number of agents.
• Communication and Coordination:
  Attaining the best possible system performance requires effective communication and coordination amongst actors. It is difficult to create communication protocols that balance data exchange without overloading the system, though.
• Non-Stationarity:
  Because of shifting customer demand patterns and varied renewable energy generation, the environment in which BTSs function is frequently non-stationary. MARL frameworks could find it difficult to quickly adjust to these changing circumstances.
• Curse of Dimensionality:
  When several BTSs, different renewable energy sources, storage capacities, and operational limits are taken into account, the state and action spaces in MARL can grow to an extremely enormous size. Inefficiencies in learning and decision making may result from this.
• Reward Design and Conflict Resolution:
  It might be difficult to create suitable reward functions that match the objectives of the entire system with the goals of individual agents. Agents attempting to maximize their individual incentives and the system’s overall advantage may come into conflict.
• Computer Resources: In order to train the agents and make decisions in real time, MARS needs a large amount of processing power. This may restrict its use in situations when resources are scarce in the actual world.
• Safety and Reliability: It is imperative to guarantee the safety and dependability of BTS operations. Strict dependability requirements and protocols may need to be incorporated into MARL frameworks, which can make the optimization more difficult.
• Privacy and Security Issues: Privacy and security issues are brought up when agents share critical operational data. In decentralized MARL systems, maintaining data secrecy and thwarting harmful assaults are crucial but difficult tasks.
• Transferability and Generalization: It is challenging to learn policies that apply to various grid environments, BTS locations, and user behaviors. MARL frameworks may find it difficult to properly transfer knowledge across many contexts and scenarios.
• Regulatory and Policy Compliance: MARL optimization frameworks are subject to extra constraints and considerations in order to comply with regulatory regulations and policies pertaining to energy usage, grid interactions, and environmental standards. We plan to investigate other optimization techniques in the future with the goal of drastically lowering end-user electricity prices.