Review and Evaluation of Multi-Agent Control Applications for Energy Management in Buildings

Abstract

The current paper presents a comprehensive review analysis of Multi-agent control methodologies for Integrated Building Energy Management Systems (IBEMSs), considering combinations of multi-diverse equipment such as Heating, Ventilation, and Air conditioning (HVAC), domestic hot water (DHW), lighting systems (LS), renewable energy sources (RES), energy storage systems (ESS) as well as electric vehicles (EVs), integrated at the building level. Grounded in the evaluation of key control methodologies—such as Model Predictive Control (MPC) and reinforcement learning (RL) along with their synergistic hybrid integration—the current study integrates a large number of impactful applications of the last decade and evaluates their contribution to the field of energy management in buildings. To this end, over seventy key scholarly papers from the 2014–2024 period have been integrated and analyzed to provide a holistic evaluation on different areas of interest, including the utilized algorithms, agent interactions, energy system types, building typologies, application types and simulation tools. Moreover, by analyzing the latest advancements in the field, a fruitful trend identification is conducted in the realm of multi-agent control for IBEMS frameworks, highlighting the most prominent solutions to achieve sustainability and energy efficiency.

1. Introduction

1.1. Motivation

According to the literature, buildings are responsible for a substantial 36% of worldwide energy usage [1]. In this context, the management of integrated building energy systems—such as HVAC, DHW, LS, RES, ESS, and EVs—is critical for enhancing energy efficiency and occupant comfort while addressing environmental concerns [2,3,4,5]. Initially, Building Energy Management Systems relied on manual controls; however, as building complexity and energy demands escalated, the limitations of these methods became apparent and automated systems became necessary to achieve greater efficiency, precision, and scalability in managing energy utilization [6,7,8]. To this end, the primary objective for engineers toward automated BEMS was to create tools able to dynamically adapt to fluctuations in environmental conditions and occupancy levels, thereby minimizing inefficiencies and optimizing energy performance—in other words, the goal was to evolve automated BEMSs into fully autonomous systems, capable of making independent decisions based on real-time data and dynamically responding to changing conditions [9,10,11]. To this end, numerous control methodologies have emerged to achieve this transformation, including machine learning algorithms. Such advanced techniques enable BEMSs to not only follow predefined rules but also to learn from historical data, predict future conditions, and make real-time adjustments [6,10].

However, the performance requirements of building occupants gradually evolved, necessitating the integration of multiple energy systems at the building level. This led to the development of Integrated Building Energy Management Systems (IBEMSs), which incorporate multiple heterogeneous energy systems (HVAC, RES, ESS, LS, DHW, and EV), portraying a System of Systems (SoS) of high complexity [12,13,14,15]. However, managing such a complex and interconnected SoS required more advanced control strategies to effectively handle the intricate energy dynamics and ensure optimal performance; as traditional centralized control methods struggled to manage the diverse and dynamic interactions within SoS frameworks, the need for more flexible and adaptive strategies became clear.

Inspired by concepts from distributed computing and artificial intelligence, multi-agent control portrayed a fruitful solution to the increasing complexity of IBEMS management and other SoS frameworks such as industrial automation [16,17,18], traffic management [19,20], swarm navigation of robots [21,22,23,24], smart grid balance [25,26,27,28,29], etc. Such approaches leverage a network of autonomous agents, each responsible for managing a specific energy subsystem, such as HVAC, lighting, or renewable energy sources. By enabling these agents to make local decisions while coordinating with one another, multi-agent control provides a more scalable and resilient solution for optimizing energy performance in complex building environments. This decentralized philosophy not only enhanced system responsiveness but also improved fault tolerance and adaptability, making it particularly effective in managing the intricate and variable energy demands of modern buildings.

Multi-agent control strategies are primarily classified into model-based and model-free control, each with distinct methodologies and applications [30,31,32]. Model-based control relies on detailed mathematical models to represent the behavior and interactions of energy subsystems. Techniques like Model Predictive Control (MPC) are commonly used, where agents utilize these models to predict future states and optimize decision-making over time [31,32]. This approach provides high precision and predictive capability but requires accurate models and substantial computational power, making it well suited for environments with stable and well-understood dynamics. On the other hand, model-free control bypasses the need for explicit models, instead using adaptive, data-driven methodologies like reinforcement learning (RL) [30,31,32,33,34]. In this approach, agents learn optimal behaviors by interacting with the environment and receiving feedback in the form of rewards or penalties. This makes model-free control highly flexible and capable of adapting to complex or changing conditions, though it may involve a longer learning process and potentially less precision during the initial phase [30,31,32,35,36]. These complementary approaches allow multi-agent systems to handle the diverse and dynamic challenges of modern IBEMS.

However, it is important to note that the choice between model-based and model-free control in multi-agent control frameworks is guided by the nature of the energy systems and the specific goals of the IBEMS. For instance, model-based control is favored when the system dynamics are well understood and can be accurately modeled, enabling precise and predictive management using methodologies [30,31,32,35,37]. Such an approach is highly effective in stable and predictable environments, where accurate models may be leveraged to optimize system performance. However, it should be noted that it commonly requires significant computational resources and relies heavily on the accuracy of the underlying models. On the other hand, model-free control is more suitable for environments characterized by complexity, uncertainty, or frequent changes. The model-free approach allows agents to learn optimal behaviors directly from data through adaptive techniques, making it highly flexible and capable of responding to dynamic conditions. However, achieving optimal performance with such adaptive control schemes commonly requires a significant amount of time, especially during the initial learning stage [36,37].

It is important to note that in the period between 2014 and 2024, and especially the last years, multi-agent control approaches in the literature comprise a noticeable amount of the overall control approaches in the literature [30]. Such a tendency is grounded on numerous reasons, such as the following: (a) The sophistication that multi-agent control for multiple Integrated BEMS (IBEMS) may offer: the need for scalable and decentralized control solutions has become more pressing as building systems have grown in complexity, and multi-agent systems naturally address this requirement by allowing distributed decision-making across large or networked equipment. (b) Advancements in computational power and edge computing: the rise of edge computing and increased computational power have made it feasible to implement complex multi-agent control in real-world settings. Contrary to old-fashioned practices, agents can now process and analyze data locally, leading to faster decision-making and reducing the reliance on centralized control systems. (c) The emergence of novel multi-agent control schemes such as RL and Deep Learning have made it possible to design more sophisticated and adaptive multi-agent systems able to handle complex tasks, learn from dynamic environments, and optimize energy management more efficiently. (d) The proliferation of IoT and sensor networks: the widespread deployment of Internet of Things (IoT) devices and sensor networks in buildings has provided real-time data, helping multi-agent methodologies to make informed, data-driven decisions about energy management, enhancing the ability to respond to changing conditions and user behaviors. (e) The increased demand for energy efficiency and sustainability: growing concerns about climate change and the need for sustainable energy practices have pushed the building industry to adopt more efficient energy management systems. Multi-agent control offers a scalable and flexible approach to optimizing energy use, integrating RES and other equipment more efficiently into the energy mix for reducing the overall carbon footprint of buildings.

In exploring multi-agent control, this review aims to delve into the most prominent model-based and model-free control strategies in IBEMS, examining their applications, effectiveness, and associated challenges. By analyzing the most impactful literature works from the past decade (2014–2024), the current paper has as its primary aim to provide insights into the utilization of these novel methodologies and their potential impact on future energy management in buildings.

1.2. Literature Analysis Approach

The current review aims to meticulously explore the influential studies on model-free and model-based multi-agent control approaches towards multifunctional IBEMS frameworks, seeking to draw out significant trends, conclusions, and future research directions in this field from the last decade. This review methodically delves into a diverse range of research works of the last decade (2014–2024), illustrating their foundational concepts, management strategies, applied algorithms, and specific implementations toward IBEMSs. By categorizing studies based on (a) the multi-agent control strategies; (b) the various combinations of energy systems; (c) their testbed building characteristics; and (d) the simulation tools that have been utilized, this study provides a comprehensive overview of established trends in the field. The systematic approach of this review ensures a detailed analysis and evaluation of each selected study.

• Article Selection: Primarily utilizing databases like Scopus and Google Scholar, this research involved a preliminary review of over 500 papers through abstracts, from which the most pertinent were chosen for in-depth examination.
• Keyword Research: Extensive keyword research was conducted, encompassing phrases like “Model-free Decentralized control in IBEMS”, “Adaptive Decentralized control in IBEMS”, “Multi-agent Control in IBEMS”, “Distributed Control in BEMS”, and specific terms related to each subsystem. This strategy ensures the capture of the diverse challenges and dynamics in optimizing IBEMS performance
• Data Collection: Data from each publication were categorized, focusing on the multi-agent control techniques used for IBEMS management and its application context. Considerations included the advantages, limitations, and practical implications, especially in optimal IBEMS management scenarios.
• Quality Assessment: Each selected study underwent a thorough quality evaluation based on citation count, the academic contribution of the authors, and the research methodologies used. This helped determine the relative significance and impact of each piece of research. It should be noted that the papers from the last decade that have been elected scored more than 10 citations according to Scopus. Such integration ensured a large number of integrated papers and also directed the evaluation and the final conclusions towards the field of multi-agent control in IBEMSs.
• Data Synthesis: Finally, the gathered insights were organized into distinct categories, facilitating straightforward comparisons and a clear understanding of the research landscape in model-based and model-free multi-agent control for IBEMS.

1.3. Previous Work

The literature exhibits numerous review works concerning the control of different BEMsS, evaluating model-based and model-free approaches in order to embrace autonomy and intelligence. In [38], Mahela et al., studied multi-agent systems (MAS) for controlling smart grids, examining their applications in energy management, pricing, scheduling, and reliability. The review highlighted the advantages of MAS in enhancing smart grid operations, including communication, security, and integration with electric vehicles and building energy systems. In [39], Naylor et al. described the gap between designed and actual building energy use, emphasizing the need to consider occupants’ influence on energy consumption. This effort reviewed the existing building control systems and explored research on advanced occupant-centric controls, comparing different methods from simple presence-based lighting to complex predictive control. The findings indicated that a balance is required between system complexity and energy-saving potential. Merabet et al. [40] evaluated different AI techniques in HVAC systems to improve energy efficiency and comfort, noting AI’s potential despite current limitations like data quality. Covering a wide range of studies, the review concluded that AI techniques contributed to significant energy and comfort improvements but also outlined challenges and future research directions for optimizing building energy use and occupant comfort with AI.

An important work by Kathirgamanathan et al. [41] explored how data-driven control, enhanced by Internet of Things technology, may improve energy management in buildings for better smart grid integration, addressing challenges in modeling diverse building stocks and emphasizing unexplored areas in model integration and energy flexibility. This work also identified research gaps and future directions for efficient grid integration. Also, Michailidis et al. [30] provided a detailed examination of the recent progress in HVAC control for energy savings and comfort in buildings, focusing on model-free methods like reinforcement learning, neural networks, and fuzzy logic for HVAC management. By analyzing key studies from recent years, the paper provided an evaluation on different fields, including HVAC use and building zoning. The review identified trends in HVAC control.

1.4. Novelty and Contribution

The current review stands out in several key ways from the existing literature on multi-agent control in IBEMS. First, it is the only one specifically focused on multi-agent control frameworks for managing various BEMS components: HVAC, DHW, LS, RES, ESS, EVs, and more. Unlike previous studies, the current approach involves a large-scale comprehensive examination and analysis of applications—notably, more that 70 research works from 2014 to 2024 have been analyzed and evaluated in order to extract fruitful conclusions for multi-agent control in the field of energy management in buildings. It should be noted that almost every work from 2014 to 2024 that scored above ten citations (>10), in the field of multi-agent control in IBEMS, has been integrated in this paper, in order to provide conclusions in the Section 5.

By providing a detailed summary of the most impactful works from the past decade, the current effort highlights the impact of each intelligent control method in the field. Through summarized tables that cover the most influential works of the last decade, potential readers may easily locate research relevant to their interests and distinguish between different studies. To this end, a thorough evaluation of different aspects of research is given: primary control strategies and their subsets, IBEMS types, application type, building type, simulation tools, etc.

1.5. Paper Structure

This paper is structured as follows: Section 1 integrates the motivation of the current review as well as the Literature Analysis Approach, the contribution and novelty, as well as the paper structure. Section 2 illustrates a general description of IBEMS frameworks, the classification of multi-agent types considering the interaction between the agents, as well as a description of the generalized process of model-based and model-free multi-agent control strategies for IBEMS. In Section 3, the general mathematical concept of the primary control strategies and algorithmic methodologies are given in detail. Next, in Section 4, a review of the most highly cited research works between 2014 and 2024 is illustrated per control strategy along with summarized tables integrating the primary characteristics of the research works. Section 5 concerns the evaluation of the different research aspects, integrating comparisons between the prevailing methodologies, the agent interactions, the IBEMS elements, the application types, and the simulation tools that have been primarily utilized. Last but not least, the Conclusions Section 6 summarizes the most significant outcomes of this work. Figure 1 illustrates graphically the structure of this paper.

2. Integrated Building Energy Management Systems

2.1. Primary IBEMS Subsystems

Managing energy at the building level through IBEMS involves coordinating various subsystems like HVAC, DHW, LSs, RES, ESS, and EVs. These subsystems are integrated to optimize energy use, comfort, and sustainability [7,30,42].

• HVACs: Essential for maintaining indoor climate by regulating temperature, humidity, and air quality, using components like furnaces, air conditioners, and heat pumps.
• DHW: Provides hot water for household and commercial use, typically involving water heaters and storage tanks to ensure a steady supply.
• LS: Lighting systems, including LEDs and smart controls, provide appropriate illumination, enhancing safety and comfort.
• RES: Systems such as solar panels and wind turbines generate power from sustainable sources, offering a clean energy alternative.
• ESS: Store excess energy from a RES for later use, ensuring a consistent energy supply during high demand or low generation periods.
• EVs: Integrated into a BEMS as mobile energy storage, EVs can be charged during low-demand periods and supply power during peak times, aiding in load management.

Figure 2 shows the interactions among energy subsystems within an IBEMS. The complexity and stochastic nature of these interactions require advanced control algorithms, real-time data processing, and robust communication networks [43]. Multi-agent control is effective in this context, allowing decentralized decision-making and optimizing each subsystem’s performance while maintaining overall coordination. The main challenge is to balance diverse energy demands and ensure the optimal performance of each subsystem without compromising the building’s energy efficiency and reliability.

2.2. Multi-Agent Control Types for IBEMS

According to the literature [44], multi-agent control in IBEMS can be segmented into four key approaches: centralized, decentralized, cooperative, and non-cooperative, each with distinct advantages and challenges. More specifically,

• Centralized Multi-Agent Control (CE): In this approach, a single central agent makes decisions based on information from all other agents, enabling globally optimal solutions and highly efficient energy management. However, it is prone to scalability issues and risks a single point of failure.
• Decentralized Multi-Agent Control (DE): Here, multiple agents operate independently, making decisions based on local observations. This offers robustness and scalability, as the system is resilient to individual agent failures. The main challenge is the potential for suboptimal global performance due to limited coordination.
• Cooperative Multi-Agent Control (CO): Agents in this approach share information and coordinate actions to achieve common goals, balancing local and global optimization. While this enhances overall efficiency, it requires reliable communication networks and complex algorithms, which can add cost and complexity.
• Non-Cooperative Multi-Agent Control (Non-CO): This approach is characterized by agents operating independently without coordination, offering simplicity and ease of implementation. However, it can lead to inefficiencies and conflicts, as agents may work at cross-purposes, reducing overall system performance.

It should be noted that in some cases the interaction between the agents may hold more than one schemes from the above classification. To this end, in some cases the algorithmic approaches may integrate cooperative and also decentralized schemes (DE/CO) or even centralized and cooperative schemes (CE/CO), etc.

The following Figure 3 illustrates graphically the four primary multi-agent types considering the behavior of multiple agents in BEMS applications:

2.3. General Description of Multi-Agent Control Processes in IBEMS Applications

In a multi-agent IBEMS control process, both model-based and model-free approaches start with data acquisition, but they differ in execution: Model-based control uses the acquired data to refine mathematical models, predict future states, and optimize decision-making. Feedback is used to adjust optimization parameters and refine existing models, ensuring precise and efficient energy management. On the other hand, Model-free control skips detailed modeling and continuously adapts the control strategies based on real-time feedback. This approach is more dynamic, learning optimal behaviors over time without relying on predefined models. More specifically Figure 4 and Figure 5, portray the generalized control processes for Mode-based and Model-free control in IBEMS, respectively:

• Model-based Control

  1.

  Data Acquisition: Collects real-time and historical data to capture the current state and improve model accuracy.

  2.

  System Modeling: Develops and refines models based on data, crucial for predicting future states.

  3.

  Future State Prediction: Uses models to predict future states and optimize control actions across subsystems.

  4.

  Control Strategy Execution: Executes optimized actions and makes real-time adjustments based on feedback.

  5.

  Performance Evaluation and Feedback: Monitors performance and refines models for continuous improvement.

Figure 4. The generalized model-based multi-agent control process for IBEMS.

Figure 4. The generalized model-based multi-agent control process for IBEMS.

• Model-free Control

  1.

  Data Acquisition: Collects real-time and historical data for pattern recognition and learning.

  2.

  Learning Algorithm Initialization: Selects and initializes model-free algorithms like reinforcement learning.

  3.

  Real-Time Learning and Adaptation: Continuously adapts decisions based on new data and learning outcomes.

  4.

  Control Strategy Execution: Executes control actions independently in each subsystem based on learning outcomes.

  5.

  Performance Evaluation and Feedback: Monitors performance and refines algorithms for ongoing improvement.

Figure 5. The generalized model-free multi-agent control process for IBEMS.

Figure 5. The generalized model-free multi-agent control process for IBEMS.

3. Mathematical Concept Multi-Agent Control Methodologies for IBEMS

According the recent literature, various control algorithms are commonly employed in multi-agent control for IBEMS, both in model-based and model-free approaches. Model-based approaches such as MPC are frequently used for their ability to leverage detailed system models for predictive optimization and coordinated decision-making [45,46]. On the other hand, model-free algorithms such as reinforcement learning (RL) are widely adopted for their flexibility and adaptability, allowing systems to learn and optimize control strategies without relying on predefined models [33]. Other common methodologies that may be implemented in both ways may concern ADMM [47] or evolutionary algorithms such as Particle Swarm Optimization (PSO) [48] and Genetic Algorithms (GAs) [49]. Such control strategies are often tailored to the specific requirements of the building’s energy management systems, providing a balance between precision and adaptability.

3.1. Reinforcement Learning

Reinforcement learning (RL) methodologies applied to multi-agent control in Integrated Building Energy Management Systems (IBEMS) are typically modeled using Markov Decision Processes (MDPs) [50]. An MDP is a mathematical framework used to describe environments in which agents make decisions, with the goal of optimizing a long-term objective, such as energy efficiency or occupant comfort [50]. An MDP is defined by the tuple $(S,A,P,R,\gamma )$, where

• S is the set of possible states. Each state $s\in S$ represents a specific configuration of the building environment, such as temperature levels, energy consumption, and occupant activities.
• A is the set of actions available to the agent. An action $a\in A$ could represent decisions like adjusting the HVAC system or controlling lighting levels.
• $P\left(s^{\prime }\right|s,a)$ is the state transition probability function, which describes the likelihood of moving to a new state $s^{\prime }$ after taking action a in state s. This probability models the dynamics of the environment.
• $R(s,a)$ is the reward function, which assigns a numerical value to each state–action pair. The reward serves as feedback to the agent, encouraging actions that lead to desirable states (e.g., high energy efficiency or occupant comfort) and discouraging undesirable ones.
• $\gamma \in [0,1]$ is the discount factor, which determines the weight given to future rewards. A lower $\gamma$ prioritizes immediate rewards, while a higher $\gamma$ encourages the agent to plan for long-term benefits.

The agent’s goal is to learn a policy $\pi :S\to A$, which dictates the best action to take in each state to maximize the expected cumulative reward over time [50]. The cumulative reward, also known as the return, is defined as follows:

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

where $G_t$ is the total reward from time step t onward, and $r_{t+k+1}$ is the immediate reward received at each future time step. This formulation emphasizes the balance between short-term and long-term rewards, controlled by the discount factor $\gamma$. In multi-agent systems, each agent i operates with its own policy ${\pi }_i\left(a_i\right|s)$, where $a_i\in A_i$ is the action chosen by agent i based on the observed state s. The interaction of multiple agents introduces complexity because the optimal policy for one agent may depend on the actions taken by others. Thus, agents must learn to cooperate or compete, depending on the system’s objectives [50]. One common approach for learning the optimal policy is Q-learning, a value-based method where agents estimate the action value function $Q(s,a)$. The function $Q(s,a)$ represents the expected cumulative reward for taking action a in state s and following the optimal policy thereafter. The Q-learning update rule is based on the Bellman equation [50]:

$$Q(s,a)\leftarrow Q(s,a)+\alpha \left[R(s,a)+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime })-Q(s,a)\right]$$

Here, $\alpha$ is the learning rate, controlling how quickly the agent updates its knowledge. The term $R(s,a)+\gamma {max}_{a^{\prime }}Q(s^{\prime },a^{\prime })$ represents the target value, which combines the immediate reward $R(s,a)$ with the estimated value of the best future action ${max}_{a^{\prime }}Q(s^{\prime },a^{\prime })$. The agent iteratively adjusts its estimate of $Q(s,a)$ to minimize the difference between the current estimate and the target value. By following this iterative process, agents in multi-agent systems gradually learn optimal or near-optimal policies that improve system performance over time. In IBEMS, these policies typically focus on optimizing energy consumption, reducing costs, and maintaining occupant comfort [50]. In more advanced approaches, actor–critic methods are used, where the policy (actor) and the value function (critic) are updated simultaneously. This approach helps agents converge more reliably and efficiently in large and complex environments, as it decouples the process of policy improvement from the process of value estimation [50]. In multi-agent settings, agents coordinate their actions, optimizing overall system performance through centralized or decentralized approaches. According to the literature, RL methodologies are classified into three approaches [36]:

• Value-based: Methods like Q-learning and Deep Q-Networks (DQNs) estimate action–state values, suitable for discrete actions but limited in complex, continuous tasks.
• Policy-based: Methods like Proximal Policy Optimization (PPO) optimize policies directly, effective for continuous control but computationally intensive.
• Actor–critic: Combining value- and policy-based methods, techniques like Soft Actor–Critic (SAC) balance exploration and exploitation, ideal for complex, multi-agent environments but requiring careful tuning.

3.2. Model Predictive Control

Model Predictive Control (MPC) is a powerful approach commonly used in multi-agent control for Integrated Building Energy Management Systems (IBEMS), where agents optimize control actions over a finite future time horizon based on a predictive model of their subsystems [45]. MPC’s advantage lies in its ability to forecast future states and adjust control actions to minimize a predefined cost function while considering system constraints, such as energy consumption limits and occupant comfort levels. In the MPC framework, at each time step, each agent i solves an optimization problem to determine the sequence of control actions $u_i\left(t\right)$ for a future time horizon T [45,51]. The control strategy is designed to minimize a cost function $f_i$, which encapsulates the objectives of the agent, such as minimizing energy consumption, reducing operational costs, or maintaining indoor comfort. The general optimization problem for each agent i can be expressed as follows:

$$\underset{u_i}{min}\sum _{t=0}^T{f}_i(x_i\left(t\right),u_i\left(t\right))$$

where $u_i\left(t\right)$ represents the control inputs (e.g., HVAC settings and lighting levels), $x_i\left(t\right)$ denotes the state variables (e.g., indoor temperature and energy storage levels), and $f_i(x_i\left(t\right),u_i\left(t\right))$ is the cost function that penalizes deviations from desired performance levels [45,51].

System Dynamics and Constraints: The optimization is subject to the system dynamics, typically modeled as a discrete-time linear system [45]:

$$x_i(t+1)=A_i{x}_i\left(t\right)+B_i{u}_i\left(t\right)$$

where

• $x_i\left(t\right)\in {\mathbb{R}}^{n_i}$ is the state vector at time t, representing the internal state of agent i’s subsystem (e.g., temperature in a zone or energy usage of a system);
• $u_i\left(t\right)\in {\mathbb{R}}^{m_i}$ is the control input vector;
• $A_i\in {\mathbb{R}}^{n_i\times n_i}$ is the state transition matrix, governing how the state evolves over time in the absence of control inputs;
• $B_i\in {\mathbb{R}}^{n_i\times m_i}$ is the control input matrix, which defines how control actions influence the state evolution.

The cost function $f_i(x_i\left(t\right),u_i\left(t\right))$ is often quadratic in practice, as this allows for efficient optimization [45,51]. A common form is as follows:

$$f_i(x_i\left(t\right),u_i\left(t\right))={(x_i\left(t\right)-x_i^{ref})}^T{Q}_i(x_i\left(t\right)-x_i^{ref})+u_i{\left(t\right)}^T{R}_i{u}_i\left(t\right)$$

where $x_i^{ref}$ is the reference state (desired temperature, for example) and $Q_i$ and $R_i$ are weight matrices that penalize deviations from the reference and large control actions, respectively. The quadratic nature of $f_i$ makes it easier to solve the optimization problem using standard convex optimization techniques, such as quadratic programming (QP).

MPC Optimization Procedure: The MPC optimization is solved over a receding horizon; although a sequence of control actions $\{u_i\left(0\right),u_i\left(1\right),\cdots ,u_i\left(T\right)\}$ is computed, only the first control action $u_i\left(0\right)$ is implemented. The process then repeats at the next time step with updated state information $x_i(t+1)$, making MPC an adaptive control strategy. This iterative process ensures that the agents can continuously adjust their actions based on real-time data, providing robustness to uncertainties and changes in the building environment [45,51].

3.3. Evolutionary Algorithms

Evolutionary algorithms (EAs) are a class of optimization methods inspired by natural evolution and are widely applied in multi-agent control for Integrated Building Energy Management Systems (IBEMS). These algorithms are particularly effective for exploring large, complex search spaces, where traditional gradient-based methods may struggle due to the non-derivable nature of the objective functions. Two prominent evolutionary approaches used in IBEMS are Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs).

Particle Swarm Optimization (PSO): PSO is a population-based optimization method where agents, known as particles, explore the search space to optimize control parameters (e.g., HVAC settings or renewable energy source (RES) integration) in a collaborative manner [48]. Each particle adjusts its position based on its own experience and that of its neighbors, making PSO particularly suitable for dynamic and uncertain environments like building energy management. The position ${\overrightarrow{x}}_i$ and velocity ${\overrightarrow{v}}_i$ of each particle i are updated using the following equations:

$${\overrightarrow{v}}_i(t+1)=w{\overrightarrow{v}}_i\left(t\right)+c_1{r}_1({\overrightarrow{p}}_i-{\overrightarrow{x}}_i\left(t\right))+c_2{r}_2(\overrightarrow{g}-{\overrightarrow{x}}_i\left(t\right))$$

$${\overrightarrow{x}}_i(t+1)={\overrightarrow{x}}_i\left(t\right)+{\overrightarrow{v}}_i(t+1)$$

where

• ${\overrightarrow{v}}_i\left(t\right)$ is the velocity of particle i at time t;
• w is the inertia weight that controls the exploration–exploitation trade-off;
• $c_1$ and $c_2$ are cognitive and social coefficients, respectively;
• $r_1$ and $r_2$ are random numbers drawn from the uniform distribution in $[0,1]$;
• ${\overrightarrow{p}}_i$ is the particle’s personal best position;
• $\overrightarrow{g}$ is the global best position found by the swarm.

PSO is simple, fast, and adaptive but can suffer from premature convergence, especially when the problem landscape contains multiple local optima [48].

Genetic Algorithms (GA): Genetic Algorithms simulate the process of natural selection, where a population of potential solutions evolves over time to optimize building energy management tasks such as scheduling energy storage systems (ESS) and optimizing domestic hot water (DHW) usage [49]. The evolutionary process follows the following steps [52]:

1.

Selection: Individuals are selected based on their fitness, with higher-fitness individuals more likely to reproduce.

2.

Crossover: Selected individuals are paired to exchange genetic material, combining their chromosomes (solution encodings) to produce offspring.

3.

Mutation: Random changes are introduced to offspring to maintain diversity and prevent premature convergence.

The overall process can be represented by the following equation:

$$\mathrm{New}\mathrm{Population}=\mathrm{Select}\left(\mathrm{Mutate}\right(\mathrm{Crossover}\left(\mathrm{Population}\right)\left)\right)$$

where the new population is formed after selection, crossover, and mutation operations are applied to the existing population. GAs are robust and flexible but can be computationally intensive, particularly in large-scale IBEMS, as the search space grows with system complexity. These evolutionary operations enable the system to converge towards optimal energy management strategies, ensuring an integrated and efficient approach to building energy management [49].

3.4. Alternating Direction Method of Multipliers

The Alternating Direction Method of Multipliers (ADMM) decomposes complex optimization problems into smaller sub-problems, each managed by individual agents controlling different building systems like HVAC, RES, and ESS [53]. Agents solve these sub-problems independently, and then iteratively coordinate to find a global solution by updating local variables $x_i$, global variables z, and dual variables ${\lambda }_i$ through a series of minimization steps. This iterative process continues until convergence, allowing ADMM to efficiently handle large-scale, distributed optimization tasks in IBEMS. Although ADMM is communication-intensive, requiring frequent data exchanges, its decomposition strategy is particularly suited for real-time applications in dynamic environments [53]. The general form of the optimization problem can be written as follows [54]:

$$\underset{x,z}{min}\sum _{i=1}^N{f}_i\left(x_i\right)+g\left(z\right),\mathrm{subject}\mathrm{to}{A}_i{x}_i+B_{i}z=c_i$$

where

• $x_i$ represents the local decision variables of agent i;
• z is a global variable shared among the agents;
• $f_i\left(x_i\right)$ is the local cost function for agent i;
• $g\left(z\right)$ is a global cost function;
• $A_i$ and $B_i$ are matrices that define the coupling between local and global variables;
• $c_i$ is a constant vector representing constraints.

ADMM operates by splitting the optimization problem into two parts: one for the local variables $x_i$ and another for the global variable z. This decomposition allows each agent to solve its own sub-problem independently, which can be computed in parallel, while a global coordination step ensures that the solutions remain consistent [54]. The iterative update steps in ADMM are as follows:

1.

Update $x_i$: Each agent solves its local optimization problem based on its current state and the global variable z.

$$x_i^{k+1}=arg\underset{x_i}{min}\left(f_i\left(x_i\right)+{\displaystyle \frac{\rho }{2}}{\parallel A_i{x}_i+B_i{z}^k-c_i+{\lambda }_i^k\parallel }_2^2\right)$$

Here, $\rho$ is a penalty parameter that controls the convergence speed, and ${\lambda }_i$ is the Lagrange multiplier associated with the constraints.

2.

Update z: A central coordinator updates the global variable z by solving

$$z^{k+1}=arg\underset{z}{min}\left(g\left(z\right)+\sum _{i=1}^N{\displaystyle \frac{\rho }{2}}{\parallel A_i{x}_i^{k+1}+B_{i}z-c_i+{\lambda }_i^k\parallel }_2^2\right)$$

3.

Update Lagrange multipliers ${\lambda }_i$: Each agent updates its local multiplier:

$${\lambda }_i^{k+1}={\lambda }_i^k+\rho \left(A_i{x}_i^{k+1}+B_i{z}^{k+1}-c_i\right)$$

These steps are repeated iteratively until the variables $x_i$, z, and ${\lambda }_i$ converge within a predefined threshold. The method’s ability to decompose large optimization problems into smaller, parallel tasks makes it highly efficient for large-scale multi-agent systems such as IBEMS, especially when coordination and computational scalability are critical [54]. ADMM’s decomposition allows for parallel computation of local updates and centralized updating of global variables, making it well suited to managing large-scale, diverse subsystems in IBEMS [53].

4. Review on Multi-Agent Control for Energy Management Systems

This section exhibits numerous highly cited multi-agent research applications related to IBEMS control optimization in an effort to discriminate them into the aforementioned sub-fields: Model-free (as RL); Model-based (as MPC or other Model-based with Evolutionary optimization, ADMM optimization, Lyapunov optimization etc.); Hybrid strategies which concern the integration of multiple approaches; and Other model-free and model-based applications do not concern any of the aforementioned control strategies.

Along with analyzing the concerning highly cited applications of the last decade (2014–2024), this section also integrates summarized tables for each model-free control sub-field. To this end, Table 1, Table 2, Table 3, Table 4, Table 5, Table 6 and Table 7 contain the following features:

• Ref.: illustrating the reference application in the first column;
• Year: illustrating the publication year of for each research application;
• Type: defining whether the applied multi-agent control strategy concerns the centralized (CE); decentralized (DE); cooperative (CO); or non-cooperative (Non-CO) interaction type between the agents;
• Method: illustrating the specific RL; MPC; ADMM; evolutionary algorithms; Lyapunov optimization; hybrid; or other multi-agent algorithmic methodologies applied in each work;
• IBEMS: illustrating the specific integrated equipment that each IBEMS integrates considering the most common devices in the building setting: HVAC, RES, ESS, LS, EVs, DHW, and Other;
• Residential: defining if the testbed application concerns a Residential Building Control Application with an “x”;
• Commercial: defining if the testbed application concerns a Commercial Building Control Application with an “x”;
• Simulation: defining if the testbed application concerns a Simulative Building Control Application with an “x”;
• Real-life: defining if the testbed application concerns a Real-world or Real-life Building Control Application with an “x”;
• Citations: portrays the number of citations—according to Scopus—of each work.

The abbreviation “N/A” represents the “not identified” elements in tables and figures. In the following subsections, the integrated research applications are described in detail regarding their motivation, their conceptual control methodology, as well and their final outcome.

Table 1. Summarized table of Model-free control applications using RL strategies (2014–2024).

Ref.	Year	Type	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[55]	2015	CO	Q-learning	RES/ESS	x		x		37
[56]	2017	CO	BL	RES/DHW/ESS	x		x		218
[57]	2017	DE/CO	eJAL	HVAC/LS/Other		x	x		50
[58]	2018	DE/CO	Q-learning	RES/ESS/Other	x			x	179
[59]	2019	DE/CO	DQN	RES/ESS/Other	x		x		34
[60]	2020	DE/CO	DQN	HVAC	x	x	x		77
[61]	2020	CO	DQN	HVAC		x	x		42
[62]	2020	CO	A2C	HVAC/RES/ESS	x		x		112
[63]	2020	DE	Q-learning	HVAC/RES/EVs	x		x		209
[64]	2020	DE	ISA	HVAC/RES/Other	x		x		43
[65]	2021	CO	MAAC	HVAC		x	x		175
[66]	2022	CO	DQN	HVAC		x	x		21
[67]	2022	CO	MAAC	HVAC		x	x		38
[68]	2022	CO	D3QN	HVAC/RES/ESS		x	x		50
[69]	2022	CO	Q-learning	HVAC	x		x		23
[70]	2022	CE/CO	BDQ	HVAC		x	x	x	48
[71]	2022	CO	SAC	RES/ESS	x	x	x		21
[72]	2022	DE	ACKTR	RES/ESS/Other	x		x		8
[73]	2022	CO	TD3	RES/ESS/Other		x	x	x	17
[74]	2023	DE/CO	PPO	RES/ESS	x	x	x		11
[75]	2023	DE/CO	DDPG	HVAC/RES/ESS	x	x	x		30
[76]	2023	DE/CO	MAAC	HVAC		x	x		18
[77]	2023	Non-CO/DE	SAC	RES/ESS	x		x		11
[78]	2023	CO	DQN	HVAC	x		x		15

Table 1. Summarized table of Model-free control applications using RL strategies (2014–2024).

Ref.	Year	Type	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[55]	2015	CO	Q-learning	RES/ESS	x		x		37
[56]	2017	CO	BL	RES/DHW/ESS	x		x		218
[57]	2017	DE/CO	eJAL	HVAC/LS/Other		x	x		50
[58]	2018	DE/CO	Q-learning	RES/ESS/Other	x			x	179
[59]	2019	DE/CO	DQN	RES/ESS/Other	x		x		34
[60]	2020	DE/CO	DQN	HVAC	x	x	x		77
[61]	2020	CO	DQN	HVAC		x	x		42
[62]	2020	CO	A2C	HVAC/RES/ESS	x		x		112
[63]	2020	DE	Q-learning	HVAC/RES/EVs	x		x		209
[64]	2020	DE	ISA	HVAC/RES/Other	x		x		43
[65]	2021	CO	MAAC	HVAC		x	x		175
[66]	2022	CO	DQN	HVAC		x	x		21
[67]	2022	CO	MAAC	HVAC		x	x		38
[68]	2022	CO	D3QN	HVAC/RES/ESS		x	x		50
[69]	2022	CO	Q-learning	HVAC	x		x		23
[70]	2022	CE/CO	BDQ	HVAC		x	x	x	48
[71]	2022	CO	SAC	RES/ESS	x	x	x		21
[72]	2022	DE	ACKTR	RES/ESS/Other	x		x		8
[73]	2022	CO	TD3	RES/ESS/Other		x	x	x	17
[74]	2023	DE/CO	PPO	RES/ESS	x	x	x		11
[75]	2023	DE/CO	DDPG	HVAC/RES/ESS	x	x	x		30
[76]	2023	DE/CO	MAAC	HVAC		x	x		18
[77]	2023	Non-CO/DE	SAC	RES/ESS	x		x		11
[78]	2023	CO	DQN	HVAC	x		x		15

Table 2. Summarized table of Model-based control applications with using MPC strategies (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[79]	2014	DE	N/A	HVAC		x	x		16
[80]	2017	DE	E-MPC	HVAC	x		x		66
[81]	2017	DE	Grey-Box	HVAC	x		x		12
[82]	2018	DE/CO	Game	HVAC	x	x	x		13
[83]	2018	CO	N/A	HVAC		x	x		34
[84]	2018	DE	EBDC	RES/EVs	x	x	x		71
[85]	2019	DE	DP	HVAC/RES/EVs	x		x		11
[86]	2020	DE	QP	HVAC/ESS	x		x		12
[87]	2020	DE	Naive	HVAC	x		x		44
[88]	2020	DE	Hierarchical	HVAC/RES/ESS	x	x	x		18
[89]	2020	DE	DP	HVAC		x	x		31
[90]	2021	DE	N/A	HVAC		x	x	x	28
[91]	2022	CO	DD	HVAC/ESS/Other		x	x	x	29

Table 2. Summarized table of Model-based control applications with using MPC strategies (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[79]	2014	DE	N/A	HVAC		x	x		16
[80]	2017	DE	E-MPC	HVAC	x		x		66
[81]	2017	DE	Grey-Box	HVAC	x		x		12
[82]	2018	DE/CO	Game	HVAC	x	x	x		13
[83]	2018	CO	N/A	HVAC		x	x		34
[84]	2018	DE	EBDC	RES/EVs	x	x	x		71
[85]	2019	DE	DP	HVAC/RES/EVs	x		x		11
[86]	2020	DE	QP	HVAC/ESS	x		x		12
[87]	2020	DE	Naive	HVAC	x		x		44
[88]	2020	DE	Hierarchical	HVAC/RES/ESS	x	x	x		18
[89]	2020	DE	DP	HVAC		x	x		31
[90]	2021	DE	N/A	HVAC		x	x	x	28
[91]	2022	CO	DD	HVAC/ESS/Other		x	x	x	29

Table 3. Summarized table of Model-based control applications with Evolutionary optimization (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[92]	2015	CO	PSO	HVAC/LS/ESS		x	x		71
[93]	2016	CO	GA	HVAC/LS/RES/ESS		x	x		69
[94]	2017	CO	PSO	HVAC/LS/RES/ESS		x	x		20
[95]	2018	CO	GA	HVAC/LS/RES/ESS		x	x		43
[96]	2023	DE/CO	PSO	HVAC/RES/ESS/EVs	x		x		16

Table 3. Summarized table of Model-based control applications with Evolutionary optimization (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[92]	2015	CO	PSO	HVAC/LS/ESS		x	x		71
[93]	2016	CO	GA	HVAC/LS/RES/ESS		x	x		69
[94]	2017	CO	PSO	HVAC/LS/RES/ESS		x	x		20
[95]	2018	CO	GA	HVAC/LS/RES/ESS		x	x		43
[96]	2023	DE/CO	PSO	HVAC/RES/ESS/EVs	x		x		16

Table 4. Summarized table of Model-based control applications with ADMM optimization (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[97]	2016	CO	ADMM	HVAC		x	x		72
[98]	2017	CO	J-ADMM	HVAC		x	x		34
[99]	2019	CO	NC-ADMM	RES	x		x		73
[100]	2020	CO	J-ADMM	HVAC		x	x		54
[101]	2021	DE/CO	J-ADMM	HVAC		x	x		21
[102]	2021	DE/CO	DC-ADMM	HVAC/RES/ESS/EVs	x	x	x		80
[103]	2022	CO	H-ADMM	HVAC		x	x		11

Table 4. Summarized table of Model-based control applications with ADMM optimization (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[97]	2016	CO	ADMM	HVAC		x	x		72
[98]	2017	CO	J-ADMM	HVAC		x	x		34
[99]	2019	CO	NC-ADMM	RES	x		x		73
[100]	2020	CO	J-ADMM	HVAC		x	x		54
[101]	2021	DE/CO	J-ADMM	HVAC		x	x		21
[102]	2021	DE/CO	DC-ADMM	HVAC/RES/ESS/EVs	x	x	x		80
[103]	2022	CO	H-ADMM	HVAC		x	x		11

Table 5. Summarized table of Model-based control applications with Lyapunov optimization (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[104]	2014	Non-CO	LCMA	RES/EES/Other	x		x		147
[105]	2014	Non-CO	Lyapunov	HVAC	x		x		87
[106]	2017	Non-CO	Lyapunov	HVAC		x	x		50

Table 5. Summarized table of Model-based control applications with Lyapunov optimization (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[104]	2014	Non-CO	LCMA	RES/EES/Other	x		x		147
[105]	2014	Non-CO	Lyapunov	HVAC	x		x		87
[106]	2017	Non-CO	Lyapunov	HVAC		x	x		50

Table 6. Summarized table of Hybrid control applications (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[107]	2014	CO	ULDC/FLC	HVAC		x		x	162
[108]	2016	DE	PSO/ANN	HVAC		x		x	73
[109]	2018	CO	CBR/ANN	HVAC		x		x	105
[110]	2018	CO	ADMM/MPC	HVAC		x	x	x	31
[111]	2020	CE	RL/ANN	HVAC	x		x		60
[112]	2020	CE	RL/ANN	HVAC		x	x		96
[113]	2021	CO	PSO/MPC	HVAC/RES/ESS	x		x	x	30
[114]	2021	CO	PSO/MPC	HVAC		x	x		19
[115]	2022	CO	RL/FLC	HVAC/LS/Other	x		x		29
[116]	2022	CO	ADMM/MPC	HVAC		x	x		19

Table 6. Summarized table of Hybrid control applications (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[107]	2014	CO	ULDC/FLC	HVAC		x		x	162
[108]	2016	DE	PSO/ANN	HVAC		x		x	73
[109]	2018	CO	CBR/ANN	HVAC		x		x	105
[110]	2018	CO	ADMM/MPC	HVAC		x	x	x	31
[111]	2020	CE	RL/ANN	HVAC	x		x		60
[112]	2020	CE	RL/ANN	HVAC		x	x		96
[113]	2021	CO	PSO/MPC	HVAC/RES/ESS	x		x	x	30
[114]	2021	CO	PSO/MPC	HVAC		x	x		19
[115]	2022	CO	RL/FLC	HVAC/LS/Other	x		x		29
[116]	2022	CO	ADMM/MPC	HVAC		x	x		19

Table 7. Summarized table of Other control applications (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[117]	2014	DE	Consensus	HVAC/EVs/Other	x			x	225
[118]	2016	CO	N/A	HVAC		x	x	x	46
[119]	2016	CO	SoC	RES/ESS	x		x	x	58
[120]	2018	CE/CO	L4GPCAO	HVAC		x		x	51
[121]	2019	DE/CO	TBSA	HVAC		x		x	85
[122]	2019	DE	CHPD	HVAC/RES/ESS		x	x		44
[123]	2020	CO	DAC	HVAC		x	x		41
[124]	2021	DE/CO	Consensus	RES/ESS/Other	x	x	x		49
[125]	2021	CE/DE/CO	RPC/APT	HVAC/DHW/RES/ESS	x		x		10
[126]	2021	CO	Hierarchical	RES/ESS/Other	x	x	x		25
[127]	2021	DE/CO	Blockchain	HVAC/LS/Other	x		x		35
[128]	2022	CO	N/A	HVAC/RES/ESS		x	x		22

Table 7. Summarized table of Other control applications (2014–2024).

Ref.	Year	Agent	Method	IBEMS	Residential	Commercial	Simulation	Real-Life	Citations
[117]	2014	DE	Consensus	HVAC/EVs/Other	x			x	225
[118]	2016	CO	N/A	HVAC		x	x	x	46
[119]	2016	CO	SoC	RES/ESS	x		x	x	58
[120]	2018	CE/CO	L4GPCAO	HVAC		x		x	51
[121]	2019	DE/CO	TBSA	HVAC		x		x	85
[122]	2019	DE	CHPD	HVAC/RES/ESS		x	x		44
[123]	2020	CO	DAC	HVAC		x	x		41
[124]	2021	DE/CO	Consensus	RES/ESS/Other	x	x	x		49
[125]	2021	CE/DE/CO	RPC/APT	HVAC/DHW/RES/ESS	x		x		10
[126]	2021	CO	Hierarchical	RES/ESS/Other	x	x	x		25
[127]	2021	DE/CO	Blockchain	HVAC/LS/Other	x		x		35
[128]	2022	CO	N/A	HVAC/RES/ESS		x	x		22

4.1. Model-Free Control Strategies

Reinforcement Learning

In 2015, Raju et al. [55] implemented a novel value-based approach–namely, the Coordinated Q-learning (CQL) approach—in a multi-agent framework for grid-connected solar microgrid battery scheduling. According to the evaluation, CQL was able to reduce grid power consumption by 15% and enhance solar power utility by 10–12%. In 2017, a policy-based RL approach was proposed by Avari et al. [56]. According to the evaluation, they proposed an ontology-driven EMS, achieving a 5% reduction in microgrid costs and maintaining a 90% user satisfaction rate. In the same year, Hurtado et al. [57] introduced another policy-based approach—namely eJAL—which concerned a cooperative and decentralized algorithm. The algorithm managed to achieve the highest fairness index and reduced overload duration by 16.3%, compared to 22.6% with Q-learning and 5.2% with the n-player game. According to the simulation results, the comfort loss using eJAL was minimal, averaging at 0.76%. The next year, Kofinas et al. [58] developed a Hybrid Q-learning multi-agent control framework for IBEMS, integrating MPC to manage different zones and HVAC components. This model-based RL approach achieved around 20% energy savings while maintaining thermal comfort at the desired levels. In 2019, Prasad et al. [59] utilized a DQN framework for energy sharing, modeling each building as an agent, achieving a 40–60 kWh improvement over non-sharing strategies.

In 2020, Gupta et al. [60] compared the performance of a novel DQN-based heating controller with traditional thermostats. Stimulative experiments demonstrated significant enhancements in thermal comfort (15–30%) and energy cost reduction (5–12%). In the same year, Nagarathinam et al. [61] employed multiple cooperative DQL agents to dynamically control both AHU and chiller set-points for optimizing HVAC operations towards energy efficiency and occupant comfort. According to the evaluation, the MARCO framework achieved a 17% reduction in energy consumption compared to traditional seasonal set-point strategies, and performed comparably to an ideal MPC with only 2% higher energy use while maintaining an absolute comfort compliance. In an actor–critic RL approach of 2020, Lee et al. [62] introduced a federated RL framework using the A2C algorithm. According to the results, the AC framework reduced electricity consumption by 20%, costs by 30%, and emissions by 20%. In the same yer, Xu et al. [63] applied Q-learning for HVAC control, based on historical and real-time data, resulting in approximately 15% energy savings and reduced operational costs. Also in 2020, Vazquez et al. [64] developed the MARLISA framework, which considers a MARL controller with an iterative sequential action selection algorithm, designed for scalable, decentralized load shaping in urban energy systems. The approach reduced the daily peak load by 15% and ramping by 35% in contrast to RBC in the concerned building. In a 2021 work, Yu et al. [65] proposed a Multi-Agent Actor–critic (MAAC) approach to minimize energy costs while considering random zone occupancy, thermal comfort, and indoor air quality. The novel control framework achieved a significant energy cost reduction of 56.50–75.25% in contrast to baseline control in simulations with 30 zones.

In 2022, Fu et al. [66] proposed a DQN framework to optimize HVAC system operation by distributing the action space across five agents—each responsible for different system components like chillers, cooling towers, and water pumps. The novel framework managed to achieve 11.1% improved energy efficiency and faster convergence. In the same year, Yu et al. [67] used a MAAC framework for coordinating HVAC and personal comfort systems in office spaces, reducing energy consumption by 0.7–4.18% and improving thermal comfort by 64.13–72.08%. In the same year, Shen et al. [68] proposed a novel multi-agent D3QN framework for energy system optimization. Such a novel approach achieved an 84% reduction in uncomfortable duration and a 43% decrease in unconsumed renewable energy. Blad et al. [69] introduced an offline Q-learning framework for HVACs, comparing Multi-Layer Perceptron (MLP) and Long short-term memory (LSTM) networks. According to the final outcomes, the LSTM models reduced prediction errors by 45% and achieved a 19.4% reduction in heating costs. Also the same year, Lei et al. [70] utilized a Branching Dueling Q-network for high-dimensional control action spaces, incorporating a tabular-based personal comfort modeling method. In a real-world office deployment, the approach achieved a 14% cooling energy reduction and 11% improvement in thermal acceptability. Pinto et al. [71] utilized a SAC algorithm for energy management in buildings with thermal storage and PV systems, leading to a 7% cost reduction and a 14% peak demand reduction. Similarly, Chu et al. [72] employed another AC methodology—namely, the ACKTR methodology—for optimizing home IBEMS, achieving a 25.37% cost reduction on a typical test day. Moreover, in 2022 Gao et al. [73] developed a hierarchical RL framework for optimizing off-grid operations, improving performance by 64.93% and reducing unsafe battery runtime by 84%. In the same year, Pigott et al. [74] introduced GridLearn, using the PPO algorithm to manage voltage regulation, resulting in a 34% reduction in overvoltages.

In 2023, a DDPG algorithm was proposed by Qiu et al. [75], abbreviated as the Fed-JPC algorithm. Such a novel algorithmic scheme incorporated federated learning to maintain privacy while optimizing energy trading and emissions reduction. Empirical results showed that the Fed-JPC outperformed traditional methods, reducing total energy and environmental costs by 5.87% and 8.02%, respectively. In the same year, Xie et al. [76] applied a MAAC methodology for demand response in grid-responsive buildings, achieving over a 6% reduction in net load demand. Also in 2023, Nweye et al. [77] introduced the MERLIN framework using SAC agents for battery control, improving electricity consumption by 20%, cost by 30%, and emissions by 20%. More recently, Homod et al. [78] introduced a DCCMARL framework for multi-chiller control, reducing energy consumption by up to 49% and improving overall building energy efficiency by 44.5%.

MARL approach classification into value-based, policy-based, and actor–critic may be found at

Table 8. Classification into value-based, policy-based, and actor–critic for RL applications.

Approach	Ref.	Achievement
Value-based	[55]	CQL reduced grid power consumption by 15% and increased solar utility by 10–12%
	[58]	Hybrid Q-learning achieved 20% energy savings
	[63]	Q-learning for HVAC control achieved 15% energy savings
	[69]	Q-learning with LSTM reduced heating costs by 19.4%
	[59]	DQN achieved a 40–60 kWh improvement in energy sharing
	[60]	DQN-based controller improved thermal comfort by 15–30%
	[61]	DQL agents reduced energy consumption by 17%
	[66]	DQN optimized HVACs, improving energy efficiency by 11.1%
	[78]	DCCMARL reduced energy consumption by 49%
	[68]	D3QN reduced uncomfortable duration by 84%
	[70]	BDQ reduced cooling energy by 14%
Policy-based	[56]	Ontology-driven EMS reduced microgrid costs by 5%
	[57]	eJAL algorithm reduced overload duration by 16.3%
	[64]	MARLISA reduced peak load by 15% and ramping by 35%
	[73]	Hierarchical RL improved off-grid operations by 64.93%
	[74]	PPO reduced overvoltages by 34%
Actor–critic	[65]	MAAC reduced energy costs by 56.50–75.25%
	[67]	MAAC reduced energy consumption by 0.7–4.18%
	[75]	Fed-JPC algorithm reduced costs and emissions by 5.87% and 8.02%
	[76]	MAAC reduced net load demand by 6%
	[71]	SAC reduced costs by 7% and peak demand by 14%
	[77]	SAC reduced electricity consumption by 20% and costs by 30%
	[62]	A2C reduced electricity by 20% and emissions by 20%
	[72]	ACKTR reduced energy cost by 25.37%

along with the primary achievement of each RL research work.

4.2. Model-Based Control Strategies

4.2.1. Model Predictive Control (MPC)

In 2014, Lauro et al. [79] proposed an MPC strategy for a commercial office building’s HVAC system, comparing decentralized and distributed configurations. Their adaptive strategy reduced energy consumption by 49% and achieved a lower average temperature error (0.035 °C vs. 0.073 °C) compared to non-adaptive methods, with a 29% improvement in comfort. In 2017, Pedersen et al. [80] introduced an Economic–MPC (E-MPC) scheme for multi-story residential buildings, achieving up to 6% cost savings and 3% CO₂ reduction. According to the evaluation, the centralized approach slightly outperformed the decentralized MPC, shifting 2 kWh/m² of energy from peak load periods. In a related study by Pedersen et al. [81], a decentralized MPC managed to achieve similar cost savings (11.1%) and comfort levels to centralized control, particularly in retrofitted buildings with insulated walls. Abobakr et al. [82] proposed a decentralized MPC strategy for thermal appliances in smart buildings, combining local MPC with a game-theoretic supervisory control. This approach reduced peak power consumption by 36% and maintained temperature regulation within limits. In 2018, Sangi et al. [83] introduced a hybrid agent-based MPC approach for large-scale commercial buildings, achieving a 2% reduction in primary energy consumption using exergy as a cost function. In the same year, Yang et al. [84] developed a decentralized MPC algorithm for coordinating EV charging with wind power in microgrids, reducing grid dependency by 68.4%. Zhuo et al. [85] presented a multi-agent control strategy for microgrids, optimizing RES, and loads. Their method reduced energy costs by 10% and enhanced computational efficiency. The next year, Lyons et al. [86] implemented MPC for a communal heating system in residential estates, achieving significant improvements in thermal comfort (2.19 Kh vs. 126.45 Kh) and cost efficiency (13.40 p vs. 13.43 p per kWh) compared to PI control. El Geneidy et al. [87] investigated energy flexibility in residential communities using MPC, showing that centralized control offered better coordination and energy efficiency, despite increased overall energy consumption (3.31–25.11%) due to preheating. Also in 2020, Wang et al. [88] proposed a multi-agent MPC scheme for optimizing renewable resource allocation and load scheduling in mixed-use communities, minimizing unserved load to 0.27% in optimal scenarios. Saletti et al. [89] enhanced MPC with dynamic programming to optimize thermal energy distribution in a school complex, reducing fuel consumption by 7% and avoiding comfort failures. The next year, Wu et al. [90] applied a multi-agent MPC approach for district heating in commercial buildings, achieving up to 4.8% annual cost savings, with a trade-off between energy savings and occupant comfort. More recently, Lefebure et al. [91] examined distributed MPC using dual decomposition, achieving near-centralized performance with 16% lower heating costs compared to decentralized control, confirming the approach’s scalability and efficiency.

4.2.2. Model-Based Control with Evolutionary-Based Optimization

The literature from 2014 to 2024 presents several multi-agent evolutionary approaches for optimizing IBEMS, with Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) being the most predominan.

Particle Swarm Optimization: Hurtado et al. [92] in 2015 integrated PSO with a multi-agent system (MAS) to optimize energy use and comfort in smart buildings. Focusing on HVAC systems, RES, and ESS, their approach dynamically adjusts building operations to support local grid voltage control. The approach achieved up to 25% energy savings while improving grid support. Altayeva et al. [94] in 2017 developed a MAS using PSO for IBEMS, dynamically allocating power based on comfort priorities. PSO achieved the required temperature values and overall comfort even with a reduction in available power, effectively handling a 30% decrease in power availability. In 2023, Ghazimirsaeid et al. [96] optimized home microgrids in green buildings using PSO, achieving a 34% reduction in the Market Clearing Price during peak times and shifting 26% of total DR+ load to non-peak intervals.

Genetic Algorithms: Shaikh et al. [93] in 2016 proposed a multi-agent system using a hybrid multi-objective genetic algorithm (HMOGA) for building energy management, including eight solar panels (each 4.5 kW) and six wind turbine generators (each 5 kW), along with batteries totaling 45 kWh storage capacity. Such a framekwork improved energy efficiency by 31.6% and the comfort index by 8.1%. In another GA study of 2018, Rehman et al. [95] proposed multi-objective GA—namely, the Non-dominated Sorting Genetic Algorithm II (NSGA-II)—to optimize and compare centralized and semi-decentralized solar district heating systems in Finnish conditions. The specific IBEMS integrated solar collectors, heat pumps, thermal energy storage, and decentralized high-temperature tanks. According to the evaluation, the decentralized approach achieved up to 35% lower life cycle costs and a renewable energy fraction close to 90%, outperforming the centralized system in both economic and energy efficiency.

4.2.3. Model-Based Control with ADMM-Based Optimization

The literature from 2014 to 2024 highlights several multi-agent ADMM approaches for optimizing IBEMS, focusing on distributed control strategies: In 2016, Cai et al. [97] proposed a multi-agent control approach utilizing ADMM that automated controller design and reduced engineering effort. Such a framework achieved a 12.2% improvement in energy efficiency and a 42.7% potential energy saving in a DX air conditioning system. Hou et al. [98] in 2017 introduced a cooperative multi-agent control strategy using Proximal Jacobian ADMM within a DMPC framework, optimizing HVAC energy consumption across multiple zones. According to the evaluation, the approached exhibited a 10.15% reduction in energy consumption. In 2019, Carli et al. [99] proposed a Non-Convex ADMM (NC-ADMM) methodology for energy scheduling in microgrids. According to the simulation results, the approach significantly reduced the overall energy costs compared to scenarios without RES exchange, indicating the algorithm’s effectiveness in practical applications. In 2020, Li et al. [100] proposed a Jacobian ADMM (J-ADMM) for decentralized HVAC control, in IoT-enabled smart buildings. The study compared the novel strategy to a baseline control, which considered a time-driven centralized optimal control with a 1-min interval, achieving up to 11.34% energy savings and a 96.44% reduction in the excessive pollutant index (EPI). The following year, Li et al. [101] extended this approach, reducing CO₂ levels and energy consumption, and extending sensor battery life from 6.23 days to 84.03 days. In 2021, Lyu et al. [102] developed a fully decentralized algorithm based on Dual-Consensus ADMM (DC-ADMM) for peer-to-peer energy sharing among smart buildings, achieving cost reductions up to USD 110.57 per building while enhancing privacy and reducing information exchange. More recently, Wang et al. [103] proposed a Hierarchical ADMM (H-ADMM) for HVAC systems, using a multi-layer architecture to represent the tree-like structure of BEMS components and zones. Such an approach divided the overall optimization task into smaller local tasks, solved recursively using a hierarchical ADMM with Nesterov acceleration to improve convergence. According to the evalution, the proposed methodology achieved 4.05–5.33% energy savings and a 69.55% reduction in computation time compared to conventional methods.

4.2.4. Model-Based Control with Lyapunov-Based Optimization

The literature from 2014 to 2024 includes a few notable multi-agent Lyapunov optimization approaches, which have shown substantial results in IBEMS optimization: Guo et al. [104] in 2013 utilized a Lyapunov-based cost minimization algorithm (LCMA) for coordinating energy consumption in smart grid neighborhoods. Such an approach was able to handle stochastic processes without needing future statistical knowledge and provided a decentralized implementation to preserve household privacy, achieving a 20% reduction in total energy costs by optimizing renewable generation, energy storage, and smart appliances without requiring future statistical knowledge. In 2014, Zheng et al. [105] applied Lyapunov optimization for distributed demand response (DR) in HVAC systems, minimizing power demand fluctuations while maintaining user comfort. The study validated its approach with practical data, including wind speed and temperature, reducing demand variability effectively. Yu et al. [106] in 2017 extended Lyapunov optimization to a real-time distributed control strategy for commercial building HVAC systems including air handling units (AHUs), variable air volume (VAV) boxes, and sensors. According to the evaluation, which considered real-world data sets like electricity prices and outdoor temperatures, the methodology managed to reduce energy costs by 26.6% and maintain temperature deviations within 1 °C.

4.3. Hybrid Strategies

From 2014 to 2024, numerous hybrid multi-agent optimization approaches have been developed, combining RL, ANNs, FLC, and MPC to enhance IBEMS efficiency. These approaches demonstrate substantial improvements over basic methods: In 2014, Jazizadeh [107] introduced a hybrid HVAC control framework combining participatory sensing, FLC, and a decentralized multi-agent strategy. The system dynamically adjusted set-points based on user comfort profiles, achieving a 39% reduction in daily airflow and increasing user satisfaction from 4.7 to 8.4 out of 10. Javed et al. [108] in 2016 proposed a decentralized multi-agent control strategy using a Random Neural Network and PSO-SQP optimization for HVAC systems in commercial buildings. The approach reduced energy consumption by 27.12% and achieved 88% occupancy estimation accuracy. Gonzalez et al. [109] in 2018 employed a hybrid approach combining a multi-agent system, case-based reasoning, and an ANN to optimize HVAC settings, resulting in a 41% energy saving in office buildings. That same year, Joe et al. [110] introduced a novel decentralized MPC strategy using PJ-ADMM for decentralized HVAC optimization—specifically, radiant floor heating and cooling—within a commercial office building environment. The methodology was evaluated in both real-life implementation and simulations, showing a 27% reduction in electricity consumption during the cooling season compared to baseline feedback control. In 2020, Lork et al. [111] proposed a hybrid control system using Bayesian Convolutional Neural Networks (BCNNs) and Q-learning, achieving a 15–20% improvement in energy savings. Zou et al. [112] combined LSTM networks with Deep Reinforcement Learning for HVAC control, reducing energy consumption by 27–30% while maintaining thermal comfort within ASHRAE standards. Rochd et al. [113] in 2021 introduced a PSO-MPC hybrid algorithm for energy management in smart homes, achieving 90% self-consumption and an 85% reduction in daily electricity costs. Li et al. [114] proposed using Improved Particle Swarm Optimization (IPSO) and Newton Raphson (NR) methods for multi-zone heating systems. Such a hybrid algorithm enhanced multi-agent control by leveraging Nash optimization for global equilibrium and sharing information between subsystems, significantly improving control accuracy. Numerical results illustrated a 19.05% improvement in control accuracy and a 35.71% reduction in mean absolute error (MAE) compared to quadratic programming for single-zone buildings and decentralized MPC for multi-zone buildings. In 2022, Homod et al. [115] introduced the DCMARL (Hybrid Deep Clustering Multi-Agent Reinforcement Learning) algorithm, which integrated RL and fuzzy logic control (FLC) to effectively handle large state and action spaces and improve learning efficiency. Such a framework was able to improve energy savings by 32% and thermal comfort by 21% compared to PID control. In 2022, Mork et al. [116] developed a multi-agent control strategy using distributed MPC for energy optimization. The approach integrated the cooperative ADMM algorithm for hydraulic coupling and non-cooperative Nash optimization for thermal coupling, applied to HVAC systems with detailed Modelica models, reducing energy consumption by 7.6% and thermal discomfort by 65.0% compared to decentralized methods.

4.4. Other Model-Free and Model-Based Strategies

Unlike RL, evolutionary algorithms, ADMM, or Lyapunov optimization, the literature from 2014 to 2024 features various independent multi-agent algorithms. These approaches excel in handling complex, nonlinear, and stochastic environments without requiring detailed system models. While computationally demanding and requiring substantial training data, their adaptability and learning capabilities make them powerful tools for efficient energy management in smart buildings: In 2014, Chen et al. [117] proposed a Distributed Direct Load Control (DDLC) algorithm using an average consensus algorithm to coordinate Energy Management Controllers (EMCs) within buildings. This system dynamically adjusted local power consumption based on real-time data, reducing demand–supply mismatches by 54.6% compared to unmanaged scenarios. In 2016, Dai et al. [118] introduced a decentralized control algorithm for HVAC systems, achieving a 6.8% reduction in power consumption and a 9.93% improvement in chiller efficiency through collaborative optimization. Hollinger et al. [119] in 2016 developed a State of Charge (SoC) decentralized control algorithm for distributed solar battery systems, ensuring local self-sufficiency while providing grid services. Their approach achieved a 58.6% self-sufficiency rate and significantly reduced the need for corrective energy. Michailidis et al. [120] evaluated a novel decentralized, agent-based, model-free Building Optimization and Control (BOC) methodology, referred as Local-for-Global Parameterized Cognitive Adaptive Optimization (L4GPCAO), applied to a real-world building. The building, located at RWTH Aachen University, was equipped with smart HVAC systems and renewable energy sources, integrated through the existing Building Management System (BES). According to the evaluation, the L4GPCAO approach demonstrated a 34.7% reduction in non-renewable energy consumption (NREC) compared to the baseline control strategy, achieving significant energy savings and improved comfort. A follow-up simulative study in 2020 [129] demonstrated further improvements, reducing non-renewable energy consumption by up to 8.96% and improving indoor comfort by up to 56.82%. Png et al. [121] in 2019 introduced the Smart-Token-Based Scheduling Algorithm (Smart-TBSA) using IoT devices to optimize HVAC systems in commercial buildings, achieving up to 20% energy savings. The same year, Xie et al. [122] proposed a decentralized control strategy for combined heat and power dispatch in integrated energy systems, reducing costs by 6.1% and wind curtailment by 51.8%. In a noticeable study of 2020, Lymperopoulos et al. [123] evaluated a Distributed Adaptive Control (DAC) scheme for multi-zone HVAC systems using online learning and information exchange between neighboring zones. The control system was applied to a six-zone building and a large school, achieving up to a 36.80% improvement in temperature tracking and a 13.68% reduction in energy consumption compared to baseline constant-gain control. The next year, Alhasnawi et al. [124] proposed a decentralized control strategy for multi-agent microgrids, enhancing frequency and voltage regulation by 30% and improving power-sharing accuracy by up to 25%. In 2021, Ahrens et al. [125] compared centralized and decentralized strategies for grid resilience, achieving a 99.91% reduction in voltage deviations and a 91.52% decrease in line congestion using decentralized Reactive Power Control (RPC). The same year, Jonban et al. [126] introduced a model-based EMS for DC microgrids in green buildings, maintaining stable DC bus voltage within 2% of the nominal value. Kolahan et al. [127] proposed a blockchain-integrated control with smart contracts to enable homes to share real-time data (Probability of the Next Hour, PNH) and collaboratively optimize energy consumption. The study simulated a neighborhood of 2,000 homes with various occupancy patterns, utilizing a validated physical model of heating, illumination, and appliance systems, reducing peak power load by 15% and total energy consumption by 11%. Lastly, Gupta et al. [128] in 2022 introduced a coordinated control strategy for ESS and IBDR programs. Each component (ESS and HVAC systems in buildings) operates based on local control signals and algorithms (PI and PD controllers) rather than a centralized coordinator, reducing the frequency deviation to within 0.1 Hz and improving the regulation capacity by 50% compared to BES alone.

5. Evaluation

5.1. Evaluation per Multi-Agent Algorithmic Methodology

Model-based approaches are predominant in multi-agent control applications due to their predictive capabilities, which are crucial for coordinating complex systems (Figure 6). However, according to the evaluation the most common algorithmic approach in multi-agent control remains RL, which predominantly concerns a model-free methodology. The following subsections elaborate on the most dominant multi-agent control methodologies as concerns model-based and model-free strategies.

5.1.1. Reinforcement Learning

The exploration of model-free algorithms and especially RL in multi-agent control systems, particularly within IBEMS, has led to a diverse set of approaches. Some algorithms have emerged as more prevalent due to their ability to handle specific challenges in multi-agent environments (Figure 7 (Left) and Figure 7 (Right)), such as coordination among agents, learning efficiency, and the handling of large state or action spaces. More specifically, Q-learning remains one of the fundamental approaches in MARL. Numerous applications [58,63,69] applied Q-learning frameworks to optimize energy consumption in BEMS, demonstrating the algorithm’s strength in handling discrete actions and providing stable, incremental improvements in control policies across different building zones and HVAC systems. Q-learning’s robustness in such structured environments has made it a popular choice, particularly when enhanced with MPC and deep learning techniques. DQNs have also become increasingly prominent, especially in complex multi-agent environments where state spaces are continuous. Such a tendency is evident in [59,60,61] showcasing the efficiency of a DQN in decentralized and cooperative settings, where agents manage their control strategies independently but contribute to a common goal. Additionally, in [66,78], the DQN multi-agent methodology was enhanced by integrating deep clustering and distributed action spaces, thereby improving learning efficiency and coordination among agents in HVAC systems. The prevalence of a DQN among all RL methodologies (Figure 7 (Right)), especially in cooperative scenarios, highlighted its flexibility and effectiveness in balancing energy efficiency and occupant comfort. Other advanced value-based methods, such as the Dueling Double Deep Q-Network (D3QN), have also been utilized in [68] to optimize renewable energy integration in building systems, leveraging its improved learning stability and coordination capabilities among agents. Moreover, the Coordinated Q-learning approach in [55] and the Branching Dueling Q-network (BDQ) in [70] further illustrated the adaptability of the value-based methods in high-dimensional action spaces, enabling efficient and reliable control in complex environments.

As concerns policy-based methods—which present a lower occurrence in RL methodologies (Figure 7 (Left))—they have demonstrated significant advantages in environments requiring continuous control and direct policy optimization. In [56,57], a policy-based multi-agent control was deployed, highlighted the effectiveness of policy-based approaches—like eJAL—in balancing grid support and occupant comfort. Similarly, policy-based methodologies [73,74] were applied to optimize off-grid operations and voltage regulation in power systems, demonstrating their scalability and robustness in real-time applications.

As concerns the actor–critic methods, it is noticeable that especially Multi-Actor Attention–Critic (MAAC) and also Soft Actor–Critic (SAC) have become increasingly prevalent in MARL due to their ability to handle continuous actions and large state spaces. Such AC methodologies are employed in [65,67,71,76,77] to manage energy consumption and demand response in grid-interactive buildings, showcasing their effectiveness in coordinating multiple agents to achieve energy efficiency and thermal comfort. It is evident that such frameworks illustrate enhanced robustness and adaptability, particularly when extended with attention mechanisms [62] and federated learning [72], facts which render them as preferable choices in complicated multi-agent scenarios.

Across these RL approaches, DQN, D3QN, and actor–critic methods, particularly SAC and MAAC, have emerged as the most prevalent algorithms due to their ability to effectively handle large, complex environments and continuous action spaces (Figure 7 (Left)). These algorithms have been chosen for their robustness and versatility in optimizing multi-agent interactions, managing state and action spaces, and ensuring stable learning processes in dynamic environments. The similarities in the adoption of these algorithms across different studies underline their effectiveness in addressing the unique challenges of multi-agent control in BEMS and related applications.

Last but not least, it should be mentioned that according to the evaluation, the non-RL model-free multi-agent control applications are limited in numbers. In the current study, such cases may be found in [117,120,123,124], where the algorithms present similarities with an RL approach learning directly from the outcomes of actions in the environment rather than solving predefined equations or optimization problems. Of course, it should be noted that there are cases where an RL-based approach may be considered model-based, since the focus is not on RL but on energy management using predefined optimization models, as in [56] or [115]. Such models include explicit mathematical formulations and optimization criteria that guide decision-making.

5.1.2. Model Predictive Control

The evaluation of algorithmic methodologies in multi-agent control for intelligent IBEMS reveals a nuanced landscape where each approach is selected based on its inherent computational strengths and optimization capabilities. MPC, EAs, and ADMM have emerged as the primary methodologies, each chosen for its specific algorithmic advantages in managing the complexities of multi-agent systems (Figure 8 (Left, Right)). Additionally, Lyapunov optimization and hybrid approaches provide distinct advantages that complement these primary methodologies. MPC is particularly favored for its capability to handle constrained optimization problems over a finite time horizon, making it well suited to real-time control in dynamic environments. The algorithmic foundation of MPC involves solving a sequence of optimization problems where the current control actions are derived by predicting future system states. This predictive control framework is highly effective in scenarios requiring anticipatory adjustments, such as the work in [80,81], where E-MPC is utilized to optimize energy cost and demand response. The flexibility of MPC is further enhanced by its integration with various optimization techniques, such as quadratic programming (QP) in the work of [86], and its application in decentralized settings as seen in [85]. The adaptive nature of MPC, as explored in [79], highlights its ability to adjust dynamically based on real-time data, optimizing both energy efficiency and occupant comfort. A common thread across these studies is the focus on balancing computational efficiency with real-time applicability, where MPC’s ability to iteratively refine predictions and control actions is crucial.

5.1.3. Model-Based Control with Evolutionary-Based Optimization

EAs, including PSO and GAs, are algorithmically robust in exploring and exploiting large and complex search spaces. The stochastic nature of these algorithms allows them to escape local optima, making them particularly useful in non-convex optimization problems commonly encountered in IBEMS. PSO, as employed in [92,94], leverages the collective behavior of particles to converge towards an optimal solution, with each particle adjusting its position based on its own experience and that of neighboring particles. Such an approach is computationally efficient and can handle multi-objective optimization, which is crucial in balancing energy use and occupant comfort. Similarly, the GAs, as applied in [93,95], utilizes principles of natural selection to iteratively evolve a population of solutions, providing a powerful mechanism for optimizing complex systems with multiple conflicting objectives. These studies highlight the adaptability of EA methods in multi-agent systems, particularly in environments where the search space is vast and the optimization landscape is rugged.

5.1.4. Model-Based Control with ADMM-Based Optimization

ADMM is algorithmically distinct in its ability to decompose complex optimization problems into smaller, more manageable sub-problems. This decomposition is particularly advantageous in distributed systems, where computational tasks can be parallelized across multiple agents. The core of ADMM’s algorithmic appeal lies in its iterative approach, where each agent solves a local sub-problem and updates a global variable to ensure consistency across the entire system. This method is effectively demonstrated in [97], where ADMM is used to coordinate multiple HVAC units by breaking down the global optimization problem into simpler sub-problems. The proximal Jacobian variant of ADMM, as utilized by Hou et al. [98,110], further enhanced the algorithm’s ability to converge quickly, even in non-convex settings.

5.1.5. Model-Based Control with Lyapunov-Based Optimization

Such an optimization strategy is less frequently applied; however, it offers a mathematically rigorous approach to ensuring system stability while optimizing performance over time. The algorithm’s strength lies in its ability to handle systems with time-varying constraints and uncertainties, making it particularly suitable for applications where maintaining stability is as critical as optimizing performance. The Lyapunov-based approach, as employed in [104,105,106], uses a drift-plus-penalty framework to balance system performance with stability, ensuring that the system remains stable while optimizing energy costs. This method does not require future statistical information, making it robust against uncertainty and highly adaptable to real-time applications.

5.1.6. Hybrid Strategies

Such schemes combine the strengths of multiple algorithms, creating a versatile framework that can address the limitations of individual methodologies. According to the evaluation, such approaches often integrate machine learning techniques, such as RL, evolutionary algorithms, and ADMM, with traditional control algorithms like MPC, ANNs, and FLC. In particular, ANNs and FLC are commonly utilized in hybrid schemes since they represent a straightforward approach to model the behavior of specific frameworks based on historical data [7,130,131]. In general, the hybridization process, as seen in [107,109], leverages the adaptive learning capabilities of RL and the predictive power of ANNs to enhance the decision-making process in multi-agent systems. Hybrid multi-agent control schemes are particularly efficient at dynamically adjusting control strategies based on real-time data, offering a high degree of flexibility and adaptability in managing complex, non-linear systems. The studies in [111,112] further demonstrate the effectiveness of combining RL with other algorithms to improve both energy efficiency and occupant comfort in dynamically changing environments.

In conclusion, the selection of MPC, evolutionary algorithms, and ADMM in multi-agent control for IBEMS is driven by their distinct algorithmic capabilities. MPC’s predictive control and adaptability, EA’s robust search mechanisms, and ADMM’s ability to decompose and parallelize optimization tasks provide a comprehensive toolkit for managing the complexities of smart building environments. Lyapunov optimization and hybrid approaches further complement these methodologies by ensuring stability and enhancing adaptability, making them essential components in the ongoing development of intelligent energy management systems. The commonalities across these methodologies include their focus on optimization under uncertainty, scalability in distributed systems, and adaptability to real-time data, all of which are critical for the successful implementation of multi-agent control in IBEMS.

5.2. Evaluation per Agent Type

In the review of multi-agent control applications, the prevalence of cooperative agents reflects the advantages of shared information and collaboration in optimizing complex environments (Figure 9). RL naturally aligns with cooperative strategies, as agents sharing observations can collectively learn to maximize overall system rewards [55,56,61,62,65,66,67,68,69,71,73,78]. Such an approach is particularly effective in scenarios where agents’ actions are interdependent, making cooperation essential for achieving optimal outcomes. Numerous RL applications, though, exhibit both decentralized and cooperative (DE/CO) characteristics, where agents retain local decision-making autonomy (decentralization) while still engaging in limited cooperation by sharing critical information [57,58,59,60,74,75,76]. Such a combined approach allows agents to benefit from both localized control and the collective knowledge of the group, leading to more robust and flexible solutions in dynamic environments. On the other hand, purely decentralized RL applications focus on scenarios where agents operate independently, relying solely on their local observations without sharing information [63,64,72]. These applications are particularly suited to environments where agents must function autonomously, such as in large-scale systems where communication overhead is a concern or in scenarios where agents operate in completely separate domains. Decentralized RL allows for scalable and flexible solutions, enabling each agent to adapt to its specific environment without the need for coordination with others. This approach is beneficial in situations where global cooperation is either unnecessary or impractical, emphasizing the autonomy and adaptability of individual agents. The distinction between these approaches in RL highlights the adaptability of the algorithm to different multi-agent types, where the degree of cooperation and decentralization can be tuned to match the specific requirements and constraints of the application. This versatility makes RL a powerful tool in multi-agent systems, capable of addressing a wide range of control challenges through varying levels of agent interaction and autonomy.

Similarly, evolutionary-based approaches, which involve population-based search processes, inherently leverage cooperation to explore and exploit the solution space more effectively, leading to the dominance of cooperative strategies [92,93,94,95,96]. More specifically, evolutionary-based approaches, such as GAs [93,95] and PSO [92,94,96], rely on a population of solutions that evolve over time through processes like selection, crossover, and mutation. Cooperation among agents is integral to these methods, as it allows for the sharing of information and traits, which enhances the exploration and exploitation of the solution space. For example, in GAs, the crossover operation enables agents to combine their best features, while in PSO, particles share information about their best positions, guiding the swarm towards optimal solutions. This inherent cooperation helps prevent the algorithm from getting stuck in local optima and ensures a more effective search for global optima, making cooperative strategies dominant in these approaches.

MPC, on the other hand, often employs decentralized strategies, adjusting its design to operate with agents that make decisions based on local information. Such a tendency suits applications where agents function independently or in loosely coupled environments, allowing for flexibility while still optimizing individual objectives [80,81,82,84,85,86,87,88,89,90]. The combination of decentralized and cooperative (DE/CO) strategies in some applications suggests a combined approach, where agents maintain autonomy but still engage in limited collaboration to enhance overall system performance [82].

In ADMM approaches, the algorithm inherently requires cooperation among agents to solve decomposed sub-problems iteratively, leading to a strong preference for cooperative strategies [97,98,99,100,103,110,116]. However, due to its distributed nature, some ADMM applications may also exhibit decentralized characteristics, where agents work on different parts of the problem while coordinating with others to ensure convergence [101,102].

Lyapunov optimization, with its focus on stability and performance under constraints, tends to favor non-cooperative strategies, where agents prioritize their individual goals to maintain system stability. This approach is consistent with the algorithm’s need to enforce strict performance criteria, often requiring agents to operate independently rather than cooperatively [104,105,106]. These tendencies illustrate how the core architecture and operational principles of each algorithm influence the choice of multi-agent optimization strategies in various control scenarios.

5.3. Evaluation per BEMS Type

According to the evaluation, HVAC equipment is the primary concern of multi-agent control for IBEMS, since they comprise a major energy consumer element in buildings, often accounting for a significant portion of energy costs (Figure 10). Additionally, the complexity and variability in heating and cooling needs make HVAC systems ideal for the advanced optimization capabilities offered by multi-agent control systems. RES—such as PV, Solar Heating, or wind turbines—commonly concern multi-agent control, since such kinds of equipment are commonly integrated in the energy mix of a building. The growing emphasis on sustainability and energy efficiency drives the adoption of such systems, which may be dynamically managed by multi-agent control frameworks to optimize energy generation and consumption. Additionally, the variability of RES aligns well with multi-agent systems’ ability to adapt to changing conditions and optimize the use of available resources. Likewise, ESS are frequently used in the energy mix for IBEMS, since they are able to foster the balance and the variability of RES by storing excess energy for use during periods of low generation. Such integration enhances energy reliability and efficiency, allowing buildings to maximize the benefits of RES and maintain a stable energy supply. EVs posses a lower occurrence in the IBEMS mix; however, it it noticeable that such energy systems are often integrated with RES and ESS to optimize energy usage and support sustainable transportation solutions. The combination of EVs with RES and ESS enables the use of renewable energy for charging, reducing the carbon footprint of EV operations. LS are an inseparable part of buildings and thus numerous applications concern a the adaptability of such systems for enhancing energy efficiency and creating more comfortable and productive environments. What is noticeable in the evaluation is that the applications concerning LS are common in evolutionary algorithm implementation along with other energy systems such as HVAC, RES, and ESS [92,93,94,95]. Last but not least, DHW systems are frequently coupled with RES, utilizing renewable energy to meet water heating demands efficiently. Such trends demonstrate the ability of multi-agent control systems to manage complex energy interactions and promote sustainability across various building functions. The following Figure 10 illustrates the occurrence of the different energy systems in the IBEMS frameworks as denoted in the literature.

A notable trend emerging from the evaluation pertains to the combinations of different kinds of equipment within IBEMS frameworks. The most frequent combination involves HVAC/RES/ESS, observed in seven instances across the reviewed literature [62,68,75,88,113,122,128]. Similarly, the integration of RES/ESS appears five times [55,71,74,77,119], while more intricate interactions, such as those involving RES/ESS/other equipment, are noted in six cases [58,59,72,73,124,126]. It is evident that as the complexity of integration increases, the occurrence diminishes. For example, combinations involving four or more energy systems, such as HVAC/LS/RES/ESS, are documented three times [93,94,95], while HVAC/RES/ESS/EVs appears just twice in the evaluated studies [96,102]. Such a trend suggests a gap in the research on more complex, multi-system IBEMS configurations. Addressing this gap may provide valuable insights into optimizing energy efficiency and system coordination in buildings where multiple subsystems—such as HVAC, RES, ESS, LS, DHW, and EVs are required to operate in an integrated and cohesive manner. Research in this area may be benefited and explore challenges like interoperability, system dynamics, and control strategies for more holistic energy management solutions.

5.4. Evaluation per Building Type

According to the evaluation, multi-agent control applications for IBEMS are more prevalent in commercial buildings, such as offices [67,70,79,84,90,98,109,110,116] and university research centers [83,120,121,129], due to the complex energy demands and potential for energy savings in such environments (Figure 11 (Left) and Figure 11 (Right)). Since commercial buildings often hold diverse and dynamic occupancy patterns, they require more sophisticated management strategies to optimize energy efficiency. Multi-agent control frameworks may effectively manage and coordinate various subsystems and their combination in the energy mix, adapting to changing conditions and user needs. Furthermore, the larger scale and higher energy consumption of commercial buildings offer significant opportunities for research into advanced control strategies, making them a prime focus for studies on improving energy efficiency and sustainability.

It should be noted, though, that the case of multi-agent control in residential settings mostly involves multi-zone, multi-storey residential ecosystems [80,81,85,86,87,88,96] as communities or green districts. Such particularity implies the integration of various and heterogeneous kinds of energy system equipment and thus preserves significant complexity as well in comparison to a conventional single-storey residential apartment. Last but not least, there are numerous cases in which multi-agent control application involved and concerned both commercial and residential structures [60,71,74,75,82,84,88,102,124]. Such a tendency highlights the versatility of multi-agent control systems in managing diverse building types by addressing the distinct energy needs and occupancy patterns of mixed-use developments, for enhancing overall energy efficiency.

The following Figure 11 illustrates the occurrence of residential and commercial buildings found in the literature (2014–2024).

5.5. Evaluation per Application Type

According to the evaluation, multi-agent control applications for IBEMS are limited in real-life implementations primarily due to the complexity and cost of deployment [58,70,73,90,91,107,109,110,113,117,118,119,120,121]. Such a tendency is reasonable considering the investment and effort requirements for infrastructure, sensors, communication networks, and control systems. Moreover, the unpredictable nature of building environments, such as varying user behaviors and external factors, poses challenges for the reliability and scalability of such control frameworks. Consequently, the vast majority of applications often rely on simulations to test and refine multi-agent strategies, as they offer a controlled environment to explore theoretical concepts without the logistical and financial constraints of physical implementation. A lot of them, though, integrate real-world historical data in an effort to enhance the accuracy and relevance of their models, allowing researchers to validate their control strategies under realistic conditions [63,104,106]. This approach helps ensure that simulations account for actual energy consumption patterns and environmental variables, improving the potential for successful real-world application. The following Figure 12, portrays the comparison between simulative and real-life applications type occurrence and percentage (%), for the last decade (2014–2024).

What is evident, though, is that other/independent algorithmic approaches illustrate a higher prevalence towards real-world applications (Table 6 and Table 7). Such a tendency may be justified, since such multi-agent approaches may be tailored to specific building conditions and constraints, providing more flexibility than standard methods like RL or MPC. Such bespoke solutions often address unique challenges and limitations that common algorithms might not handle effectively, increasing their suitability for real-life practical deployment [117,118,119,120,121].

5.6. Evaluation per Simulation Tool

In the framework of IBEMS, advanced simulation tools play a crucial role in developing and testing multi-agent control strategies. Such tools allow researchers to model complex building environments, simulate various control scenarios, and optimize energy management systems using a combination of traditional modeling techniques and advanced machine learning approaches. Such a combination of simulation tools and machine learning paves the way for intelligent and sustainable energy management systems, reducing energy consumption and enhancing building performance.

More specifically, platforms like OpenAI Gym have been widely utilized in order to enhance the development and testing of multi-agent control strategies for IBEMS. Since OpenAI Gym supports the implementation of RL algorithms, it has helped researchers to create simulated environments that mimic real-world building operations [71,72,74,75,76]. In particular, the utilization of the CityLearn open-source environment, as denoted in [77,132], leveraged OpenAI Gym’s ability to provide a standardized setting for developing and testing RL algorithms, facilitating the exploration of decentralized and cooperative control strategies.

EnergyPlus is another building energy software that is commonly utilized in order to efficiently model heating, cooling, lighting, ventilation, and other energy flows within a building, allowing researchers to simulate and analyze the energy performance of building systems [66,86,123]. According to the evaluation, EnergyPlus is often used in combination with other tools to simulate building environments where agents may learn and adapt their control strategies. For instance, Xie et al. [76] utilized EnergyPlus along with OpenAI Gym for multi-agent experiments, while others [61,80,81] combined the tool with the Building Controls Virtual Test Bed (BCVTB) for space modeling and energy performance analysis.

Modelica is another open-source, object-oriented modeling language commonly used for simulating complex multi-domain systems, capturing interactions between mechanical, electrical, and thermal components [70]. The tool does not explicitly focus on building energy systems, but posses more general-purpose features than EnergyPlus. In particular, the use of Dymola—a powerful simulation tool that provides a graphical user interface and advanced simulation algorithms for developing and analyzing models written in Modelica—has also been commonly denoted in the literature. More specifically, Blad et al. [69] utilized Modelica/Dymola for simulating data-driven models in building control, while Michailidis et al. [129] and Sangi et al. [83] used the tools to apply novel multi-agent control scenarios where complex system interactions were required to be modeled, such as integrating HVAC, lighting, and control systems.

Similar to Dymola, TRNSYS is often used for studying the dynamic performance of systems and optimizing energy flows in BEMS through control strategies, including those implemented by multi-agent systems [95,100,101]. Such a tool was specifically designed for simulating transient energy systems and building performance, focusing on dynamic interactions over time. It excels in modeling RES, HVACs, and building energy consumption.

Last but not least, the Java Agent Development Framework (JADE) is also commonly denoted in the literature, portraying a software framework that simplifies the development of multi-agent systems. It is widely used in academic research for implementing agent-based models due to its flexibility and ease of integration with other software environments. For instance, Hurtado et al. [57,92] utilized the JADE with MATLAB for enabling smart building systems, while Kofinas et al. [58] used the tool for developing FLC in multi-agent systems.

According to the evaluation, it is evident that OpenAI Gym and CityLearn are particularly favored for reinforcement learning experiments, while EnergyPlus and Modelica/Dymola are preferred for detailed energy simulations and dynamic system modeling. The following Figure 13 portrays the occurrence of each simulation tool as denoted in the literature for multi-agent control simulation experiments.

6. Conclusions

This review critically examines the most significant applications of multi-agent control strategies for IBEMS over the past decade (2014–2024). By drawing fruitful conclusions from the evaluation of various algorithmic methodologies, the evaluation underscores the critical importance of selecting appropriate methodologies to effectively address the unique challenges of smart buildings. The most significant conclusions drawn from the evaluation are as follows:

• Prevalence of Algorithmic Control Methodologies: RL remains the most prominent algorithmic approach for multi-agent control in IBEMS, with methods such as Q-learning, Deep Q-Networks, and actor–critic approaches (e.g., MAAC) being the most frequently used. Such strategies excelled at handling complex environments with large state spaces, and their effectiveness in optimizing energy management and occupant comfort makes them a top choice for multi-agent scenarios. However, it is observed that, overall, model-based control strategies are more commonly used in the literature than model-free ones. In particular, MPC and other model-based approaches, such as ADDM-based and PSO-based, have been shown to offer strong predictive capabilities and real-time adaptability, making them essential for managing dynamic environments where anticipatory control is required. The dominance of model-based control methodologies, however, highlights an opportunity for modern research to delve deeper into model-free methodologies, which offer more flexible and adaptive solutions that are potentially less reliant on predefined system models. This trend could also be supported by the rise of IoT technologies, enabling more data-driven, real-time control without the need for complex system modeling.
• Rising Adoption of Hybrid Methodologies: Hybrid control schemes, integrating methodologies like RL, MPC, EAs, and ANN/FLC, are becoming increasingly popular due to their ability to leverage the strengths of multiple techniques. Such methods offer enhanced flexibility and adaptability, particularly in dynamically changing environments where multiple control objectives must be optimized simultaneously. According to the evaluation, hybrid methodologies hold strong potential for the future, not only due to their ability to synergize strengths from different algorithmic paradigms but also because they provide a framework for adaptive control in increasingly complex and uncertain environments. To this end, as the complexity of IBEMS frameworks continues to grow—requiring robust coordination and real-time adaptability—hybrid methodologies will play a crucial role in optimizing energy management, improving system efficiency, and enhancing occupant comfort.
• Prevalence of Cooperative Agent Type Strategies: According to the evaluation, cooperative agents represent a particularly advantageous scheme in multi-agent control systems, especially in RL and evolutionary algorithms. Cooperation enables agents to share critical information and align their objectives, which is crucial in interdependent systems like HVAC, RES, and ESS. By coordinating actions, cooperative agents can optimize energy management more effectively than decentralized agents, which operate independently and may struggle in complex environments with high interdependencies. The ability to exchange experiences accelerates learning in RL, leading to better collective outcomes. Similarly, in evolutionary algorithms, cooperation prevents premature convergence by promoting diverse exploration of the solution space. In contrast, purely decentralized approaches may underperform in scenarios requiring high coordination, highlighting the superiority of cooperative methods in managing complex, distributed energy systems.
• Constraints on Multi-Equipment Integration in IBEMS: The evaluation highlights HVAC systems as the predominant focus of multi-agent control in IBEMS due to their high energy consumption. RES, often combined with ESS, also feature prominently, contributing to sustainable building energy management. A key trend identified concerns the relatively low frequency of applications involving more than four types of systems. While multi-system interactions involving HVAC, RES, ESS, and EVs remain limited, such integrations are essential in real-world applications for achieving comprehensive energy efficiency. As IBEMS grow more complex with diverse subsystems, the need for advanced multi-agent control methods capable of managing this complexity, ensuring real-time adaptability, and facilitating effective inter-agent coordination, becomes increasingly critical.
• Limited Real-life Implementations: Despite the potential of multi-agent control systems, real-world applications are still limited due to the high cost, complexity, and logistical challenges of deployment. However, it is noticeable that real-life implementations tend to favor novel approaches–like Hybrid or Other approaches—because they offer greater flexibility and customization to specific building conditions. Unlike conventional approaches, customized approaches may provide a better handling of real-world complexities, uncertainties, and system-specific constraints, making them more practical for deployment in diverse and dynamic environments. Such a fact is also supported by numerous literature works, where comparisons often highlight that novel approaches outperform conventional methods. Another key observation is that real-life implementations predominantly focus on commercial buildings, underscoring the need for more extensive research and deployment at the residential level. Given the unique energy demands and dynamics of residential environments, there is a clear opportunity to expand multi-agent control systems to optimize energy use and sustainability in homes, which remains underexplored in comparison to commercial real-world applications.

In conclusion, the ongoing advancements in multi-agent control for energy management in buildings reflect a strong commitment to developing more efficient, adaptable, and intelligent energy management systems. By leveraging the complementary strengths of model-free and model-based approaches, future research and practical applications will be well equipped to meet the evolving demands of smart buildings, contributing to the creation of more sustainable and responsive urban environments.