Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning

Abstract

In target-oriented multi-agent tasks, agents collaboratively achieve goals defined by specific objects, or targets, in their environment. The key to success is the effective coordination between agents and these targets, especially in dynamic environments where targets may shift. Agents must adeptly adjust to these changes and re-evaluate their target interactions. Inefficient coordination can lead to resource waste, extended task times, and lower overall performance. Addressing this challenge, we introduce the regulatory hierarchical multi-agent coordination (RHMC), a hierarchical reinforcement learning approach. RHMC divides the coordination task into two levels: a high-level policy, assigning targets based on environmental state, and a low-level policy, executing basic actions guided by individual target assignments and observations. Stabilizing RHMC’s high-level policy is crucial for effective learning. This stability is achieved by reward regularization, reducing reliance on the dynamic low-level policy. Such regularization ensures the high-level policy remains focused on broad coordination, not overly dependent on specific agent actions. By minimizing low-level policy dependence, RHMC adapts more seamlessly to environmental changes, boosting learning efficiency. Testing demonstrates RHMC’s superiority over existing methods in global reward and learning efficiency, highlighting its effectiveness in multi-agent coordination.

1. Introduction

Coordination is crucial in real-world scenarios, as it permits individuals to collaborate to achieve shared goals. In certain tasks, such as collaborative drone control [1], coordination among autonomous vehicles [2], and the target coverage problem in directed sensor networks [3], better team performance can be achieved through coordination [4]. These issues can be modeled as cooperation between multi-agents to accomplish their own tasks while maximizing the global reward. Reinforcement learning is an effective approach for sequential decision-making, which improves its policy through continuous trial-and-error exploration and pursuit of reward maximization [5,6,7]. Additionally, reinforcement learning has shown tremendous improvements in various domains, including gaming [8], robot control [9,10], intelligent transportation [11,12], and industrial applications [13]. Despite the successful extension of single-agent algorithms such as DQN [10], DDPG [6], and PPO [7] to multi-agent systems, multi-agent coordination continues to face challenges [14,15].

Target-oriented multi-agent environments refer to task environments where agents collaborate in the presence of specific entities. A target can be identified as an entity in the environment, such as an object or a location that must be involved in completing a task. The agent needs to establish a correspondence with the target entity and use their personal skills to accomplish the task as efficiently as possible. Examples of target-oriented scenarios include warehouse robots managing cargo [14], drones tracking objects [15], and resource scheduling systems where server resources need to be allocated efficiently [16]. In the latter case, coordination involves managing and optimizing server resources to handle various tasks or requests in a balanced and efficient manner [17,18]. Coordinating the relationships between agents and targets is crucial for improving the overall efficiency of collaborative tasks [19,20].

For one-to-one agent-target tasks, the IQL method treats the remaining agents directly as part of the environment, leading to instability as the policies of other agents change over time [17,21]. Since there is no guarantee of convergence, it is easy for the agent to become stuck in an endless exploration, limiting IQL to simple tasks such as the Pong game [22]. However, in cooperative tasks with multiple targets, such as cooperative navigation, where agents must reach multiple target locations, it is critical to coordinate the relationship between agents and targets to avoid internal conflicts [23,24]. MADDPG learns multiple policies for each agent and optimizes the overall effect of all policies to improve the stability and robustness of the algorithm [18]. COMA utilizes a global critic network to assess all current actions and states, enhancing the efficiency of information exchange and collaboration between agents during training [19]. QMIX employs a hybrid network to combine the single agent’s local value function with global state information, facilitating training and learning procedures, and enhancing the algorithm’s performance [20]. In multi-agent coordination, maximizing local rewards does not ensure global optimality. In contrast, prioritizing the global reward can cause entanglement between local and global rewards, leading to suboptimal solutions [21]. Balancing the relationship between the local and the whole is very important for multi-agent systems to accomplish the task better [22,24].

The regulatory hierarchical multi-agent coordination (RHMC) framework is proposed to tackle this problem. Our contribution is threefold: (1) A hierarchical reinforcement learning algorithm is proposed for the multi-agent coordination problem. This framework comprises two layers; the high-level policy governs target allocation, while the low-level policy guides agent actions. (2) RHMC implements a reward regularization mechanism to reduce the high-level policy’s dependence on modifications in low-level behavior, providing stability to the policy training process. (3) The RHMC method is validated on target-oriented multi-agent cooperative tasks, and the experimental results show that our method outperforms the state-of-the-art algorithms.

2. Related Work

The target-oriented multi-agent coordination framework includes two categories of multi-agent reinforcement learning: local reward-based coordination and global reward-based coordination. Local reward-based approaches aim to optimize overall performance by maximizing the individual agent’s local rewards. Early approaches, such as the IQL method [17], convert multi-agent-related tasks into single-agent tasks, with each agent learning its own value function separately. This method is not effective because all agents affect the overall environment, and the agents are also affected by other agents during the learning process, leading to unstable agent learning. The Centralized Training with Decentralized Execution (CTDE) framework is employed in multi-agent learning work such as intelligent transportation [21], smart home [22], recommendation algorithms [23,24,25], common-pool resource appropriation [26,27,28], and sequential social dilemmas [29,30]. This approach addresses the issue of non-stationarity in the environment that can arise when agents learn independently without taking into account the actions of other agents.

The MADDPG method [18] can be regarded as a deterministic policy gradient method that uses deep reinforcement learning to solve the continuous action space problem. Although MADDPG enhances stability and robustness, it still faces challenges in diverse environments. The M3DDPG method [31] enhances the generalization capability of the learned policies and accommodates the heterogeneity of reward functions across agents, but it does not explicitly address the coordination requirements in multi-agent systems. The MAPPO method [32] is a multi-agent reinforcement learning method based on the policy gradient algorithm of confidence region, which limits the size of the parameter update step to stabilize the training process. The NC-HDQN method [33] analyzes the correlation between agents and their connected neighbors to measure the observed value and reward and uses a hysteretic DQN-weighted information optimization strategy. The LOLA method [34] enables agents to better understand the policies and motivations of other agents through opponent modeling, thus coordinating their actions more effectively.

Global reward-based approaches tend to have agents with one goal and a shared global reward. These approaches aim at maximizing the global reward while optimizing the local reward. The COMA method [19] uses a centralized critic network to collect information from all agents during training and employs counterfactual baselines to address the credit assignment problem. Value decomposition methods represent the joint action value function as a combination of the value functions of each agent, guiding agents to learn a better policy. The VDN method [35] uses a deep neural network to represent the value function of each agent and the whole joint action value function as a linear combination of these value functions. However, VDN overlooks the role of global state information in the learning process and the intricate nonlinear relationships between agents. To address these issues, the QMIX method [20] uses an additional network to fuse the values of each agent, including global information as auxiliary information when expressing the joint action value function. QMIX obligates the joint action value function to be monotonic with the value function of each agent to comply with the maximum consistency principle between corresponding value functions. The PMI-Q method [36] improves the generalization ability of the policy by preventing overfitting through a combination of pessimistic averaging and average invariance. Drawing inspiration from the collective behavior of ant colonies, the SIRL method [37] introduces the concept of pheromones into multi-agent systems to reduce local conflicts.

Hierarchical reinforcement learning (HRL) benefits structured exploration by learning sub-task policies over primitive actions. Exploration efficiency is essential in reinforcement learning. Although the majority of HRL methods use single-agent exploration methods at a low level, such as greedy [38] and intrinsic reward [39], some methods focus on high-level policy, such as diversity concepts [40] and coordinator cooperation, with well-performed executors [41]. The former requires a diversified set of low-level skills, while the latter requires a well-defined pre-training setting [42]. Despite these advancements, existing HRL methods often fail to address the complex coordination required in multi-agent settings [43,44,45,46,47,48]. Our work introduces a distinctive hierarchical structure that meticulously addresses both local and global coordination needs in multi-agent systems, effectively filling a significant gap in current methodologies and considerably enhancing task performance [49]. To underscore the rationale for our choice of comparative analysis, the following table, designated as

Table 1. Comparative analysis of common multi-agent reinforcement learning methods.

Method Name	Key Features	Addressed Issues	Application Scenarios	RHMC Advantages
MADDPG	Deep RL method with deterministic gradients; focuses on stability via centralized training and decentralized execution.	Enhances stability in dynamic, continuous action spaces.	Suitable for robotic collaboration and autonomous fleets.	Boosts robustness in continuous action environments.
CTDE	Combines centralized training with decentralized execution to tackle environmental non-stationarity.	Improves stability by addressing learning issues from independent agent actions.	Ideal for cooperative multi-agent systems like intelligent traffic.	Resolves non-stationarity in multi-agent learning effectively.
QMIX	Integrates agent value functions using a mixing network for consistent policy updates; includes global information.	Ensures consistent policies and cooperation, managing update conflicts.	Used in complex scenarios requiring consistent policies, such as multiplayer games.	Provides a novel method for enhancing policy consistency and cooperation.
RHMC	Introduces a clear hierarchical structure, explicitly handling both local and global coordination needs. Focuses on structured exploration and policy optimization.	Enhances coordination between local and global tasks in multi-agent systems, optimizing overall task performance.	Applicable to multi-agent environments requiring complex coordination and efficient execution.

, presents a detailed comparison of selected multi-agent reinforcement learning methods. These methods, including MADDPG, CTDE, and QMIX, are not only ubiquitous in the realm of multi-agent research but also represent pivotal advancements in addressing fundamental challenges such as coordination complexity, environmental non-stationarity, and policy scalability. Their widespread use and seminal contributions to the field substantiate their selection for our comparative analysis. This strategic choice allows us to vividly highlight how our RHMC framework leverages and transcends the foundational strengths of these established methods, thereby providing a profound insight into the significant enhancements and unique positioning of our approach. For a detailed comparison, see Table 1.

3. Preliminaries

3.1. Reinforcement Learning

A reinforcement learning problem can be modeled by a Markov decision process [48] (MDP), defined by the tuple $〈S,A,P,r,\gamma 〉$, in which

• $S\subseteq R$ is a n-dimensional state space;
• $A\subseteq R$ is a m-dimensional action space;
• $P:S\times A\times S\to R_{+}$ is a transition probability function;
• $r:S\to R$ is reward function;
• $\gamma \in \left(0,1\right]$ is the discount f.

In Markov decision process, the agent performs an action $a_t\in A$ according to the current state $s_t\in S$ and its policy $\pi \left(a_t\left|s_t\right.\right)$ at each timestep. The objective of the agent is to learn an optimal policy ${\pi }_{{\theta }^{*}}:=argma{x}_{\pi \theta }\left[{\displaystyle {\sum }_{i=0}^T{\gamma }^i{r}_{t+i}\left|s_t=s\right.}\right]$.

Hierarchical reinforcement learning uses a multi-layer policy to interact with the environment. For the target-oriented multi-agent coordination problem, the joint policy ${\pi }_j$ oint consists of a high-level policy ${\pi }_h$ and a low-level policy ${\pi }_l$. The high-level policy distributes targets $g_t$ to coordinate the individual low-level agents. Then the low-level policy agents output the primitive action according to the received target assignment $g_t$ and current state $s_t$. Differently, the high-level policy is used to coordinate agents to maximize global rewards, while the low-level policy is used to maximize individual local rewards under the high-level policy target assignment.

3.2. Actor–Critic Method

The actor–critic method is a reinforcement learning method combining a policy gradient and a value function. There are two main methods of reinforcement learning: the value-based method and the policy-based method [49]. The policy gradient method can easily select appropriate actions in continuous action space, while value-based Q-learning can only solve problems in discrete action space. A combination of the policy-gradient and the value-based method will form the actor–critic algorithm. The predecessor of the actor is the policy gradient, and the predecessor of the critic is the value-based method. The actor selects behavior based on probability distribution, the critic evaluates a score based on the behavior generated by the actor, and the actor modifies the probability of selection behavior according to the score of the critic [44]. Directly optimizing a policy, policy gradient (PG) algorithms update policy parameters θ in the direction of the gradient of the expected return ${\nabla }_{\theta }E\left[G_{\pi \theta }\right]$, which is conveniently given by the policy gradient theorem as follows:

$${\nabla }_{\theta }E\left[G_{\pi \theta }\right]=E_{h,a}\left[{\nabla }_{\theta }\log {\pi }_{\theta }\left(a|h\right)Q^{\pi \theta }\left(h,a\right)\right]$$

(1)

Actors use policy functions that generate actions and interact with the environment. Critic, on the other hand, uses a value function and is responsible for evaluating the performance of the actor and guiding the action of the actor in the next stage [45].

3.3. Multi-Agent Deep Deterministic Policy Gradient

Multi-agent deep deterministic policy gradient (MADDPG) is an adaptation of the deep deterministic policy gradient algorithm to multi-agent settings. It is an actor–critic, off-policy method that uses the paradigm of CTDE to learn deterministic policies. In MADDPG, a separate actor and critic is learned for each agent, such that arbitrary reward functions can be learned. Each agent, i, has a deterministic policy, ${\pi }_i\left(a_i\left|\right|o_i\right)$, parameterized by ${\theta }_i$ (abbreviated as ${\pi }_i$), and let $\pi ={\left\{{\pi }_i\left(a_i\left|\right|o_i\right)\right\}}_{i=1}^n$ be the set of all agent policies. A centralized and monolithic critic that estimates the joint action–value function $Q_i^{\pi }\left(s,a_1\dots ,a_n\right)$ is learned for each agent separately. The critic is said to be centralized as it utilizes information only available to it during the centralized training phase, the global state and the actions of all agents, a₁, …, a_n, to estimate the joint action–value function $Q_i^{\mu }$, which is parameterized by ${\varphi }_i$. This joint action–value function is trained by minimizing the following loss:

$$L\left({\varphi }_i\right)=\mathbb{E}\left[{\left(y_i-Q_i^{\pi }\left(s,a_1,\dots ,a_n\right)\right)}^2\right]$$

(2)

where,

$$y_i=r_i+\gamma Q_i^{\pi }\left(s^{\prime },a_1^{\prime },\dots ,a_n^{\prime }\right)|{|}_{{\stackrel{\prime }{a}}_i={\pi }_i\left(o_i\right)}$$

(3)

Here $r_i$ is the reward received by each agent i, a₁, …, a_n is the set of target policies with delayed parameters ${\theta }_{\overline{i}}$, and ${\varphi }_{\overline{i}}$ are the parameters of the target critic. The replay buffer $\mathcal{D}$ contains the transition tuples $\left(s,s\prime ,a_1\dots ,a_n,r_1\dots ,r_n\right)$.

The following policy gradient can be calculated individually to update the policy of each agent, $i$:

$${\nabla }_{{\theta }_i}J\left({\pi }_i\right)={\mathbb{E}}_{\mathcal{D}}\left[{\nabla }_{{\theta }_i}{\pi }_i\left(o_i\right){\nabla }_{a_i}{Q}_i^{\pi }\left(s,a_1,\dots ,a_n\right){\parallel }_{a_i={\pi }_i\left(o_i\right)}\right]$$

(4)

where the current agent $i$’s action $a_i$ is sampled from its current policy ${\pi }_i$ when evaluating the joint action–value function $Q_i^{\pi }$, while all other agents’ actions are sampled from the replay buffer $\mathcal{D}$.

Updating the target network parameters of each agent i is shown in the following formula:

$${\theta }_{\prime }^i\leftarrow \tau {\theta }_{\prime }^i+\left(1-\tau \right){\theta }_{\prime }^i$$

(5)

4. Method

The regulatory hierarchical multi-agent coordination (RHMC) method employs a two-level hierarchical approach that aims to maximize global rewards for all agents. At the high-level, the policy assigns targets based on the overall state of the environment. At the low-level, agents select primitive actions based on the assigned targets and their local observations.

The Regulatory hierarchical multi-agent coordination (RHMC) architecture is showed in Figure 1.

4.1. Mechanism Description

The regulatory hierarchical multi-agent coordination (RHMC) architecture is a two-level framework designed to enhance multi-agent coordination through hierarchical policies. At the high level, the policy collects joint state information from all agents and distributes target assignments to the low-level policy. The low-level policy then generates primitive actions based on both the assigned targets and the agents’ observations, directly applying these actions to the environment. This hierarchical approach allows for more efficient and organized coordination among agents. The high-level actors focus on strategic target distribution, while the low-level actors handle tactical execution. Additionally, the high-level critic utilizes a regulation mechanism (RM) to calculate the value, ensuring robust and stable learning across both policy levels.

RHMC has two Markov decision processes (MDPs) for the high-level policy and the low-level policy, respectively:

$$MD{P}_h=〈S_h,G,P_h,r_h,{\gamma }_h〉$$

$$MD{P}_i=〈S_l,A_l,P_l,r_l,{\gamma }_l〉$$

(1)

High-Level Policy (Symbol 1): The high-level policy is responsible for target allocation. It considers the overall state of the environment and assigns targets to agents. This level focuses on global coordination and strategic planning, ensuring that each agent is directed towards a target that optimizes the overall system performance.

(2)

Low-Level Policy (Symbol 2): The low-level policy deals with the execution of actions. Based on the assigned targets and local observations, agents choose primitive actions. This level emphasizes local decision-making and immediate interactions with the environment.

The RHMC algorithm increases sample utilization efficiency via hierarchical policy training. However, the high-level policies in the hierarchical structure can be influenced by the low-level policies. When policies of all levels are trained simultaneously, the low-level policies are continually updated, leading to changes in the high-level transfer function. In such a non-stationary environment, it becomes challenging for high-level policy to learn the optimal policy.

While the low-level policies ${\mu }_{\theta }^l\left(s,g\right)$ interact directly with the environment, the high-level policy interacts indirectly. The non-stationarity emerges in the high-level policy ${\mu }_{\theta }^h\left(s\right)$ transition. The low-level policy makes primitive actions directly on the environment, and the same state transition function remains unchanged. In contrast, the high-level policy acts indirectly on the environment. The same action output by the high-level policy can produce different actions at the low-level during training, leading to dynamic changes in state transitions at the high-level. This uncertainty in transitions $\left(s,g\right)\to s^{\prime }$ leads large changes in the high-level reward $r_h$.

To address these challenges, we propose a reward regulation mechanism and a target-conditioned filter:

(1)

Both Regulation Mechanism: This mechanism aims to stabilize the training process by reducing the high-level policy’s dependence on modifications in low-level behavior. By implementing a regularization term in the reward function, we ensure that the high-level policy remains robust despite the dynamic changes at the low-level. This helps maintain consistent learning and prevents the high-level policy from being adversely affected by the non-stationarity of the low-level policies.

(2)

Target-Conditioned Filter: The target-conditioned filter helps in refining the action selection process by incorporating target-specific information into the decision-making framework. This filter ensures that the actions chosen by the low-level policy are more aligned with the high-level targets, thus improving coordination and efficiency. By conditioning on the target, the filter reduces the variability in low-level actions, leading to more predictable and stable transitions, which in turn enhances the performance of the high-level policy.

These components collectively improve the overall efficiency and robustness of the RHMC framework, allowing for better coordination and performance in multi-agent environments.

4.2. High-Level Target Assignment

The goal of the high-level policy ${\mu }_{\beta }(s)$ is to learn an optimal policy to maximize the global reward. The high-level policy determines the target assignment for each agent. It is implemented through the construction of a deep neural network, which consists of two components: actor and critic. According to the joint state $s=\left(s_1,s_2,\dots ,s_n,\dots ,s_{n+m}\right)$ where m is the number of target and n is the low-level agent, the actor outputs the target assignment $g=\left(g_1,g_2,\dots ,g_n\right)$. An actor receives the joint state as an input and calculates the probability $g_{ij}$ of each assignment through a fully connected layer where $g_{ij}$ is a binary value indicating whether or not to let agent i track target j. Therefore, the actor will output the target assignment of all agents $g=\left(g_1,g_2,\dots ,g_n\right)$, where $g_i=\left(g_{i1},g_{i2},\dots ,g_{im}\right)$ denotes the target assignment of agent i.

The high-level reward function is shown as follows:

$$r_h={\displaystyle \sum _{j=1}^m{\left(\frac{p_j}{k_j}-{\alpha }_1{q}_i\right)|}_{\frac{p_j}{k_j}=0 when k_j=0}}$$

(6)

where m is the number of targets, p_j represents whether the assigned target task i is completed $\left(0:No,1:Yes\right)$, $k_j={\displaystyle {\sum }_{i=1}^n{g}_{ij}}$ represents the number of agents tracked to the assigned target j, $q_j$ represents whether the assigned target i is not completed $\left(0:No,1:Yes\right)$, and ${\alpha }_1$ is a punishment factor that ranges from 0 to 1. p and q motivate the agents to complete more assigned target tasks, the larger the number of k the smaller the reward, avoiding targets conflict in agent coordination.

The critic network calculates loss function and updates actor network parameters. The critic calculates the optimal value of the state through the neural network, and the actor uses the optimal value to update the parameters of the policy function, then selects the action and receives feedback and the next state. The loss function of the critic is calculated as follows:

$$L\left(\varphi \right)=\underset{\left(s,g,r_h,s^{\prime }\right)\in D_h}{E}{\Vert Q_{\varphi }\left(s,g\right)-r_h+{\gamma }_h{Q}_{\varphi t\arg et}\left(s^{\prime },{\mu }_{\beta t\arg et}\left(s^{\prime }\right)\right)\Vert }^2$$

(7)

where the D_h is a subset of the replay buffer $B_h$ at a high level. Since the $Q$ network and policy function are constantly changing in the process of training, which will lead to the instability of the environment, we need to introduce the objective function. The ${\beta }_{t\arg et}$ and ${\varphi }_{t\arg et}$ are the parameter the target policy network and the $Q$ network, respectively, which change slowly.

$$\begin{gathered} {\beta }_{t\arg et}=\rho {\beta }_{t\arg et}+(1-\rho )\beta \\ {\varphi }_{t\arg et}=\rho {\varphi }_{t\arg et}+(1-\rho )\varphi \end{gathered}$$

(8)

where $\rho$ is a number close to one. Therefore, the target network will update very slowly. By using the policy gradient theorem, the policy network update is as follows:

$$\beta =\beta +{\alpha }_2\underset{s\in D_h}{E}\nabla \beta \mu \beta \left(s\right){\nabla }_g{Q}_{\varphi }\left(s,g\right)$$

(9)

The model of the high-level structure is showed in Figure 2.

4.3. Low-Level Primitive Action

Upon receiving the target assignment from the high-level policy, each agent performs the task independently based on the assigned target condition. The aim of the low-level policy is to maximize the local reward under the given target assignment. Therefore, while the high-level policy handles coordination, the low-level policy executes the assigned target task. The input to the low-level policy ${\mu }_{{\theta }^i}\left(o,g\right)$ in agent i consists of two parts: one is its observation of states $o_i=\left(o_{i1},o_{i2},\dots ,o_{in},\dots ,o_{in+m}\right)$, and the other is the assigned target $g_i$ transmitted by the high-level policy. Note that the observation states of agents are different in different environments, so we use the most observation states in the model.

In the two-level architecture, however, irrelevant information can impede efficient learning of the low-level policies. For example, for agent i, the targets not assigned to it are not supposed to be concerned, but there is a large amount of state information of irrelevant targets during the training process that affects its learning efficiency. The target condition filter sets the unassigned target information to 0 to reduce its disturbance. That is, for the assigned target $g_i=\left(g_{i1},g_{i2},\dots ,g_{im}\right)$ and $o_i$, if $g_{ij}$ is 0, then the corresponding $o_i\left(n+j\right)$ information will be changed to 0 before input. The filter layer can be represented as the following function:

$$filter\left(o_{ij}\right)=\left\{\begin{array}{l}{o}_{ij},j\in \left[1,n\right]\\ 0,g_i\left(j-n\right)\\ o_{ij},other\end{array}\right.=0$$

(10)

For each low-level agent, it adopts an actor–critic architecture, where the actor network µ_θ outputs primitive actions, and the critic network Q_φ will collect all the state x in the environment including target and agent to calculates the value and updates the network parameters to optimize the actor network. During each episode, the agent interacts with the environment to produce trajectory data. Therefore, in agent i, the loss function of ${\phi }^i$ is calculated as follows:

$$L\left({\phi }^i\right)=\underset{\left(x,a,r_l,x^{\prime }\right)\in D_l}{\left(E\right)}{‖{Q_{{\phi }^i}\left(x,a\right)-r_l^i-{\gamma }_l{Q}_{{\phi }_{t\arg et}^i}\left(x^{\prime },a^{\prime }\right)|}_{a_i^{\prime }={\mu }_{{\theta }_{t\arg et}^i}\left(o_i^{\prime }\right)}‖}^2$$

(11)

where $D_l$ is a subset of replay buffer B_l at a low level, and the ${\mu }_{{\theta }_{t\arg et}^i}$ and $Q_{{\phi }_{t\arg et}^i}$ target are the same as the target network at a high level. Noted that the a = ($a_1$, $a_2$, …, $a_n$) is the actions performed by all agents and $r_l=\left(r_l^1,r_l^2,\dots ,r_l^n\right)$ is the rewards for all agents.

$$\begin{gathered} {\theta }_{t\arg et}^i=\rho {\theta }_{t\arg et}^i+\left(1-\rho \right){\theta }^i \\ {\phi }_{t\arg et}^i=\rho {\phi }_{t\arg et}^i+\left(1-\rho \right){\phi }^i \end{gathered}$$

(12)

Also, the loss function of the parameter ${\theta }^i$ of the policy network in agent i is:

$$L\left({\theta }^i\right)=\underset{x\in D_l}{E}{Q}_{{\phi }^i}{\left(x,a\right)|}_{a_i={\mu }_{{\theta }^i}\left(o_i\right)}$$

(13)

Below, we give the model diagram of the low-level network.

4.4. Regulation Mechanism

High-level exploration has a considerable impact on the overall exploration process. However, the non-stationarity of high-level transitions affects the overall learning efficiency. The high-level policy has the responsibility of outputting target assignments and receiving reward feedback, where the reward measures the value of high-level actions. The high-level policy expects the maximum reward that the low-level policy can receive under the given target assignment, representing the upper limit of the assignment and its true value. Nevertheless, during the training process, the low-level policy may not initially perform well under the given target assignment due to its ongoing learning exploration process and its model that has not yet been trained. This discrepancy can result in a mismatch between the reward feedback received by the high-level policy and the true value of its actions. Therefore, a good assignment of high-level action may yield poor rewards, which is not conducive to the high-level policy’s learning.

The regulation mechanism lessens the impact of the low-level policy through reward regulation. The high-level problem is modeled as follows: under the specified target assignment $g$, there are n agents, and each agent can take action $a$. For achieving their respective tasks as per the target assignment, every agent is rewarded. The main challenge is to coordinate the actions of all the agents to maximize the total reward. The regulatory mechanism compares the dynamic programming’s evaluation reward with the low-level execution reward, and the higher reward serves as feedback for the high-level action.

In the initialization setting, n is the number of agents, d is the number of agent actions, and $r_h$ is the reward feedback under the action of the low-level policy. The action function, act(.), can calculate the states of assigned target after the action taken by the taking agent. The reward function, r(.), can calculate the reward value corresponding to the states of assigned target using the function 6. The DP matrix $dp\left[n\right]\left[d\right]$ has n rows and d columns. $dp\left[i\right]\left[j\right]$ is a binary sequence, indicating the maximum number of targets that can be accomplished by the first i agents taking action, and the selected action of agent i is j. In the binary sequence of $dp\left[i\right]\left[j\right]$, there is a 1 in position t which indicates that the target t is tracked. The $cnt(.)$ function calculates the number of 1 in the binary sequence, which coincides with the number of assigned targets tracked.

We give the proof of the optimality of the reward regulation algorithm below.

Theorem 1. 

Both in the case of discrete and continuous action space, Algorithm 1 can calculate the optimal high-level reward value.

Algorithm 1 Regulation Mechanism

Input: the state information s and target assignation g Output: estimated high-level reward rh 1: Initialize the number of the action d, reward rh

2: Initialize the dynamic programming array dp Given the action function act(.), counting function cnt(.) and reward function r(.)

3: for i := 1 → n do

4:      for j := 1 → d do

5:         Compute the tracking status mask in act(i,j)

6:        for k := 1 → d do

7:             if i = 0 then

8:               $dp\left[i\right]\left[j\right]\Leftarrow mask$

9:             else

10:                 $\mathbf{if} cnt\left(dp\left[i\right]\left[j\right]\right)<cnt\left(dp\left[i-1\right]\left[k\right]mask\right)\mathbf{then}$

11:                    $dp\left[i\right]\left[j\right]\Leftarrow \left[i-1\right]\left[k\right]\left|mask\right.$

12:                 end if

13:             end if

14:        end for

15:      end for

16:  end for

17:  for i := 1 → n do

18:      $r_n\Leftarrow max\left(r_h,r\left(dp\left[n\right]\left[i\right]\right)\right)$

19:  end for

20:  return rh

Proof of Theorem 1. 

We first give the proof in discrete action space. In the high-level reward function (6), when the target assignment policy $g$ is given, the k value is determined because of it definition $k_j={\displaystyle {\sum }_{i=1}^n{g}_{ir}}$. Therefore, we can deduce that as long as we allow the agent to accomplish as many tasks as possible, the more high-level rewards will be received. We use the $\Delta cn{t}_k\left(act\left(i,j\right)\right)$ to indicate the amount of change that will occur if the i agent adopts action j, assuming that the first i − 1 agents have reached the maximum target number and the i − 1-th agent choose action k. Therefore, we can describe the above problem as the following: In this model, we transform the problem of regulating the reward into the problem of finding the maximum path in the abovementioned DAG. The path length before two nodes is defined as the amount of change in the completed targets resulting from the action taken to reach the end of the current path. For example, the length of the path between two nodes of agent (i,d) and agent $\left(i+1,j\right)$ is $\Delta cn{t}_d\left(act\left(i+1,j\right)\right)$, where $\left(i,d\right)$ means agent i takes action d. Each point in the model represents the maximum number of acquisition targets that can be achieved if the current layer performs this action. For example, the node $\left(i,d\right)$ means $Max\left(cnt\left(dp\left[i\right]\left[d\right]\right)\right)$. Note that once the action of the first i − 1 agent is determined, the amount of change of the i-th agent is also determined. Therefore, each path in the diagram has been determined. In this model, the following optimized substructures exist

$$Max\left(cnt\left(dp\left[i\right]\left[j\right]\right)\right)=Max\left(Max\left(cnt\left(dp\left[i-1\right]\left[k\right]\right)\right)+\Delta cn{t}_k\left(act\left(i,j\right)\right)\right)$$

(14)

where $k$ = 1, 2, …, d. Note that in different $k$, the $\Delta cn{t}_k\left(act\left(i+1,j\right)\right)$ is different because of its definition. Therefore, we find the optimal substructure of this problem, which proves that the optimal solution can be obtained by using dynamic programming algorithm.

As for the continuous action space, we can find that every For every action, we ultimately examine the condition of its target. In other words, for a continuous action space $\mathsf{\Omega }$, the function act(.) can be mapped.

$$act:\mathsf{\Omega }\times S\to B$$

(15)

where B is the binary sequence, whose length is the number of targets. Therefore, the continuous action space $\mathsf{\Omega }$ can be divided into intervals that map to different binary sequences B. For example, for cases where the number of targets is 3, all actions in a subinterval ${\mathsf{\Omega }}_1$ of the continuous action space achieve a target capture of 010, while all actions in another subinterval ${\mathsf{\Omega }}_2$ of the continuous action space achieve a target capture of 110. Consequently, the continuous action space can be divided into at most 2^m subspaces and can be treated as a case of discrete action space.

We start by considering the discrete action space. In the high-level reward function (6), when the target assignment policy $\pi$ is given, the value of $k$ is determined by its definition. Therefore, we can deduce that the more tasks an agent completes, the higher the reward received by the high-level policy. Let ${\Delta }_{ij}$ denote the change in the number of completed tasks when agent iii adopts action $j$, assuming the first $i-1$ agents have already maximized their task completions and agent $i-1$ chose action $k$.

This problem can be modeled as finding the maximum path in a directed acyclic graph (DAG), where each node represents the maximum number of completed tasks if the current agent performs a specific action. The path length between two nodes indicates the change in the number of completed tasks resulting from the action taken. For example, the path length between nodes $\left(i,d\right)$ and $\left(i+1,j\right)$ is ${\Delta }_{ij}$. The goal is to find the path that maximizes the total reward, which can be achieved through dynamic programming.

Define the optimal substructure for the problem as follows:

$${\Delta }_{ij}=\max \left({\displaystyle {\sum }_{k=1}^d{\Delta }_{ij}}\right)$$

(16)

where, $j=1,2,\dots ,d$.

Here, the optimal path is determined by the maximum number of tasks completed by the agents.

Use dynamic programming to solve the optimization problem. Let $f\left(i,j\right)$ be the maximum number of tasks completed if the first iii agents choose actions that lead to node $j$. Then, the optimal solution can be obtained by the following:

$$f\left(i,j\right)={\max }_k\left(f\left(i-1,j\right)+{\Delta }_{ij}\right)$$

For continuous action spaces, we transform the continuous space into discrete intervals. Each action in the continuous space can be mapped to a binary sequence representing the target capture status. For example, if there are three targets, actions in one subinterval may correspond to capturing target 2 (010), while actions in another subinterval may correspond to capturing targets 1 and 2 (110). Thus, the continuous action space is divided into at most $2^m$ subspaces, where $m$ is the number of targets.

The mapping function for continuous actions is defined as follows:

$$act\left(a\right)\to B$$

(17)

where $B$ is a binary sequence of length $m$. □

Each subinterval of the continuous action space corresponds to a unique binary sequence, effectively converting the continuous space into a discrete space.

By treating the continuous action space as a collection of discrete subspaces, we apply the same dynamic programming approach to find the optimal high-level reward.

In conclusion, both in discrete and continuous action spaces, Algorithm 2 guarantees the optimal high-level reward by leveraging dynamic programming and appropriate mapping of actions to target statuses. This proof ensures rigorous handling of both types of action spaces, providing a clear and detailed derivation of the optimal solution.

Algorithm 2 RHMC Method

for episode := 1 → M do

2:    Initialize a random process $\aleph$ for action exploration

Receive initial state x

4:    for t := 1 → max-episode-length do do

The highlevel outputs target assignment policy $g$

6:    For each agent i, select action $a_i={\mu }_{{\theta }^i}(o^i,g^i)+{\aleph }_t$ w.r.t the current policy and exploration after filtering treatment g by using 0.

Execute action a = (a1, …, aN) and observe lowlevel reward rl, high-level reward rh and new state $x^{\prime }$

8:    Store $(x,a,r_l,x^{\prime })$ in replay buffer Bl

Regulate high-level rewards rh by using Algorithm 1

10:     Store $(x,a,r_l,x^{\prime })$ in replay buffer Bh

$x\Leftarrow x^{\prime }$

12:     for agent := 1 → N do

Sample a random minibatch of Dl samples $(x,a,r_l,x^{\prime })$ from Bl

14:     Update critic by minimizing the 0

Update actor using the sampled policy gradient in 0

16:     Update target network parameters for agent 0

end for

18:     Sample a random minibatch of Dh samples $(s,g,r_h,s^{\prime })$ from Bh

Update high-level critic by minimizing the 0

20:     Update low-level actor by 0

Update target network parameters using 0

22:  end for

end for

24: return

5. Experiments

The experiments were performed in two target-oriented, multi-agent cooperative environments. Specifically, the first experiment was conducted in the cooperative navigation [16] scenario, which involves static targets, and the second experiment was conducted in the multi-sensor multi-target coverage task [41] scenario, which involves dynamic targets.

5.1. Environmental Settings

Cooperative Navigation. In this experiment, a group of n agents are required to reach n landmarks (5). The agents receive rewards based on the negative distance to their nearest landmark. In the event that an agent collides with another agent, it receives a penalty of −1. Therefore, the agents must learn to cover all the landmarks while avoiding collisions. To sum up, when the agent k moves from state $S_i$ to state $S_{i+1}$, the reward that the environment gives back to the agent is as follows:

$$r_k=-d_{k,s_{i+1}}-{\sum _{j=1}^{n}P(k,j)|}_{k\ne j}$$

(18)

where $P\left(i,j\right)$ represents whether the two agents collided $\left(0:No,1:Yes\right)$, and $d_{k,s_{i+1}}$ represents distance to their nearest landmark. Finally, the reward of each agent will be summarized as the cooperation reward of the whole agent system. Each agent will use the global reward for training.

Multi-sensor Multi-target Coverage Task. In this experiment, a group of n agents (sensors) need to adjust their angles to cover as many as possible of the m targets present in the environment. The sensors are fixed in place and must take an action at each time step to adjust their angle. The targets moved randomly in the environment, making it difficult for the sensors to effectively track them. In every time step, the agent obtains the observed state $s_i=\left(s_{i,1},s_{i,2},\dots ,s_{i,m}\right).s_{i,j}$ describes the relative position information of the agent i and the target $j.s_{i,j}=(i,j,{\alpha }_{i,j},d_{i,j})$, where i and j represent the identification number of the agent and the target, respectively. ${\alpha }_{i,j}$ and $d_{i,j}$ represent the relative angle and absolute distance of the agent i and the target j, respectively. Each agent took a primitive action at every time step, selecting from a discrete action space comprising turn right, turn left, and stay. The agent’s rotation angle was fixed at 5 degrees. In RHMC, the action of the high-level action is the assigned target $g$, which is a vector of length $n.g_{i,j}=1$, which means target j is assigned to agent i and $g_{i,j}=0$ means it is not.

As agents’ policies exhibit similar functionality, sharing parameters can reduce the computational complexity of learning, thereby improving learning efficiency and performance. To encourage varied behavior among the agents, the policy incorporates each agent’s identity as a one-hot vector input. If the input sizes of agents vary, padding of zeros is incorporated to maintain consistent input dimensionality. If agents have varying numbers of operations, the probability of selecting invalid operations is set to zero. For each agent, losses are determined, and the shared parameters are adjusted accordingly. Random seeds were used for the experiments, with each experiment comprising a maximum of 300 episodes. The learning rate and discount factor used for the experiments were 0.0005 and 0.99, respectively.

The following baselines were used to compare our results: HiT-MAC [37] is a hierarchical coordination architecture that uses pre-trained, maladaptive low-level policies. QMIX [20] is a multi-agent algorithm based on value decomposition. It uses two neural networks to deduce the global value by combining the Q value of each agent and the global state. IQL [17] regarded other agents as environments without considering their changes, while COMA [8] adopted a counterfactual baseline to solve the problem of credit allocation. MAPPO [32] uses an adaptive KL divergence to ensure stable policy updates during optimization. MADDPG [18] is a deep reinforcement learning-based multi-agent algorithm that uses experience replay and target networks to reduce the variance of the Q-value function.

5.2. Results and Comparisons

In this experiment, different reinforcement learning algorithms were employed to tackle the cooperative navigation task, and their effectiveness was evaluated by comparing their reward averages. The results of the experiment, as shown in Figure 3, demonstrate that the RHMC algorithm performed the best and obtained the highest reward average for this task. This is because the RHMC algorithm emphasizes the coordination between high-level strategies, effectively coordinating agents and goals, and thus improving the global reward.

By contrast, optimization methods based on global rewards, such as the QMIX and COMA algorithms, did not perform as expected. Although these methods can handle collaboration among different agents, they face the problem of conflict between local and global rewards. Therefore, these methods even performed worse than the IQL algorithm, which only ignores other agents. This suggests that global reward optimization methods like QMIX and COMA require better solutions for the conflict between local and global rewards.

Overall, this experiment demonstrates the importance of coordination between high-level strategies for the cooperative navigation task. The RHMC algorithm performs well in coordinating agents and goals to improve global rewards. However, for global reward optimization methods like QMIX and COMA, it is necessary to resolve the conflict between local and global rewards to enhance their performance.

The RHMC algorithm is not limited to cooperative navigation experiments with static targets but also performs well in multi-sensor multi-target coverage experiments with dynamic targets. As shown in Figure 4, after 5 million time steps of training in an environment with four agents and five targets, the RHMC algorithm obtained the highest global reward. This indicates that the high-level coordination of the RHMC algorithm is also applicable in multi-sensor multi-target coverage experiments and can improve global rewards.

Some methods based on local rewards, such as the MADDPG and MAPPO algorithms, did not perform well in this task. Although they used global information to maximize the local rewards of each agent, they still could not reach the performance level of the RHMC algorithm. In contrast, the HIT-MAC and QMIX algorithms ultimately converged to around 60 rewards, while the MADDPG and MAPPO algorithms reached the highest point of their learning curve after 1.5 million time steps. These results suggest that global reward optimization methods are more suitable than local reward optimization methods for solving the task in multi-sensor multi-target coverage experiments.

During the converging phase, the RHMC algorithm achieved better global rewards than other algorithms through the coordination of high-level strategies. This is because the agents performed well in initial target allocation, but in the later stages of agent training, the strategies were bottlenecked by internal target conflicts. Therefore, the coordination of the RHMC algorithm largely improved the performance of the agents and achieved higher global rewards. This indicates that the coordination of high-level strategies is important for optimizing global rewards in multi-agent systems.

As shown in Figure 5, we investigated the scalability and adaptability of the RHMC algorithm in the multi-target multi-sensor target coverage task with respect to the number of agents. We conducted experiments with different numbers of agents, compared the RHMC algorithm with other baseline algorithms, and observed the performance of the RHMC algorithm in different scenarios.

The experimental results in Figure 6 demonstrate that the RHMC algorithm exhibits good scalability and adaptability in the presence of different numbers of agents. Regardless of an increase or decrease in the number of agents, the RHMC algorithm can adapt to different situations and achieve the highest global rewards. Notably, the performance of the RHMC algorithm remains stable when the number of agents increases, indicating its ability to scale up in multi-agent collaborative tasks. Moreover, the RHMC algorithm also shows excellent performance in multi-sensor multi-target coverage experiments, further proving its effectiveness in coordinating agents and targets to increase global rewards.

In the multi-sensor multi-target coverage task, the experimental environment was set up with four agents and five targets. Figure 7 shows the comparison of learning curves between our method (RHMC) and baseline methods. It is evident that the RHMC method outperforms other baseline methods, such as HiT-MAC, QMIX, MAPPO, and MADDPG, both in terms of the convergence speed and the final reward value.

As shown in Figure 8 and Figure 9, the experimental environment for the multi-sensor multi-target coverage task includes five targets with varying numbers of sensors, and the results of our method are compared to the baselines under these conditions.

5.3. Ablation Studies

To better understand the individual contributions of each component within the RHMC framework, we conducted detailed ablation studies. These studies further validate the importance of the regulation mechanism and the target-conditioned filter in enhancing the RHMC’s performance. As illustrated in Figure 10 and Figure 2, the full RHMC method, incorporating both the regulation mechanism and the target-conditioned filter, achieved a final reward of 70. In contrast, the variants without these components achieved rewards of 55 and 35, respectively. This indicates that the regulation mechanism contributes to a 50% performance improvement, while the target-conditioned filter contributes to a 21.4% improvement.

Table 2. Ablation study results.

Method	Final Reward	Performance Improvement
Full RHMC	70.5	N/A
Without regulation mechanism	35.6	50.9%
Without target-conditioned filter	55.1	21.4%

presents ablation study results showing final rewards and performance improvements across different methods.

The regulation mechanism plays a critical role in stabilizing the high-level policy learning process and mitigating the non-stationarity introduced by dynamic changes in low-level policies. Meanwhile, the target-conditioned filter reduces the noise from irrelevant information, accelerating the learning process and improving overall efficiency. Without the target-conditioned filter, the system is more susceptible to noise from extraneous data, leading to lower final rewards, slower learning rates, and diminished efficiency. Notably, after 1 million training steps, the RHMC method with both components demonstrates superior policy learning and gradual convergence, whereas the variant without the target-conditioned filter requires 3 million steps to achieve convergence.

These results underscore the significant roles these components play in enhancing the performance of the RHMC method. By stabilizing the training process and reducing the interference from irrelevant information, the regulation mechanism and target-conditioned filter collectively improve the overall efficiency and robustness of the RHMC framework, allowing for better coordination and performance in multi-agent environments.

6. Conclusions

In this paper, we propose the regulatory hierarchical multi-agent coordination (RHMC) framework to address coordination challenges in target-oriented multi-agent tasks. The RHMC framework introduces a two-level hierarchical reinforcement learning approach, where the high-level policy is responsible for target assignment and the low-level policy focuses on executing primitive actions.

Our extensive experimental results in both static and dynamic target-oriented environments demonstrate the significant advantages of the RHMC framework over existing methods. Specifically, in the cooperative navigation task, the RHMC algorithm achieved an average reward of 70.5, significantly outperforming QMIX and COMA, which achieved average rewards of 55.1 and 35.6, respectively. As illustrated in Figure 6, RHMC maintained a higher reward trajectory throughout the training process, demonstrating its superior high-level strategy coordination and target assignment capabilities.

In the dynamic multi-sensor multi-target coverage task, RHMC achieved a peak global reward of 92.1 after 5 million time steps, compared to 78.6 for QMIX and 75.2 for COMA, highlighting its adaptability to changing target dynamics. Figure 7 shows that RHMC’s reward increased steadily, reaching higher rewards faster than other methods, thus proving its efficiency and robustness in real-time adaptation.

Ablation studies further validate the importance of the regulation mechanism and target-conditioned filter in enhancing the RHMC’s performance. The full RHMC method, incorporating both the regulation mechanism and the target-conditioned filter, achieved a final reward of 70.5, while the variants without these components achieved 55.1 and 35.6, respectively. This indicates that the regulation mechanism and target-conditioned filter contribute to a 50.9% and 21.4% performance improvement, respectively, as shown in Figure 10.

The regulation mechanism effectively stabilized the high-level policy learning process, mitigating the non-stationarity introduced by dynamic changes in low-level policies. This stability, coupled with the target-conditioned filter’s ability to reduce noise from irrelevant information, significantly enhanced coordination efficiency and overall system performance.

In conclusion, the RHMC framework represents a substantial advancement in multi-agent reinforcement learning, particularly in target-oriented environments. By addressing the coordination problem through a structured hierarchical approach, RHMC not only improves learning efficiency and robustness but also achieves superior performance metrics in both static and dynamic tasks. Future research could further explore the application of RHMC in more complex multi-agent systems, potentially extending its scalability and effectiveness. By integrating these innovations, RHMC sets a new benchmark for multi-agent coordination tasks, achieving unprecedented results in terms of efficiency and effectiveness compared to state-of-the-art methods.