Reinforcement Learning with Value Function Decomposition for Hierarchical Multi-Agent Consensus Control

Abstract

A hierarchical consensus control algorithm based on value function decomposition is proposed for hierarchical multi-agent systems. To implement the consensus control algorithm, the reward function of the multi-agent systems can be decomposed, and two value functions can be obtained by analyzing the communication content and the corresponding control objective of each layer in the hierarchical multi-agent systems. Therefore, for each agent in the systems, a dual-critic network and a single-actor network structure are applied to realize the objective of each layer. In addition, the target network is introduced to prevent overfitting in the critic network and improve the stability of the online learning process. During the updating of network parameters, a soft updating mechanism and experience replay buffer are introduced to slow down the update rate of the network and improve the utilization rate of training data. The convergence and stability of the consensus control algorithm with the soft updating mechanism are analyzed theoretically. Finally, the correctness of the theoretical analysis and the effectiveness of the algorithm were verified by two experiments.

1. Introduction

In recent years, due to the intense development of technology and the rapid improvement of data processing capabilities, many practical systems in the field of engineering have become increasingly complex. Therefore, the study of large and intricate systems has emerged as a prominent research focus. At present, the main research trend is to model complex network systems and divide them into multiple simple and identical subsystems [1]. The hierarchical multi-agent system is an architecture for organizing and managing multiple agents [2]. The agents of the systems can be divided into different layers, each of them having different responsibilities and control objectives. Typically, higher-layer agents are responsible for global policy making, while lower layer agents perform more specific tasks or operations. Therefore, the model of hierarchical multi-agent systems can be described by multiple subsystems. This holds significant practical relevance; that is, we can combine many of the same simple agents to form a large and complex system, or we can realize the complex tasks that cannot be completed by a single agent. In the control process of robotic swarms [3], hierarchical multi-agent systems can be employed to coordinate multiple robots to accomplish complex tasks. For example, in search-and-rescue missions [4], higher-layer agents can plan the search area, while lower layer agents carry out specific search actions. In intelligent traffic management [5], higher-layer agents can be responsible for overall traffic flow management, whereas lower layer agents control specific traffic signals or vehicles. In complex supply chain systems [6], higher-layer agents can perform global supply chain optimization, while lower layer agents handle specific tasks such as production, transportation, or inventory management. By decomposing complex tasks and allocating them to different layers of agents, hierarchical multi-agent systems can enhance the efficiency and flexibility of the system. Over the past decade, the analysis of hierarchical multi-agent systems and the development of distributed control schemes have become focal issues in the field of control and obtained numerous research outcomes [7].

For large-scale multi-agent systems, the model is usually described by a hierarchical structure because large-scale systems can usually be divided into some groups, and the number of agents in different groups and the communication topology may be different. The research on the consensus control of hierarchical multi-agent systems is the basis of all the problems mentioned above. Consensus control is critical for ensuring the stability and performance of the entire system. Williams et al. [8] outlined the concept of the hierarchy, which is used to describe relationships between subeformations, for example, “formations of formations”. This concept could be valuable in fields like robotics, swarm intelligence, and military applications. The authors provide a strong theoretical foundation by discussing the dynamics of stability within hierarchical formations. This is important for ensuring that the structures they propose are robust and applicable in real-world scenarios. However, it may pose challenges in large-scale systems with unpredictable environmental factors. Smith et al. [9] studied the application of the hierarchical structure to the circular tracking of vehicle networks, and better results can be obtained compared with the traditional circular tracking algorithm; the convergence rate of vehicle groups to the center of mass is significantly increased. Hara et al. [10] proposed a general hierarchical multi-agent model with a fractal structure and studied the stability, global convergence, and low rank of the interconnection structure between different layers. Consensus in hierarchical systems with low-rank interconnections has practical implications in various fields, such as decentralized control systems and distributed optimization. However, like many theoretical papers, it may rely on certain simplifying assumptions that might not hold in real-world scenarios, potentially limiting the generalizability of the findings. Tsubakino and Hara et al. [11] propose a hierarchical characterization method based on eigenvectors for low-rank, interconnected, multi-agent, dynamical systems. This eigenvector-based approach could have practical implications in areas such as robotic coordination, decentralized decision-making, and networked systems, where efficient and stable intergroup communication is crucial. However, while low-rank interconnections aim to reduce complexity, eigenvector computations in large, dynamic systems may still be computationally expensive, especially in real-time applications where swift decisions are needed. Sang et al. [12] studied the group consensus problem based on hierarchical containment control in linear multi-agent systems. The influence of time delay in dynamic environments and systems on consensus is considered in this paper. A robust control strategy is designed to ensure that the system can achieve stable containment control and consensus in the presence of dynamic changes and communication delays. Wang et al. [13] proposed a new hierarchical control framework for distributed control of multi-agent systems. The framework divides the system into multiple levels, each responsible for a different control task. The framework is designed to improve the scalability and robustness of the system and simplify the control design of complex systems.

Hierarchical reinforcement learning is one of the effective methods of solving large-scale and multi-task reinforcement learning problems [14]. Through task decomposition, strategies are learned according to the goals of each subtask, and the learned strategies are combined to obtain a global strategy, which can effectively solve the dimensional disaster problem in large-scale systems. Makar et al. [15] propose a hierarchical reinforcement learning algorithm based on value function decomposition to solve the multi-agent discrete control problem. The combination of hierarchical structures with multi-agent reinforcement learning represents a novel framework. Hierarchical approaches allow for the decomposition of complex tasks into subtasks, making it easier to handle environments with multiple agents and large state-action spaces. However, implementing a hierarchical algorithm is often more challenging. Designing the appropriate hierarchy of tasks and sub-tasks and determining how agents should transition between them can require significant domain knowledge and expertise. To solve large-scale and complex continuous control problems, Yang et al. proposed the hierarchical Deep Deterministic Policy Gradient (DDPG) algorithm [16], which is a dual-critic network and multi-actor network structure. The critic network is divided into two layers. In the first layer, a Multi-DDPG algorithm structure is used to solve simple task problems, and the second layer focuses on solving complex combined tasks. At the same time, hierarchical reinforcement learning can also solve the problem of switching topology in the multi-agent systems. For example, Wang et al. [17] proposed a two-layer reinforcement learning framework to solve the output consensus problem of multi-agent systems under switching topology conditions. The first layer uses the Q-learning algorithm, wherein each agent selects the optimal action according to the current state, and the second layer selects the appropriate action strategy according to the current topology to ensure the output consensus of the entire system. This two-layer structure gives the control strategy strong adaptability and robustness and can be adjusted quickly when the topology changes.

The research concerning hierarchical consensus control mentioned above is largely based on the specific sequence of the tasks, which means that the performance of the algorithm convergence may be insufficient. However, in this paper, a hierarchical consensus control algorithm based on value function decomposition is proposed for hierarchical multi-agent systems. According to the communication content of each layer of the multi-agent systems and the corresponding control objectives, the reward function is decomposed, and two value functions are obtained. Specifically, for each agent in the system, the dual-critic network and single-actor network structure are designed. The updating of the dual-critic network is based on the decomposition of value functions in different tasks, and the two decomposed value functions have no logical order. It is also unnecessary to consider the training order of the double evaluation network during the training process. In addition, this paper introduces the target network to avoid overfitting in the critic network and improve the stability of the online learning process. In the updating of network parameters, a soft updating mechanism and experience replay buffer are introduced to slow down the update rate of the network and improve the utilization rate of training samples. The main contributions of this paper are as follows:

(1)

For hierarchical multi-agent systems, a hierarchical consensus control algorithm based on value function decomposition is proposed. Firstly, the structure of the algorithm is that of a distributed actor–critic network, which ensures that the distributed characteristics of multi-agent systems are fully utilized. In addition, a value function decomposition algorithm is introduced to ensure the simultaneous optimization of the control objectives of agents at different levels, which gives the training process a certain robustness.

(2)

The convergence and stability analysis of the consensus control algorithm with a soft update mechanism are proposed. It is proved that for each agent, the action value function estimated by the critic network can converge to the optimal value, the policy output from the actor network can converge to the corresponding optimal value, and the multi-agent system can be asymptotically stable.

In this paper, the implementation of the hierarchical consensus control algorithm based on value function decomposition is presented, and the convergence and stability of the algorithm are analyzed. Finally, the correctness of the theoretical analysis and the effectiveness of the algorithm were verified by two experiments. Multi-agent systems with and without leaders are both considered in the Section 5.

2. Preliminaries

2.1. Graph Theory

The communication of the leader–follower multi-agent system can be represented by a weighted directed graph $G(\mathcal{A})=(V,E,\mathcal{A})$, which is composed of a set of N agents $V=\{v_1,v_2,\dots ,v_N\}$, a set of edges $E=\{e_{ij}=(v_i,v_j)\}\subset V\times V$, and a non-negative adjacency matrix $\mathcal{A}=[a_{ij}]$. $a_{ij}>0$ if and only if $(v_i,v_j)\in E$, which means agent i and j can communicate with each other; otherwise, $a_{ij}=0$. For $\forall i=1,2,\dots ,N$, $a_{ii}=0$. The in-degree matrix $D={[d_{ij}]}_{N\times N}$ is a diagonal matrix with $d_{ii}={\displaystyle \sum _{j\in {\mathcal{N}}_i}}{a}_{ij}$, and the Laplacian matrix is defined as $L=D-\mathcal{A}$. The communication between the leader and followers is modeled by the diagonal matrix $\mathcal{B}={[b_{ii}]}_{N\times N}$, and $b_{ii}>0$ means there is a directed path between the leader and the ith follower; otherwise, $b_{ii}=0$. If a digraph contains a directed spanning tree, it means there exists a directed path from the leader to any other follower.

2.2. Problem Formulation

Consider the hierarchical(two-layer) discrete multi-agent system [18]. The structure is shown in Figure 1. The system contains $N=p\times q$ agents:

$$x_i(k+1)=A{x}_i(k)+B{u}_i(k)i=1,2,\dots ,N,$$

(1)

where $x_i(k)\in {\mathbb{R}}^n$ is the system state, and $u_i(k)\in {\mathbb{R}}^m$ is the control signal values. The system matrix parameters are $A\in {\mathbb{R}}^{n\times n}$ and $B\in {\mathbb{R}}^{n\times m}$. The number of agents is N, which is divided into q groups, and each group has p agents. Based on the graph theory mentioned above, the communication of p agents in the bottom $H2$ groups can be represented by the adjacency matrix ${\mathcal{A}}_1$ of dimensions $p\times p$. The communication between the q groups of the top layer $H1$ can be represented by the adjacency matrix ${\mathcal{A}}_2$ of dimensions $q\times q$. The Laplace matrices are $L_1\in R^{p\times p}$ and $L_2\in R^{q\times q}$, corresponding to the adjacent matrices ${\mathcal{A}}_1$ and ${\mathcal{A}}_2$, respectively. The communication mode between groups and whether the agents in each group can receive inter-group information are crucial to the consensus of hierarchical multi-agent systems. In this paper, it is assumed that the agents in each group can receive information outside the group in the way of the topological structure ${\mathcal{A}}_2$. And the communication information between groups is fixed as the average value of all agents in each group, $g_j$ [19]; that is, for each group of agents in the bottom layer, $g_j={\displaystyle \frac{1}{p}}{x}_i(k),i\in p,j\in q$.

Assumption 1.

There is no time delay in intra-group communication and inter-group communication in hierarchical multi-agent systems. The number and topology of agents in each group are the same.

In the two-layer discrete multi-agent systems, the agents in the same group are relatively concentrated. For example, the communication distance between robots or vehicles composed of communication nodes in the group is close, so the communication delay is small and can be ignored. The communication distance between groups is often very far because the communication line is long, the signal transmission capacity is limited, and the communication time delay of each group of agents can not be ignored. In order to reduce the difficulty of the algorithm, the problem of communication time delay is not considered in this paper.

In hierarchical multi-agent systems, the number and topology of agents within each group can indeed be the same, but this is not always necessary. If each group has the same number of agents, it simplifies the system design and interaction protocols. This uniformity makes coordination easier since each group can follow a similar communication and decision-making process. If each group shares the same topology, it typically means that the connections and interactions between agents are similar across groups. The specific number and topology of agents depend on the problem’s requirements.

Definition 1.

The two-layer discrete multi-agent system (1) reaches hierarchical consensus, meaning that for any initial state, there is $\forall i,j$,${lim}_{k\to \infty }\parallel x_i(k)-x_j(k)\parallel =0$.

If the hierarchical multi-agent systems has a leader agent, it can be defined as follows:

$$x_0(k+1)=A{x}_0(k),$$

(2)

where $x_0(k)\in {\mathbb{R}}^n$ is the reference signal value of multi-agent systems.

Definition 2.

The two-layer discrete multi-agent systems (1) and (2) can reach hierarchical consensus, meaning that for any initial state, there is $\forall i$, ${lim}_{k\to \infty }\parallel x_i(k)-x_0(k)\parallel =0$.

Considering the communication characteristics of hierarchical multi-agent systems, each agent can obtain intra-group and inter-group information, so the local tracking error for agent i in each group is as follows:

$$e_i(k)=\sum _{j\in q}[a_{1,ij}(x_i(k)-x_j(k))+b_{ii}(x_i(k)-x_0(k))]+\sum _{j\in p}{a}_{2,ij}(g_i(k)-g_j(k)).$$

(3)

According to the equation above, the global tracking errors are defined below; that is, $e(k)=[e_1^T(k),e_2^T(k),\dots ,e_N^T(k)]\in {\mathbb{R}}^{Nn}$:

$$e(k)=(I_q\otimes (L_1+\mathcal{B})\otimes I_n)(X(k)-X_0(k))+{\displaystyle \frac{1}{p}}(L_2\otimes {\Delta }_1\otimes I_n)X(k),$$

(4)

where ⊗ is the Kronecker product, $I_n$ is the n-dimensional unit matrix, $I_q$ is the q-dimensional unit matrix, the state of the h group agent is $X_h(k)={[x_{(h-1)p+1}^T(k),x_{(h-1)p+2}^T(k),\dots ,x_{hm}^T(k)]}^T$, the state vector for all agents is defined as $X(k)={[x_1^T(k),x_2^T(k),\dots ,x_N^T(k)]}^T\in {\mathbb{R}}^{Nn}$, the state vector of the leader agent is $X_0(k)={[x_0^T(k),x_0^T(k),\dots ,x_0^T(k)]}^T\in {\mathbb{R}}^{Nn}$, ${\displaystyle \frac{1}{p}}(L_2\otimes {\Delta }_1\otimes I_n)X(k)$ is the value of inter-group state error, and ${\Delta }_1={[1,\dots ,1]}^T[1,\dots ,1]\in {\mathbb{R}}^{p\times p}$ is the information interaction between each group structure. If the multi-agent systems has no leader, then both $b_{ii}$ and $\mathcal{B}$ are zero.

According to (4), the global tracking errors can be defined as follows:

$$e(k)=e_1(k)+e_2(k),$$

(5)

where $e_1(k)=(I_q\otimes (L_1+\mathcal{B})\otimes I_n)(X(k)-X_0(k))$, $e_2(k)={\displaystyle \frac{1}{p}}(L_2\otimes {\Delta }_1\otimes I_n)X(k)$.

For hierarchical multi-agent systems with a leader agent, in order to reach consensus, the global consensus error of the whole system is defined as follows:

$$ϵ(k)=X(k)-X_0(k)+{\displaystyle \frac{1}{p}}(L_2\otimes {\Delta }_1\otimes I_n)X(k),$$

(6)

where ${ϵ}_1(k)=X(k)-X_0(k)$, ${ϵ}_2(k)={\displaystyle \frac{1}{p}}(L_2\otimes {\Delta }_1\otimes I_n)X(k)$.

Lemma 1

([20]). If the matrix $(L_1+\mathcal{B})$ is singular, then $\parallel {ϵ}_1(k)\parallel \le \parallel e_1(k)\parallel /{\lambda }_{min}(L_1+\mathcal{B})$, where ${\lambda }_{min}(L_1+\mathcal{B})$ is the minimum singular value of matrix $(L_1+\mathcal{B})$.

Lemma 1 shows that if the global tracking error is small enough, then the global consensus error can be arbitrarily small. To ensure that the inequality in Lemma 1 holds, the following hypothesis is given:

Assumption 2.

The communication graphs $G({\mathcal{A}}_1)$ and $G({\mathcal{A}}_2)$ of the multi-agent systems are connected.

2.3. Optimal Consensus Control Based on Value Function Decomposition

Considering the two-layer multi-agent systems, the performance index function of agent i in any group can be defined as follows:

$$\begin{array}{cc}\hfill J_i(e_i(k),{\underline{u}}_i(k))=& \sum _{t=k}^{\infty }{\gamma }^{t-k}{r}_i(e_i(k),u_i(k))\hfill \\ \hfill =& r_i(e_i(k),u_i(k))+\gamma J_i(e_i(k+1),u_i(k+1)),\hfill \end{array}$$

(7)

where ${\underline{u}}_i(k)=(u_i(k),u_i(k+1),\dots ,u_i(\infty ))$ is the control sequence composed of all the control quantities of agent i from the current moment, and $0<\gamma <1$ is the discount factor. The reward value of environment is $r_i(e_i(k),u_i(k))$.

The objective of the optimal consensus control is to minimize the performance index function (7). Therefore, according to Bellman’s principle, the optimal value of the state value function $J_i^{*}(e_i(k))$ satisfies the following equation:

$$\begin{array}{c}\hfill J_i^{*}(e_i(k))=\underset{u_i(k)}{min}\{r_i(e_i(k),u_i(k))+\gamma J_i^{*}(e_i(k+1))\}.\end{array}$$

(8)

The DTHJB equation for agent i can be expressed as follows:

$$\begin{array}{c}\hfill J_i^{*}(e_i(k))=r_i(e_i(k),u_i^{*}(k))+\gamma J_i^{*}(e_i(k+1)).\end{array}$$

(9)

Therefore, the DTHJB equation for hierarchical multi-agent systems can be expressed as follows:

$$\begin{array}{c}\hfill J^{*}(e(k))=R(e(k),u^{*}(k))+\gamma J^{*}(e(k+1)).\end{array}$$

(10)

The action value function Q and the optimal action value function $Q^{*}$ are introduced. Then, the optimal action value function of agent i is as follows:

$$\begin{array}{c}\hfill Q_i^{*}(e_i(k),u_i(k))=r_i(e_i(k),u_i(k))+\underset{u_i(k+1)}{min}\gamma Q_i^{*}(e_i(k+1),u_i(k+1)).\end{array}$$

(11)

By minimizing the action value function, the optimal control value $u^{*}$ can be obtained directly as follows:

$$u_i^{*}(e_i(k))=arg\underset{u_i(k)}{min}{Q}_i^{*}(e_i(k),u_i(k)).$$

(12)

For two-layer discrete multi-agent systems, a hierarchical reinforcement learning algorithm can be introduced to decompose the performance index function and action value function according to the control objectives of different layers. The following content gives the conditions that need to be satisfied by the decomposition of the action value function, so as to provide the basis for the implementation of the algorithm in the next section.

Theorem 1

([21]). Suppose the reward function r can be decomposed into M reward functions, namely, $r(e,u)={\sum }_{k=1}^M{r}_k(e,u)$. Then, the performance index function and action value function can be decomposed into $J(e)={\sum }_{k=1}^M{J}_k(e)$ and $Q(e,u)={\sum }_{k=1}^M{Q}_k(e,u)$.

Proof of Theorem 1.

The performance index function $J(e)$ is expressed as follows:

$$J(e)=\sum _{l=0}^{\infty }{\gamma }^l{r}_{t+l+1}(e,u).$$

(13)

According to the decomposed form of the reward function in Theorem 1, the performance index function can be decomposed into the following:

$$\begin{array}{cc}\hfill J(e)& =\sum _{l=0}^{\infty }{\gamma }^l\sum _{k=1}^M{r}_{t+l+1,k}(e,u)\hfill \\ \hfill & =\sum _{k=1}^M{\gamma }^l\sum _{l=0}^{\infty }{r}_{t+l+1,k}(e,u)\hfill \\ \hfill & =\sum _{k=1}^M{J}_k(e).\hfill \end{array}$$

(14)

Similarly, the decomposition form of the action value function can be obtained:

$$Q(e,u)=\sum _{k=1}^M{Q}_k(e,u).$$

(15)

   □

From Theorem 1, we can know that the action value function can be decomposed into several sub-functions for different objectives, and there is no sequential relationship between the objectives. How to optimize these sub-functions separately is also a problem to be studied below.

3. Proposed Algorithm

3.1. Consensus Algorithm Based on Action Value Function Decomposition

For hierarchical multi-agent systems which contain N agents, the overall consensus algorithm structure based on action value function decomposition in order to obtain the optimal control value (12) for each agent is shown in Figure 2. The reward function of agent i is decomposed into two rewards corresponding to the consensus control objective with the intra-group $r_{i,1}(e_i(k),u_i(k))$ and the inter-group $r_{i,2}(e_i(k),u_i(k))$. That is, the reward function is expressed as follows:

$$r_i(e_i(k),u_i(k))=r_{i,1}(e_i(k),u_i(k))+r_{i,2}(e_i(k),u_i(k)),$$

(16)

where $r_{i,1}(e_i(k),u_i(k))=e_{i,1}^T(k)Q_i{e}_{i,1}(k)+u_i^T(k)P_i{u}_i(k)$, $r_{i,2}(e_i(k),u_i(k))=e_{i,2}^T(k)Q_i{e}_{i,2}(k)+u_i^T(k)P_i{u}_i(k)$, $Q_i(k)>0$, and $P_i(k)>0$ are positive definite matrices. According to Theorem 1, the action value function of agent i can be decomposed into the following:

$$Q_i(e_i(k),u_i(k))=Q_{i,1}(e_i(k),u_i(k))+Q_{i,2}(e_i(k),u_i(k)).$$

(17)

Therefore, for agent i, the initial value of the action value function is defined as $Q_{i,1}^0(e_i(k),u_i(k))$ and $Q_{i,2}^0(e_i(k),u_i(k))$, and the corresponding control value is as follows:

$$u_i^0(e_i(k))=arg\underset{u_i(k)}{min}{Q}_i^0(e_i(k),u_i(k)).$$

(18)

The objective of the hierarchical consensus control is to achieve consensus states of all agents; that is, for each agent i, the intra-group error $e_{i,1}$ and the inter-group error $e_{i,2}$ tend toward zero. Therefore, the action value function is decomposed and updated according to the reward function decomposition. Based on Formula (18), each sub-function of the action value function can be calculated as follows:

$$\begin{array}{cc}\hfill Q_{i,1}^1(e_i(k),u_i(k))=& r_{i,1}(e_i(k),u_i(k))+\underset{u_i(k+1)}{min}\gamma Q_{i,1}^0(e_i(k+1),u_i(k+1)),\hfill \\ \hfill Q_{i,2}^1(e_i(k),u_i(k))=& r_{i,2}(e_i(k),u_i(k))+\underset{u_i(k+1)}{min}\gamma Q_{i,2}^0(e_i(k+1),u_i(k+1)).\hfill \end{array}$$

(19)

Similarly, the control value of the m-th iteration is as follows:

$$u_i^m(e_i(k))=arg\underset{u_i(k)}{min}{Q}_i^m(e_i(k),u_i(k)).$$

(20)

The corresponding decomposed action value function is as follows:

$$\begin{array}{cc}\hfill Q_{i,1}^{m+1}(e_i(k),u_i(k))=& r_{i,1}(e_i(k),u_i(k))+\underset{u_i(k+1)}{min}\gamma Q_{i,1}^m(e_i(k+1),u_i(k+1)),\hfill \\ \hfill Q_{i,2}^{m+1}(e_i(k),u_i(k))=& r_{i,2}(e_i(k),u_i(k))+\underset{u_i(k+1)}{min}\gamma Q_{i,2}^m(e_i(k+1),u_i(k+1)),\hfill \end{array}$$

(21)

where $m=1,2,\dots$ indicates the number of iterations. The algorithm is based on Bellman’s principle of optimality and adopts the iterative method to solve the problem. For each agent i, the optimal Q function and corresponding control value can be obtained by iteratively updating Formulas (20) and (21). The iterative algorithm presented in this paper can deal with the consensus control problem of hierarchical multi-agent systems with and without leaders. However, how to obtain optimal control and minimize the Q function is a problem that needs to be solved below.

3.2. Algorithm Implementation

For each agent i, the hierarchical multi-agent consensus algorithm based on action value function decomposition is implemented by the dual-critic network and single-actor network. The corresponding offline training process is shown in Figure 3. The structure is similar to the hierarchical DDPG, but the difference is that the training of the dual-critic network has no primary and secondary order. The dual-critic network learns the control objectives of each layer and aggregates the learned knowledge into the same actor network to obtain the corresponding control value. For the actor network of each agent, the input is the local tracking error $e_i(k)$, and the output is the control policy $u_i(k)$, which can be obtained by minimizing the action value function $Q_i(e_i(k),u_i(k))=Q_{i,1}(e_i(k),u_i(k))+Q_{i,2}(e_i(k),u_i(k))$. According to Theorem 1, each agent has a dual-critic network ${\theta }^{C_1}$ and ${\theta }^{C_2}$, output ${\widehat{Q}}_{i,1}(e_i(k),u_i(k))$, and ${\widehat{Q}}_{i,2}(e_i(k),u_i(k))$, which are used to evaluate the control effect of current control values on the intra-group consensus and inter-group consensus of hierarchical multi-agent systems. For the updating of the critic network, according to the Bellman equation, the temporal difference error is defined as follows:

$$\begin{array}{cc}\hfill E_{td,1}=& r_{i,1}(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_{i,1}(e_i(k+1),u_i(k))-{\widehat{Q}}_{i,1}(e_i(k),u_i(k)),\hfill \\ \hfill E_{td,2}=& r_{i,2}(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_{i,2}(e_i(k+1),u_i(k))-{\widehat{Q}}_{i,2}(e_i(k),u_i(k)),\hfill \end{array}$$

(22)

where $y_1(k)=r_{i,1}(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_{i,1}(e_i(k+1),u_i(k))$ and $y_2(k)=r_{i,2}(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_{i,2}(e_i(k+1),u_i(k))$ are the target value of the dual-critic network. The parameters of the critic network are updated according to the criterion of gradient descent by minimizing the square value of the temporal difference error.

The traditional learning process of the critic network often tends to be unstable or even divergent. One reason is that the learning process of the critic network is based on the traditional Q-learning algorithm, which approximates the Q value by taking the maximum operation, which may suffer great performance loss. Meanwhile, the observation data used for the critic network training is limited and cannot fully reflect the dynamic characteristics of the system; therefore, the neural network may overfit, which leads to unstable training. Therefore, the target networks ${\theta }^{T_1}$ and ${\theta }^{T_2}$ and soft update mechanism are introduced into the training process of the critic network. The main idea is to slow down the parameter updating of the neural network to avoid overfitting of incomplete observation data so that the parameters of the target network gradually approximate the critic network. Thus, the accuracy of the output of the critic network is ensured. The setting of the initial weight of the target network is generally consistent with that of the critic network. The updating of the target networks ${\theta }^{T_1}$ and ${\theta }^{T_2}$ is as follows:

$$\begin{array}{cc}\hfill {\widehat{Q}}_{i,1}(e_i(k),u_i(k)|{\theta }_{k+1}^{T_1})& =(1-\tau ){\widehat{Q}}_{i,1}(e_i(k),u_i(k)|{\theta }_k^{T_1})+\tau {\widehat{Q}}_{i,1}(e_i(k),u_i(k)|{\theta }_{k+1}^{C_1}),\hfill \\ \hfill {\widehat{Q}}_{i,2}(e_i(k),u_i(k)|{\theta }_{k+1}^{T_2})& =(1-\tau ){\widehat{Q}}_{i,2}(e_i(k),u_i(k)|{\theta }_k^{T_2})+\tau {\widehat{Q}}_{i,2}(e_i(k),u_i(k)|{\theta }_{k+1}^{C_2}),\hfill \end{array}$$

(23)

where $\tau \in (0,1]$ indicates the soft update rate, and when $\tau =1$, the update process is the same as that of the traditional actor–critic network. The value of $\tau$ is generally small, which slows down the parameter updating of the target network. Accordingly, the temporal difference errors used to critic network updates are as follows:

$$\begin{array}{cc}\hfill E_{td,1}=& r_{i,1}(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_{i,1}(e_i(k+1),u_i|{\theta }_k^{T_1})-{\widehat{Q}}_{i,1}(e_i(k),u_i(k)|{\theta }_{k+1}^{C_1}),\hfill \\ \hfill E_{td,2}=& r_{i,2}(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_{i,2}(e_i(k+1),u_i|{\theta }_k^{T_1})-{\widehat{Q}}_{i,2}(e_i(k),u_i(k)|{\theta }_{k+1}^{C_2}).\hfill \end{array}$$

(24)

The hierarchical multi-agent consensus algorithm proposed in this paper based on action value function decomposition adopts an experience replay buffer and batch processing technology in the training process, which fully improves the utilization rate of historical data. Figure 3 shows the use of the experience replay buffer in the training process. The implementation of the hierarchical multi-agent consensus algorithm based on action value function decomposition is shown in Algorithm 1. For each agent i at the time step k, the data tuples $(e_i(k),u_i(k),r_{i,1}(k),r_{i,2}(k),e_i(k+1))$ are put into the experience replay buffer. Tuples $(e_i(k)$, $u_i(k))$ are used as the input of the critic network by training the weights of the critic network which can output $Q_{i,1}(e_i(k),u_i(k))$ and $Q_{i,2}(e_i(k),u_i(k))$, and the objective of the critic network is to minimize a batch of temporal difference error, based on Formula (24). It is defined as follows:

$$\begin{array}{c}\hfill T{D}_{i,1}={\displaystyle \frac{1}{\mathcal{M}}}\sum _n{(E_{td,1}(n))}^2,\\ \hfill T{D}_{i,2}={\displaystyle \frac{1}{\mathcal{M}}}\sum _n{(E_{td,2}(n))}^2,\end{array}$$

(25)

where $n=1,\dots ,\mathcal{M}$. $\mathcal{M}$ is the number of tuples taken from the experience replay buffer.

Algorithm 1 Hierarchical consensus algorithm based on action value function decomposition

1: Initialization:   2: Randomly initialize the weights ${\theta }^{C_1},{\theta }^{C_2}$ of the critic network ${\widehat{Q}}_{i,1},{\widehat{Q}}_{i,2}$   3: Randomly initialize the weights ${\theta }^{T_1},{\theta }^{T_2}$ of the target network   4: Randomly initialize the weights ${\theta }^{u_i}$ of theaction network $u_i$   5: Initialize the experience replay buffer $\mathcal{R}$ with the space $\mathcal{M}$   6: Learning process:   7: for Current training Episode $=1,M$ do   8:    Initialize the exploration noise sequence   9:    Randomly initialize the system state 10:    Local tracking error $e_i(1)$ is obtained 11:    for Timestep k $=1,T$ do 12:   Calculate control input $u_i(k)$ based on the noise 13:   Execute the control input $u_i(k)$ and observe the new state $r_{i,1}(k),r_{i,2}(k)$ and $e_i(k+1)$ 14:   Store the data tuples $(e_i(k),u_i(k),r_{i,1}(k),r_{i,2}(k),e_i(k+1))$ into the experience replay buffer $\mathcal{R}$ 15:   Sample the $\mathcal{M}$ training data from the experience replay buffer $\mathcal{R}$ 16:   The weight parameters ${\theta }^{C_1},{\theta }^{C_2}$ of the critic network are updated according to Formula (25) 17:   Update the weight parameters ${\theta }^{T_1},{\theta }^{T_2}$ of the target network according to Formula (23) 18:   Update the weight parameters ${\theta }^{u_i}$ of the actor network according to (20) 19:    end for 20: end for

4. Theoretical Analysis

4.1. Convergence Analysis of the Proposed Algorithm

For agent i, the update mechanism (24) and (23) are simplified as follows:

$$\begin{array}{cc}\hfill {\widehat{Q}}_i^C(e_i(k),u_i(k))& =r_i(e_i(k),u_i(k))+\gamma min{\widehat{Q}}_i^T(e_i(k+1),u_i),\hfill \\ \hfill {\widehat{Q}}_i^T(e_i(k),u_i(k))& =(1-\tau ){\widehat{Q}}_i^T(e_i(k),u_i(k))+\tau {\widehat{Q}}_i^C(e_i(k),u_i(k)),\hfill \end{array}$$

(26)

where the critic network of agent i is ${\widehat{Q}}_i^C(e_i(k),u_i(k))={\widehat{Q}}_{i,1}^{C_1}(e_i(k),u_i(k))+{\widehat{Q}}_{i,2}^{C_1}(e_i(k),u_i(k))$, and the target network is ${\widehat{Q}}_i^T(e_i(k),u_i(k))={\widehat{Q}}_{i,1}^{T_1}(e_i(k),u_i(k))+{\widehat{Q}}_{i,2}^{T_1}(e_i(k),u_i(k))$. Considering the whole hierarchical multi-agent systems, and the control value is defined as $u(k)={[u_1(k),u_2(k),\dots ,u_N(k)]}^T$, the overall optimization objective function is $Q(e(k),u(k))={[Q_1(e_1(k),u_1(k))\dots Q_N(e_N(k),u_N(k))]}^T$. For simplicity of expression, we use u to represent $u(k)$, and Q represents $Q(e(k),u(k))$. When the layered multi-agent systems reach consensus, the optimal performance index is $Q^{*}$ and the optimal control signal is $u^{*}$.

Theorem 2.

Let $Q_1^0$ and $Q_2^0$ represent the randomly initialized $2N$ critic networks, and each function is a bounded continuous function. Then, according to the action value function sequence $Q_m$ generated by Formula (26) and the control sequence $u_m$ generated by Formula (20), if the discount factor satisfies $\gamma \in (0,1)$, when the iteration number $m\to \infty$, there is $Q_m\to Q^{*}$ and $u_m\to u^{*}$.

Proof of Theorem 2.

For the layered multi-agent systems, the global tracking error is $e(k)={[e_1^T(k),e_2^T(k),\dots ,e_N^T(k)]}^T$, and according to Formula (26), we introduce two operators $S_q$ and $T_q$:

$$\begin{array}{cc}\hfill T_{q}Q(e,u)& =R(e,u)+\gamma \underset{u}{min}Q(e^{\prime },u^{\prime }),\hfill \\ \hfill S_{q}Q(e,u)& =(1-\tau )Q(x,u)+\tau T_{q}Q(e,u),\hfill \end{array}$$

(27)

where $R(e,u)={[r_1(k),r_2(k),\dots ,r_N(k)]}^T$. Based on the two mappings above, Formula (26) can be expressed as follows:

$$\begin{array}{c}\hfill {\widehat{Q}}_{k+1}^C(e,u)=T_q{\widehat{Q}}_k^T(e,u),\\ \hfill {\widehat{Q}}_{k+1}^T(e,u)=S_q{\widehat{Q}}_k^T(e,u).\end{array}$$

(28)

First, the monotonicity of the operator $T_q$ is proved; that is, let $f(e,u),g(e,u)$ be the estimates of two action-valued functions. For all $(e,u)$, when $f(e,u)\ge g(e,u)$, then $T_{q}f(e,u)\ge T_{q}g(e,u)$. Let $u^{*}=argminf(e,u)$, then

$$\begin{array}{cc}\hfill T_{q}f(e,u)& =R(e,u)+\gamma f(e,u^{*})\hfill \\ \hfill & \ge R(e,u)+\gamma g(e,u^{*})\hfill \\ \hfill & \ge R(e,u)+\gamma ming(e,u)\hfill \\ \hfill & =T_{q}g(e,u),\hfill \end{array}$$

(29)

Therefore, it can be proved that the operator $T_q$ is monotonic.

If the hierarchical multi-agent systems can reach consensus, the optimal value of $Q^{*}$ must be the fixed point of mapping $S_q$; that is, $Q^{*}=S_q{Q}^{*}$. In order to prove that the mapping $S_q$ has a unique fixed point, $\Vert .\Vert$ is simplified to represent ${\Vert .\Vert }_{sup}$, and two bounded continuous functions f and g are defined. Let ${\mathit{I}}_N={[1,1,\dots ,1]}^T$, then we can obtain $f\le g+\Vert f-g\Vert$:

$$\begin{array}{cc}\hfill S_{q}f& =(1-\tau )f+\tau T_{q}f\hfill \\ \hfill & \le (1-\tau )(g+{\mathit{I}}_N\Vert f-g\Vert )+\tau T_q(g+{\mathit{I}}_N\Vert f-g\Vert )\hfill \\ \hfill & =S_{q}g+(1-\tau +\tau \gamma )({\mathit{I}}_N\Vert f-g\Vert ).\hfill \end{array}$$

(30)

Similarly, we can obtain the following formula:

$$\begin{array}{cc}\hfill S_{q}g-S_{q}f& \le (1-\tau +\tau \gamma ){\mathit{I}}_N\Vert g-f\Vert .\hfill \end{array}$$

(31)

According to the Pareto optimality [22], we can obtain:

$$\begin{array}{cc}\hfill \Vert S_{q}f-S_{q}g\Vert & \le (1-\tau +\tau \gamma )\Vert g-f\Vert .\hfill \end{array}$$

(32)

Therefore, the above formula satisfies Banach’s fixed point theorem [23], and the mapping $S_q$ is a strictly compressed mapping with a unique fixed point. It is easy to verify that the optimal action value function $Q^{*}$ is the fixed point of $S_q$, and therefore it must be unique.

Based on the above analysis, the following proves that when $m\to \infty$, there is $Q^m\to Q^{*}$:

$$\begin{array}{cc}\hfill \Vert Q_{m+1}^T-Q^{*}\Vert & =\Vert S_q{Q}_m^T-S_q{Q}^{*}\Vert \hfill \\ \hfill & \le (1-\tau +\tau \gamma )\Vert Q_m^T-Q^{*}\Vert \hfill \\ \hfill & =(1-\tau +\tau \gamma )\Vert Q_m^T-Q_{m+1}^T+Q_{m+1}^T-Q^{*}\Vert \hfill \\ \hfill & \le (1-\tau +\tau \gamma )\Vert Q_m^T-Q_{m+1}^T\Vert \hfill \\ & +(1-\tau +\tau \gamma )\Vert Q_{m+1}^T-Q^{*}\Vert ,\hfill \end{array}$$

(33)

Then, let $\delta =1-\tau +\tau \gamma$; the above formula can be written as follows:

$$\begin{array}{cc}\hfill \Vert Q_{m+1}^T-Q^{*}\Vert & \le {\displaystyle \frac{\delta }{1-\delta }}\Vert Q_m^T-Q_{m+1}^T\Vert \hfill \\ \hfill & \le {\displaystyle \frac{{\delta }^{m+1}}{1-\delta }}\Vert Q_1^T-Q_0^T\Vert ,\hfill \end{array}$$

(34)

According to the analysis above, when $m\to \infty$, then $Q_m^T\to Q^{*}$. When the discount factor $\gamma \in (0,1)$, the performance index function $Q_m$ and the control policy $u_m$ can reach the optimal value. That is, $Q_m^C\to Q^{*}$, $Q_m^T\to Q^{*}$ and $u_m\to u^{*}$. □

4.2. Stability Analysis

Theorem 3.

Assuming that the Assumption 2 is true, the hierarchical multi-agent consensus algorithm based on action value function decomposition can make the global tracking error $e(k)$ of the multi-agent system (1) and (2) asymptotically stable, and the states of all agents can reach consensus.

Proof of Theorem 3.

The hierarchical multi-agent consensus algorithm is based on action value function decomposition, which considers the DTHJB equation of the multi-agent systems below:

$$\begin{array}{c}\hfill J^{*}(e(k))=R(e(k),u^{*}(k))+\gamma J^{*}(e(k+1)),\end{array}$$

(35)

Then, multiplying both sides by ${\gamma }^k{\mathit{I}}_N^T$ gives the following:

$$\begin{array}{c}\hfill {\gamma }^k{\mathit{I}}_N^T{J}^{*}(e(k))-{\gamma }^{k+1}{\mathit{I}}_N^T{J}^{*}(e(k+1))={\gamma }^k{\mathit{I}}_N^{T}R(e(k),u^{*}(k)).\end{array}$$

(36)

In order to prove the stability of the system, the first difference of the Lyapunov function is defined as follows:

$$\begin{array}{c}\hfill \Delta ({\gamma }^k{\mathit{I}}_N^T{J}^{*}(e(k)))={\gamma }^{k+1}{\mathit{I}}_N^T{J}^{*}(e(k+1))-{\gamma }^k{\mathit{I}}_N^T{J}^{*}(e(k)).\end{array}$$

(37)

Based on Formula (36), we can obtain the following:

$$\begin{array}{c}\hfill \Delta ({\gamma }^k{\mathit{I}}_N^T{J}^{*}(e(k)))=-{\gamma }^k{\mathit{I}}_N^{T}R(e(k),u^{*}(k))\le 0.\end{array}$$

(38)

Since the reward function R is the quadratic or absolute form of the global tracking error, there is $\Delta ({\gamma }^k{\mathit{I}}_N^T{J}^{*}(e(k))=0$ if and only if $e(k)=0$. Therefore, the global tracking error $e(k)$ of the multi-agent systems is asymptotically stable. The state of the entire multi-agent systems can reach consensus. □

5. Simulation

In this section, two simulation examples are used to verify the effectiveness of the proposed algorithm. For each agent, the dual-critic network, the actor network, and the target network all adopt the backpropagation three-layer neural network. The hidden layer of each agent contains 30 linear rectification units and is fully connected to the input layer. The output layer of each agent is a fully connected linear layer, and the weights of all networks are updated according to Algorithm 1. The Adaptive Moment Estimations (Adam) optimization method was used to update the weights. In order to ensure the implementation of the algorithm and the consensus control effect, hyperparameters of the neural network were employed and are given in

Table 1. Parameter settings in the experiment.

Parameter Name	Mumber
learning rate of critic network ($l_c$)	0.001
learning rate of action network ($l_a$)	0.001
discount factor ($\gamma$)	0.995
soft update rate ($\tau$)	0.001
experience reply buffer ($\mathcal{R}$)	50,000
batch number ($\mathcal{M}$)	128
hidden layer nodes of ciric network ($N_{ch}$)	30
hidden layer nodes of action network ($N_{ah}$)	30

. The capacity of the experience replay buffer was set to 50,000.

5.1. Multi-Agent Systems without Leader

Considering the topology structure of the multi-agent systems as Figure 4, the system contains 12 agents, which are divided into 3 groups, and each group has 4 agents. It can be seen from the figure that the topology structure of the agents in each group is a connected graph, and the dynamic performance of a single agent is defined as follows:

$$x_i(k+1)=A{x}_i(k)+B{u}_i(k),$$

(39)

where the system matrix is $A=\left[\begin{array}{cc}0.995& 0.09983\\ -0.09983& 0.995\end{array}\right]$ and $B=\left[\begin{array}{c}0.2\\ 0.1\end{array}\right]$. The initial value of the system is randomly selected as $[-1,1]$, and the safe value of the system state is set to $\pm 2.0$. The reward function is decomposed according to the intra-group error and inter-group error, and Q is taken as the unit matrix, $R=0.1$. To verify Theorem 2, we set $\gamma =0.995$, and the offline training method was used, that is, the offline implementation of Algorithm 1. We set the soft update rate of the target network to $\tau =0.001$ and used the experience replay buffer to maximize the use of historical data. The number of iterative steps in each episode during the training process was 500, and the total number of performed episodes was 300.

Figure 5 shows the control effect of the algorithm. In a randomly given initial state, the state of all agents can reach consensus after 200 time steps. Figure 6a,b show the convergence process of the intra-group error, and Figure 6c,d show the convergence process of the inter-group error. From all the simulation results, it can be seen that all the agents can reach the same state in 150 time steps.

5.2. Multi-Agent Systems with Leader

Consider the topology structure of multi-agent systems as Figure 7; the system contains 1 virtual leader agent and 12 follower agents. The 12 follower agents are divided into 3 groups, and each group has 4 agents. It can be seen from the figure that the topology graph of the agents in each group is connected. The multi-agent systems are defined as follows:

$$x_0(k+1)=A{x}_0(k),x_i(k+1)=A{x}_i(k)+B{u}_i(k),$$

(40)

where the system matrix is $A=\left[\begin{array}{cc}0.995& 0.09983\\ -0.09983& 0.995\end{array}\right]$ and $B=\left[\begin{array}{c}0.2\\ 0.1\end{array}\right]$. The initial value of the system is randomly selected as $[-1, 1]$, and the safe value of the system state is set as $\pm 2.0$. The reward function is decomposed according to intra-group error and inter-group error, and Q is taken as the unit matrix, $R=0.1$. To verify Theorem 2, we set $\gamma =1$ and used the offline training method, that is, the offline implementation of Algorithm 1, and set the soft update rate of the target network to $\tau =0.001$. The number of iterative steps in each episode during the training process was 3000, and the total number of performed episodes was 100.

Figure 8 shows the control effect of the algorithm. In a random given initial state, all agents can follow the leader agent in 400 time steps. Figure 9a,b show the convergence process of the intra-group error, and Figure 9c,d show the convergence process of the inter-group error. It can be seen from all the simulation graphs that the consensus control algorithm based on action value function decomposition proposed in this paper can make all the agent states reach consensus.

6. Conclusions

In this paper, a hierarchical consensus control algorithm based on value function decomposition is proposed for hierarchical multi-agent systems. According to the communication content of each layer and the corresponding control objective of the multi-agent systems, the value function is decomposed according to the reward function of each layer. For each agent in the system, a dual-critic network and single-actor network are adopted. The target network is introduced to avoid overfitting in the critic network and improve the stability of the online learning process. The soft update mechanism and experience reply buffer are introduced in the network parameter update process to improve the utilization rate of training samples. In this paper, the convergence and stability of consensus control algorithm with a soft update mechanism are analyzed theoretically. Finally, the correctness of the theoretical analysis and the effectiveness of the algorithm were verified by two experiments. The value function decomposition method in this paper is used for hierarchical consensus control, but the multi-agent systems studied are homogeneous, and for heterogeneous multi-agent systems, achieving consensus control with this undifferentiated value decomposition method is of great research value. At the same time, in this paper, it is assumed that there is no time delay during the information interaction between the groups of agents, and communication delay often exists in actual systems. Therefore, the design of an effective hierarchical algorithm for multi-agent systems with communication delay must also be urgently studied.