Multi-Agent Planning under Uncertainty with Monte Carlo Q-Value Function

Abstract

Decentralized partially observable Markov decision processes (Dec-POMDPs) are general multi-agent models for planning under uncertainty, but are intractable to solve. Doubly exponential growth of the search space as the horizon increases makes a brute-force search impossible. Heuristic methods can guide the search towards the right direction quickly and have been successful in different domains. In this paper, we propose a new Q-value function representation—Monte Carlo Q-value function ${\mathrm{Q}}_{\mathrm{MC}}$, which is proved to be an upper bound of the optimal Q-value function ${\mathrm{Q}}^{*}$. We introduce two Monte Carlo tree search enhancements—heavy playout for a simulation policy and adaptive samples—to speed up computation of ${\mathrm{Q}}_{\mathrm{MC}}$. Then, we present a clustering and expansion with Monte-Carlo algorithm (CEMC)—an offline planning algorithm using ${\mathrm{Q}}_{\mathrm{MC}}$ as Q-value function, which is based on the generalized multi-agent A* with incremental clustering and expansion (GMAA*-ICE or ICE). CEMC calculates Q-value functions as required, without computing and storing all Q-value functions. An extended policy pruning strategy is used in CEMC. Finally, we present empirical results demonstrating that CEMC outperforms the best heuristic algorithm with a compact Q-value presentation in term of runtime for the same horizon, and has less memory usage for larger problems.

1. Introduction

Cooperative decision-making under uncertainty is a common problem in many scenarios such as sensor networks and airport service robots. Consider a scenario where a robot soccer team cooperates to beat the opponent. A total of 11 robot players dash in a simulated field, kicking, passing, and shooting a ball to score a goal. Each of them could only acquire his local observation and every action taken is uncertain, which means passing may be unsuccessful or shooting may be biased. Another example is congestion control, which is an important task in the Internet. The router’s buffer accommodates transient packets to guarantee a stable network link. The distributed routers need to come up with a strategy to avoid congestion. These kinds of problems can be modeled as a decentralized partially observable Markov decision process (Dec-POMDP), which provides a general multi-agent model for decision-making under uncertainty. However, Dec-POMDPs are provably intractable (NEXP-Complete, nondeterministic exponential time, it is known that P $\subseteq$ NP $\subseteq$ PSPACE $\subseteq$ EXP $\subseteq$ NEXP) [1] and difficult to solve with the number of joint policies growing explosively [2]. One of the primary reasons is that the search space of Dec-POMDPs grows exponentially with the number of actions and observations, and doubly exponentially with the horizon of the problem [3]. The optimal policy can be extracted from the optimal Q-value function ${\mathrm{Q}}^{*}$ for Dec-POMDPs, but solving ${\mathrm{Q}}^{*}$ is also proved to be NEXP-Complete [4].

In a previous work [5], Monte Carlo tree search (MCTS) [6] is introduced into the MMDP (Multiagent MDP) model, which is a simplified model for Dec-POMDPs, to solve ad hoc agent teams decision-making. MCTS is a sample-based tree search algorithm using Monte Carlo simulations as state evaluations, which has been successful in Computer Go [7]. MCTS will converge to the optimal value if exploration (find new nodes that are not sampled) and exploitation (find promising nodes that are already sampled) are traded off appropriately when given sufficient time and can be stopped at any time to get an approximate result.

To tackle the doubly exponential problem, we propose an offline algorithm called a clustering and expansion with Monte-Carlo algorithm (CEMC) based on incremental clustering and expansion (ICE), a version of generalized multi-agent A* (GMAA*) with incremental clustering and expansion, which is the best heuristic method in this literature. MCTS is integrated into the CEMC framework to fast obtain the approximate Q-value function. MCTS is an anytime algorithm, and its efficiency is up to sample times and the degree of accuracy of the Q-value function evaluation. Thus, we propose an adaptive sample method according to the horizon and the number of joint actions and observations. The best upper bound found in the MCTS algorithm is used to prune the nodes that will not be the optimal ones in any case. The term MCTS in this paper denotes the specific algorithm that uses UCB1 (a version of upper confidence bound) as the tree selection policy proposed by Kocsis and Szepesvári [6,8].

The key contribution is twofold: We present a new Q-value function representation—Monte Carlo Q-value function ${\mathrm{Q}}_{\mathrm{MC}}$ and an offline planning algorithm CEMC. Heavy playout for simulation policy and adaptive sample are introduced to speed up the computation of ${\mathrm{Q}}_{\mathrm{MC}}$. The algorithm CEMC calculates the Q-value functions as required, without computing and storing all Q-value functions and it uses an extended pruning strategy for the policy pool. We present empirical results demonstrating that CEMC outperforms the best heuristic algorithm with a compact Q-value presentation in term of runtime for the same horizon, and has less memory usage for larger problems.

The remainder of this paper is structured as follows. We first introduce some related work in Section 2. Then, we outline some background knowledge on Dec-POMDPs, ICE algorithm, approximate Q-value functions, MCTS, and collaborative Bayesian Games in Section 3. Section 4 gives the Q-value function representation and proves that ${\mathrm{Q}}_{\mathrm{MC}}$ is an upper bound of the optimal Q-value function. Then, an adaptive sample and heavy rollout technique are proposed. Section 5 presents the core algorithm CEMC and a pruning strategy. We empirically prove the efficiency and effectiveness of CEMC in Section 6 and discuss the results. We conclude and outline the future research in Section 7.

2. Related Work

The Dec-POMDP is an extension of the POMDP model which has been proved to be doubly exponential [1]. The brute force search becomes intractable when the number of agents or the problem size grows. In recent years, a lot of work has focused on fast computing an optimal policy of larger horizon, such as dynamic programming [9,10,11,12], heuristic search [2,3,4,13,14,15], and mixed integer linear programming [16].

GMAA* framework [4] is one of the state-of-the-art exact algorithms in the heuristic search literature, which is based on multi-agent A* (MAA*) [2] and uses an approximate heuristic function to guide the search in the whole search space. GMAA*-ICE (or ICE) [3] is a heuristically guided algorithm based on a GMAA* framework [4] and has demonstrated impressive performance over GMAA*. Two successful components used in ICE are lossless clustering [13] and incremental expansion [14]. Recently, a Bayesian learning method [17] and approximate inferences and heuristics [18] are proposed for solving Dec-POMDP.

Silver and Veness [19] proposed a Monte-Carlo algorithm—POMCP—for online planning in large POMDPs. Unlike our approach, POMCP combines a Monte-Carlo update of the agent’s belief state with a Monte-Carlo tree search from the current belief state. Importantly, POMCP is suitable for single-agent partially-observable and unfactored Markov decision process (MDP). Amato and Oliehoek [20] proposed a scalable approach based on sample-based planning called factored-value partially observable Monte Carlo planning (FV-POMCP), to combat the intractability of an increasing number of agents. However, FV-POMCP requires a decomposition of the value function into a set of overlapping factors.

3. Background

Consider a scenario where a number of agents inhabit in a particular environment modeled with a Dec-POMDP, where the perception of the agents is uncertain and they must work together to achieve their goal. At each discrete time step $t=0,1,\dots ,H-1$ every agent takes an action and the combination of these actions influences the world, making it transform to the next state. The goal of the agents is to try to come up with a plan that maximizes their expected long-term rewards. The goal or task need to be completed with cooperation, because no single agent is competent enough. Therefore, the plan computed should be decentralized.

In this section, we briefly review some background of the decentralized POMDPs model, ICE, approximate Q-value functions, MCTS, and collaborative Bayesian games. We refer the readers to the previous work [2,3,4,21,22] about the detailed background.

3.1. Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) Model

Dec-POMDP generalizes POMDP to a multi-agent version where rewards are common and based on joint actions, but observations are individualistic [23,24]. The detail definition is as follows.

Definition 1.

A decentralized partially observable Markov decision process is defined as a 9-tuple <I, S, A, T, $𝒪$, O, R, b⁰, h>, where

• $I=\{1,\dots ,n\}$ is the finite set of agents;
• $S=\{s_0,\dots ,s_{\left|S\right|-1}\}$ is the finite set of states, where $s_0$ is the initial state;
• $A={\times }_{i=1}^n{A}_i$ is the set of joint actions where $\overrightarrow{a}=<a_1,a_2,\dots ,a_n>$, and $A_i$ is the finite set of individual actions of agent $i$. The component $a_i$ in the joint action $\overrightarrow{a}$ belongs to $A_i$;
• $T$ is the transition function where $P(s^{\prime }|s,\overrightarrow{a})$ denotes the probability of transiting to the new state $s^{\prime }$ when taking the joint action $\overrightarrow{a}$ in state $s$;
• $𝒪={\times }_{i=1}^n{𝒪}_i$ is the set of joint observations where $\overrightarrow{o}$ = <o₁, ..., o_n> ∈ $𝒪$, and $𝒪$_i is the finite set of individual observations of agent i. The component $o_i$ in the joint observation $\overrightarrow{o}$ belongs to $𝒪$ⁱ;
• O is the observation function where P($\overrightarrow{o}|\overrightarrow{a},s^{\prime }$) denotes the probability of observing $\overrightarrow{o}$ when the system state transfer to $s^{\prime }$ taking the joint action $\overrightarrow{a}$;
• $R$ is the reward function, where $R(s,\overrightarrow{a})$ denotes the immediate reward when taking the joint action $\overrightarrow{a}$ in state $s$;
• $b^0$ is the initial state distribution;
• $h$ is the finite horizon.

At every step $t$, the environment is in a particular state $s$, which emits a joint observation ${\overrightarrow{o}}^t$ according to the observation model. Each agent obtains its individual observation component $o_i^t$. Each agent selects its individual action $a_i^t$, forming a joint action ${\overrightarrow{a}}^t$, which leads to a state $s^{\prime }$ according to the transition model. Every agent receives an immediate reward $r$.

The action-observation history ${\theta }_i^t$ for agent $i$ is the sequence of actions taken and observations received, which is denoted as a sequence $(o_i^0,a_i^0,o_i^1,\dots ,a_i^{t-1},o_i^t)$. Thus, the joint action-observation history ${\overrightarrow{\theta }}^t$ is $<{\theta }_1^t,{\theta }_2^t,\dots ,{\theta }_n^t>$. A policy for agent $i$ is a mapping from history to actions ${\pi }_i:{\theta }_i\to A_i$, and a joint policy is denoted as $\pi =<{\pi }_1,{\pi }_2,\dots ,{\pi }_n>$ which specifies a policy for each agent. The goal of solving Dec-POMDPs is to calculate an optimal joint policy to maximize the expected long-term accumulated reward, which is defined as

$$E({\displaystyle \sum _{i=1}^{h-1}R(s^t,{\overrightarrow{a}}^t)}|\pi ,b^0)$$

(1)

The value of an optimal joint policy ${\pi }^{*}$ can be defined as follows:

$$V^t({\pi }^{*})={\displaystyle \sum _{{\overrightarrow{\theta }}^t}P({\overrightarrow{\theta }}^t|b^0)Q^{*}({\overrightarrow{\theta }}^t,{\pi }^{*}({\overrightarrow{\theta }}^t))}$$

(2)

where ${\pi }^{*}({\overrightarrow{\theta }}^t)$ denotes the joint action that ${\pi }^{*}$ specifies for ${\overrightarrow{\theta }}^t$, and

$$Q^{*}({\overrightarrow{\theta }}^t,\overrightarrow{a})=R({\overrightarrow{\theta }}^t,\overrightarrow{a})+{\displaystyle \sum _{{\overrightarrow{o}}^{t+1}\in O}P({\overrightarrow{o}}^{t+1}|{\overrightarrow{\theta }}^t,\overrightarrow{a})}{Q}^{*}({\overrightarrow{\theta }}^{t+1},{\pi }^{*}({\overrightarrow{\theta }}^{t+1}))$$

(3)

is the optimal Q-value function.

The policy ${\pi }_i$ of agent $i$ specifies actions for all steps of Dec-POMDPs, and ${\pi }_i$ can be represented as a sequence of decision rules ${\pi }_i=({\delta }_i^0,{\delta }_i^1,\dots ,{\delta }_i^{h-1})$. The decision rule ${\delta }_i^t$ for agent $i$ at step $t$ is a mapping from the length-$t$ history to action ${\delta }_i^t:{\theta }_i^t\to A_i$. The partial joint policy ${\phi }^t$ is defined as a sequence of decision rules specified for steps $0,\dots ,t-1$, which is denoted as ${\phi }^t=({\delta }^0,{\delta }^1,\dots ,{\delta }^{t-1})$.

3.2. Generalized MAA* with Incremental Clustering and Expansion (GMAA*-ICE)

GMAA*-ICE (or ICE) [3] is a heuristically guided algorithm based on GMAA* framework [4], which generalizes the first multi-agent A* search MAA* [2]. It uses $\widehat{V}({\phi }^t)=V^{0\dots t-1}({\phi }^t)+H^{t\dots h-1}({\phi }^t)$ to guide A* search over partially specified joint policies to construct a tree from top–down, where $V^{0\dots t-1}({\phi }^t)$ is the actual expected reward achieved over the first $t$ steps and $H^{t\dots h-1}({\phi }^t)$ is a heuristic value for the remaining h-t steps. ICE uses an admissible heuristic to guarantee that it can find an optimal policy.

The key contributions of ICE are lossless clustering and incremental expansion. The authors introduced a probabilistic equivalence criterion that can be used to decide whether two individual action-observation histories, ${\theta }_{i,a}$, ${\theta }_{i,b}$ for agent $i$, can be clustered. Especially in problems that have many histories satisfying probabilistic equivalence, ICE can significantly speed up planning. Incremental expansion defines a method of incrementally constructing a search tree by expanding the most promising child ${\phi }^{t+1}=({\phi }^t,{\beta }^{*})$ where ${\beta }^{*}$ is the optimal solution of collaborative Bayesian games. The author defined a node priority which helps find the tight lower bound early by first expanding deeper nodes [2].

ICE [3] is our baseline algorithm, but our methods are different from it as we integrate Q-value function computing with policy solving, so CEMC calculates the Q-value functions as required, and does not have to store and maintain a large matrix of the Q-value function.

3.3. Approximate Q-Value Functions

As mentioned above, ICE uses $\widehat{V}({\phi }^t)=V^{0\dots t-1}({\phi }^t)+H^{t\dots h-1}({\phi }^t)$ to guide the search and $H^{t\dots h-1}({\phi }^t)$ is a heuristically estimated value using approximate Q-value functions. There are three well-known approximate Q-value functions, ${\mathrm{Q}}_{\mathrm{MDP}}$, ${\mathrm{Q}}_{\mathrm{POMDP}}$ and ${\mathrm{Q}}_{\mathrm{BG}}$, which are all approximations for the optimal Q-value function ${\mathrm{Q}}^{*}$. Although the optimal policy can be extracted from ${\mathrm{Q}}^{*}$, it is costly and impractical to compute.

Szer and Charpillet [25] used ${\mathrm{Q}}_{\mathrm{MDP}}$ to solve Dec-POMDP that is approximated as an underlying MDP. This underlying MDP is defined by a single agent which takes joint actions but has the same state set, transition function, and reward function as original Dec-POMDPs. It assumes that the state can be fully observed and the single agent gains more information than Dec-POMDPs do.

${\mathrm{Q}}_{\mathrm{POMDP}}$ [25] is another approximation for ${\mathrm{Q}}^{*}$. In this case, Dec-POMDPs are treated as underlying POMDPs defined by a single agent which takes joint actions and receives joint observations but has the same state set, transition function, observation function, and reward function. It assumes that underlying POMDPs are settings of centralized control, which allow for better coordination. This means that the single agent knows the joint action-observation history while the agents in Dec-POMDPs can only gain local information such as individual actions and observations.

${\mathrm{Q}}_{\mathrm{BG}}$ is proposed by Oliehoek et al. [4], which assumes that the agents know the joint action-observation history ${\overrightarrow{\theta }}^{t-1}$ and the joint action ${\overrightarrow{a}}^{t-1}$ for the previous time step $t-1$. This assumption defines a Bayesian games (BGs) for each (${\overrightarrow{\theta }}^{t-1}$,$a$) pair. The actions taken by agents can be solved by constructing BGs.

${\mathrm{Q}}_{\mathrm{MDP}}$, ${\mathrm{Q}}_{\mathrm{POMDP}}$ and ${\mathrm{Q}}_{\mathrm{BG}}$ are all proved to be upper bounds of ${\mathrm{Q}}^{*}$ [4]. These approximate Q-value functions form a hierarchy of upper bounds to ${\mathrm{Q}}^{*}$:

$${\mathrm{Q}}^{*}<{\mathrm{Q}}_{\mathrm{BG}}<{\mathrm{Q}}_{\mathrm{POMDP}}<{\mathrm{Q}}_{\mathrm{MDP}}$$

(4)

This means that all these approximate Q-value functions are admissible heuristic for Dec-POMDPs.

Apart from the Q-value functions mentioned above, Oliehoek et al. [18] proposed a factored Q-value function as a heuristic payoff function for factored Dec-POMDP. The precondition of a decomposition is that the Q-value function of every joint policy is factored. Each local component of this kind of Q-value function depends on a subset of state factors and the action-observation histories and actions of a subset of agents.

3.4. Monte Carlo Tree Search (MCTS)

MCTS builds a search tree incrementally and asymmetrically in a given domain by taking a random sample. Each node in the tree denotes a state and links to child nodes which represent actions taken. It works by repeating selection, expansion, simulation, and backpropagation on a loop until some predefined computational budget, usually runtime or memory, is reached [26].

Many extensions have been reviewed in the literature [22]. MCTS is used to find the optimal feature subset through the feature space in [27]. In this paper, we use plain UCT (upper confidence bounds for trees) [6] as the tree policy for computing the Q-value function. Plain UCT is MCTS with UCB1 (a version of upper confidence bound), the tree policy of which is:

$${\overline{x}}_i+2c\sqrt{\frac{2\ln n}{n_i}}$$

(5)

where ${\overline{x}}_i$ is the mean value; $n$ is the number of times that the parent node has been visited; $n_i$ is the number of times child $i$ has been visited, and $c>0$ is a constant.

The role of a tree policy is to select and create a node, which is carried out in the selection and expansion phase. A default policy is in the simulation phase. Its function is to play out the domain to estimate a state value. In a previous paper [28], four criteria for selecting actions are listed: (i) max child, (ii) robust child, (iii) max-robust child, and (iv) secure child. If the search terminates, one can choose one of the four selection mechanisms above for selecting an action to take. Our tree policy chooses the highest value node in the selection phase and only one node is expanded in the expansion phase. In the default policy (simulation phase), we replaced a random sample with simulation enhancement best-first policy. The advantage of randomly selecting actions is that it is simple, requires no domain knowledge, and can quickly simulate many times. However, it is not likely to be realistic compared to rational players [22]. The enhancement we proposed can use domain knowledge to simulate realistically. The details will be introduced in Section 3.3.

3.5. Collaborative Bayesian Games

A Bayesian game (BG) [29] is an augmented normal form game in which a number of agents interact in a one-shot manner, and each of them holds some private information. This private information defines the type of the agent. A collaborative Bayesian game (CBG) is a BG in which agents receive identical payoffs. A CBG is defined as follows:

Definition 2.

A collaborative Bayesian game (CBG) [3,4] $B(b^0,{\phi }^t)=<I,A,\mathsf{\Theta },\mathrm{Pr}(\cdot ),u>$ modeling stage $t$ of a Dec-POMDP, given initial state distribution $b^0$ and past joint policy ${\phi }^t$, consists of:

• $I=\{1,2,\dots ,n\}$ is the finite set of agents;
• $A$ is the set of joint actions;
• $\mathsf{\Theta }$ is the set of joint types, each of which specifies a type for each agent $\theta =<{\theta }_1,{\theta }_2,\dots ,{\theta }_n>$;
• $\mathrm{Pr}(\cdot )$ is a probability distribution over joint types;
• $u$ is a heuristic payoff function mapping joint types and joint actions to a real number: $u(\theta ,a)$.

That is, given $b^0$, for each ${\phi }^t$, it is possible to construct a CBG $B(b^0,{\phi }^t)$. Therefore, $B(b^0,{\phi }^t)$ represents a decision-making problem for stage $t$, if ${\phi }^t$ was followed for the first $t$ stages starting from $b^0$. We simply write $B({\phi }^t)$ below.

In a CBG, each agent uses a Bayesian game policy ${\beta }_i$ that maps individual types to actions ${\beta }_i({\theta }_i)=a_i$. If $\beta$ denotes the joint policy for the CBG, it corresponds to a joint decision rule: $\beta \leftrightarrow {\delta }^t$. Oliehoek et al. [4] model Dec-POMDP with series of CBGs, or in other words, a stage of a Dec-POMDP can be modeled with a CBG. Therefore, we can solve a Dec-POMDP problem stage by stage, which facilitates A*-like search algorithms.

4. Monte Carlo Q-Value Function

In this section, we propose a new Q-value function representation—Monte Carlo Q-value function ${\mathrm{Q}}_{\mathrm{MC}}$—and give proof of an upper bound of the optimal Q-value function ${\mathrm{Q}}^{*}$. ${\mathrm{Q}}_{\mathrm{MC}}$ combines the precision of tree search and the anytime character of sample-based Monte Carlo, and has more flexibility than other Q-value functions do. Then, we present two MCTS enhancements—heavy playout for simulation policy and adaptive sample—to speed up the computation of ${\mathrm{Q}}_{\mathrm{MC}}$.

4.1. ${\mathrm{Q}}_{\mathrm{MC}}$

In reinforcement learning [30], the Q-value function $Q(s,a)$ is defined as the expected return starting from state $s$, taking action $a$, and then following policy $\pi$:

$$\begin{array}{ll}{Q}^{\pi }(s,a)& ={\mathbb{E}}_{\pi }[R_{t+1}+\gamma Q^{\pi }(S_{t+1},a^{\prime })|S_t=s,A_t=a]\\ & ={\displaystyle \sum _{s^{\prime },r}}p(s^{\prime },r|s,a)[r+\gamma Q(s^{\prime },a^{\prime })].\end{array}$$

(6)

Different from the former, the state in ${\mathrm{Q}}_{\mathrm{MC}}$ is the joint history ${\overrightarrow{\theta }}_t$ at time step $t$. We define ${\mathrm{Q}}_{\mathrm{MC}}$ as:

$${\mathrm{Q}}_{\mathrm{MC}}({\overrightarrow{\theta }}^t,\overrightarrow{a})=R({\overrightarrow{\theta }}^t,\overrightarrow{a})+{\displaystyle \sum _{{\overrightarrow{o}}^{t+1}\in O}P({\overrightarrow{o}}^{t+1}|{\overrightarrow{\theta }}^t,\overrightarrow{a})}\underset{MC({\overrightarrow{o}}^{t+1})}{\max }{Q}_{\mathrm{MC}}({\overrightarrow{\theta }}^{t+1},MC({\overrightarrow{o}}^{t+1}))$$

(7)

where MC is the Monte Carlo method, which returns the optimal action $MC({\overrightarrow{o}}^{t+1})$ when observing ${\overrightarrow{o}}^{t+1}$. Because of the partially observability, the probability multiplied by ${\mathrm{Q}}_{\mathrm{MC}}$ is a conditional probability of all observations at ${\overrightarrow{\theta }}_t$ and $\overrightarrow{a}$.

The solution of ${\mathrm{Q}}_{\mathrm{MC}}$ can be illustrated in Figure 1. The black filled nodes are joint histories, and the solid links and dotted links are respectively joint actions taken and joint observations received by agents. Each joint history node contains two statistical values—$N(\overrightarrow{\theta })$, visiting count of node $\overrightarrow{\theta }$ and $N(\overrightarrow{\theta },\overrightarrow{a})$, number of taking $\overrightarrow{a}$ at $\overrightarrow{\theta }$.

At the selection phase of ${\mathrm{Q}}_{\mathrm{MC}}$, UCB1 is used to choose the best action $a^{*}$

$$a^{*}=\underset{\overrightarrow{a}\in A}{\mathrm{argmax}}\left\{R+\widehat{R}+c\sqrt{\frac{2\ln N(\overrightarrow{\theta })}{N(\overrightarrow{\theta },\overrightarrow{a})}}\right\}$$

(8)

where $R$ and $\widehat{R}$ are respectively the immediate and future reward at .., and the term added to it is called a UCT bonus, which is used to balance exploitation and exploration. $R+\widehat{R}$ is the mean value of reward. The $c$ parameter is a constant ($c>0$), which is used to tune how much influence the bonus has on the action selection. It is set bigger for problems with more branches and smaller for deeper ones. We did not consider adaptive setting of parameter $c$ in this paper.

At the expansion phase of ${\mathrm{Q}}_{\mathrm{MC}}$, it expands only one node. At the simulation phase, ${\mathrm{Q}}_{\mathrm{MC}}$ replaces the default random policy with heavy playout—an enhancement of Monte Carlo, which is presented in Section 3.3. At the backpropagation phase, ${\mathrm{Q}}_{\mathrm{MC}}$ updates the values or statistics of nodes that are visited in this iteration with the result of the finished play.

Lemma 1.

${\mathrm{Q}}_{\mathrm{MC}}$is an upper bound of the optimal Q-value function Q*.

Proof 

See appendix. ☐

From the representation of ${\mathrm{Q}}_{\mathrm{MC}}$ we can see that it treats Dec-POMDPs as models in which a single agent takes joint actions and receives joint observations, but has the same state set $S$, transition function $T$, observation function $O$ and reward function $R$. The assumption makes more information available than the underlying Dec-POMDPs. ${\mathrm{Q}}_{\mathrm{MC}}$ is an overestimation of the optimal Q-value function and makes A*-like search find the optimal policy. The pseudocode of computing ${\mathrm{Q}}_{\mathrm{MC}}$ is shown in Algorithms 1 and 2.

4.2. Adaptive Sample

Due to the anytime character of MCTS, sufficient sample times ($K$) can make ${\mathrm{Q}}_{\mathrm{MC}}$ converge to ${\mathrm{Q}}^{*}$, but it is costly to calculate a converged ${\mathrm{Q}}_{\mathrm{MC}}$. However, a small $K$ leads to a loose Q-value function or even a wrong policy. Thus, in this article, we present a method for an adaptive sample, which is a function of joint actions and time steps. That is:

$$K=m\frac{\left|A\right|}{{(t+1)}^2}$$

(9)

where $m$ is a constant, determined by the specific problem, $\left|A\right|$ is the number of joint actions and $t$ is a time step.

$K$ is in proportion to joint actions and inverse proportion to the square of time steps. This means that problems with more joint actions need larger $K$ to make ${\mathrm{Q}}_{\mathrm{MC}}$ converge. With the tree of ${\mathrm{Q}}_{\mathrm{MC}}$ constructed from top–down, $K$ shrinks gradually. We use this technique to balance the accuracy of convergence and computing time.

4.3. Best-First Simulation Policy

Simulation policy uses a default policy to play a game quickly. After that, MCTS propagates back the simulation results. The default policy is typically domain dependent. As there is no value like $Q({\overrightarrow{\theta }}^t,\overrightarrow{a})$ for guiding the action selection, uniformly choosing available actions is a common scheme. However, the heuristic knowledge of the problem can be involved into the default policy to strengthen simulation. Silver [31] discusses some factors that influence simulation performance. In this paper, we extend the rollout phase by incorporating domain knowledge into playouts, which are called heavy playouts [32]. If it is not the last time step, the algorithm recursively calls for the Rollout function. Then it returns the best immediate reward (immR) and backpropagates the value obtained and updates $N(\overrightarrow{\theta })$ and $N(\overrightarrow{\theta },\overrightarrow{a})$. This Best-First Simulation Policy (BFSP) returns the promising node first to ensure that the heuristic value is admissible or a guaranteed overestimation. The detail is shown in Algorithm 3.

Especially we would like to note that the selectBestA() function selects the best action combining an idea of reinforcement learning and the history heuristic. Reinforcement learning (RL) reinforces the actions that make success. The history heuristic assumes that an action good in one position may be good in another. Joint actions, which acquire the highest simulating value, will be given more chances in the later simulation. Concretely, we keep an average accumulated value $v(\overrightarrow{a})$ of each joint action in a playout. Apart from $v(\overrightarrow{a})$, a separate visit of each action $n_i$ and total visits of all actions, $n$ in the simulation phase is also stored. An action is selected by a softmax function with inputs $z=v(\overrightarrow{a})\cdot \frac{n_i}{n}$. This implies that the larger $z$ will correspond to a larger probability. If it is at an initial state or the chance of nodes are equal, a random legal action will be selected. For example, if a problem has nine joint actions, in the start of a simulation phase, an action is selected through this softmax function. After the simulation ends, its accumulated value to the terminal is recorded and the visit number increases by one. In another simulation phase, the statistics are reused, which we call the history heuristic.

5. Multi Agent Planning Using ${\mathrm{Q}}_{\mathrm{MC}}$

In this section, we present our offline planning algorithm—CEMC—which is based on an ICE framework. Then, we introduce a new policy pruning strategy to speed up planning. CEMC integrates Q-value function computing and plan solving, and calculates the Q-value functions as required, which avoids the need to compute and store all Q-value functions. Thus, CEMC does not calculate and maintain a large matrix for Q-value, which achieves improvement in runtime and memory.

5.1. Offline Planning Algorithm—CEMC

Offline algorithms generate a complete policy before execution begins. The policy specifies what actions to take in any possible runtime situation. Before presenting CEMC, we first introduce two successful techniques used in ICE, which is also adopted by CEMC.

The first one is lossless clustering. Two individual action-observation histories ${\theta }_{i,a}$, ${\theta }_{i,b}$ for agent $i$ can be losslessly clustered when they satisfy probabilistic equivalence criterion:

$$\forall {\overrightarrow{\theta }}_{\ne i},\forall s\begin{array}{cc},& P(s,{\overrightarrow{\theta }}_{\ne i}|{\theta }_{i,a})\end{array}=P(s,{\overrightarrow{\theta }}_{\ne i}|{\theta }_{i,b})$$

(10)

where ${\overrightarrow{\theta }}_{\ne i}$ is the joint action-observation histories of all agents except agent $i$. This means that the probability distribution over the other agent’s joint action-observation histories are identical for both ${\theta }_{i,a}$, ${\theta }_{i,b}$.

The second one is incremental expansion which defines a method of incrementally constructing a search tree by expanding the most promising child ${\phi }^{t+1}=({\phi }^t,{\beta }^{*})$ where ${\beta }^{*}$ is the optimal solution of CBGs. A child is selected by node priority. Node $q$ in the search tree is a 3-tuple $<{\phi }^t,\widehat{v},h>$. Node priority $q<q^{\prime }$ if:

$$\{\begin{array}{l}\widehat{v}<{\widehat{v}}^{\prime }\hfill \\ h<h^{\prime }\hfill \end{array}\begin{array}{cc},& \\ ,& \end{array}\begin{array}{l}\mathrm{if} \widehat{v}\ne {\widehat{v}}^{\prime }\hfill \\ \mathrm{otherwise} \mathrm{if} h\ne h^{\prime }\begin{array}{l}\begin{array}{l}\hfill \end{array}\begin{array}{l}.\hfill \end{array}\hfill \end{array}\hfill \end{array}$$

(11)

That is, if the heuristic values are equal, the greater depth will be chosen first. This helps find the tight lower bound early by first expanding deeper nodes and converge quickly.

The CEMC algorithm (Algorithm 4) starts planning with an unspecified partial joint policy ${\phi }^0$ in an initialized policy pool. It greedily selects the best partial joint policy in the pool according to the node priority(Select(L)). Then, it proceeds by constructing a CBG using lossless clustering (ClusterBG()). After that, the CBG is solved, which returns a 2-tupe $\langle {\beta }^t,\widehat{V}({\beta }^t)\rangle$ where ${\beta }^t$ is a Bayesian game policy, and $\widehat{V}({\beta }^t)$ is given by the function Solver():

$$\widehat{V}({\beta }^t)={\displaystyle \sum _{{\overrightarrow{\theta }}^t}P({\overrightarrow{\theta }}^t|b^0,{\phi }^t)}{\mathrm{Q}}_{\mathrm{MC}}({\overrightarrow{\theta }}^t,\beta ({\overrightarrow{\theta }}^t))$$

(12)

where $P({\overrightarrow{\theta }}^t|b^0,{\phi }^t)$ is the probability distribution over joint histories and ${\mathrm{Q}}_{\mathrm{MC}}({\overrightarrow{\theta }}^t,\beta ({\overrightarrow{\theta }}^t))$ is the Q-value function computed by Algorithm 1.

The set of child nodes $N_{\mathrm{IE}}$ is expanded using the incremental expansion technique and only the promising child node ${\beta }^{t,*}$ will be put into $N_{\mathrm{IE}}$(IncExp()). The heuristic value $\widehat{v}$ of the node is given by the function ComputePolicyAndValue():

$$\widehat{V}({\phi }^{t+1})=V^{0\dots t-1}({\phi }^t)+\widehat{V}({\beta }^t)$$

(13)

where $V^{0\dots t-1}({\phi }^t)$ is the actual expected reward of ${\phi }^t$ and $\widehat{V}({\beta }^t)$ is given by Equation (12).

We use two pruning strategies during planning. If time step $t<h-1$, nodes with values less than the best Q-value function ${\mathrm{Q}}_{\mathrm{MC}}^{*}$ will be pruned in advance. If $t=h-1$, CEMC computes fully expanded policies. Nodes with values less than the best lower bound $\underset{\_}{v}$ can be pruned. CEMC ends when the policy pool becomes empty or the highest value in the policy pool is less than that of the best full policy.

The main idea of CEMC is that it integrates the Q-value function computing with plan solving, which benefits from the anytime character of ${\mathrm{Q}}_{\mathrm{MC}}$. The original ICE computes and stores the Q-value functions in a $N_{jaoh}\times N_{ja}$ matrix, where $N_{jaoh}$ is the number of joint action-observation histories and $N_{ja}$ is the number of joint actions. It is costly to calculate complete Q-value functions in advance. For example, for the benchmark problem, Mars Rovers (Appendix

Table A1. Domain parameters and number of $\pi$ for different $h$.

Problem	Domain Parameters				$\mathbf{num}. \mathbf{of} \mathit{\pi }$$\mathbf{for} \mathit{h}$
	n	|S|	|A|	|O|	h = 2	h = 5	h = 10
Dec-Tiger	2	2	3	2	6561	3.43 × 1030	1.39 × 10977
Recycling Robots	2	4	3	2	6561	3.43 × 1030	1.39 × 10977
Grid-Small	2	16	5	2	390625	5.42 × 1044	3.09 × 101431
Box Pushing	2	100	4	5	3.34 × 107	5.23 × 10940	1.25 × 102,939,746
Mars Rovers	2	256	6	8	1.69 × 1014	1.88 × 107285	2.57 × 10238,723,869
Broadcast Channel	2	4	2	2	256	1.84 × 1019	3.23 × 10616

), there are about 2.82 × 10¹⁴ joint action-observation histories and 36 joint actions. It needs to compute about 1.02 × 10¹⁶ $(\overrightarrow{\theta },\overrightarrow{a})$ value pairs. The algorithm CEMC calculates the Q-value functions only when required, without computing and storing all Q-value functions.

5.2. Policy Pruning Strategy

Pruning is split into hard pruning, which removes branches of the tree permanently, and soft pruning, which temporarily removes branches, but then adds them back into the search at some later point [33]. In this paper, we use hard pruning as our pruning strategy.

The ICE algorithm maintains a policy pool. A node in the pool is denoted as a tuple $<\phi ,\widehat{v},h>$, where $\phi$ is a partial policy, $\widehat{v}$ is its value, and $h$ is its depth. We know the value of a node is an actual value plus an estimated value: $\phi =V^{0\dots t-1}({\phi }^t)+\widehat{V}({\beta }^t)$. The estimated value $\widehat{V}({\beta }^t)$ is an overestimation of the actual value from this node to a terminal. Actually, the value of a node $\phi$ is the value of a path or a policy across this node. Therefore, with the search descending deep down to the terminal or the finite horizon, the value of a node is also descending (note that the actual value becomes larger and larger, and the estimated value becomes close to the actual value). When the search reaches the terminal, the value of the terminal node can be seen as the best lower bound ever found. If the value of a node in the policy pool is smaller than the best lower bound, it would never be the optimal. Therefore, it uses the best lower bound $\underset{\_}{v}$ to prune the policy pool. However, for some problems in the literature, the policy pool shrinks slowly, which wastes a lot of time in computing useless nodes.

We use an example to illustrate it. Consider, for example, Figure 2 is a search tree. The policy pool is a priority list, which has several nodes, such as [$\langle {\phi }^2,10.125,2\rangle$,$\langle {\phi }^{1^{\prime }},10.125,1\rangle$,$\langle {\phi }^{2^{\prime }},9.721,2\rangle$,$\langle {\phi }^{2^{″}},6.182,2\rangle$,…]. The superscript of $\phi$ is its time step or its depth. At each loop the ICE algorithm chooses the first node in the pool and expands it. If the algorithm reaches the last step, it returns the fully expanded node ($\langle {\phi }^3,5.363,3\rangle$) as the best lower bound to prune the policy pool. However, node $\langle {\phi }^{2^{″}},6.182,2\rangle$ > node $\langle {\phi }^3,5.363,3\rangle$ according to the node priority, and it will not be pruned. Assuming that the child of the node ${\phi }^{2^{″}}$ is $\langle {\phi }^{3^{\prime }},4.182,3\rangle$, this fully expanded policy is obviously not the optimal one and should have been pruned in advance, but node $\langle {\phi }^{2^{″}},6.182,2\rangle$ will be still expanded in these iterations. Not pruning in advance makes the search algorithm go the wrong direction and this is to be solved in our policy pruning strategy.

Thus, these useless nodes like $\langle {\phi }^{2^{″}},6.182,2\rangle$ should be removed as early as possible. We improve the pruning function using the max Q-value function at time step $t$ ${\mathrm{Q}}_{\mathrm{MC}}^{t,*}$, combined with the best lower bound $\underset{\_}{v}$, that is:

$$\{\begin{array}{l}\{q\in L|\widehat{v}>v^{*}\}\hfill \\ \{q\in L|\widehat{v}>\underset{\_}{v}\}\hfill \end{array}\begin{array}{cc},& \\ ,& \end{array}\begin{array}{l}\mathrm{if} t<h-1\hfill \\ \mathrm{if} t=h-1\hfill \end{array}$$

(14)

where $t$ is the deepest level in the policy pool, $v^{*}$ is computed through ${\mathrm{Q}}_{\mathrm{MC}}^{t,*}$:

$$v^{*}=V^{0\dots t-1}({\phi }^t)+\underset{a}{\max }{\mathrm{Q}}_{\mathrm{MC}}({\overrightarrow{\theta }}^t,a)$$

(15)

The Equation (14) illustrates that if $t<h-1$, all nodes with values less than $v^{*}$ can be pruned and if CEMC reaches the last step, any node with values less than $\underset{\_}{v}$ can be pruned. Note that pruning using $v^{*}$ is done during the planning, it does not have to wait for full expansion. This technique can reduce the number of nodes in the policy pool and make the algorithm converge quickly, which is tested experimentally in Section 5.

6. Experiments

This section experimentally demonstrates and validates the effectiveness of all techniques proposed: the new Q-value function representation ${\mathrm{Q}}_{\mathrm{MC}}$, the adaptive sample, best-first simulation policy (BFSP), policy pruning strategy, and the core algorithm CEMC. We give several benchmark problems in the literature, followed by experimental data. Finally, we analyze and discuss the results.

6.1. Experiments Setup

We give some benchmark problems, including Dec-Tiger, Grid-Small, Recycling Robots, Box Pushing, Mars Rovers, Broadcast Channel. All these problems have two agents, with different numbers of states, actions and observations, as shown in Table A1 (see Appendix A), which lists the number of joint policies for different horizons. These domains can be downloaded at MASPlan (http://masplan.org/).

The problem, Broadcast Channel, has two agents, each of them must choose whether or not to send a message in a Broadcast Channel. If both agents send, a collision occurs. Therefore, they must cooperate to send messages. The problem, Recycling Robots, also has two agents in the environment, in which both of them move around using their motors, pick up cans, and place them in a specified place using their mechanical arms. They must complete the task cooperatively. The benchmark Mars Rovers involve two rovers performing certain scientific experiments by choosing to check, drill, or sample. They can move in four directions. Some locations need only one agent and some locations require their cooperation.

Experiments were run on Ubuntu 16.04 with Intel Core i5 CPU and 6 GB memory. CEMC was implemented in C++ based on the MADP Toolbox [34]. Every average result was run over 100 times. We compared CEMC with the state-of-the-art exact heuristic algorithm ICE with a compact Q-value function implementation ${\mathrm{Q}}_{\mathrm{BG}}$. Five statistics are taken into consideration: time of computing Q-value function $t_Q$, time of calculating the optimal joint policy $t_p$, overall time $t_o$, value of optimal policy $V^{*}$, and value of random policy $V_r$. We would like to note that ICE may not get the same performance contrasted with the original paper due to the computing resources, but all results including ICE and CEMC were tested in the same environment, so the time and scalability results are directly comparable.

6.2. Experiment Results

6.2.1. Comparing CEMC and ICE

Table 1 shows performance results for ICE and CEMC. “-” indicates unknown values, caused by getting the wrong results or a time limit violation (less than 3600 s is valid). “*” indicates out of memory. Table 1 clearly shows that CEMC outperforms the state-of-the-art heuristic algorithm with ${\mathrm{Q}}_{\mathrm{BG}}$. Especially in solving scalable problems such as Grid-Small, Box Pushing, and Mars Rovers, CEMC shows significant improvement without computing and storing all Q-values. CEMC allows for getting the needed Q-value to guide A* search without pre-computation of all Q-value function.

From Table 1 we can see that in some problems like Grid-Small, Box Pushing, and Mars Rovers, CEMC is significantly better than ICE, while in some problems like Dec-Tiger, Recycling Robots, and Broadcast Channel the difference between both may not be as obvious as the three previous ones. Actually, the number of joint decision rules for time step $t$ in the search tree is

$$\mathrm{O}({\left|A\right|}^{n({\left|𝒪\right|}^t)})$$

(16)

which means the complexity is exponential with the number of actions and observations, and doubly exponential with the time step. The search tree contains the action-observation histories, the actions taken, and the observations received. Therefore, the number of branches is related to the action and observation space. Although the domain Mars Rovers has a larger state space, it has little influence on the scale of the search tree. What it influences is the efficiency of state transition and the computing time of the probability like $P(o|a,s^{\prime })$.

6.2.2. Analysis of Policy Pruning Strategy

To illustrate the effectiveness of the extended policy pruning strategy, we list the max number of policies in the policy pool for different horizons. Figure 3 shows a comparison of the original ICE and CEMC. Note that the original ICE maintains a larger policy pool than CEMC. For problems with approximatively $\widehat{V}({\phi }^t)$, ICE expands nodes back and forth to construct a tree carefully. This will significantly increase the search time. We use Equation (14) to prune the policy pool, and the promising nodes are left.

6.2.3. Convergence of ${\mathrm{Q}}_{\mathrm{MC}}$

More sample times can make the approximate Q-value function ${\mathrm{Q}}_{\mathrm{MC}}$ converge and be close to ${\mathrm{Q}}^{*}$, but it needs more runtime. To illustrate the convergence of ${\mathrm{Q}}_{\mathrm{MC}}$, we gathered the statistics of the max ${\mathrm{Q}}_{\mathrm{MC}}$ of Dec-Tiger with the sample time $K$ increasing, as shown in Figure 4. The max values of ${\mathrm{Q}}_{\mathrm{MDP}}$, ${\mathrm{Q}}_{\mathrm{POMDP}}$ and ${\mathrm{Q}}_{\mathrm{BG}}$ for Dec-Tiger with horizon $h$ = 3 and $h$ = 4 are also listed. We can see from the figure that the max ${\mathrm{Q}}_{\mathrm{MC}}$ declines and converges as $K$ increases. Note that ${\mathrm{Q}}_{\mathrm{MC}}$ is as compact as ${\mathrm{Q}}_{\mathrm{POMDP}}$ with horizon $h$ = 3. In horizon $h$ = 4, ${\mathrm{Q}}_{\mathrm{MC}}$ is more compact than ${\mathrm{Q}}_{\mathrm{POMDP}}$ but looser than ${\mathrm{Q}}_{\mathrm{BG}}$. The goal of Figure 4 is to show that ${\mathrm{Q}}_{\mathrm{MC}}$ is an upper bound of ${\mathrm{Q}}^{*}$ and we can use a less compact Q-value function to reduce time of computing, but it still can guide A* to search the optimal policy.

6.3. Analysis and Discussion

CEMC outperforms the state-of-the-art exact heuristic algorithm ICE with compact Q-value function ${\mathrm{Q}}_{\mathrm{BG}}$ in terms of runtime and memory. There are three reasons for CEMC’s performance.

Firstly, we propose a new Q-value function representation ${\mathrm{Q}}_{\mathrm{MC}}$, which is an upper bound for the optimal Q-value function ${\mathrm{Q}}^{*}$. Q_MC is an anytime Q-value function, which makes it possible to compute as required. Adaptive sample and heavy rollout speed up the computation of ${\mathrm{Q}}_{\mathrm{MC}}$.

Secondly, CEMC calculates the Q-value function as required and does not maintain a large matrix to store all $Q({\overrightarrow{\theta }}^t,\overrightarrow{a})$. For problems with a large joint action-observation space and horizon, maintaining the matrix will run out of memory. In addition, it can be seen from problems, Box Pushing and Mars Rovers, in Table 1 that it is costly to calculate the Q-value function (time of computing Q-value for ICE is more than 1100 s). If not every $({\overrightarrow{\theta }}^t,\overrightarrow{a})$ pair is computed, it will save much time. A tighter upper bound can speed up planning, as it allows the algorithm to prune the policy pool more [4], but a tighter value function increases time of computing Q-value function, which accordingly increases the overall time (total time for computing Q-value and planning).

Thirdly, an extended policy pruning strategy is used in CEMC. The pruning strategy uses the best lower bound ever found $\underset{\_}{v}$ and the best accumulated Q-value function at time step $t$ ${\mathrm{Q}}_{\mathrm{MC}}^{t,*}$ to prune the policy pool. This technique avoids expanding those nodes that will not be the optimal ones in any case.

7. Conclusions

In this paper, we combine the advantages of the heuristic algorithm and MCTS, and propose a new Q-value function representation and a novel offline algorithm CEMC, generalizing the prior method. In particular, we demonstrate two important results: (i) Computation of the Q-value function as required is an important reason for the CEMC’s performance. This permits us no longer to maintain a large matrix to store all Q-value function pairs, which saves much memory; (ii) There is no need to compute Q-value functions as compact as possible. Experimental results demonstrate that it is costly to compute compact Q-value functions. Other techniques such as an adaptive sample, heavy rollout, and policy pruning strategy are introduced to speed up the algorithm. In the future work, we plan to analyze the relationship between the loose extent and sample times in order to find the smallest $K$ to optimally solving Dec-POMDPs.