Environmental-Impact-Based Multi-Agent Reinforcement Learning

Abstract

To promote cooperation and strengthen the individual impact on the collective outcome in social dilemmas, we propose the Environmental-impact Multi-Agent Reinforcement Learning (EMuReL) method where each agent estimates the “environmental impact” of every other agent, that is, the difference in the current environment state compared to the hypothetical environment in the absence of that other agent. Inspired by the inequity aversion model, the agent then compares its own reward with that of its fellows multiplied by their environmental impacts. If its reward exceeds the scaled reward of one of its fellows, the agent takes “social responsibility” toward that fellow by reducing its own reward. Therefore, the less influential an agent is in reaching the current state, the more social responsibility is taken by other agents. Experiments in the Cleanup (resp. Harvest) test environment demonstrated that agents trained based on EMuReL learned to cooperate more effectively and obtained $54\%$ ($39\%$) and $20\%$ ($44\%$) more total rewards while preserving the same cooperation levels compared to when they were trained based on the two state-of-the-art reward reshaping methods: inequity aversion and social influence.

1. Introduction

Many real-world problems are selfish multi-agent systems with shared facilities, referred to as the common pool resource problems [1]. The agents choose from a number of available actions to use the shared facility and accordingly earn “rewards” from the environment. Using the common resource to maximize its own reward, each agent’s goal will be in potential conflict with that of its fellows. Examples include water resource management [2], real-time traffic flow control [3], coordination of autonomous vehicles [4], and multi-player video games [5]. In the autonomous vehicle problem, streets, intersections, and highways are shared common resources. Each vehicle plans a path to reach its given destination and is rewarded based on how fast it gets there. Ideally, each vehicle should choose the shortest path, but this may lead to congestion. So, to maximize their individual rewards, they need to cooperate. Such situations raise a type of Sequential Social Dilemma (SSD) [6] where the most rewarding strategy for each agent (in the short-term) is to “defect” and exploit the common resource as much as possible, resulting in mutual defection among all agents; however, this results in the depletion of the common resource, and each agent earns (in the long-term) much less than if they would have all “cooperated” and used the common resource up to a certain limit. In SSDs, cooperative or defective behaviors exist not only as atomic actions but also as being temporally extended over each agent’s decision-making “policy”, that is, a strategy that determines what action to choose at each environment state. The question, then, is how to design the agents’ policies to have cooperative agents that maximize the collective reward, i.e., to maximize total reward while minimizing unfairly low rewards earned by cooperators who facilitate the high rewards earned by the “free-riders” who defect.

The many often stochastic factors impacting the environment and their unknown underlying mechanisms challenge the use of traditional modeling techniques in solving SSDs. For example, in the autonomous vehicle problem, the number of vehicles in each lane is variable, and great uncertainty is involved in the drivers’ behavior. Moreover, the observation and action spaces of each agent are too large to search through exhaustively or near exhaustively to find the optimal policies. Even if the optimal policies are found, they are not generalizable—a new environment would need new policies.

To tackle these modeling complexities, in addition to model-based structures, Multi-Agent Reinforcement Learning (MARL) proposes a model-free structure where intelligent agents require no prior knowledge about the environment and “learn” the optimal policies directly from interacting with the environment. During the interactions, each agent observes the environment state and executes one of the available actions based on its policy. Consequently, the environment transfers to a new state and provides the agent an extrinsic reward indicating the immediate desirability of the environment state. The agent utilizes these interactions to estimate a state value function that assigns to each state the long-term expected reward that the agent obtains when starting from that state and following its policy. According to this value function, the agent improves its policy to choose those actions that result in the highest-value states [7]. The learned policies may be generalized to or sometimes readily used in unseen environments of the problem.

However, learning policies in SSDs by using only extrinsic rewards has limited performance because each extrinsic reward is agent (rather than collective)-specific and typically selfishly maximized by exploiting other agents, which can lead to mutual defection. This is because a cooperative equilibrium either does not exist or is difficult to find [8,9]. One way to mitigate social dilemmas is to reshape the rewards so that defecting behaviors are no longer an equilibrium. To this end, researchers have enriched extrinsic rewards by the so-called intrinsic rewards [10] that capture aspects of the agents’ behavior that are not necessarily encoded by the extrinsic reward. For each agent k at time step t, this results in its reshaped reward $r_t^k$, defined as a linear combination of its extrinsic reward $e_t^k$ and intrinsic reward $i_t^k$:

$$r_t^k=\alpha e_t^k+\beta i_t^k.$$

for constant scalers $\alpha$ and $\beta$.

Insights into the nature of human social behavior have led to the introduction of two main categories of intrinsic rewards. The first focuses on self-preferencing attributes, including empowerment [11], social influence [12], and curiosity [13,14]. The second is motivated by social welfare emotions, including envy and guilt [15], and empathy [16]. For example, in the inequity aversion (IA) model [15], which is based on envy and guilt, each agent compares its own and fellows’ extrinsic rewards to detect inequities and balances its selfish desire for earning rewards by keeping the differences as small as possible. A disadvantageous inequity occurs when agent k experiences “envy” by earning a lower reward than another agent j, that is, if $e_t^j-e_t^k>0$. Conversely, an advantageous inequity happens when the agent experiences “guilt” by earning a higher reward than another agent j, that is, if $e_t^k-e_t^j>0$. To incorporate its feelings of envy and guilt, agent k averages these comparisons over all other agents, resulting in the following intrinsic reward:

$$\begin{array}{c}\hfill i_t^k=-{\displaystyle \frac{{\alpha }_k}{N-1}}\sum _{j\ne k}max(e_t^j-e_t^k,0)-{\displaystyle \frac{{\beta }_k}{N-1}}\sum _{j\ne k}max(e_t^k-e_t^j,0),\end{array}$$

(1)

where N is the total number of agents and parameters ${\alpha }_k$,${\beta }_k\in \mathbb{R}$ control the agent’s aversion to disadvantageous and advantageous inequities, respectively. Regardless of whether it earns more or less than others, the agent decreases its obtained reward.

Here, the feeling of guilt can be interpreted as a feeling of social responsibility; that is, the agent wants to revise its policy to reduce the reward inequity of itself compared to changing the others who are “worse-performing”. However, the IA model measures the agents’ performance only by their obtained rewards and ignores their roles in establishing the current environment state that resulted in those rewards. This may lead to equal treatments of defectors and cooperators that earn the same reward despite their different contributions to the obtained rewards. Namely, rather than only those who earned less than itself, each agent should also “feel responsible” for those who earned more but did not contribute to reaching the current rewarding environment state. Even among others who are equally less-earned, a higher social responsibility should be dedicated to those who also contributed less to reaching the current state. Similarly, compared to other higher-earning agents, the agent feels that it should work “harder” (by finding a better policy) to earn as much as they do, and this feeling should be stronger toward those who played a more effective role in changing the state.

To take into effect the agents’ roles in reaching the environment states, we propose an environmental impact of each agent defined by the difference between the current local state and the hypothetical one where that agent would have been absent in the previous state. In the autonomous vehicle example, a vehicle’s impact can be the difference in the current congestion in the presence and absence of that vehicle. When performing the comparisons in reshaping the reward function, each agent k computes the impact $d_t^{k,j}$ of every other agent j and scales that agent’s reward $e_t^j$ by its impact $d_t^{k,j}$. We demonstrate the effectiveness of the proposed impact criterion through experiments in the Cleanup and Harvest environments [15,17,18]. The results demonstrate that agents trained by the EMuReL method can learn to cooperate more effectively compared with the IA and Social Influence (SI) [12] methods, that is, they earn a higher collective reward and have a slightly higher cooperation level.

The rest of this paper is organized as follows: the MARL setting in an SSD environment and the basic formulation of the proposed Environmental Impact-based Multi-Agent Reinforcement Learning (EMuReL) approach are explained in Section 2 and Section 3, respectively. Related work is presented in Section 4. Section 5 describes the experiments and comparison results. The discussion is given in Section 6.

2. Background: MARL and Markov Games

A Markov game [19] is a standard framework for modeling MARL problems such as SSDs. It is defined on a state space $\mathcal{S}$, a collection of N agents’ action sets $\mathcal{A}=\{{\mathcal{A}}^1,...,{\mathcal{A}}^N\}$, and a state transition distribution $\mathcal{T}$ preserving the Markov property, that is, the next state of the environment is independent of the past states and depends only on the current environment state and the agents’ applied actions [20,21]. The global state of the environment at time step t is given by $s_t\in \mathcal{S}$. Each agent k observes the environment to some extent as a local state $s_t^k$ and selects an action $a_t^k\in {\mathcal{A}}^k$ based on its policy ${\pi }^k$, that is, the probability distribution of selecting each of the available actions. A joint action ${\mathit{a}}_t$ is the stacking of all agents’ actions $[a_t^1,...,a_t^k,...,a_t^N]$. Applying the joint action ${\mathit{a}}_t$ at the global state $s_t$ causes a transition in the environment state according to the transition distribution $\mathcal{T}\left(s_{t+1}\right|s_t,{\mathit{a}}_t)$. Each agent k then receives a reward $r_{t+1}^k$, that is, originally, the agents’ extrinsic reward $e_{t+1}^k$, or can be reshaped by utilizing some intrinsic reward $i_{t+1}^k$. Using a Reinforcement Leaning (RL) algorithm, the agent evaluates its policy by calculating its associated value function $V^{{\pi }^k}\left(s_t^k\right)=\mathbb{E}\left[R_t^k\right|s_t^k,{\pi }^k]$ over every state $s_t^k$, where $R_t^k={\sum }_{i=0}^{\infty }{\gamma }^i{r}_{t+i+1}^k$ is the cumulative discounted future rewards starting from the current local state $s_t^k$ following policy ${\pi }^k$, where $\gamma \in [0,1]$ is the discount factor. Then, agent k learns to improve its policy such that in each state $s_t^k$, the agent chooses the action that transfers the environment to the next state $s_{t+1}^k$ where $V^{{\pi }^k}$ is maximized.

3. The EMuReL Approach

To define intrinsic rewards, we use the same equation as in the IA model (1) with two main differences. First, as in [15], to take into account the temporally distributed, rather than the single-instant reward, we replace the extrinsic reward $e_t^j$ with the temporal smoothed extrinsic rewards $w_t^j$ defined by

$$\begin{array}{c}\hfill w_t^j=\gamma \lambda w_{t-1}^j+e_t^j\forall t\ge 1,w_0^j=0,\end{array}$$

(2)

where $\lambda \in [0,1]$ is a hyperparameter. Second, when performing the IA comparisons, agent k scales the reward of every agent j by $d_t^{k,j}\in [0,1]$, that is, agent j’s environmental impact in establishing agent k’s local state at time t. Hence, agent k’s intrinsic reward is

$$\begin{array}{c}\hfill i_t^k=-{\displaystyle \frac{{\alpha }_k}{N-1}}\sum _{j\ne k}max(d_t^{k,j}{w}_t^j-w_t^k,0)-{\displaystyle \frac{{\beta }_k}{N-1}}\sum _{j\ne k}max(w_t^k-d_t^{k,j}{w}_t^j,0),\end{array}$$

(3)

where ${\alpha }_k,{\beta }_k\in \mathbb{R}$. The definition of the impact $d_t^{k,j}$ lies in the answer to the following question:

Agent k: “How impactful was the presence of agent j in reaching my current local environment state $s_t^k$?”

So, agent k needs to compare its current local state $s_t^k$ in the presence and absence of agent j. This requires estimating the typically large size and detailed state $s_t^k$ in the absence of agent j with a reasonable accuracy, which may not be practical. Moreover, $s_t^k$ is often an image with many features that may not have been caused by the agents’ actions. Therefore, rather than $s_t^k$, we use a reduced-dimension feature encoding function $\varphi (s_t^k;{\theta }_{\varphi })$, briefly denoted as $\varphi \left(s_t^k\right)$, that encodes the raw state $s_t^k$ into a feature vector by using a deep neural network (Figure A2). This function encodes only those features of the environment state that are influenced by the agents’ actions [14].

Now, inspired by the dropouts technique used in neural network regularization [22] to be able to extract the features of the local state $s_t^k$ in the absence of agent j, we need to estimate $\varphi \left(s_t^k\right)$ when agent j is omitted. Thus, we first define the estimated feature encoding function $\widehat{\varphi }$ that, instead of $s_t^k$, takes the previous joint action ${\mathit{a}}_{t-1}$ and features $\varphi \left(s_{t-1}^k\right)$ as its input and estimates $\varphi \left(s_t^k\right)$ as its output, i.e.,

$$\begin{array}{c}\hfill \widehat{\varphi }(\varphi \left(s_{t-1}^k\right),{\mathit{a}}_{t-1})\approx \varphi \left(s_t^k\right).\end{array}$$

Then, by omitting $a_{t-1}^j$, resulting in the reduced joint action denoted by ${\mathit{a}}_{t-1}^{-j}$, we obtain the estimated features at state $s_t^k$ in the absence of agent j (Figure 1). Taking the norm of the difference of the two estimated feature results in the impact of agent j in view of agent k:

$$\begin{array}{c}\hfill d_t^{k,j}={\displaystyle \frac{1}{2}}{∥\widehat{\varphi }(\varphi \left(s_{t-1}^k\right),{\mathit{a}}_{t-1})-{\widehat{\varphi }}^{-j}(\varphi \left(s_{t-1}^k\right),{\mathit{a}}_{t-1}^{-j}))∥}_2^2,\end{array}$$

where ${\parallel ·\parallel }_2$ is the Euclidean norm and ${\widehat{\varphi }}^{-j}$ is the same as $\widehat{\varphi }$ but when agent j is eliminated. These disparities are scaled using unity-based normalization to bring all values into the range $[0,1]$. We emphasize that the elimination of an agent is different from when the agent is present but performs no operation (NOOP).

To learn the feature encoding function $\varphi$ and its estimation $\widehat{\varphi }$, inspired by the neural network structure of the Social Curiosity Module (SCM) [8] for multi-agent systems, we extend the Intrinsic Curiosity Module (ICM) [14] in the single-agent setup to our multi-agent RL system. We propose the Extended ICM (EICM) by embedding two associated predictors inside the architecture of each agent k (Figure A2).

The first is the forward model representing $\widehat{\varphi }$ by a neural network function f with parameter ${\theta }_F$ that predicts the encoded state $\varphi \left(s_t^k\right)$ based on the previous joint action ${\mathit{a}}_{t-1}$ and encoded state $\varphi \left(s_{t-1}^k\right)$. However, the state $s_t^k$ is the result of applying the agents’ actions ${\mathit{a}}_{t-1}$ on the previous global state $s_{t-1}$, not local state $s_{t-1}^k$. Hence, the forward model needs the global state $s_{t-1}$ or an estimation of it. To this end, we exploit the Model of Other Agents (MOA) [12], which is a neural network embedded in each agent’s architecture, to predict the other agents’ actions based on their previous actions and the previous state, i.e., $P({\mathit{a}}_t\mid s_{t-1}^k,{\mathit{a}}_{t-1})$. Via its internal LSTM state $u_{t-1}^k$, MOA implicitly models the state transition function $\mathcal{T}\left(s_t\right|s_{t-1},{\mathit{a}}_{t-1})$ to estimate the global state $s_t$ (the green nodes in the gray rectangle in Figure A2) or $s_{t-1}$ when the time is shifted negatively by one unit, i.e., $\mathcal{T}\left(s_{t-1}\right|s_{t-2},{\mathit{a}}_{t-2})$. By passing this internal LSTM state to the forward model, we implicitly provide an estimate of the global state $s_{t-1}$. Therefore, the estimation of the encoded state $\varphi \left(s_{t-1}^k\right)$ can be written as follows in terms of the neural network function f:

$$\begin{array}{cc}\hfill \widehat{\varphi }(\varphi \left(s_{t-1}^k\right),{\mathit{a}}_{t-1})& =f(\varphi \left(s_{t-1}^k\right),u_{t-1}^k,{\mathit{a}}_{t-1};{\theta }_F).\hfill \end{array}$$

During the training of the forward model, parameters ${\theta }_F$ and ${\theta }_{\varphi }$ are learned to minimize $L_F$ defined as the discrepancy between the predicted and actual encoded local states:

$$\begin{array}{c}\hfill L_F(\varphi \left(s_t^k\right),\widehat{\varphi }\left(\varphi \left(s_{t-1}^k\right),{\mathit{a}}_{t-1}\right)={\displaystyle \frac{1}{2}}{∥\widehat{\varphi }(\varphi \left(s_{t-1}^k\right),{\mathit{a}}_{t-1})-\varphi \left(s_t^k\right)∥}_2^2.\end{array}$$

(4)

Second, to lead the forward model to extract those features that are more relevant to the agents’ actions, the actions are estimated using the so called inverse model and parameters ${\theta }_{\varphi }$ are tuned by minimizing the loss of the actions estimation. The inverse model is a neural network function g with parameters ${\theta }_I$ that predicts the applied joint action ${\mathit{a}}_{t-1}$ given $\varphi \left(s_{t-1}^k\right)$ and $\varphi \left(s_t^k\right)$ as follows:

$$\begin{array}{cc}\hfill {\widehat{\mathit{a}}}_{t-1}& =g(\varphi \left(s_{t-1}^k\right),\varphi \left(s_t^k\right),u_{t-1}^k;{\theta }_I).\hfill \end{array}$$

Then, parameters ${\theta }_I,{\theta }_{\varphi }$ are learned to minimize $L_I({\widehat{\mathit{a}}}_{t-1},{\mathit{a}}_{t-1})$, where $L_I$ is a cross entropy over the predicted and actual actions of the agents:

$$\begin{array}{c}\hfill L_I({\widehat{\mathit{a}}}_{t-1},{\mathit{a}}_{t-1})=-\sum _{j=1}^N{a}_{t-1}^{j}log\left({\widehat{a}}_{t-1}^j\right).\end{array}$$

(5)

The inverse model assists in learning a feature space that encodes only the relevant information to predict the actions of the agents, and the forward model makes the representation of these learned features more predictable. Thus, the feature space has no incentive to encode environmental features that are not affected by the agents’ actions.

4. Related Work

There are several MARL approaches to design intrinsic rewards in order to improve cooperation in social dilemmas in the literature. An emotional intrinsic reward is defined based on each agent’s perception about the cooperativeness of its neighbors in social dilemmas [23,24]. This approach assumes the agents’ actions are distinguished in two cooperative and non-cooperative groups.

The IA model [15] is based on inequity aversion, allowing each agent to know others’ rewards, and penalizing those with a much lower or higher reward. Causal influence in the SI method [12] measures the influence of each agent’s actions. It computes the policy change of an agent with respect to the action of another agent. An intrinsic reward based on the episodic memory is defined to learn exploratory policies [25]. Each agent is encouraged to revisit all states in its environment by utilizing k-nearest neighbors over the agent’s recent experience in a memory of all controllable states visited in the current episode. ICM [14] is a curiosity-driven exploration model that predicts the feature representation of the next environment states and uses the prediction error in the feature space as the curiosity-based intrinsic reward.

Other types of environment “impacts”, such as auxiliary reward functions [26] and measures of deviation for some baseline state [27], are defined in the literature to avoid undesired irreversible changes to the environment.

Two influence-based methods are proposed by Wang et al. [28] to solve the multi-agent exploration problem. These methods exploit the interactions among agents and measure the amount of one agent’s influence on another agent’s exploration processes by using mutual information between agents’ trajectories.

5. Experiments

In this section, two SSD games and the result of applying the proposed EMuReL as well as its comparison with some baseline methods are presented.

5.1. Experimental Setup

Multi-agent sequential social dilemmas are divided into two main categories: (1) public goods dilemmas, where providing a shared resource requires each agent to pay a personal cost, and (2) commons dilemmas, where defecting causes the depletion of a shared resource [29]. The Cleanup and Harvest games are examples of these dilemmas, respectively (See Figure A1). In the Cleanup game, there are two geographically separated areas on a two-dimensional grid environment: an apple field and a river. A group of agents moves inside the field and collects apples to obtain rewards. For producing apples, the agents should use their cleaning beam and clean up some of the waste that appears in a river over time. A higher waste level means a lower apple reproducing-rate in the field. Similar to the Cleanup game, the Harvest game includes a discrete grid area as an apple field with a number of agents that collect apples to earn rewards. However, the growth rate of new apples is determined by the apple density in each area of the field. The higher the apple density in an area, the faster new apples grow in that area. When all apples in an area are harvested, none will ever grow back. So, agents should choose an appropriate harvesting rate to both maximize their reward and keep an ongoing apple reproduction rate. In both games, the extrinsic reward function is the same: $+1$ is the reward for collecting each apple. The agents are equipped with a punishment beam as an action to fire others with the cost of −1. An agent hit by this beam will lose 50 rewards. The Cleanup and Harvest environments were developed by Vinitsky et al. [18] as open-source Python codes.

As illustrated in the Schelling diagrams by Hughes et al. [15], these games are SSDs. In the Cleanup game, if an agent defects by staying in the apple field longer without cleaning up the river, it can obtain a higher reward. However, too many agents defecting makes the apple field depleted, resulting in lower future rewards. So, increasing the number of cooperative agents improves the overall long-term reward for a single cooperator. In the Harvest game, if an agent defects and collects all nearby apples quickly, it can receive a higher reward. However, continuing in this manner causes an unproductive empty field in the long term.

To learn the policy and value function, a decentralized setting of MARL with no communication access is used. Each agent uses a neural network for each of the policy and value function based on the structure of the SI method [12], where each network consists of (i) a convolutional layer to handle image inputs ($15\times 15$ pixels images), (ii) some fully connected layers to mix signals of information between each input dimension and each output, (iii) a Long Short Term Memory (LSTM) recurrent layer to create an internal memory, and (iv) some linear layers to provide the full range of output values. To learn the parameters of the two neural networks, the policy gradient algorithm Proximal Policy Optimization (PPO) [30] and the Asynchronous Advantage Actor-Critic (A3C) [31] were used. These algorithms take an actor–critic structure that optimizes the parameters of neural networks of the policy (actor) and the value function (critic) with respect to the gradient of the actor estimated by the critic using gradient ascent.

Here, the results of four methods were compared: (i) the baseline method that uses only the extrinsic rewards; (ii) the IA method [15] that utilizes the reshaped rewards based on the IA intrinsic rewards (1) and the temporal smoothed extrinsic rewards (2), i.e., the same as (3) but when $d_t^{k,j}=1$; (iii) the SI method [12] that enriches the extrinsic rewards with intrinsic rewards derived from social empowerment, having a causal influence on other agents to make them change their policies; and (iv) our proposed EMuReL method that enriches the IA intrinsic rewards with the environmental impacts. The SCM algorithm with the same parameter setting as in Heemskerk [8] was adapted to the proposed EMuReL method in the Cleanup and Harvest environments. Two commonly used algorithms in these environments are PPO and A3C. As the baseline and SI methods with the PPO algorithm are reported to achieve higher collective rewards than with the A3C algorithm in the Cleanup environment [8,12], we chose PPO for this environment. For the Harvest environment, we tested both PPO and A3C algorithms.

For the hyperparameters, the default values recommended by the authors of the PPO algorithm were used [30]. The training batch size and the PPO minibatch size as two effective hyperparameters were set to 96,000 and 24,000, respectively. According to the results reported by Hughes et al. [15], the advantageous-IA agents are more effective in the Cleanup game, and having more agents who are averse to inequity facilitates cooperation. Thus, in this environment, for the IA and EMuReL methods, we considered 5 advantageous-IA agents with $\beta$ equal to $0.05$ and set $\alpha =0$. We conducted 15 experiments of each method with random seeds and without optimizing the hyperparameters. For the Harvest environment, we evaluated both advantageous-IA and disadvantageous-IA agents and conducted 5 experiments of each method. We considered disadvantageous-IA agents with $\alpha$ equal to 5 and set $\beta =0$. Every experiment was performed on a Linux server with 3 CPUs, a P100 Pascal GPU, and 100G RAM and took between 10 and 28 days considering different methods and environments.

5.2. Experimental Results

The total reward received by all agents is considered as a measure that explains how well the agents learned to cooperate [15]. According to

Table 1. Collected rewards in Cleanup and Harvest environments. The collective reward obtained by agents in the last 192,000 and 1,600,000 steps is averaged over 13 and 5 experiments in the Cleanup and Harvest environments, respectively. The results of the IA and EMuReL methods are reported for the advantageous-IA agents. The largest number in each column is in boldface.

Methods	Environment (Algorithm)
	Cleanup (PPO)	Harvest (A3C)
Baseline	484.4	622.2
IA	431.7	494.7
SI	554.2	475.5
EMuReL	663.2	685.5

, when trained based on the EMuReL method, the agents could earn $53.6\%$ and $38.6\%$ more total rewards than when they were trained based on the IA model, $19.7\%$ and $44.2\%$ more than when they were trained based on SI method, and $36.9\%$ and $10.2\%$ more than when they were trained by using only the extrinsic rewards in the Cleanup and Harvest environments, respectively ($percentageincrease={\displaystyle \frac{newvalue-originalvalue}{originalvalue}}\times 100$). According to subplot (a) in Figure 2, the EMuReL method persistently outperformed the other methods starting from step $2.15\times {10}^8$ and had the minimum performance variance. Moreover, the EMuReL method improved the agents’ cooperation while maintaining the equality between the obtained rewards of the agents better than other methods (subplot (b) in Figure 2). The mean equality of all methods is calculated by using the Gini coefficient as $Equality=1-{\displaystyle \frac{{\sum }_{i=1}^N{\sum }_{j=1}^N|R^i-R^j|}{2N{\sum }_{i=1}^N{R}^i}}$ [15].

All four methods performed almost identically in the Harvest environment using the PPO algorithm (see Figure A4 and Figure A5 in Appendix A.1). The same results were reported by Heemskerk [8], comparing the baseline, SI, and SCM methods when the PPO algorithm is applied. Since the results reported by Jaques et al. [12], Hughes et al. [15] for training the agents based on A3C algorithm achieved higher collective rewards compared to our PPO results, we also used the A3C instead of the PPO algorithm to examine the performance of the EMuReL method. We compare the results with four agents in Figure 3. This plot shows that the EMuReL method with advantageous-IA agents (called advantageous EMuReL method) outperformed other methods.

5.3. Ablation Results

An ablation system was also used to test the contribution of each key ingredient of the EMuReL method. The blue curve in Figure 4 shows the result of the EMuReL method where the internal LSTM state $v_t^k$ of agent k’s actor–critic structure was used instead of the internal LTSM state $u_t^k$ of its MOA structure. The red curve depicts the mean collective rewards of the EMuReL method where $e_t^k$ (resp. $e_t^j$) were used instead of $w_t^k$ (resp. $w_t^j$) in (3) (Figure A7 in Appendix A.1 shows the result of each individual experiment). According to this reward comparison, the inclusion of each of the key ingredients of the EMuReL approach, especially the MOA structure, is necessary and contributes to the obtained performance.

6. Discussion

Researchers have developed several MARL methods to train cooperative agents in SSD problems [32,33]. They introduced intrinsic rewards as a stimulus that is internally computed by agents to accelerate and improve their learning process [34]. IA and SI are two state-of-the-art examples of reward reshaping methods that are based on two social concepts: inequity aversion and social influence [12,15]. We propose the EMuReL method based on the social responsibility concept. To this end, we define the environmental impact as a criterion to measure the role of each agent in reaching the current environment state and, in turn, making the agents continuously measure the cooperativeness of their fellows. We incorporate these impacts into the reward function of the IA method. So, in the advantageous case, the more the agents play an impactful role in reaching the current environment state, the less each agent feels socially responsible for them and, as a result, the less it penalizes itself with negative internal rewards induced by the IA method.

To compute the agents’ impacts, inspired by the SCM method [8], we propose the EICM structure, which extends the single-agent curiosity setup of the ICM method [14] to the MARL setting. The EICM structure utilizes the representation function $\varphi$ to isolate the impact of each agent’s action on the current environment state and to make the agents “curious” about the behavior of others by assessing their environmental impacts. The EMuReL method achieved better results compared to the IA and SI methods in the Cleanup and Harvest environments.

In the Cleanup environment, as [15] explains, the (advantageous) IA method encourages the agents to contribute and clean up the river. Cleaning up the waste causes producing more apples in the field and makes other agents more successful in collecting more apples. So, it reduces the negative rewards fed by the IA method. Here, incorporating the environmental impacts in the IA method improves the agents’ policies. Depending on what action each agent chooses, its impact on the environment is remarkably different in this environment. If the agent chooses one of the movement actions without collecting apples, it creates the minimum change in the environment state, which is simply the change of its location. Such agents are subject to social responsibility by the other agents. If, however, the movement action is with collecting an apple, the field conditions additionally change. The most impactful agents are those who clean the river. Such an agent, in addition to changing the condition of the river on a large scale, changes the condition of the field by producing several apples depending on the growth rate of the apples.

After completing the learning phase of the EMuReL method in the Cleanup environment, the behavior of the agents were investigated in the rendered videos for the polices learned in one of the last episodes. It was observed that one of the agents learned a circular movement pattern in the environment so that it traveled the entire length of the river and then entered the apple field. On the way back to the river, it collected the apples. This agent also followed a specific pattern to clean up the river. It cleaned the maximum possible width of the river whenever it applied the cleaning beam and cleaned the whole width of the river every time it crossed the river. When returning to the river area, it chose the shortest path according to the position of the apples in the field. The other agents maneuvered in the apple field and collected apples. None of the agents in their behavior pattern used the fining beam; neither did they prevent each other from moving in the direction of the apples.

Limitations and Future Work

The proposed algorithm is distributed in the sense that no central agent is needed to learn the policies–each agent has its own MOA model. Nevertheless, the agents require access to all agents’ actions. Relaxing this constraint to the case where the state in the absence of a certain agent is predicted using the actions of the local agents only is the subject of future work. Other limitations include lacking comparisons to other more recent MARL algorithms and environments, long required training time, and considering only a discrete (rather than continuous) action space.