Integrating Production Planning with Truck-Dispatching Decisions through Reinforcement Learning While Managing Uncertainty

Abstract

This paper presents a new truck dispatching policy approach that is adaptive given different mining complex configurations in order to deliver supply material extracted by the shovels to the processors. The method aims to improve adherence to the operational plan and fleet utilization in a mining complex context. Several sources of operational uncertainty arising from the loading, hauling and dumping activities can influence the dispatching strategy. Given a fixed sequence of extraction of the mining blocks provided by the short-term plan, a discrete event simulator model emulates the interaction arising from these mining operations. The continuous repetition of this simulator and a reward function, associating a score value to each dispatching decision, generate sample experiences to train a deep Q-learning reinforcement learning model. The model learns from past dispatching experience, such that when a new task is required, a well-informed decision can be quickly taken. The approach is tested at a copper–gold mining complex, characterized by uncertainties in equipment performance and geological attributes, and the results show improvements in terms of production targets, metal production, and fleet management.

1. Introduction

In short-term mine production planning, the truck dispatching activities aim to deliver the supply material, in terms of quantity and quality, being extracted from the mining fronts by the shovels to a destination (e.g., processing facility, stockpile, waste dump). The dispatching decisions considerably impact the efficiency of the operation and are of extreme importance as a large portion of the mining costs are associated with truck-shovel activities [1,2,3,4]. Truck dispatching is included under fleet optimization, which also comprises equipment allocation, positioning shovels at mining facies and defining the number of trucks required [2,5,6]. Typically, the truck dispatching and allocation tasks are formulated as a mathematical programming approach whose objective function aims to minimize equipment waiting times and maximize production [7,8,9,10,11]. Some methods also use heuristic rules to simplify the decision-making strategy [12,13,14]. In general, a limiting aspect of the structure of these conventional optimization methods is related to the need to reoptimize the model if the configuration of the mining complex is modified, for example, if a piece of fleet equipment breaks. Alternatively, reinforcement learning (RL) methods [15] provide means to make informed decisions under a variety of situations without retraining, as these methods learn from interacting with an environment and adapt to maximize a specific reward function. The ability to offer fast solutions given multiple conditions of the mining complex is a step towards generating real-time truck dispatching responses. Additionally, most methods dealing with fleet optimization are applied to single mines, whereas an industrial mining complex is a set of integrated operations and facilities transforming geological resource supply into sellable products. A mining complex can include multiple mines, stockpiles, tailing dams, processing routes, transportation systems, equipment types and sources of uncertainty [16,17,18,19,20,21,22,23,24,25,26,27].

The truck-dispatching model described herein can be viewed as a particular application belonging to the field of material delivery and logistics in supply chains, commonly modelled as vehicle routing problems and variants [28,29,30]. Dynamic vehicle routing problems [31,32] are an interesting field which allows for the inclusion of stochastic demands [33] and situations where the customer’s requests are revealed dynamically [34]. These elements can also be observed in truck-dispatching activities in mining complexes, given that different processors have uncertain performances and that production targets may change, given the characteristics of the feeding materials. One particularity of the truck-dispatching model herein is that the trips performed between shovels and destinations usually have short lengths and are repeated multiple times. Another important aspect is that uncertainty arises from the geological properties of the transported materials and the performance of different equipment. Over the last two decades, there is an effort to develop frameworks accommodating uncertainties in relevant parameters within the mining complex operations to support more informed fleet management decisions. Not accounting for the complexity and uncertainties inherent to operational aspects misrepresent queue times, cycling times and other elements, which inevitably translates to deviation from production targets [6,35]. Ta et al. [9] allocate the shovels by a goal programming approach, including uncertainties in truckload and cycle times. Few other approaches optimize fleet management and production scheduling in mining complexes under both geological and equipment uncertainty [22,36,37].

A common strategy to model the stochastic interactions between equipment and processors in an operating mining environment is through the use of discrete event simulation (DES) approaches [35,38,39,40,41,42,43]. The DES allows for replacing an extensive mathematical description or rule concerning stochastic events by introducing randomness and probabilistic parameters related to a sequence of activities. The environment is characterized numerically by a set of observable variables of interest, such that each event modifies the state of the environment [44]. This simulation strategy can be combined with ideas from optimization approaches. Jaoua et al. [45] describe a detailed truck-dispatching control simulation, emulating real-time decisions, coupled with a simulated annealing-based optimization that minimizes the difference between tonnage delivered and associated targets. Torkamani and Askari-Nasab [35] propose a mixed integer programming model to allocate shovels to mining facies and establish the number of required truck trips. The solution’s performance is assessed by a DES model that includes stochastic parameters such as truck speed, loading and dumping times, and equipment failure behavior. Chaowasakoo et al. [46] study the impact of the match factor to determine the overall efficiency of truck-shovel operations, combining a DES and selected heuristics maximizing production. Afrapoli et al. [47] propose a mixed integer goal programming to reduce shovel and truck idle times and deviations from production targets. A simulator of the mining operations triggers the model to be reoptimized every time a truck requires a new allocation. Afrapoli et al. [11] combine a DES with a stochastic integer programming framework to minimize equipment waiting times.

It is challenging to formulate all the dynamic and uncertain nature of the truck-shovel operation into a mathematical formulation. The daily operations in a mining complex are highly uncertain; for example, equipment failure, lack of staff or weather conditions can cause deviations in production targets and cause modifications in the dispatching policy. These events change the performance of the mining complex; thus, the related mathematical programming model needs to be reoptimized. The use of DES of the mining complex facilitates the modelling of such events. Note that some of the above mentioned approaches simulate the mining operations to assess the dispatching performance or improve it, using heuristic approaches. This strategy can generate good solutions, but the models do not learn from previous configurations of the mining complex.

Unlike the in the mentioned heuristic approaches, RL-based methods can take advantage of a mining complex simulator to define agents (decision-makers) that interact with this environment based on actions and rewards. The repetition of such interaction provides these agents with high learning abilities, which enables fast responses when a new assignment is required. Recent approaches have achieved high-level performances over multiple environments that require complex and dynamic tasks [48,49,50,51,52,53,54,55]. They have also been applied to some short-term mine planning aspects showing interesting results [56,57,58].

This paper presents a truck-dispatching policy based on deep Q-learning, one of the most popular RL approaches, in order to improve daily production and overall fleet performance, based on the work in Hasselt et al. [50]. A DES is used to model daily operational aspects, such as loading, hauling and dumping activities, generating samples, to improve the proposed truck dispatching policy. A case study applies the method to a copper–gold mining complex, which considers equipment uncertainty, modelled from historical data, and orebody simulations [59,60,61,62,63] that assess the uncertainty and variability related to metal content within the resource model. Conclusion and future work follow.

2. Method

The proposed method adapts the deep double Q-learning neural network (DDQN) method [50] for dispatching trucks in a mining environment. The RL agents continually take actions over the environment and receive rewards associated with their performances [15]. Herein, each mining truck is considered an agent; therefore, these terms are used interchangeably throughout this paper. The DES, described in Section 2.1, receives the decisions made by the agents, simulates the related material flow and a reward value evaluating each action. Section 2.2 defines the reward function and how the agents interact with the RL environment; where the observed states and rewards compose the samples used to train the DDQN. Subsequently, Section 2.3 presents the training algorithm based on Hasselt et al. [50], updating the agent’s parameters (neural network weights). Figure 1 illustrates the workflow showing the interaction between the DES and the DDQN policy.

2.1. Discrete Event Simulator

The discrete event simulator presented in this work assumes a predefined sequence of extraction, the destination policy of each mining block and the shovel allocation. It also presumes that the shortest paths between shovels and destinations have been defined. Figure 2 illustrates this relationship where the black arrow is the predefined destination path for the block being extracted by the shovel. After the truck delivers the material to the dumping point (waste dump, processing plant or leaching pad, for example), a dispatching policy must define the next shovel assignment. The red arrow illustrates the path options for dispatching.

To simulate the operational interactions between shovels, trucks and dumping locations present in the mining complex, the DES considers the following major events:

Shovel Loading Event: The shovel loads the truck with an adequate number of loads. The total time required for this operation is stochastic, and once the truck is loaded, it leaves the shovel as the destination, triggering the “Truck Moving Event.” If the shovel must move to a new extraction point, it incurs a delay, representing the time taken to reposition the equipment. After the truck leaves the loading point, this event can trigger itself if there is another truck waiting in the queue.

Truck Moving Event: This event represents the truck going from a shovel to a dumping location, or vice versa. Each travelling time is sampled from a distribution approximated from historical data. Travelling empty or loaded impacts on the truck speed, meaning that time values are sampled from different distributions in these situations. When the truck arrives at the loading point and the shovel is available, this event triggers a “Shovel Loading Event”; otherwise, it joins the queue of trucks. If the truck arrives at the dumping location, the event performs similarly; if the destination is empty, this event triggers a “Truck Dumping Event,” otherwise, the truck joins the queue of trucks.

Truck Dumping Event: This event represents the truck delivering the material to its destination, to a waste dump or a processing plant, for example. The time to dump is stochastic, and after the event is resolved, a “Truck Moving Event” is triggered to send the truck back to be loaded. Here, a new decision can be made, sending the truck to a different shovel. Similar to the “Shovel Loading Event,” once this event is finished, it can trigger itself if another truck is in the queue waiting for dumping.

Truck Breaking Event: Represents a truck stopping its activities due to maintenance or small failures. In this event, a truck is removed from the DES regardless of its current assignment. No action can be performed until it is fixed and can be returned to the operation.

Shovel Breaking Event: Represents the shovel becoming inaccessible for a certain period due to small failures or maintenance. No material is extracted during this period, and no trucks are sent to this location, being re-routed until the equipment is ready to be operational again.

Figure 3a shows a diagram illustrating a possible sequence of events that can be triggered. In the figure, the solid lines represent the events triggered immediately after the end of a particular event. The dashed lines are related to events that can be triggered if trucks are waiting in the queue. To ensure the sequence respects a chronological ordering, a priority queue is maintained, where each event is ranked by its starting time, as illustrated in Figure 3b.

The DES starts with all the trucks being positioned at their respective shovel. This configuration triggers a “Shovel Loading Event,” and the DES simulates the subsequent events and how much material flows from the extraction point to their destinations by the trucks. Once the truck dumps, a new decision is taken according to the DDQN policy. The DES proceeds by simulating the subsequent operations triggered by this assignment. This is repeated until the predefined time horizon, which represents $N_{days}$ of simulated activities, is reached by the DES. All events that occur between the beginning and the end of the DES constitute an episode. Subsequent episodes start by re-positioning the trucks at their initial shovel allocation.

2.2. Agent–Environment Interaction

2.2.1. Definitions

The framework considers $N_{trucks}$ trucks interacting with the DES. At every time step $t\in T$, after dumping the material into the adequate location, a new assignment for truck $i\in N_{trucks}$ is requested. The truck-agent $i$ observes the current state $S_t^i\in S$, where $S_t^i$ represents the perception of truck $i$ on how the mining complex is performing at step $t$ and takes an action $A_t^i\in A$, defining the next shovel to which the truck will be linked. The state $S_t^i$ is a vector encoding all attributes relevant to characterize the current status of the mining complex. Figure 4 illustrates these attributes describing the state space, such as current queue sizes, current GPS location of trucks and shovels, and processing plant requirements. This state information is encoded in a vector and inputted into the DDQN neural network, which outputs action-values, one for each shovel, representing the probability that the truck be dispatched to a shovel-dumping point path. A more detailed characterization of the state $S_t^i$ is given in Appendix A.

2.2.2. Reward Function

Once the agent outputs the action $A_t^i$, the DES emulates how the mining complex environment evolves by simulating, for example, new cycle times, the formation of queues, taking into consideration all other trucks in operation. The environment, then, replies to this agent’s decision with a reward function, represented by Equation (1):

$$R_t^i=per{c}_t^i-p{q}_t^i$$

(1)

where $per{c}_t^i$ is the reward associated with delivering material to the mill and accomplishing a percentage of the destination’s requirement (e.g., mill’s daily target in tons/day). $p{q}_t^i$ is the penalty associated with spending time in queues at both shovels and destinations. This term guides solutions with smaller queue formation while ensuring higher productivity.

In this multi-agent setting, each truck receives a reward $R_t$, which is the sum of each truck $R_t^i$, as shown in Equation (2), to ensure that all agents aim to maximize the same reward function.

$$R_t={\displaystyle \sum }_i^{N_{trucks}}{R}_t^i$$

(2)

During each episode, the agent performs $N_{steps}$ actions, the discounted sum of rewards is the called return presented by Equation (3):

$$G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots +{\gamma }^{N_{steps}-t-1}{R}_{N_{steps}}={\displaystyle \sum }_{k=t+1}^{N_{steps}}{\gamma }^{k-t-1}{R}_k$$

(3)

where $\gamma$ is a discounting factor parameter, which defines how much actions taken far in the future impact the objective function [15]. Equation (4) defines the objective, which is to obtain high-level control by training the agent to take improved actions so that the trucks can fulfil the production planning targets and minimize queue formation.

$$\underset{a\in A}{\max }E\left[G_t|S=S_t^i,A=A_t^i\right]$$

(4)

The environment is characterized by uncertainties related to loading, moving, dumping times of the equipment, breakdowns of both trucks and shovels. This makes it very difficult to define all possible transition probabilities between states $\left(p\left(S_{t+1}^i|S=S_t^i,A=A_t^i\right)\right)$ to obtain the expected value defined in Equation (4). Therefore, these transition probabilities are replaced by the Monte Carlo approach used in the form of the DES.

The framework allows for future actions to be rapidly taken since providing the input vector $S_t$ to the neural network and outputting the corresponding action is a fast operation. This means that the speed at which the decisions can be made depends more on how quickly the attributes related to the state of the mining complex can be collected, which has been recently substantially improved with the new sensors installed throughout the operation.

2.3. Deep Double Q-Learning (DDQN)

The approach used in the current study is the double deep Q-learning (DDQN) approach based on the work of Hasselt et al. [50]. Q-function ${\mathit{Q}}_{\mathit{i}}\left(S_t^i,A_t^i,w_t^i\right)$ is the action-value function, shown in Equation (5), which outputs values representing the likelihood of truck $i$ choosing action $A_t^i$, given the encoded state $S_t^i$ and the set of neural-network weights $w_t^i$, illustrated by Figure 4.

$${\mathit{Q}}_{\mathit{i}}\left(S_t^i,A_t^i,w_t^i\right)=E\left[G_t|S=S_t^i,A=A_t^i\right]$$

(5)

Denote $Q_i^{*}\left(s_t^i,a_t^i,w^i\right)$ to be the theoretical optimal action-value function. Equation (6) presents the optimal policy ${\pi }^{*}\left(S_t^i\right)$ for the state $S_t^i$, which is obtained by using the action-function greedily:

$${\pi }^{*}\left(S_t^i\right)=\underset{a^{\prime }\in A}{\mathrm{argmax}}{Q}_i^{*}\left(S_t^i,a^{\prime },w^i\right)$$

(6)

Note that, using Equation (6), the approach directly maximizes the reward function described in Equation (4). This is accomplished by updating the ${\mathit{Q}}_{\mathit{i}}\left(S_t^i,A_t^i,w_t^i\right)$ function to approximate the optimal action-value function $\left({\mathit{Q}}_{\mathit{i}}\left(S_t^i,A_t^i,w_t^i\right)\to Q_i^{*}\left(S_t^i,A_t^i,w^i\right)\right)$.

By letting agent $i$ interact with the environment, given the state $S_t^i$, the agent chooses $A_t^i$, following a current dispatching policy ${\pi }_i\left(S_t^i\right)=\underset{a^{\prime }\in A}{\mathrm{argmax}}{\mathit{Q}}_{\mathit{i}}\left(S_t^i,a^{\prime },w_t^i\right)$, the environment then returns the reward $R_t$ and a next state $S_{t+1}^i$. The sample experience $e_k^i=\left(S_t^i, A_t^i,R_t,S_{t+1}^i\right)$ is stored in a memory buffer, $D_K^i=\left\{e_1^i,e_2^i,\dots ,e_K^i\right\}$, which is increased as the agent interacts with the environment for additional episodes. A maximum size limits this buffer, and once it is reached, the new sample $e_k^i$ replaces the oldest one. This is a known strategy called experience replay, which helps stabilize the learning process [48,50,64].

In the beginning, ${\mathit{Q}}_{\mathit{i}}\left(S_t^i,A_t^i,w_t^i\right)$ is randomly initialized, then a memory tuple $e_k^i$ is repeatedly uniformly sampled from the memory buffer $D_K^i$, and the related $e_k^i=\left(S_t^i, A_t^i,R_t,S_{t+1}^i\right)$ values are used to estimate the expected future return ${\overline{G}}_t$, as shown in Equation (7):

$${\overline{G}}_t=\{\begin{array}{l}{R}_t, \hspace{1em}\hspace{1em}if episode terminates at t+1\\ R_t+\gamma {\overline{\mathit{Q}}}_{\mathit{i}}\left(S_t^i,\underset{a^{\prime }\in A}{ \mathrm{argmax} }{\mathit{Q}}_{\mathit{i}}\left(S_{t+1}^i,a^{\prime },w_t^i\right), {\overline{w}}^i\right), otherwise\end{array}$$

(7)

Additionally, gradient descent is performed on ${\left({\overline{G}}_t-{\mathit{Q}}_{\mathit{i}}\left(S_t^i,A_t^i,w_t^i\right)\right)}^2$ with respect to the parameter weights $w_t^i$. Note that a different Q-function, ${\overline{\mathit{Q}}}_{\mathit{i}}\left(·\right)$, is used to predict the future reward; this is simply the ${\mathit{Q}}_{\mathit{i}}\left(·\right)$ with the old weight parameters. Such an approach is also used to stabilize the agent’s learning, as noisy environments can result in a slow learning process [50]. After $N_{Updt}$ steps, the weights $w_t^i$ are copied to ${\overline{w}}^i$, as follows: ${\overline{\mathit{Q}}}_{\mathit{i}}\left(·\right)={\mathit{Q}}_{\mathit{i}}\left(·\right)$.

During training, the agent $i$ follows the greedy policy ${\pi }_i\left(S_t^i\right)$ meaning that it acts greedily with respect to its current knowledge. If gradient descent is performed with samples coming solely from ${\pi }_i\left(S_t^i\right)$, the method inevitably would reach a local maximum very soon. Thus, to avoid being trapped in a local maximum, in $ϵ\%$ of the time, the agent takes random actions exploring the solution space, sampling it from a uniform distribution $A_t^i~U\left(A\right)$. In $\left(100-ϵ\right)\%$ of the time, the agent follows the current policy $A_t^i~{\pi }_i\left(S_t^i\right)$. To take advantage of long-term gains, after every $N_{steps\_reduce}$ steps this value is reduced by a factor $reduce\_factor\in \left[0,1\right]$. In summary, the algorithm is presented as follows:

Algorithm 1 Proposed learning algorithm.
Initialize the action-functions.${\mathit{Q}}_{\mathit{i}}\left(·\right)$ and ${\overline{\mathit{Q}}}_{\mathit{i}}\left(·\right)$ by assigning initial weights to $w_t^i$ and ${\overline{w}}^i$.
Set $n{1}_{counter}=0$ and $n{2}_{counter}=0$.
Initialize the DES, with the trucks at their initial locations (e.g., queueing them at the shovel).
Repeat for each episode:
Given the current truck-shovel allocation, the DES simulates the supply material    being transferred from mining facies to the processors or waste dump by the trucks.
Once the truck $i$ dumps the material, a new allocation must be provided.
At this point, the agent collects the information about the state $S_t^i$.
Sample $u ~U\left(0,100\right)$
If $u<ϵ\%$
The truck-agent $i$ acts randomly $A_t^i~U\left(A\right)$
Else:
The truck-agent $i$ acts greedily $A_t^i~{\pi }_i\left(S_t^i\right)$
Taking action $A_t^i$, observe $R_t$ and a new state $S_{t+1}^i$.
Store the tuple $e_k^i=\left(S_t^i, A_t^i, R_t, S_{t+1}^i\right)$ in the memory buffer $D_K^i$
Sample a batch of experiences $e_k^i=\left(S_t^i, A_t^i, R_t, S_{t+1}^i\right)$, of size $batch\_size$, from $D_K^i$:
For each transition sampled, calculate the respective ${\overline{G}}_t$ from Equation (7).
Perform gradient descent on ${\left({\mathit{Q}}_{\mathit{1}}^{\mathit{i}}\left(s_{t+1},a^{\prime },w_1^i\right)-{\overline{G}}_t\right)}^2$ according to Equation (8):
$w_{1,next}^i\leftarrow w_{1,old}^i-\alpha \left({\mathit{Q}}_{\mathit{1}}^{\mathit{i}}\left(s_{t+1},a^{\prime },w_1^i\right)-G_t\right){\nabla }_{w_1^i}{\mathit{Q}}_{\mathit{1}}^{\mathit{i}}\left(s_{t+1},a^{\prime },w_1^i\right)$	(8)
$n{1}_{counter}\leftarrow n{1}_{counter}+1$.
$n{2}_{counter}\leftarrow n{2}_{counter}+1$.
If $n{1}_{counter}\ge N_{Updt}$:
${\overline{w}}^i\leftarrow w_t^i$.
$n{1}_{counter}\leftarrow 0$.
If $n{2}_{counter}\ge N_{step\_reduce}$:
$ϵ\leftarrow ϵ\ast reduce\_factor$.
$n{2}_{counter}\leftarrow 0$.

3. Case Study at a Copper—Gold Mining Complex

3.1. Description and Implementation Aspects

The proposed framework is implemented at a copper–gold mining complex, summarized in Figure 5. The mining complex comprises two open-pits, whose supply material is extracted by four shovels and transported by twelve trucks to the appropriate destinations: waste dump, mill or leach pad.

Table 1. Mining complex equipment and processors.

Equipment	Description
Trucks 12 in total	6 of payload capacity of 200 tons 3 of payload capacity of 150 tons 3 of payload capacity of 250 tons
Shovel 4 in total	2 of bucket payload of 80 tons 1 of bucket payload of 60 tons 1 of bucket payload of 85 tons
Mill	Capacity 80,000 ton/day, with 2 crushers.
Leach Pad	Capacity 20,000 ton/day, with one crusher.
Waste Dump	1 Waste Dump with no limitation on capacity.

presents information regarding the mining equipment and processors. The shovels are placed at the mining facies following pre-defined extraction sequences, where the destination of each block was also pre-established beforehand. The mining complex shares the truck fleet between pits A and B. The waste dump receives waste material from both mines, whereas the leach pad material only processes supply material from pit B due to mineralogical characteristics. The truck going to the leach pad dumps the material into a crusher, then transported it to the leach pad. Regarding the milling material, each pit is associated with a crusher, and the trucks haul the high-grade material extracted from a pit and deliver it to the corresponding crusher. Next, a conveyor belt transfers this material to the mill combining the material from the two sources. Both the mill and the leach pad are responsible for producing copper products and gold ounces to be sold.

The discrete event simulation, described in Section 2.1, emulates the loading, hauling and dumping operations in the mining complex. Each event is governed by uncertainties that impact the truck cycling times.

Table 2. Definition of stochastic variables considered in the mining complex.

Stochastic Variable	Probability Distribution
Loaded truck speed (km/h)	Normal (17, 4)
Empty truck speed (km/h)	Normal (35, 6)
Dumping + maneuver time (min)	Normal (1, 0.15)
Shovel bucketing load time (min)	Normal (1.1, 0.2)
Truck mean time between failures (h)	Poisson (36)
Truck mean time to repair (h)	Poisson (5)
Shovel mean time between failures (h)	Poisson (42)
Shovel mean time to repair (h)	Poisson (4)

presents distributions used for the related uncertainty characterization. For simplicity, these stochastic distributions are approximated from historical data; however, a more interesting approach would have been to use the distribution directly from historical data. When the truck dumps material into a destination, a new dispatching decision must be taken by the DDQN dispatching policy. This generates samples that are used to train the DDQN dispatching policy. During the training phase, each episode lasts the equivalent of 3 days of continuous production, where the truck-agent interacts with the discrete event mine simulator environment, taking actions and collecting rewards. In total, the computational time needed for training, for the present case study, is around 4 h. For the comparison (testing) phase, the method was exposed to five consecutive days of production. This acts as a validation step, ensuring that the agents observe the mining complex’s configurations which were unseen during training. The results presented show the five days of production, and the performance obtained illustrates that the method does not overfit regarding the three days of operation but maintain a consistent strategy for the additional days.

Note that although the DDQN policy provides dispatching decisions considering a different context from the one it was trained, the new situations cannot be totally different. It is assumed that in new situations, the DDQN experiences are sampled from the same distribution observed during training. In the case where the sequence of extraction changes considerably and new mining areas as well as other destinations are prioritized, the model needs to be retrained.

Two baselines are presented to compare the performance of the proposed approach. The first one, referred to as fixed policy, is a strategy that continually dispatches the truck to the same shovel path throughout the episode. The performance comparison between the DDQN and fixed policy is denoted Case 1. The second approach, referred to as greedy policy, sends trucks to needy shovels with the shortest waiting times to decrease idle shovel time, denoted Case 2. Both cases start with the same initial placement of the trucks.

The environment is stochastic, in the sense that testing the same policy for multiple episodes generates different results. Therefore, for the results presented here, episodes of 5 days of continuous production are repeated 10 times for each dispatching policy. To assess uncertainty outcomes beyond the ones arising from operational aspects, geological uncertainty is also included in the assessment by considering 10 orebody simulations (Boucher and Dimitrakopoulos; 2009) characterizing the spatial uncertainty and variability of copper and gold grades in the mineral deposit. The graphs display results in P10, P50 and P90 percentile, corresponding to the probability of 10, 50 and 90%, respectively, of being below the value presented.

3.2. Results and Comparisons

Figure 6 presents the daily throughput obtained by running the DES over the five days of production, which is achieved by accumulating all material processed by the mill within each day. Note that here the P10, P50 and P90 are only due the equipment uncertainty. Overall, the proposed model delivers more material to the mill when compared to both cases. The DDQN method adapts the dispatching to move trucks around, relocating them to the shovels that are more in need, which constantly results in higher throughput.

The throughput in day five drops compared to previous days, mostly due to a smaller availability of trucks as the DES considers failures in the trucks; Figure 7 presents the average number of trucks available per day. During the initial three days, the availability of trucks hovers between 10 and 12 trucks, but this rate drops in the last 2 days, which decreases the production. However, the trained policy can still provide a higher feed rate at the mill, even in this adversity. The availability of trucks on days 4 and 5 is smaller than the period for which the DDQN based method was trained, which shows an adapting capability of the dispatching approach.

The framework is also efficient in avoiding queue formation. Figure 8 presents the average queue sizes at the mill and the sulphide leach. The queue at different locations is recorded hourly and averaged over each day. The plot shows that, for most of the days, the proposed approach generates smaller queues. Combined with the higher throughput obtained, this reduction in queue sizes demonstrates better fleet management. For example, during the initial three days, the DDQN approach improves the dispatching strategy by forming smaller queues at the mill. At the same time, the amount of material being delivered is continuously higher. On the 4th day, the proposed approach generates a larger queue size at the mill, which is compensated by having considerably higher throughput at this location.

Figure 9 displays the cumulative total copper recovered at the mining complex over the five days. Interestingly, during the first three days of DES simulation, corresponding to the training period of the DDQN approach, the total recovered copper profile between the proposed method and the baselines is similar. However, this difference is more pronounced over the last two days, which represents the situation that the trained method has not seen. This results in 16% more copper recovered than the fixed policy and 12% more than the greedy strategy. This difference in results is even larger when the total gold recovered is compared. The DDQN method generates a 20 and 23% higher gold profile in Case 1 and Case 2, respectively, Figure 10.

4. Conclusions

This paper presents a new multi-agent truck-dispatching framework based on a reinforcement learning framework. The approach involves the interaction between a DES, simulating the operational events in a mining complex, and a truck-dispatching policy based on the DDQN method. Given a pre-defined schedule in terms of the sequence of extraction and destination policies for the mining blocks, the method improves the real-time truck-dispatching performance. The DES mimics daily operations, including loading, transportation and dumping, and equipment failures. A truck delivers the material to a processor or waste dump, and the truck-dispatcher provides it with a different shovel path. At this point, the truck receives information about the mining complex, such as other truck locations via GPS tracking, the amount of material feeding the processing plant and queue sizes at different locations. This state information is encoded into a vector, characterizing the state of the mining complex. This vector is inputted into the DDQN neural network, which outputs action values, describing the likelihood to send the truck to each shovel. Each dispatching decision yields a reward, which is received by the agent, as a performance evaluation. Initially, the truck-agent acts randomly; as the agent experiences many situations during training, the dispatching policy is improved. Thus, when new dispatching decisions are requested, an assignment is quickly obtained by the output of the DDQN agent. It differs from previous methods that solve a different optimization repeatedly during dispatching. Instead, the only requirement is to collect information regarding the state of the mining complex. With the digitalization of the mines, obtaining the required information can be done quickly.

The method is applied to a copper–gold mining complex composed of two pits, three crushers, one waste dump, one mill and one leach-pad processing stream. The fleet is composed of four shovels, and twelve trucks that can travel between the two pits. The DDQN-based method is trained for the equivalent of three days, while the results are presented for five days of production. Two dispatching baseline policies are used for comparison to assess the capabilities of the proposed method: fixed truck-shovel allocations and a greedy approach that dispatches trucks to needy shovels with the smallest queue sizes. The results show that the DDQN-based method provides the mill processing stream with higher throughput while generating shorter queues at different destinations, which shows a better fleet utilization. Over the five days of production, the proposed policy produces 12 to 16% more copper and 20 to 23% more gold than the baseline policies. Overall, the reinforcement learning approach has shown to be effective in training truck-dispatching agents, improving real-time decision-making. However, future work needs explore the development of new approaches that address the impact and adaptation of truck-dispatching decisions to changes and re-optimization of short-term extraction sequences given to the acquisition of new information in real-time and uncertainty in the properties of the materials mind.