Solving the Vehicle Routing Problem with Stochastic Travel Cost Using Deep Reinforcement Learning

Abstract

The Vehicle Routing Problem (VRP) is a classic combinatorial optimization problem commonly encountered in the fields of transportation and logistics. This paper focuses on a variant of the VRP, namely the Vehicle Routing Problem with Stochastic Travel Cost (VRP-STC). In VRP-STC, the introduction of stochastic travel costs increases the complexity of the problem, rendering traditional algorithms unsuitable for solving it. In this paper, the GAT-AM model combining Graph Attention Networks (GAT) and multi-head Attention Mechanism (AM) is employed. The GAT-AM model uses an encoder–decoder architecture and employs a deep reinforcement learning algorithm. The GAT in the encoder learns feature representations of nodes in different subspaces, while the decoder uses multi-head AM to construct policies through both greedy and sampling decoding methods. This increases solution diversity, thereby finding high-quality solutions. The REINFORCE with Rollout Baseline algorithm is used to train the learnable parameters within the neural network. Test results show that the advantages of GAT-AM become greater as problem complexity increases, with the optimal solution generally unattainable through traditional algorithms within an acceptable timeframe.

1. Introduction

Combinatorial Optimization (CO) problems optimize a classical and intricate category of decision-making quandaries. These problems involve the fastidious selection of a subset of components from a limited set of choices, with the overarching objective of maximizing or minimizing a foreordained objective function. Over recent years, the significant ascendancy of deep learning (DL) has markedly propelled progress in conventional domains such as image classification, speech recognition, and computer vision. This advancement can be ascribed to the profound adaptability in feature acquisition demonstrated by DL models. These models exhibit the capacity to autonomously glean intricate abstract feature representations from raw data, thus endowing them with adaptability across a myriad of complex tasks. The end-to-end learning paradigm inherent in DL engenders a streamlined trajectory for model formulation. By directly linking inputs to outputs, it obviates the necessity for elaborate feature engineering processes, thereby amplifying the efficiency and precision of data manipulation. In recent years, the ability of DL systems to tackle CO problems has been extensively demonstrated across various applications [1,2]. Despite the significant potential of DL methods in CO problems, their application faces several limitations. These models typically demand a substantial amount of labeled data for training, which are often difficult to acquire. They also demand substantial computational resources, including high-performance GPUs and extensive memory capacities, leading to increased computational costs. Moreover, the generalization ability of DL models can be unstable with unseen data, affecting the reliability of real-world solutions. Additionally, the complexity of these CO problems often necessitates specific model architectures and hyperparameter tuning, complicating design and implementation [2]. DL methods are prone to getting trapped in local optima rather than finding global optima, leading to suboptimal solutions [3]. Lastly, the training duration can be lengthy, especially for complex problems, delaying practical application. Hence, it is crucial to evaluate these advantages and limitations comprehensively and consider integrating other methodologies to enhance solution effectiveness. Based on the characteristics of CO problems, reinforcement learning (RL) [4] is well suited for them because it does not require labeled data like supervised learning does. Furthermore, the performance of deep reinforcement learning (DRL) in discrete sequence decision-making problems such as Alpha Go Zero [5] and Atari [6] demonstrates its powerful learning and optimization capabilities. DRL has been employed to address CO problems, such as the Traveling Salesman Problem (TSP), Vehicle Routing Problem (VRP), and Facility Location Problem (FLP). Encouraging outcomes have been achieved with regards to computational efficiency and generalization. While certain conventional heuristic approaches excel in resolving particular challenges, they frequently entail considerable time expenditures when tackling large-scale instances [7]. Conversely, with the trained neural network, rapid results can be generated, thereby alleviating a substantial portion of the time burden.

DRL is an approach that integrates DL and RL to tackle intractable problems. Deep Q-learning Networks [6] mark the initial revelation of DRL, which expands upon Q-learning [8] to better address high-dimensional state spaces and complex environments. By amalgamating various classical RL algorithms such as REINFORCE [9], Actor-Critic [10], and DPG [11], it has led to the emergence of many renowned DRL algorithms, including TRPO [12], DDPG [13], PPO [14], A3C [15], and others. These algorithms find application across numerous domains, including robotics [16], finance [17], recommendation systems [18], and various decision-making contexts. Another noteworthy accomplishment is observed in works such as AlphaGo [19], AlphaZero [20], and MuZero [21], which represent a series of achievements in mastering board games and electronic games.

In the realm of using DL to address CO problems, Vinyals et al. [22] are among the first to propose Pointer Networks (Ptr-Net) for this purpose. Their method utilized a sequence-to-sequence Ptr-Net supervised learning framework to tackle TSP. Due to its reliance on supervised learning methods, it necessitated the creation of large labeled datasets beforehand, thereby limiting its application in larger-scale CO problems. RL has been employed to train deep neural networks for addressing CO problems [23,24,25,26]. Inspired by the Transformer model, Kool et al. [27] proposed an attention model as a solution for multiple CO problems. It is noteworthy that their method not only surpasses all previous Ptr-Net models in performance but also closely approximates the optimal solutions achieved by solvers such as Gurobi. Lu et al. [23] introduced a framework that utilizes RL to select among various classical improvement and perturbation operators, effectively addressing the VRP with capacity constraints. Moreover, for graph-based CO problems, more sophisticated models, such as Graph Neural Networks (GNNs), have been utilized to address these challenges [24,27,28]. GNNs adopt a more global perspective by representing CO problems as graphs. By learning relationships between nodes, GNNs can automatically extract and leverage features within the spatial layout of cities. For instance, Li et al. [29] utilized Graph Convolutional Networks (GCNs) to address the graph-based CO problems, such as the Minimum Vertex Cover. Their method entails directly predicting the probabilities of selecting all candidate nodes, rather than iteratively generating solution nodes. In summary, they employ GCNs to directly output the probabilities of all feasible nodes, rather than adopting a step-by-step construction approach for solutions. Drodi et al. [30] employed GAT to address various graph-based CO problems. They claimed that their model possesses the capability to generalize from random point training to real city scenarios. Lodi et al. [31] first utilized machine learning methods to foresee the percentage of facilities in the current solution that require adjustment. Subsequently, they integrated linear constraints associated with this percentage into the mixed-integer programming model and utilized classical solvers to derive solutions. Nevertheless, this paper concentrates primarily on supervised learning, leveraging the robust capabilities of classical solvers to achieve significant benefits.

The inherent randomness and unpredictability of road traffic conditions call for the application of fuzzy logic concepts in urban logistics. In practical systems, travel time varies due to numerous factors like traffic congestion, weather, and road conditions, which are intrinsically challenging to quantify and measure precisely. However, these variables can be effectively described using linguistic terms grounded in human judgment, such as “good” or “bad”. In the study conducted by Xidias et al. [32], the Mamdani method was employed due to its widespread use in fuzzy logic systems. The Mamdani-type fuzzy inference system proposed here utilizes two input variables—actual distance and traffic conditions—and produces one output variable, virtual distance. The actual distance reflects the length of the path between successive request points, derived from a map image. Road traffic conditions reflect the current state of the road and are closely linked to the traffic flow of autonomous vehicles. For example, traffic congestion, characterized by slower speeds, longer travel times, and increased vehicle queues, can be described as poor. The virtual distance is derived by adjusting the actual distance to account for the prevailing traffic conditions. Consequently, the virtual distance is typically greater than the actual distance, depending on the severity of the traffic conditions. Luo et al. [33] represented the traffic congestion factor using speed profiles for different departure times. The travel time function in their study is a stepwise linear continuous function, following a first-come-first-served rule. Undoubtedly, speeds vary between normal and peak hours, impacting the required travel time. This approach provides a robust method for studying the randomness of traffic conditions.

In this paper, our focus lies on the study of the Vehicle Routing Problem with Stochastic Travel Cost (VRP-STC). This problem introduces uncertainty factors into the classical VRP, thereby adding complexity to the problem. In this problem setting, vehicles are constrained not only by their capacity limits but also by the fact that the travel cost between each pair of nodes is not fixed but rather a stochastic variable. This reflects real-life traffic conditions, such as congestion, accidents, or weather-related uncertainties. Similar to the majority of CO problems, solving it entails an NP-hard complexity. Consequently, numerous heuristic approaches have been proposed for addressing this problem, with the majority relying on mathematical programming or metaheuristic techniques [7,34]. We propose a learnable decision-making approach wherein the current partial solution is dynamically updated based on a DRL model. This approach aims to generate feasible solutions that adhere to problem constraints. In detail, we employ an enhanced GAT to handle instances of the VRP-STC and obtain node embeddings and graph embeddings. Subsequently, we utilize multi-head AM to guide the consecutive development of attainable solutions. We will refer to this model as the GAT-AM model. Finally, we utilize the REINFORCE with Rollout Baseline algorithm [27] to train the learnable parameters within the neural network. The main contributions of this paper are as follows. (1) We investigate the VRP with an emphasis on traffic congestion, incorporating limiting distributions to address this issue. (2) We propose a novel encoder using GAT with added residual connections for improved model performance. (3) The results of extensive experiments validate and highlight the effectiveness of the proposed GAT-AM model.

2. Preliminary

2.1. VRP-STC

The VRP-STC represents a sophisticated extension of the classical VRP paradigm by integrating stochastic elements into the calculation of travel costs. In this problem setting, vehicles are not only subject to capacity constraints but also encounter varying travel costs between each pair of nodes, characterized by stochastic variables. This dynamic reflects real-world traffic conditions, encompassing uncertainties such as congestion, accidents, or weather-related factors.

Let C = {0, 1, …, n} denote the set of nodes, where 0 typically represents the depot; V = {1, 2, …, m} is the set of vehicles; Q is the maximum capacity of each vehicle; q_i is the demand of customers i (i = 1, …, n). C_ij represents the generalized cost from node i to node j, which is a stochastic variable in this context (in VRP, the C_ij is fixed and known in advance); $x_{ij}^v$ is the binary decision variable indicating whether or not vehicle v travels directly from node i to node j (1 if yes, and 0 otherwise).

The aim is to reduce the total expected delivery cost by factoring in its stochastic characteristics. The formulation of the objective function is as follows:

$$min\hspace{1em}E\left[{\displaystyle \sum _{v=1}^m}\hspace{1em}{\displaystyle \sum _{i=0}^n}\hspace{1em}{\displaystyle \sum _{j=0,j\ne \mathrm{i}}^n{x}_{ij}^v}{\mathrm{C}}_{ij}\right]$$

(1)

Subject to

$${\displaystyle \sum _{i=1}^n{q}_i}{x}_{ij}^v\le Q,\hspace{1em}\forall v\in V,\forall j\in C$$

(2)

$${\displaystyle \sum _{v=1}^m{\displaystyle \sum _{j=0,j\ne i}^n{x}_{ij}^v}}=1,\hspace{1em}\forall i\in C\backslash \left\{0\right\}$$

(3)

$${\displaystyle \sum _{j=1}^n{x}_{0j}^v=}{\displaystyle \sum _{i=1}^n{x}_{i0}^v}=1,\hspace{1em}\forall v\in V$$

(4)

$${\displaystyle \sum _{i\in C,i\ne j}{x}_{ij}^v}-{\displaystyle \sum _{h\in C,h\ne j}{x}_{jh}^v}=0,\hspace{1em}\forall v\in V,\forall j\in C\backslash \left\{0\right\}$$

(5)

Constraint (2) ensures that each vehicle’s total payload must not exceed Q. Constraint (3) specifies that each customer must be visited exactly once by a vehicle. Constraint (4) requires each vehicle to depart from the depot and return to the depot after servicing all assigned customers. Constraint (5) states that, for each vehicle and each customer, the number of vehicles entering and leaving the customer must be equal.

2.2. Reinforcement Learning Framework

RL represents a methodology tailored for addressing successive decision-making quandaries. Primarily, through systematic exploration of the environment and iterative trial-and-error endeavors, the agent endeavors to acquire proficiency in interacting with the environment to optimize long-term rewards. As is well known, RL has five key components, as portrayed in [4]: S: State space, A: Action space, p (s′, r | s, a): State transition function, r (s, a): Reward function, γ: Discount factor. In the context of CO problems, the “state” denotes the status of an ongoing solution, which can be either a partial solution within the construction framework or a feasible solution within the improvement framework. The “action” pertains to selecting an element from the candidate set to augment the partial solution in the construction framework or applying an operator to modify the current feasible solution in the improvement framework. When addressing CO problems within the construction framework, the state transition function is deterministic. The “reward” typically measures the difference in the objective function between two states, thus serving as an indicator of the improvement achieved by a specific action. Within the construction framework, this setup aligns with the finite-horizon task of the Markov Decision Process (MDP) framework [35], indicating that interactions with the environment will eventually come to a conclusion.

3. Method

3.1. Formulation of DRL

We set discount factor γ = 1. MDPs provide the mathematical framework essential for reinforcement learning problems. They enable agents to make decisions and adapt their learning in the face of uncertainty. The specific forms of the MDP elements are as follows:

• State: Status s_t = (R_t, C_t) is a part of the solution, for instance, G (Q, q, R) created at time step t. Here, R_t (for t ≠ 0) is the group of customers receiving services, which includes all of the chosen customer locations up to step t, C_t is the set of candidate nodes at step t, Q is the maximum capacity of each vehicle, q is the demand of customers.
• Action: Action a_t indicates that at step t, the candidate node π_t is chosen from the candidate node set C_t and added to the service customer set R_t.
• Transition: With the action a_t, a modern fractional solution is obtained as the following state, i.e., s_t+1 = (R_t+1, C_t+1). Within the updated state, R_t+1 includes π_t in addition to the nodes chosen so far, whereas C_t+1 consists of the candidate nodes from C_t with π_t expelled.
• Reward: To minimize the total cost, we define the value of the objective function at step t as Obj_t = min E(cost), and the reward at step t as r_t = Obj_t−1 − Obj_t.
• Policy: The strategy P_θ is parameterized using θ within the GAT-AM model. At each step t, a candidate node is automatically chosen as the service customer node until all service customer nodes are chosen, resulting in the final solution π = {π₁, π₂, …, π_n,} generated by the policy.

3.2. Model

In this section, we introduce our GAT-AM model, which consists of two components: an encoder and a decoder. The encoder utilizes graph attention mechanisms, leveraging graph neural networks to parameterize and learn node representations. The decoding method is based on the learned features. We devise a multi-head AM-based customer allocation strategy and adapt it to an RL framework. The entire model structure flowchart is depicted in Figure 1.

3.3. Encoder

In GAT [36], node features serve as the input to each layer of the network, $x=\left\{{\overrightarrow{x}}_1,{\overrightarrow{x}}_2,\dots ,{\overrightarrow{x}}_N\right\},{\overrightarrow{x}}_i\in {ℝ}^F$, N is the number of nodes, and F is the number of features per node. Feature extraction is conducted to obtain the features $x^{\prime \prime \prime }=\left\{{\overrightarrow{x}}_1^{\prime \prime \prime },{\overrightarrow{x}}_2^{\prime \prime \prime },\dots ,{\overrightarrow{x}}_N^{\prime \prime \prime }\right\},{\overrightarrow{x}}_i^{\prime \prime \prime }\in {ℝ}^{F^{\prime }}$ (Fʹ and F are not necessarily of the same size) of each node, where the original input is parameterized by learnable weight matrix $W\in {ℝ}^{F\times F^{\prime }}$; at each node, a self-attention mechanism is performed using a shared Attention Mechanism (AM) $a:{ℝ}^F\times {ℝ}^{F^{\prime }}\to ℝ$ to compute initial attention coefficients e_ij. Here, e_ij represents the importance of node j with respect to node i, $j\in {\mathcal{N}}_i$(${\mathcal{N}}_i$ denotes the set of neighboring nodes of node i). Subsequently, we employ the softmax function to normalize them and obtain α_ij for ease of computation. The formula is as follows:

$$e_{ij}=a\left(W{\overrightarrow{x}}_i,W{\overrightarrow{x}}_j\right)$$

(6)

$${\alpha }_{ij}=softmax\left(e_{ij}\right)={\displaystyle \frac{\exp (e_{ij})}{{\displaystyle \sum _{k\in {\mathcal{N}}_i}\exp (e_{ik})}}}$$

(7)

The AM typically consists of a single-layer feedforward neural network parameterized by a weight vector $\overrightarrow{\alpha }\in {ℝ}^{2F^{\prime }}$, employing LeakyReLU for non-linear activation. Here, T denotes the transpose operation, and ‖ represents the concatenation operation. Thus, the attention coefficients for a single head (as illustrated in Figure 2a) can be represented as:

$${\alpha }_{ij}={\displaystyle \frac{exp(LeakyReLU({\overrightarrow{\alpha }}^T[W\overrightarrow{x_i}||W\overrightarrow{x_j}]))}{{\displaystyle \sum _{k\in {\mathcal{N}}_i}}exp(LeakyReLU({\overrightarrow{\alpha }}^T[W\overrightarrow{x_i}||W\overrightarrow{x_k}]))}}$$

(8)

Once the activated attention coefficients are obtained, we use these coefficients in conjunction with the corresponding node features to calculate the updated features for each node, which serve as the final output of each layer:

$${\overrightarrow{x}}_i^{\prime }=\sigma \left({\displaystyle \sum _{k\in {\mathcal{N}}_i}{\alpha }_{ij}W{\overrightarrow{x}}_j}\right)$$

(9)

We perform a Skip Connection [37] by adding the features obtained through the single-head AM to the original features, integrating the information:

$${\overrightarrow{x}}_i^{\prime \prime }={\overrightarrow{x}}_i+{\overrightarrow{x}}_i^{\prime }$$

(10)

At this stage, we acquire features after the single-head AM and Skip Connection. Subsequently, we proceed with the computation of the multi-head AM to enhance stability and efficiency. The multi-head AM entails executing AM independently for each head, followed by concatenating the features obtained from each head. The computational formula is expressed as follows:

$${\overrightarrow{x}}_i^{\prime \prime \prime }=\begin{array}{c}K\\ \left|\right|\\ k=1\end{array}\sigma \left({\displaystyle \sum _{j\in {\mathcal{N}}_i}{\alpha }_{ij}^k{W}^k{\overrightarrow{x}}_j+}{\overrightarrow{x}}_i\right)$$

(11)

where ‖ denotes the concatenation operation, ${\alpha }_{ij}^k$ represents the normalized attention coefficients obtained from the k-th head, and W^k is the weight matrix corresponding to the linear transformation of the k-th head. The aggregation process of a multi-head graph attention layer is illustrated in Figure 2b. Simultaneously, we compute the mean of all node features as the aggregated feature $\overline{{\overrightarrow{x}}^{\prime \prime \prime }}$ of the output graph:

$$\overline{{\overrightarrow{x}}^{\prime \prime \prime }}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{\overrightarrow{x}}_i^{\prime \prime \prime }}$$

(12)

Thus, the node features generated by the encoder, along with the aggregated graph features, will be fed into the decoder, as illustrated in Figure 3.

3.4. Decoder

The decoding process in this paper follows multi-head AM, as employed in Transformer [38]. Decoding occurs sequentially. At time step $t\in \left\{1,\dots ,T\right\}$, the decoder generates the current time step’s selected node π_t′ by leveraging both the node embeddings from the encoder and the node π_t selected at the preceding time step tʹ ˂ t. Simultaneously, employing a designated contextual node (c) extends the graph’s information, thereby representing the contextual connections for decoding. The decoder employs an attention sub-layer on top of the encoder to refine the information of contextual nodes. Finally, using a self-attention mechanism, it computes the probability of selecting each node and iteratively determines the final path. Figure 4 illustrates the decoding process.

For the sake of capacity constraints, we monitor the remaining demand ${\widehat{\delta }}_{i,t}$ for nodes $i\in \left\{1,\dots ,n\right\}$, and the remaining vehicle capacity ${\widehat{D}}_t$ at time step t. At t = 1, we initialize ${\widehat{\delta }}_{i,t}={\widehat{\delta }}_i$ and ${\widehat{D}}_t$=1. Subsequently, we update them as follows (where π_t denotes the index of the node selected at decoding step t):

$${\widehat{\delta }}_{i,t+1}=\left\{\begin{array}{cc}max(0,{\widehat{\delta }}_{i,t}-{\widehat{D}}_t)& {\pi }_t=i\\ {\widehat{\delta }}_{i,t}& {\pi }_t\ne i\end{array}\right.$$

(13)

$${\widehat{D}}_{t+1}=\left\{\begin{array}{cc}max({\widehat{D}}_t-{\widehat{\delta }}_{{\pi }_t,t},0)& {\pi }_t\ne 0\\ 1& {\pi }_t=0\end{array}\right.$$

(14)

Context embedding: At time step t, the context of the decoder comprises the node at the current or last position π_t−1 and the remaining capacity ${\widehat{D}}_t$. When t = 1, corresponding to the route’s initiation at the depot, we omit the need to furnish information about the initial node since the route is expected to terminate at the depot:

$$x_{(c)}^{(L)}=\left\{\begin{array}{cc}\left[{\overline{x}}^{(L)},x_{{\pi }_{t-1}}^{(L)},{\widehat{D}}_t\right]& t>1\\ \left[{\overline{x}}^{(L)},x_0^{(L)},{\widehat{D}}_t\right]& t=1\end{array}\right.$$

(15)

where [·, ·, ·] denotes the horizontal concatenation operator. The embedding of the contextual node is denoted as $x_{\left(c\right)}^{\left(L\right)}$ (with a dimension size of 3·d_x), where the superscript L aligns with the node embedding $x_i^{\left(L\right)}$ (indicating the embedding of the L-th layer).

To compute the updated embedding of the contextual node $x_{\left(c\right)}^{(L+1)}$ using the multi-head AM, where the Keys are derived from the node embeddings $x_i^{\left(L\right)}$, and the Queries (for each head) are from the embedding of the contextual node $x_{\left(c\right)}^{\left(L\right)}$ for convenience, the superscript (L) will be omitted hereafter:

$$q_{(c)}=W^Q{x}_{\left(c\right)},k_i=W^K{x}_i$$

(16)

Compute the relevance between the contextual node and all other nodes based on the Keys and Queries:

$$u_{(c)j}=\left\{\begin{array}{ll}{\displaystyle \frac{q_{(c)}^T{k}_j}{\sqrt{d_k}}}& ,ifj\ne {\pi }_{t^{\prime }},\forall t^{\prime }<t\\ -\infty & \hspace{1em}otherwise\end{array}\right.$$

(17)

Here, $d_k={\displaystyle \frac{d_x}{M}}$, where d_k is the dimensionality of query/key, and M is the number of attention heads.

Calculate probabilities: In the last layer of the decoder, generate the probability of selecting each node. Utilize a single-head AM (M = 1) in this layer. Unlike the earlier updated contextual node embeddings, we solely employ Formula (18) to calculate node correlations, where we utilize tanh to clip the results within the range of [−C, C] (C = 10):

$$u_{(c)j}=\left\{\begin{array}{cc}C•\tanh \left({\displaystyle \frac{q_{\left(c\right)}^T{k}_j}{\sqrt{d_k}}}\right)& ,ifj\ne {\pi }_{t^{\prime }},\forall t^{\prime }<t\\ -\infty & otherwise\end{array}\right.$$

(18)

The obtained correlation coefficients are referred to as unnormalized log-probabilities (logits). Finally, employ softmax to calculate the output probability Vector p:

$$p_i=p_{\theta }\left({\pi }_t=i|s,{\pi }_{1:t-1}\right)={\displaystyle \frac{e^{u_{\left(c\right)i}}}{{\displaystyle \sum _j{e}^{u_{\left(c\right)j}}}}}$$

(19)

3.5. Algorithm

We define a solution (cost) π = (π₁, …, π_n) as a permutation of nodes, where ${\pi }_t\in \left\{1,\dots ,n\right\}$ and ${\pi }_t\ne {\pi }_{t^{\prime }}$, $\forall t\ne t^{\prime }$. Our encoder–decoder model defines a stochastic policy P(π|s) for selecting a solution π given a problem instance s. This policy is decomposed and parameterized by θ:

$$p_{\theta }\left(\pi |s\right)={\displaystyle \prod _{t=1}^n{p}_{\theta }\left({\pi }_t|s,{\pi }_{1:t-1}\right)}$$

(20)

Given an instance s, we define a probability distribution P_θ(π|s), from which we can sample a solution (cost)π|s. To train our model, we define the loss function $\mathcal{L}(\theta |s)$ as the expected value of the cost C(π) under the distribution P_θ(π|s): $\mathcal{L}(\theta |s)={\mathbb{E}}_{p_{\theta }(\pi |s)}[C(\pi )]$. We employ the REINFORCE gradient estimator with a baseline b(s) to optimize $\mathcal{L}$ using gradient descent algorithm:

$$\nabla \mathcal{L}\left(\theta |s\right)={\mathbb{E}}_{p_{\theta }(\pi |s)}[(C\left(\pi \right)-b\left(s\right))\nabla {\log }_{p_{\theta }(\pi |s)}]$$

(21)

We propose using a rollout baseline analogous to the self-critical training method by Rennie et al. [39], with periodic updates to the baseline policy. This baseline, denoted as b(s), represents the cost of a solution derived from a deterministic greedy rollout of the policy defined by the best model to date. To stabilize the baseline as the model evolves during training, we periodically freeze the greedy rollout policy P_θ^Baseline for a fixed number of steps per epoch, akin to the freezing of the target Q-network in DQN [6]. Stronger algorithms define stronger baselines. Hence, at the end of each epoch, we compare the current training policy with the baseline policy (using greedy decoding), and only replace the parameters θ^Baseline of the baseline policy when there is a significant improvement based on a paired t-test (α = 5%) across 10,000 individual (evaluation) instances. In the event of an update to the baseline policy, we sample fresh evaluation instances to mitigate the risk of overfitting. By using greedy rollout as the baseline b(s), if the sampled solution π outperforms the greedy rollout, the function C(π) − b(s) becomes negative, leading to reinforcement of the action. Conversely, if the sampled solution performs worse, the opposite effect occurs. This mechanism trains the model to enhance its greedy performance. The algorithm procedure is detailed in Algorithm 1, and the training process is illustrated in Figure 5.

Algorithm 1. REINFORCE with Rollout Baseline

Input: number of epochs E, steps per epoch T, batch size B, significance α  Initialize θ, θBaseline ← θ  For epoch = 1, …, E do    For step = 1, …, T do $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{s}_i\leftarrow RandomInstance()\forall i\in \left\{1,\dots ,B\right\}$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\pi }_i\leftarrow SampleRollout(s_i,P_{\theta })\forall i\in \left\{1,\dots ,B\right\}$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\pi }_i^{Baseline}\leftarrow GreedyRollout\left(s_i,p_{{\theta }^{Baseline}}\right)\forall i\in \left\{1,\dots ,B\right\}$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\nabla \mathcal{L}\leftarrow {\displaystyle \sum _{i=1}^B(C\left({\pi }_i\right)-C({\pi }_i^{Baseline})){\nabla }_{\theta }{\log }_{p_{\theta }\left({\pi }_i\right)}}$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\theta \leftarrow Adam(\theta ,\nabla \mathcal{L})$     End For    If Test (Pθ, PθBaseline) < α then      θBaseline ← θ    End If  End For

3.6. Stochastic Travel Costs

To integrate stochastic travel costs [40] into the model, this study introduces three distinct traffic conditions: light, normal, and heavy, represented as Λ = {0,1,2}. By utilizing a 3 × 3 transformation matrix, as the incidence of faults increases, it suggests that the current high flow may potentially surpass the current low flow in the future. Additionally, we employ matrices with constrained distributions, indicating that, over the long term, each traffic condition appears in the road network with a fixed probability. Specifically, these matrices are designed such that rows and columns represent light, normal, and heavy traffic states:

$$P_{\mathrm{heavy}}=\left[\begin{array}{ccc}0.3& 0.5& 0.2\\ 0.2& 0.3& 0.5\\ 0.1& 0.2& 0.7\end{array}\right]$$

(22)

$$P_{light}=\left[\begin{array}{ccc}0.7& 0.2& 0.1\\ 0.5& 0.3& 0.2\\ 0.2& 0.5& 0.3\end{array}\right]$$

(23)

The limiting distributions of the matrices are [0.16, 0.27, 0.57] and [0.57, 0.27, 0.16], respectively. Under light traffic conditions, the vehicle travel cost between nodes is half of that under normal traffic conditions, whereas under heavy traffic conditions, it is double the cost observed in normal conditions. We incorporate stochastic information by averaging the traffic states according to the limiting distribution. For instance, the vehicle travel cost from node i to node j under heavy traffic conditions becomes:

$$c_{ij}^{\prime }=0.16\cdot 0.5\cdot c_{ij}+0.27\cdot 1\cdot c_{ij}+0.57\cdot 2\cdot c_{ij}$$

(24)

Given the existence of limiting distributions for traffic conditions, through repeated simulations, the stochasticity under traffic conditions will ultimately converge towards an “average value”, surpassing the limiting distribution. To better align the stochastic variables with the model in this paper, we utilize the distance between two points to represent the travel cost. Therefore, the travel cost computed after obtaining the policy solution from the decoder becomes:

$$C_{(\pi )}^{\prime }=0.16\cdot 0.5\cdot C(\pi )+0.27\cdot 1\cdot C(\pi )+0.57\cdot 2\cdot \mathrm{C}(\pi )$$

(25)

Hence, we only need to substitute C′(π) in (25) into C(π) in (21) and proceed accordingly.

4. Experiments

In this section, we perform experiments on VRP and VRP-STC at various scales to assess the performance of the proposed GAT-AM model. Specifically, we train a total of six models, encompassing both randomized and non-randomized variants, across different scales: n = 20, 50, 100. Node coordinates are randomly sampled from a uniform distribution within the unit square [0, 1] × [0, 1]. We conduct a comparative analysis by evaluating the results obtained from exact methods, heuristic methods, and DRL-based methods.

4.1. Experimental Settings

During the training phase, we train on randomly generated training instances for 100 epochs. In each epoch, we process 2500 batches of 1024 instances. For the two-dimensional coordinates of customer nodes, we denote d = 512. Our model architecture features a single encoder with three layers of GAT, eight attention heads (M = 8), and one decoder. Additionally, we employ the Adam Optimizer [41] with a fixed learning rate of η = 10⁻³ to train the strategy network. In the first epoch, we utilize a rollout baseline with an exponential decay factor β = 0.8, a practical approach for ensuring stable training [27]. During the testing phase, we generate 1000 instances for each VRP scale using the same distribution as during training, with the random seed number fixed at seed = 1235. We utilize either greedy decoding or sampling decoding techniques. In greedy decoding, we meticulously select the optimal action at each step, guided by the model’s output. Conversely, in sampling decoding, we sample 1280 solutions randomly and subsequently report the most optimal among them. Augmenting the sample size has the potential to improve solution quality, albeit at the expense of heightened computational demands. Increasing the sample size may lead to improved solution quality, although this improvement comes with an associated increase in computational complexity. Similarly, for VRP-STC, we follow the same procedure. This maintains a clear distinction between the training and testing instances. This approach allows us to effectively evaluate the performance of GAT-AM and all baselines across different problem scales for both VRP and VRP-STC.

4.2. Baseline Methods and Evaluation Metrics

We compare the proposed GAT-AM model, without randomness, with three baselines, including the heuristic algorithm LKH3 [42], OR Tools [43], and the DRL-based method AM [27]. We specifically compare the GAT-AM model, with randomness added, with the AM model. We will utilize Objective Obj, Gap (%), and Time as performance metrics for evaluation. Considering that the objective of all methods is to minimize the objective function, Obj signifies the optimal value of the objective function, denoting the minimal attainable objective function value associated with the optimal solution. Additionally, we consider the Obj obtained from LKH3 as the benchmark (which is consistently optimal for small-scale instances), and we evaluate the accuracy and stability of each method by analyzing the average gap (Gap). The Gap measures the relative difference between the objective function value achieved by a given method and the optimal objective function value determined by LKH3. It is computed using the following formula:

$$Gap={\displaystyle \frac{{\displaystyle \sum _{i=1}^T}\left(\frac{{\mathrm{method}}_{ob{j}_i}}{LKH{3}_{ob{j}_i}}-1\right)}{T}}\times 100\%.$$

(26)

Computational time is a crucial factor in our analysis. However, due to various influences, such as differences between Python3.9 and C++ GCC 11.2.0, as well as variations in hardware GPU/CPU models, comparing running times directly proves challenging. To address this, we measure “Time” as the average duration required to solve 1000 instances. All methods are trained and tested on a PC equipped with a single GPU (RTX 3090) and an Intel^® Core™ i7-8700K CPU operating at 3.70 GHz.

4.3. Comparison Analysis

In

Table 1. GAT-AM vs. baselines.

	VRP20			VRP50			VRP100
Method	Obj	Gap	Time	Obj	Gap	Time	Obj	Gap	Time
LKH3	6.14	0.00%	(7 h)	10.38	0.00%	(7 h)	15.65	0.00%	(13 h)
OR Tools	6.42	4.84%	(-)	11.22	8.12%	(-)	17.14	9.34%	(-)
AM (greedy)	6.40	4.57%	(1 s)	10.98	5.78%	(3 s)	16.80	7.34%	(8 s)
AM (sampling)	6.25	2.12%	(6 m)	10.62	2.31%	(28 m)	16.23	3.72%	(2 h)
GAT-AM (greedy)	6.35	3.76%	(1 s)	10.88	4.82%	(2 s)	16.13	2.89%	(5 s)
GAT-AM (sampling)	6.18	0.98%	(5 m)	10.51	1.25%	(11 m)	15.89	1.53%	(23 m)

, we summarize the comparison results between the GAT-AM model and three comparative methods on the non-randomized problem. In

Table 2. GAT-AM vs. AM.

	VRP-STC20		VRP-STC50		VRP-STC100
Method	Obj	Time	Obj	Time	Obj	Time
AM (greedy)	9.54	(2 s)	16.36	(8 s)	25.08	(23 s)
AM (sampling)	9.31	(13 m)	15.98	(1 h)	24.68	(4.5 h)
GAT-AM (greedy)	9.34	(2 s)	16.01	(6 s)	24.67	(13 s)
GAT-AM (sampling)	9.18	(7 m)	15.68	(16 m)	24.36	(46 m)

, we compare the results of the GAT-AM model with the AM model on the randomized problem. The comparison between Table 1 and Table 2 reveals that the Obj and Time for problems with added randomness are significantly higher than those without randomness. This result could be interpreted as randomized problems having higher travel costs compared to non-randomized ones. In comparison to the VRP, the computational complexity of VRP-STC is higher, leading to an increase in solving time. Table 1 clearly delineates that among traditional heuristic approaches, LKH3 consistently yields the optimal solutions across three distinct problem scales, hence establishing it as the benchmark method. It is obvious that when examining Obj and Gap, the outcomes attained by DRL-based methods closely resemble those achieved by traditional algorithms. Furthermore, in terms of computational time, methods based on DRL demonstrate the fastest resolution speed. It is noteworthy that with the escalation of problem scale, the computational time of traditional heuristic methods undergoes exponential growth, while the solving time of DRL-based methods demonstrates nearly linear progression. Moreover, it is evident from the comparison results of DRL-based methods that GAT-AM surpasses AM in both Obj and Gap, while also exhibiting shorter computational time. Through the comparison presented in Table 2, it is obvious that our GAT-AM model outperforms the AM model. The GAT demonstrates greater efficacy in addressing problems with stochastic traversal costs, leveraging its specialized network architecture tailored for graph data to unearth deeper layers of information. Following the introduction of stochastic elements, this advantage becomes even more pronounced.

4.4. Model Convergence Performance

Figure 6a–c demonstrate that the proposed model exhibits superior convergence performance across VRP problems of different scales. Compared to the AM model, the GAT-AM model achieves near-optimal results within the initial epochs. As the node number increases, the convergence rate of the AM model declines due to the exponential growth in the complexity of the graph structure. This complexity hampers the traditional Transformer architecture’s ability to capture intricate node relationships during the early training stages. In contrast, the GAT leverages an attention mechanism to automatically learn and optimize node connections. This enables the network to assign varying weights to different node pairs based on the dataset characteristics and task requirements. By utilizing continuous numerical attention weights instead of binary node connections, GAT enhances its expressive capacity. The parallel computation of attention values further contributes to the high computational efficiency of the GAT. GAT provides personalized feature representations for each node by incorporating both the features of the node itself and its neighbors during attention calculation. Moreover, GAT’s focus on capturing the local structural information of nodes is crucial for determining the optimal solution to VRPs.

As illustrated in Figure 7, the introduction of random travel costs into the model induces varying degrees of oscillation in the cost convergence curves for both the GAT-AM and AM models. The GAT-AM model exhibits noticeably poorer convergence performance during the first 40 epochs compared to the scenario without random transit costs. Nonetheless, with an increasing number of epochs, the cost curve typically shows a tendency to stabilize. This phenomenon is attributed to the model’s extensive exploration due to the introduced randomness during the initial stages, leading to significant cost fluctuations. Nevertheless, with sufficient exploration, the model identifies more optimal solutions, resulting in stabilized costs. This observation aligns with the hypothesis made when configuring the random model; with an increase in the number of training samples, the model’s final cost is expected to approach the “average value” of the random matrix.

In comparison to the AM model, the GAT-AM model consistently demonstrates superior convergence performance, with a smoother training cost curve. This indicates that the GAT-AM model effectively addresses the challenges posed by randomness. The GAT’s ability to capture relational information between nodes, coupled with the Transformer model’s capacity to autonomously learn the dependencies and sequences of paths, enables the formulation of more accurate and adaptive vehicle routing solutions. Through the modelling and learning of random travel costs, the proposed approach yields more precise and robust vehicle routing planning outcomes.

4.5. Visualization

In this subsection, we present a data simulation example to provide a more intuitive demonstration of the solution outcomes. Utilizing GAT-AM, Figure 8 and Figure 9 showcase exemplary solutions for VRP and VRP-STC with different node numbers (n = 20, 50, and 100), achieved via models incorporating sampling decoding techniques. Figure 8 illustrates the solutions to the VRP for different node numbers. The lines of different colors represent distinct vehicles, while the black squares represent depots, and circular nodes denote customers. From Figure 8, it is evident that with smaller node sizes, fewer vehicles are employed. For instance, in VRP20, only three vehicles suffice to fulfill the demands of all customers. As the node size expands, the number of vehicles gradually increases. In VRP50, six vehicles are utilized. However, the number of vehicles does not proportionally increase with the node size. In VRP100, despite doubling the number of customers compared to VRP50, the number of vehicles only increases by two. This is because each vehicle also aims to serve as many customers as possible to achieve the objective of minimizing the total number of vehicles required. Figure 9 displays the solutions to VRP-STC for different scales. The lines of different colors represent distinct vehicles. Comparing Figure 9 with the results of VRP in Figure 8, it can be observed that in both node sizes 20 and 50, the solutions for VRP-STC entail one additional vehicle compared to the VRP solutions. As the node size expands to 100, although the number of vehicles remains the same, the final vehicle is loaded with only a small amount of cargo. In this case, while the vehicles are not fully loaded, they compensate for the increased cost of using multiple vehicles by traveling shorter distances.

5. Conclusions

In this research endeavor, we present the GAT-AM model, which combines AM and DRL techniques. This model exhibits robust performance in addressing the challenges posed by both the VRP and the VRP-STC. The architecture of the GAT-AM model comprises an encoder and a decoder. The encoder utilizes GAT to effectively learn feature representations of customer nodes across diverse subspaces. Meanwhile, the decoder employs multi-head AM to refine solution construction strategies, aiming to generate a broader range of solutions and thereby explore high-quality solutions more effectively. During model training, the REINFORCE with Rollout Baseline algorithm is employed, which effectively mitigates the variance of policy gradients by incorporating a baseline. This optimization technique accelerates the learning process and enhances training efficiency. Additionally, the decoder’s use of a dedicated context mechanism and masking policy restricts the choice of subsequent nodes, ensuring that the solutions generated remain feasible. This refinement significantly augments the likelihood of the model discovering high-quality solutions within polynomial time complexity.

The comparative experiments encompassed three baseline methods for VRP across various scales and conditions. For VRP-STC, we conducted a distinct comparison between GAT-AM and AM. The experimental results indicate that, compared to existing heuristic algorithms, the proposed model exhibits a significant improvement in both training and testing speed, thereby confirming its efficiency. Notably, when dealing with large-scale instances, the proposed model can furnish solutions within a reasonable time complexity, underscoring the immense potential of DRL in addressing CO problems.

Although this paper has achieved certain accomplishments, there are still several avenues worth further exploration in solving VRPs:

• Algorithmic Refinement and Generalization: The model currently exhibits certain limitations. Enhancing the algorithm’s generalization capacity to accommodate a broader spectrum of environments and various problem scenarios represents a pivotal area for future investigation.
• Real-time Dynamic Planning: With the growing demand for practical applications, implementing real-time dynamic planning within the model to accommodate the evolving logistics demands and traffic conditions is a pressing challenge awaiting resolution.
• Multi-objective Optimization: The present model primarily focuses on minimizing the total travel cost. In the future, exploration could be extended to achieve a balance among multiple objectives, such as service level assurance and minimizing environmental impacts.

In conclusion, while this paper has made some progress, there remains vast research space in the realms of deep learning and combinatorial optimization problem solving. Future work will continue to advance the integration of theory and practice, driving the intelligent upgrading of the transportation industry, and providing robust technological support and theoretical guidance for scientific research and industrial development in related fields.