An Improved Ant Colony Algorithm with Deep Reinforcement Learning for the Robust Multiobjective AGV Routing Problem in Assembly Workshops

Abstract

Vehicle routing problems (VRPs) are challenging problems. Many variants of the VRP have been proposed. However, few studies on VRP have combined robustness and just-in-time (JIT) requirements with uncertainty. To solve the problem, this paper proposes the just-in-time-based robust multiobjective vehicle routing problem with time windows (JIT-RMOVRPTW) for the assembly workshop. Based on the conflict between uncertain time and JIT requirements, a JIT strategy was proposed. To measure the robustness of the solution, a metric was designed as the objective. Afterwards, a two-stage nondominated sorting ant colony algorithm with deep reinforcement learning (NSACOWDRL) was proposed. In stage I, ACO combines with NSGA-III to obtain the Pareto frontier. Based on the model, a pheromone update strategy and a transfer probability formula were designed. DDQN was introduced as a local search algorithm which trains networks through Pareto solutions to participate in probabilistic selection and nondominated sorting. In stage II, the Pareto frontier was quantified in feasibility by Monte Carlo simulation, and tested by diversity-robust selection based on uniformly distributed weights in the solution space to select robust Pareto solutions that take diversity into account. The effectiveness of NSACOWDRL was demonstrated through comparative experiments with other algorithms on instances. The impact of JIT strategy is analyzed and the effect of networks on the NSACOWDRL is further discussed.

1. Introduction

The vehicle routing problem (VRP) was proposed by Dantzig and Ramser in 1959 [1]; this problem aims to find an optimal route based on constraints and objectives. The problem is widely used in traffic management, logistics transportation, and other fields. This problem has been extensively studied. Several VRP variants have been proposed by applying the VRPs in different environments, such as the capacity-constrained vehicle routing problem (CVRP) and vehicle routing problem with time window (VRPTW). Many researchers have performed systematic literature reviews on the VRP.

Liu et al. [2] extracted data and analyzed relevant literature within 2018–2022 to address the problems associated with VRPTW. Li et al. [3] analyzed the articles which solve VRPs by Learning-Based Optimization (LBO) algorithms from various aspects, evaluated the performance of representative LBO algorithms, summarized the types of problems to which LBO is applicable, and proposed directions for improvement. Asghari et al. [4] investigated the contributions related to the green VRP and proposed a classification scheme based on the variants considered in the scientific literature, so as to provide a comprehensive and structured survey of the state of knowledge, as well as to discuss the most important features of the problems and to indicate future directions. Gunawan et al. [5] provided and categorized a list of VRP datasets. Salehi Sarbijan et al. [6] systematically reviewed and analyzed recent research on the VRP from 2001 to 2022, focusing on new and emerging topics in VRPs. Based on the basic VRP, Zhang et al. [7] classified VRPs according to their characteristics and practical applications, gave the unified description and mathematical model of each type of problem, and then analyzed the solution methods of each type of VRP. Ni et al. [8] comprehensively analyzed and summarized the literature related to VRPs from 1959 to 2022 by using the knowledge graph, and classified the VRP models and their solutions.

However, the complexity of the VRP increases with the additional constraints of real life. The VRP is NP-hard, so finding the optimal solution is difficult. Therefore, the method proposed to solve the VRP aims to find the approximate optimal solution via an efficient computational approach.

VRPs for AGVs are solved by using exact algorithms and intelligent evolutionary algorithms. The exact algorithms include the branch-and-bound method, integer linear programming method, and dynamic programming method, which are suitable for solving small-scale VRPs with simple structures. Soysal [9] et al. proposed a simulation-based restricted dynamic programming approach to solve the VRP associated with traffic emissions. Reihaneh [10] proposed a branch-and-price algorithm that solves the vehicle routing with demand allocation problem. However, the exact algorithm is prone to falling into a local optimal solution or taking too long to compute when solving large-scale VRPs.

In contrast, intelligent evolutionary algorithms can find satisfactory approximate optimal solutions in a limited time when solving large-scale VRPs. Therefore, many experts have improved and designed intelligent evolutionary algorithms. Yong Wang [11] et al. proposed an improved multiobjective genetic algorithm with tabu search (IMOGA-TS), which combines local and global searches with the improvement of solutions at each iteration by TS to solve a collaborative multidepot EV routing problem with time windows and shared charging stations (CMEVRPTW-SCS). Pierre [12] proposed a multiobjective genetic algorithm with stochastic partially optimized cyclic shift crossover to solve the multiobjective VRPTW.

The multiobjective model is always transformed into a single-objective model by weighting or other conversion methods. In the VRP, the total objective is often composed of multiple objectives with mutual exclusivity, which means that improving the performance of one objective may lead to degradation of the performance of other objectives. These objectives may not be linearly related and cannot be transformed into a single-objective problem with weights. Therefore, there is a limitation in using a single-objective model to solve the VRP.

Multiobjective optimization problems (MOPs) are proposed for optimizing multiple objectives which are always mutually exclusive at the same time. In recent decades, researchers have proposed many methods for solving MOPs. Multiobjective evolutionary algorithms (MOEAs) are the mainstream methods for solving MOPs. Jinlong Wang [13] proposed an enhanced algorithm based on SPEA2 (ESPEA) to solve the pickup vehicle scheduling problem with mixed steel storage (PVSP-MSS) which optimizes the makespan of pickup vehicles and the makespan of steel logistics parks at the same time. Yadian Geng [14] proposed an improved hyperplane-assisted evolutionary algorithm (IhpaEA) to solve the distributed hybrid flow shop scheduling problem (DHFS), which takes maximum completion time and energy consumption as objectives.

Based on different algorithmic strategies, most MOEAs can be classified into the following categories: (1) based on their dominant relationship, such as the NSGA-II [15], SPEA [1], and PESA-II [16]; (2) based on decomposition, such as the MOEA/D [17], MOEA/D-M2M [18], and RVEA [19]; and (3) based on performance evaluation indicators, such as the IBEA [20], SMSEMOA [21], and DNMOEA/HI [22].

From the above algorithms, Pareto solutions can be obtained. However, for MOPs, the number of Pareto solutions increases as the number of objectives increases. For practical scenarios, not all solutions make sense, especially for uncertain time-based VRPs. Uncertain times require Pareto solutions that are most resistant to uncertain times compared to deterministic times, and uncertainty in VRPs is rarely considered in these algorithms. To solve this problem, robust optimization has been proposed, where the researcher searches for robust solutions with the strongest resistance to interference. Robust optimization [23] means that the solution and performance results remain relatively unchanged when exposed to uncertain conditions. The solution is usually evaluated using the most unfavorable uncertainty. Xia [24] et al. proposed a new method that sequentially approaches the robust Pareto front (SARPF) from nominal Pareto points to solve MOPs with uncertainties. Jin et al. [25] introduced and discussed existing methods for dealing with different uncertainties and studied the relationships between different categories of uncertainty. Scheffermann [26] et al. introduced and compared the NSGA-II with an improved predator–prey algorithm for the VRPTW with uncertain travel times. He [27] et al. used a robust multiobjective optimization evolutionary algorithm (RMOEA) to solve robust multiobjective optimization problems (RMOPs), which consists of two parts: multiobjective optimization using the improved NSGA-II and robust optimization to search for a robust optimal frontier. The robust optimization in this paper is based on Monte Carlo simulation to simulate the practical environment in which the results are approached to the practical situation through a large number of simulation experiments. The number of feasible times of Monte Carlo simulation under the disturbance of uncertainty coefficients is taken as the criterion for assessing the robustness of the solution.

In the VRP, the AGV operation time, total travel distance of the AGVs or number of AGVs are often used as objectives. However, the just-in-time (JIT) of the material distribution is often ignored. In addition, the literature discusses the robustness of VRPs, but few papers have combined robustness with JIT problems. JITs pursue just-in-time arrivals, and uncertain times can seriously affect the realization of JIT, so combining JITs with robust optimization can improve the possibility of realizing JITs.

To solve the above issues, in this paper, the multiobjective robust VRP model that simultaneously considers uncertainty and JIT strategy is proposed. In addition, an improved ACO algorithm with deep reinforcement learning for assembly workshops is proposed. This study aims to make the following contributions.

(1) A just-in-time-based multiobjective robust vehicle routing problem with time windows (JIT-RMOVRPTW) is constructed which simultaneously optimizes three objectives. The model takes into account the uncertain time by introducing uncertainty coefficients. By defining the robustness metric that combines uncertain time to measure the robustness of the solution, the robustness optimization is defined as a dedicated optimization objective that can filter robust solutions in a better way.

(2) The JIT-RMOVRPTW considers the two conflicting goals of JIT and robustness at the same time. In this model, the JIT strategy under uncertainty is proposed. The traditional time window is eliminated, and the JIT time, which is used as the deadline for the workstation, is proposed. The model eliminates the fixed departure time of AGVs, so that AGVs can flexibly adjust the depot time according to the JIT time to eliminate the waiting time at the workstation generated by early arrival of AGVs.

(3) A two-stage nondominated sorting ant colony algorithm with deep reinforcement learning (NSACOWDRL) is proposed to improve the robustness of the solutions by double screening. The NSACOWDRL divides the problem into two stages: a multiobjective optimization problem and a robust optimization problem. In the first stage, an initial search is performed with robustness metrics as one of the objectives. The small habitat conservation strategy based on reference points used in NSGA-III [28] is introduced in routing selection. The nondominant rank, the elite path strategy, and the max-min pheromone strategy are introduced in the pheromone update strategy. Furthermore, DDQN is introduced as a local search algorithm. The network is trained by the Pareto solutions which guide the learning direction of the network, after which the trained network is used to participate in nondominated sorting. Moreover, a probabilistic formula based on the network and objectives is designed. In the second stage, the feasibility of the solutions is quantified through Monte Carlo simulation, then the solutions are partitioned according to the uniformly distributed weights in the solution space. Each partitioned solution set is screened separately for robustness. The set of screened solutions is the robust Pareto frontier that takes into account the diversity of solutions.

(4) The performance of the proposed algorithm is validated by comparative experiments. The paper also further discusses the effect of network structure on the performance of the algorithm.

The remainder of this paper is organized as follows: In Section II, the background is introduced. Section III introduces the related VRP problem in the assembly workshop. In Section IV, the NSACOWDRL is described in detail. The experimental design and analysis are described in Section V. Finally, conclusions and suggestions for future work are given in Section VI.

2. Background

2.1. Vehicle Routing Problem

With further study of these elements, many VRP models, such as the green VRP (GVRP) and the dynamic VRP (DVRP), have been proposed and investigated. Moreover, depending on the objectives of the VRP, the problems can be categorized into single-objective VRPs and multiobjective VRPs.

Many models and algorithms have been proposed for single-objective VRP. Kaijun [29] et al. constructed a new VRPTW model of multiple distribution centers based on urban rail transportation, and a novel concentration-immune algorithm particle swarm optimization (C-IAPSO). Zhang [30] et al. proposed and established a reverse logistics VRP with backhauls (RL-VRPB) for waste packaging recycling route planning. An improved scatter search algorithm (ISS) was proposed, which introduces vehicle residual space recovery capability and the local search strategy. To solve the DVRP, Sabar [31] et al. proposed a population-based iterated local search (ILS) algorithm that integrates skewed variable neighborhood descent (SVND), a quality-and-diversity updating strategy and an adaptive multioperator perturbation procedure. Feng [32] et al. proposed a new algorithm by transferring knowledge from useful customer representations that are captured from past solved VRPs. Li [33] et al. utilized a novel neural network integrated with a heterogeneous attention mechanism to solve the pickup and delivery problem in the VRP problem. Jia [34] et al. proposed a novel bilevel ACO algorithm to solve the capacitated EV routing problem (CEVRP). The algorithm divides the problem into two subproblems: first, an order-first split-second max-min ant system algorithm is proposed to solve problems which ignore the electricity constraint. Second, a new removal heuristic combining a restricted enumeration method is designed to generate the charging schedule in the generated routes to satisfy the electricity constraint.

However, these VRP algorithms do not consider the multiobjective characteristics of real problems and lack multidimensional consideration of real problems. Therefore, multiobjective VRPs as well as many variants of multiobjective algorithms have been proposed. Nayera Elgharably [35] et al. proposed a new hybrid search genetic algorithm that combines the SPEA with a resultant local search heuristic to solve the stochastic GVRP, which simultaneously considers economic, environmental, and social aspects. Arash Motaghedi-Larijani [36] proposed four metaheuristic algorithms that involve combining different multiobjective algorithms to solve routing and scheduling vehicles on a cross-dock, minimizing the total shipping and tardiness costs and minimizing the number of outbound open doors as the objectives. Nan Yin [37] proposed M-NSGA-II, which combines the multifactorial evolutionary algorithm (MFEA) with NSGA-II to solve the VRP in low-carbon intelligent transportation. This approach considers carbon emissions, the number of vehicles, and distribution cost funding. Mazdarani [38] proposed a hybrid multiobjective genetic algorithm to solve the bi-objective overlapped links VRP (OLVRP) for valuable goods transportation, which takes the total cost and risk of routing as objectives. Wang et al. [39] established a five-objective multidepot VRP with time windows (MDVRPTW) and proposed a two-stage MOEA to solve the problem. In stage I, the algorithm focuses on finding extreme solutions and forms a coarse Pareto front. Stage II extends the found solutions for approximating the whole Pareto front. The two-stage strategy provides a new method for balancing convergence and diversity when solving MOPs.

Nevertheless, these multiobjective algorithms seldom consider the uncertainty of the real problem, which is an unavoidable problem in practical applications. Uncertainty disrupts problem-solving significantly. With the study of uncertainty problems, researchers have turned to incorporating robustness into VRPs that are insensitive to disturbances and can maintain the optimization performance at an acceptable level under disturbances. Robustness means maintaining feasibility as well as maintaining the best possible performance under uncertainty constraints. The goal of robust optimization is to make full use of the search space to find the best optimal solution while minimizing the effect of uncertainty.

Deb et al. [40] proposed a novel, robust, two-stage planning model for charging station placement that is determined by fuzzy inference considering distance, road traffic, and grid stability. A hybrid algorithm combining chicken swarm optimization and the teaching-learning-based optimization (CSO TLBO) algorithm was proposed to obtain the Pareto front, and fuzzy decision-making was used to compare the Pareto optimal solutions. Muñoz [41] et al. proposed a novel strategy to solve the VRPTW with uncertain demands using adaptive credibility thresholds, which try to find good solutions with relaxed capacity constraints. For this purpose, a new split decodification algorithm that combines a genetic algorithm with local search strategies was designed. Wang [42] et al. proposed a robust routing optimization model for multivehicle electric delivery that considers random changes in demand and transportation time at different customer points to minimize the distribution cost and carbon emissions; additionally, an improved genetic algorithm was proposed to solve this problem. Jiahui Duan [43] et al. designed a new form of disturbance on travel time that is determined by the maximum disturbance degree and constructed a robust multiobjective VRPTW, which adopts two conflicting objectives to capture the uncertainty characteristics. Moreover, a robust multiobjective particle swarm optimization approach that adopts a coding method involving priority assignment and decoding based on a greedy strategy is proposed. A new metric, Rob, was designed to measure the robustness of the solutions. To take full advantage of the problem characteristics, the algorithm adopts problem-based and route-based local searches.

In the current literature, robustness methods rarely take into account the JIT objective of the VRP, and there are fewer studies on the combination of robust VRPs with MOPs. In this paper, the solutions obtained by the algorithm are subjected to an uncertainty test. Robust solutions that take into account diversity are obtained based on the distribution of the solutions in the solution space.

2.2. Ant Colony Algorithm

The ant colony algorithm (ACO) [44] is a widely studied intelligent evolutionary algorithm. The ACO algorithm consists of two main steps: calculation of the state transfer probabilities and updating of the routing pheromone concentrations:

(1) The node transfer probabilities:

When the ant selects the next transfer node, it selects nodes by a roulette wheel after calculating the transfer probability of each node. To prevent ants from repeatedly selecting nodes, the selected node is added to the tabu list.

(2) Updating of routing pheromone concentrations:

The ACO algorithm mimics the behavior of ants that leave pheromones on each route on which they walk and introduces a pheromone volatilization mechanism to prevent the algorithm from falling into local optima. After all the ants have explored all the nodes, the pheromone concentration of each route is updated according to the pheromone update strategy.

The ACO algorithm is used in various fields, such as assignment problems, scheduling problems, and resource allocation. Fengyu Chen [45] et al. proposed the product ant colony optimization (PACO) algorithm which combines the ACO algorithm with production products to improve the productivity of textile manufacturers and balance the total operating time of all machines in each process. Alwin M. [46] et al. combined the binary ant colony algorithm (BACA) with other search optimization algorithms, such as hill climbing (HC) and simulated annealing (SA) to ensure the optimal Wireless Sensor Networks (WSNs) Coverage.

The ACO algorithm has good performance and effectiveness in solving VRPs. Huang [47] et al. employed an ACO to solve the feeder vehicle routing problem (FVRP). Li [48] et al. applied an improved ACO which used an innovative approach in updating the pheromone to solve the multidepot GVRP (MDGVRP). Current research on the improvement of the ACO for VRPs has focused on the state transfer strategy, pheromone update strategy, introduction of heuristic rules, individual learning strategy, and integration with other algorithms. Jiao [49] et al. proposed a polymorphic ACO with the adaptive state transfer strategy and the adaptive information updating strategy to improve the global search capability of the algorithm. Teng Ren [50] proposed a new improved ACO to solve the VRP with split pickup and delivery of multicategory goods. The algorithm adopts a TS operator that contains five neighborhood operators to improve the local search ability and introduces SA mechanisms to update the global pheromones. Luo [51] et al. proposed an improved ACO which adopted a pseudo-random state transition rule and avoided the blindness search at early planning through the unequal allocation initial pheromone. Li [52] proposed the multiobjective ant colony system algorithm for epidemic situations (MOACS4E) to solve a new multiobjective VRP model for epidemic situations that considers the traditional travel cost and the prevention cost of the VRP in epidemic situations. MOACS4E adopted two ant colonies to optimize the problem and proposed a new pheromone fusion-based solution generation method. In addition to this, algorithms such as the simulated annealing algorithm [53] and the artificial fish swarm algorithm [54] were used to combine with the ACO.

Deep reinforcement learning (DRL) is currently one of the important research directions in the field of machine learning. Common reinforcement learning algorithms are Deep Q-Networks (DQNs), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO). DRL can realize end-to-end learning by directly interacting with the environment. With excellent feature extraction ability to realize the migration between different tasks, it can effectively overcome various perturbations in the system, as well as have good ability to solve high-dimensional and large-scale problems. However, it has the problems of insufficient ability to explore the environment, poor robustness, and susceptibility to deceptive gradients caused by deceptive rewards; in addition, the training process of DRL is quite complicated in the face of real problems. Evolutionary algorithms have the advantages of better global search capability, good robustness, and parallelism, but it is difficult to carry out migration learning in the face of perturbations. Overcoming the shortcomings of each by combining reinforcement learning and evolutionary algorithms to achieve further optimization of the algorithms is one of the current research directions. NICOLÁS FRÍAS [55] proposed the four innovative hybrid algorithms which combine Machine Learning (ML) clustering techniques with metaheuristic approaches inspired by an ACO to solve the energy minimizing VRP (EMVRP).

Neural networks are good at learning but not at searching, while evolutionary algorithms are good at searching but not at learning. Evolutionary algorithms combined with DRL can be used to make up for the lack of DRL. Thus, in this paper, an ACO is combined with a Double DQN (DDQN). A DDQN is introduced to overcome the problem that the ACO is prone to fall into local optimization to improve the optimization ability of the ACO; meanwhile, the good searching ability of the ACO on the VRP is utilized to guide the DDQN for learning and searching.

3. Problem Description and Mathematical Models

3.1. Problem Description

In this paper, the multiobjective multi-AGV routing planning mathematical model JIT-RMOVRPTW was constructed. The VRP in the assembly workshop is described as follows:

$i,j$ indicates the identification of workstations; $i,j\in V,V=\left\{\mathrm{0,1},\dots ,n\right\}$ represents the set of workstations; and when $i$ or $j$ is 0, it represents the raw material library. There are K AGVs with the same performance in the raw material library. Each of them has a maximum load capacity of Q and an average travel speed of $v$. The maximum travel distance of the AGV is unlimited. Materials are delivered by material carts, which are pulled by the tail hook of the AGV. Each AGV can transport up to $R$ carts. The cart has a maximum capacity of $Q_r$. There are $N$ workstations in the assembly workshop, and a total of $M$ materials are needed. The correspondence between workstations and materials is known, the information about the materials is known, and $q_i$ is the total number of material boxes required for the workstation $i$. The position of each workstation is known, and $d_{ij}$ is the distance between stations. AGVs are dispatched to deliver $M$ materials to $N$ workstations. In AGV delivery, disturbed by random events such as traffic jams, the transportation time is uncertain, and the disturbance time is defined as $t_i^{~}$. In the JIT-RMOVRPTW, the traditional time window ${[a}_i,b_i]$ is canceled, and materials need to be delivered on time by the deadline ${JIT}_i$. The materials of each workstation require $s_i$ to be unloaded. $s_i$ is the unloading time, determined by the number of boxes of materials to be delivered to the station.

The VRP problem in the assembly workshop is based on the following assumptions:

(1) There is only one raw material library in the entire workshop, and the raw material library is able to meet the material requirements of all the workstations.

(2) The material demand at each station is less than the load capacity of the AGV, and the unloading time of the AGV at each station is known.

(3) All the AGVs depart from the same raw material library and must return to the raw material library after unloading the transported material.

(4) AGV acceleration and deceleration, charging, etc., are not considered during vehicle operation.

(5) Materials are placed in standard material boxes, and the material requirements of each workstation cannot be split. The AGV transports materials through the material carts. The boxes of the materials required for the same workstation are loaded in the same material cart, which is not mixed with other workstation materials.

3.2. Mathematical Model

Based on the traditional VRPTW, the JIT-RMOVRPTW adds uncertain time constraints and designs a new metric alpha based on uncertain time as an objective to measure the robustness of the solution. In addition, the JIT-RMOVRPTW considers JIT delivery. JIT time is used as the deadline, and the AGV adjusts the departure time according to the JIT time of the workstation to achieve the objective of the JIT.

The JIT-RMOVRPTW can be formulated as follows:

The three objectives for the JIT-RMOVRPTW, each of which needs to be minimized, can be stated as follows:

$f_d$ (Total travel distance):

$$\mathit{min}{f}_d =\sum _{k=1}^K\sum _{i=0}^N\sum _{j=0}^N{d}_{ij}{x}_{ijk}$$

(1)

$f_{alpha}$ (Robustness metrics):

$$\mathit{min}{f}_{alpha }={\displaystyle \frac{1}{\prod _{i=1}^N{alpha}_i}}$$

(2)

$f_{JIT}$ (JIT penalty time):

$$\mathit{min}{f}_{JIT}=\sum _{k=1}^K{w}_k$$

(3)

$f$ (Overall objectives):

$$\mathit{min}f =\left(f_d,f_{alpha},f_{JIT}\right)$$

(4)

The constraints associated with the JIT-RMOVRPTW are added as follows:

• Capacity constraints: The number of material boxes on each workstation must not exceed the maximum capacity of the material cart. The number of material boxes delivered by each AGV should not exceed the maximum load capacity of the AGV. $E_i$ indicates the kinds of materials required at station $i$. $q_{ie}$ indicates the number of material boxes of material $e$ for workstation $i$.

  $$q_i=\sum _{e=1}^{E_i}{q}_{ie},\left(i=\mathrm{1,2},\dots ,n\right)$$

  (5)

  $$q_i\le Q_r$$

  (6)

  $$\sum _{i=0}^N{q}_i{y}_{ik}\le Q,\left(k=\mathrm{1,2},\dots ,m\right)$$

  (7)
• Constraints on the number of material carts: The number of delivery workstations for each AGV must not exceed the number of carts for its own material. The number of material carts of each AGV must not exceed the maximum number of material carts of the AGV. $R_k$ indicates the number of material carts of the $kth$ AGV.

  $$\sum _{i=0}^N{y}_{ik}\le R_k\le R,\left(k=\mathrm{1,2},\dots ,m\right)$$

  (8)
• Service time: The service time for each station is determined by the number of material boxes to be delivered to that station. $T$ indicates the time taken to unload a material box.

  $$s_{i }=\left\{\begin{array}{l}T{q}_i,(i=\mathrm{1,2},\dots ,n)\\ 0,(i=0)\end{array}\right.$$

  (9)
• Distribution constraints: Each workstation can be delivered only once.

  $$\sum _{k=1}^K{y}_{ik}=1,\left(i=\mathrm{1,2},\dots ,n\right)$$

  (10)
• AGV arrival number constraints: Only one AGV arrives at each workstation.

  $$\sum _{i=0}^n{x}_{ijk }=y_{jk},\left(j=\mathrm{1,2},\dots ,n; k=\mathrm{1,2},\dots ,m\right)$$

  (11)

  $$\sum _{j=0}^n{x}_{ijk } {=y}_{ik},\left(i=\mathrm{1,2},\dots ,n; k=\mathrm{1,2},\dots ,m\right)$$

  (12)
• Material requirement constraints: Material requirements are met at each workstation.

  $$\sum _{k=1}^K\sum _{i=0}^N{x}_{ijk}=1\left(j=\mathrm{1,2},\dots ,n\right)$$

  (13)
• Loop constraints: Each subloop starts at the raw material library and ends at the raw material library.

  $$\sum _{k=1}^K\sum _{j=1}^N{x}_{0jk}=\sum _{k=1}^K\sum _{i=1}^N{x}_{i0k}$$

  (14)
• Workstation demand time: The JIT problem contradicts the robust problem, so the lead time is used as the safety time for the vehicle. JIT-RMOVRPTW eliminates the traditional time windows, with ${JIT}_i$ serving as the deadline for the JIT problem (as shown in Figure 1). $p$ is the slack time. As shown in Equation (17), for the station $i$, the larger the $q_i$, the higher the JIT requirement.

  $$b_i=\mathit{min}\{b_{ie}\}$$

  (15)

  $$a_i=\mathit{max}\{a_{ie}\},(a_{ie}\le b_i)$$

  (16)

  $${JIT}_i=b_i-\left(b_i-a_i\right)\ast p\ast \left(1-{\displaystyle \frac{q_i}{Q}}\right)$$

  (17)
• The arrival time of the AGVs:

  $$t_j=t_i+{\displaystyle \frac{d_{ij}}{v}}{x}_{ijk}+s_i,\left(i=\mathrm{0,1},2,\dots ,n; j=\mathrm{0,1},2,\dots ,n\right)$$

  (18)
• Uncertainty constraints: $t_i^{~}$ is the uncertainty time of arriving at station $i$. $\zeta$ is the overall random disturbance. $\overline{D}$ is the average distance between workstations. $\widetilde{u_{ij}}$ is the characteristic coefficient between stations $i$ and $j$. $\widetilde{u_{0i}}$ has unique characteristics, so it is discussed separately in Equation (19).

  $$\widetilde{u_{ij}}=\left\{\begin{array}{l}{d}_{ij}/\overline{D},(i=\mathrm{1,2},\dots ,n; j=\mathrm{1,2},\dots ,n)\\ 1,(i=0; j=\mathrm{1,2},\dots ,n)\end{array}\right.$$

  (19)

  $$t_j^{~}=t_j+\widetilde{u_{ij}}\overline{\zeta },\left(\overline{\zeta }={\displaystyle \frac{\zeta }{n}}\right)$$

  (20)
• Feasibility of workstations: To measure the robustness of the solution, the metric alpha is proposed in JIT-RMOVRPTW. Alpha is specifically defined according to different random distributions that different scenarios have. In this paper, $\zeta$ is taken as a continuous probability distribution of a triangular distribution with a lower limit of $min$, a peak location of $mid$, and an upper limit of $max$, which can be expressed as [$min,max,mid$]. Equation (21) represents the range of random arrival times of the station $j$. Equation (22) represents the degree of uncertainty from station $i$ to station $j$.

  $$tj~=ti~+\mathrm{d}ijvxijk+si+\mathrm{u}ij\zeta , i=0,1,2,\dots , n; j=0,1,2,\dots , n$$

  (21)

  $${interval}_{ij}^{~}=\mathit{max}\left(\widetilde{u_{ij}}\overline{\zeta }\right)-\mathit{min}\left(\widetilde{u_{ij}}\overline{\zeta }\right)$$

  (22)

${alpha}_i$ represents the feasibility of station $i$. The value of ${alpha}_i$ is shown by Equation (23). The maximum value of alpha is set to 2 to prevent too large an alpha from affecting the realization of the JIT.

$${\displaystyle \begin{array}{c}{alpha}_i=\\ \left\{\begin{array}{ll}0& ; min\left(t_i^{~}\right)>b_i\\ {\displaystyle \frac{{\left(\left|b_i-\mathit{max}\left(t_i^{~}\right)\right|-\mathit{min}\left(\widetilde{u_{ij}}\overline{\zeta }\right)\right)}^2}{{interval}_{ij}^{~}\ast \left(\mathrm{mid}\left(\widetilde{u_{ij}}\overline{\zeta }\right)-\mathit{min}\left(\widetilde{u_{ij}}\overline{\zeta }\right)\right)}}& ; max\left(t_i^{~}\right)>b_i and min\left(t_i^{~}\right)<b_i and \left|b_i-\mathit{max}\left(t_i^{~}\right)\right|\le mid\left(\widetilde{u_{ij}}\overline{\zeta }\right) \\ 1-{\displaystyle \frac{{\left(\mathit{max}\left(\widetilde{u_{ij}}\overline{\zeta }\right)-\left|b_i-\mathit{max}\left(t_i^{~}\right)\right|\right)}^2}{{interval}_{ij}^{~}\ast \left(\mathit{max}\left(\widetilde{u_{ij}}\overline{\zeta }\right)-\mathrm{mid}\left(\widetilde{u_{ij}}\overline{\zeta }\right) \right)}}& ; max\left(t_i^{~}\right)>b_i and min\left(t_i^{~}\right)<b_i and \left|b_i-\mathit{max}\left(t_i^{~}\right)\right|>mid\left(\widetilde{u_{ij}}\overline{\zeta }\right) \\ {\displaystyle \frac{b_i-\max \left(t_i^{~}\right)}{{interval}_{ij}^{~}}}& ; max\left(t_i^{~}\right)<b_i and {\displaystyle \frac{b_i-max\left(t_i^{~}\right)}{{interval}_{ij}^{~}}}\le 2\\ 2& ; {\displaystyle \frac{b_i-max\left(t_i^{~}\right)}{{interval}_{ij}^{~}}}>2\end{array}\right.\end{array}}$$

(23)

12.

Time window constraints:

$$t_i\le {JIT}_i,\left(i=\mathrm{1,2},\dots ,n\right)$$

(24)

13.

Advanced time for materials:

$$t_i\le {JIT}_i,\left(i=\mathrm{1,2},\dots ,n\right)w_{ie}={JIT}_{ie}-t_i\left(i=\mathrm{1,2},\dots n; e=\mathrm{1,2},\dots ,E_i\right)$$

(25)

14.

Penalty time: $w_{kmin}$ indicates the minimum advance time of the material required for station $i$, which is delivered by the $kth$ AGV. $w_k$ indicates the total penalty time of each AGV. Equation (28) indicates that the departure time from the raw library of the $kth$ AGV is delayed for $w_{kmin}$ to satisfy the smallest demand time of the material required for workstation $i$ delivered by the $kth$ AGV in time. $N_k$ indicates the set of stations distributed by the $kth$ AGV.

$$w_i=\sum _{e=1}^{E_i}{w}_{ie}\left(i=\mathrm{1,2},\dots n\right)\#\left(26\right)$$

(26)

$$w_{kmin}=\min \left\{w_{ie}\right\},\left(i{\in N}_k,e\in E_i\right)$$

(27)

$$w_k=\sum _{i=1}^{N_k}\left(w_i-w_{kmin}\right)$$

(28)

15.

Decision-making variables:

$$x_{ijk}=\left\{\begin{array}{l}1, kth AGVis responsible for the distribution from workstation i to j\\ 0, other\end{array}\right.$$

(29)

$$y_{ik}=\left\{\begin{array}{l}1,Delivery service for station i is completed by the kth vehicle\\ 0,other\end{array}\right.$$

(30)

Thus, the JIT-RMOVRPTW can be summarized by three objectives defined by Equations (1)–(4), subject to constraint conditions defined by Equations (5)–(30).

4. Proposed Algorithm

In this section, the overall NSACOWDRL process is first described. Then, the six components of NSACOWDRL are introduced, which include (1) solution construction, (2) nondominated sorting based on reference points, (3) the pheromone updating strategy, (4) local search based on the DDQN, (5) the probabilistic transfer formula, and (6) the robust optimization method. The optimization process of the NSACOWDRL is given in Figure 2.

4.1. Framework of the NSACOWDRL

The framework of the NSACOWDRL is given in Figure 3. The NSACOWDRL consists of two stages: fully searching the solution space to generate Pareto solutions and robust selection of Pareto solutions.

(1) Stage 1: Solution Space Characteristic Exploration and Pareto Solution Generation

Step 1: Initialize the algorithm parameters, including pheromone matrix, distance matrix, neural networks, etc. The initial number of iterations is 0.

Step 2: Obtain the set of feasible routes according to the solution construction strategy (see Section 4.2 for details).

Step 3: The routings are stratified by nondominated sorting based on the reference points, and N routings are selected for this iteration.

Step 4: The pheromone matrix is updated according to the pheromone update strategy.

Step 5: When the iteration is a specific value, the network of the DDQN is trained through the nondominated solutions, and the N paths are obtained by the trained network to participate in the next iteration through the selection of the actions.

Step 6: The number of iterations plus 1.

Step 7: If the maximum number of iterations is reached, output the solution set, otherwise skip to step 2.

(2) Stage 2: Robust Multiobjective Optimization

Step 1: The Pareto solutions obtained in stage 1 are partitioned by weights that are uniformly distributed in the solution space.

Step 2: Each solution is assigned to the weight closest to this solution.

Step 3: Through Monte Carlo simulation, the Monte Carlo feasible times for each solution is obtained to quantify the feasibility of the solution.

Step 4: The set of solutions belonging to each weight is then selected according to feasibility, respectively, to obtain the set of Pareto solutions for each weight. Robust solutions that take diversity into account are obtained.

4.2. Solution Construction

NSACOWDRL constructs solutions by imitating the behavior of ant colonies foraging for food with the following steps.

Step 1: There are $m$ ants in the raw material bank. Record the number of ants as 0.

Step 2: Each ant sets out independently.

Step 3: The ant calculates the workstations that satisfy the constraints based on the current information and obtains the set of feasible workstations.

Step 4: If the feasible set is empty and the ant does not visit all the workstations, the ant returns to the raw material library and sets out again. Otherwise, calculate the node transfer probability according to the transfer probability formula and select the next node from the feasible set via roulette. Then, update the feasible set.

Step 5: Repeat Steps 3–4 until the ant distributes all the empty workstations, and record the route of the ant. The number of ants plus 1.

Step 6: Repeat Steps 2–5 until the number of ants is $m$. The feasible routes are obtained.

The pseudo-code for solution construction is as follows (Algorithm 1):

Algorithm 1: Solution Construction

Input: $V=\left\{\mathrm{0,1},\dots ,n\right\}:\mathrm{s}\mathrm{e}\mathrm{t} \mathrm{o}\mathrm{f} \mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}$; $d_{ij}:\mathrm{t}\mathrm{h}\mathrm{e} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e} \mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{n} \mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s};$ $Q:\mathrm{t}\mathrm{h}\mathrm{e} \mathrm{m}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{m} \mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d} \mathrm{c}\mathrm{a}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}$; $m:the number of ants$; $K=0$; While $K$ < $m$  Set of undistributed workstations: $S=${1,2,…,n}; depart from the raw materials warehouse: $i$←0.  $route=\varnothing$;   While $S!=\mathrm{\varnothing }$    calculates the workstations that satisfy the constraints to obtains the set of feasible workstations, named $A$.    if $A!=\mathrm{\varnothing }$     $\mathrm{calculate} \mathrm{the} \mathrm{node} \mathrm{transfer} \mathrm{probability} \mathrm{according} \mathrm{to} \mathrm{the} \mathrm{transfer} \mathrm{probability} \mathrm{formula} \mathrm{and} \mathrm{select} \mathrm{the} \mathrm{next} \mathrm{node} j$ $\mathrm{from} A$ via roulette.     $S$ = $S$ -[$j$]; $A$ = $A$ -[$j$]; $i$←$j$; Update $A$. Append $i$ to $route(K+1)$    else     Return to raw material library: $j=$0; Update $A$    end   end  $K=K+1$; end Output: $route:\mathrm{t}\mathrm{h}\mathrm{e} \mathrm{s}\mathrm{e}\mathrm{t} \mathrm{o}\mathrm{f} \mathrm{f}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e} \mathrm{r}\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{s}$;

4.3. Nondominated Sorting Based on Reference Points

NSACOWDRL uses nondominated sorting based on reference points to obtain Pareto solutions with good diversity. Nondominated routing indicates that each of the routes is better than the others for some objectives. Searching for nondominated routing can expand the search range in the solution space, which can increase the probability of selecting better routing and improve the diversity of the algorithm.

In the NSACOWDRL, during the selection of half of the individuals in each generation, the paths obtained from the two iterations before and after are selected in a nondominated way to obtain more elite generation in such a way that it is elitist for high-quality individuals and at the same time, it will sufficiently bias in favor of the individuals with better nondominated ranks.

4.3.1. Nondominated Sorting

After one iteration of the NSACOWDRL, a set of routings can be obtained. To improve the diversity of routings, a set of feasible routings named $Q_{ur}$ is obtained by mixing the routings of the current generation with the routings of the previous generation. Then, $Q_{ur}$ is stratified by nondominated sorting via the following steps:

First, $n_i$ represents the number of individuals who dominate individual $i$. All individuals in $Q_{ur}$ with $n_i$ = 0 are found by a two-by-two comparison. They are given the nondominance rank 1. These individuals are stored in the set $R_1$;

Then, find the individuals that are dominated by the individuals in $R_1$ and store them in the set $S_1$. Subtract 1 from $n_j$ for each individual in $S_1$. If ($n_j$ − 1) equals 0, individual $j$ is given nondominance rank 2 and is stored in the set $R_2$.

The above operation is repeated for the remaining individuals in $Q_{ur}$ until all individuals have been assigned a nondominance rank representing the importance of the routing.

4.3.2. Diversity Conservation

In the $Q_{ur}$ containing a large number of solutions, there will be many ranks, which leads to the existence of individuals with nondominated ranks that are too low and which are less important. These individuals not only reduce the running efficiency of the algorithm but also cause pheromone concentration changes in the poorer individuals to be recorded, thus increasing the search difficulty of the algorithm or even leading to a fall into a local optimum.

To eliminate this difficulty, a routing selection strategy based on reference points is adopted. The solutions in $Q_{ur}$ are selected according to their impact on the diversity of the algorithm. The uniformly distributed reference points are generated through the hyperplane. The individual is selected based on the association between the solutions and reference points. This strategy uses well-distributed reference points to maintain the diversity of populations. The steps are described below:

(1) The individuals starting from nondominance rank 1 are added to set $D$ sequentially until the size of $D$ is larger than $N_{max}$ after adding the individuals with nondominance rank L to $D$. $N_{max}$ is the maximum number of nondominated solutions. The individuals with nondominated ranks from 1 to (L-1) are added to set $G$. Individuals with nondominant rank L are added to set $G^{*}$.

(2) Establish reference points on the hyperplane:

The hyperplane has the same tilt for all coordinate axes and an intercept of one on each axis. For the problem with M objectives, reference points are established on an (M-1)-dimensional hyperplane. For the three-objective problem, to generate reference points, each coordinate axis is uniformly segmented into H segments. The plane perpendicular to the coordinate axis is established through the segmented points of the coordinate axis The reference point is the intersection of planes from different coordinate axes with the hyperplane. The number of reference points is determined by Equation (31). Figure 4 shows the hyperplane of three objectives and reference points on the hyperplane when H is set to 5 (a detailed discussion of the reference points is given in Section 5.4).

$$P=C_{M+H-1}^H$$

(31)

(3) Normalize the individuals:

First, calculate the ideal point $z^{\mathrm{m}\mathrm{i}\mathrm{n}}$, which is the point consisting of the minimum value of all individuals in set $D$ on each objective. All individuals in set $D$ are subtracted from the ideal point.

Then, calculate the extreme points, which are the points with large values in one objective and small values in other objectives. M objectives have M extreme points. To find the extreme point on the $ith$ objective, the value of the $ith$ objective of all individuals is divided by a factor of 1, and the other objective values are divided by a factor of 10e−6. Then, excluding the $ith$ objective value, the largest value of each individual is found. Comparing these values, the point corresponding to the smallest of these values is the extreme value point on the $ith$ objective.

The (M-1) dimensional hyperplane is formed using M extreme points. Then, the intercept between the hyperplane and the coordinate axes is calculated. Figure 5 shows the two-dimensional hyperplane formed using three extreme points. $z_{max}^i$ represents the extreme point of the $ith$ objective. $a_i$ represents the intercept of the $ith$ coordinate axis.

Finally, the normalized objective value is calculated by Equation (32)

$$f_i^n\left(x\right)={\displaystyle \frac{f_i\left(x\right)-z_i^{min}}{a_i-z_i^{min}}}, i=\mathrm{1,2},\dots ,M$$

(32)

where $f_i\left(x\right)$ represents the value of the individual in the $ith$ objective. $z_i^{min}$ represents the value of the ideal point in the $ith$ objective.

(4) Link individuals to reference points:

The reference point is connected to the origin of the coordinates to form the connecting line. The vertical distance from each individual in set $D$ to each connecting line is calculated. The individual is then associated with the reference line closest to it. Each individual will correspond to a reference point. $Pj$ represents the number of individuals in set $G$ associated with reference point j.

(5) Select individuals:

The individual selection operation is executed according to the number $Pj$ from smallest to largest. Starting from the reference point with the smallest $Pj$ and if more than one reference point has the smallest $Pj$, choose a reference point randomly among these reference points. If $Pj$ = 0 and there are individuals associated with the selected reference point in the set $G^{\ast }$, the individual with the smallest distance from the reference point is selected and added to $G$. If there is no individual associated with it in $G^{*}$, this reference point is no longer considered in the rest of the operations. If $Pj$ ≥ 1 and there are individuals associated with this reference point in $G^{*}$, an associated individual is randomly selected to join $G$. The above operation is repeated until the size of $G$ equals $N_{max}$.

This method involves a small habitat protection operation. The method preserves the diversity of the algorithm by selecting solutions within the region where the solution’s distribution of the previous Pareto front is sparser. This selection operation uses hyperplane generation reference points and can be easily generalized to high-dimensional multiobjective problems to solve the difficulty of maintaining the diversity of solution sets in high-dimensional multiobjective problems.

4.4. Pheromone Update Strategy

The pheromone update strategy guides the algorithm update through a positive feedback mechanism; however, if the positive feedback mechanism is too strong, local convergence problems can easily occur, and if the mechanism is too weak, the algorithm will converge more slowly. To improve the convergence and diversity of the algorithms, this paper proposes a pheromone update strategy that combines the nondominated ranks with the elite routing strategy and the max-min pheromone update strategy. In this strategy, the pheromones of the routes are updated according to their nondominated rank, which comprehensively considers the influence of multiple objectives. Then, the elite paths with better nondominated ranks are updated twice to improve the convergence of the algorithm. Finally, to avoid the algorithm falling into a local optimum due to an excessively large difference between pheromone concentrations, the max-min pheromone strategy is adopted to limit the range of pheromone concentrations.

The update strategy is as follows:

(1) The pheromone update process is shown in Equations (33)–(35):

$${\tau }_{ij}\left(t+1\right)=\left(1-\rho \right){\tau }_{ij}\left(t\right)+\Delta {\tau }_{ij}\left(t,t+1\right)+\Delta {{\tau }_{ij}}^{\mathrm{*}}\left(t,t+1\right)$$

(33)

$$\Delta {\tau }_{ij}\left(t,t+1\right)=\sum _{k=1}^m\Delta {\tau }_{ij}^k\left(t,t+1\right)$$

(34)

$$\Delta {{\tau }_{ij}}^{\mathrm{*}}\left(t,t+1\right)=\sum _{k=1}^m\Delta {\tau }_{ij}^{\ast k}\left(t,t+1\right)$$

(35)

where $\rho$ ($0<\rho <1$) represents the pheromone volatilization rate. $\Delta {\tau }_{ij}(t,t+1)$ represents the pheromone increment of path (i, j) as the sum of the pheromone increments left by all ants that traverse path (i, j) in the cycle. $\mathsf{\Delta }{\tau }_{ij}^k$ represents the amount of pheromone released by ant $k$ to edge $\left(i,j\right)$. $\Delta {{\tau }_{ij}}^{\mathrm{*}}$ represents the pheromone increment of the elite path $\left(i,j\right)$. $\mathsf{\Delta }{\tau }_{ij}^{\ast k}$ represents the amount of pheromone released by elite ant $\ast k$ to edge $\left(i,j\right)$.

(2) Common ant pheromone update strategy: $\mathsf{\Delta }{\tau }_{ij}^k$ is updated for all routings according to the nondominance rank.

$$\Delta {{\tau }_{ij}}^k\left(t,t+1\right)=\left\{\begin{array}{l}{C}_P\ast C_N/n, n=1, 2,3,4 \\ 0,other\end{array}\right.$$

(36)

where $k$ represents the $kth$ ant. $C_P$ represents the pheromone constant, $C_N$ represents the nondominant constant, and $n$ represents the nondominated rank of the routing in this iteration.

(3) Elite routing pheromone update strategy: $\mathsf{\Delta }{\tau }_{ij}^{\ast k}$ updated pheromone concentrations for elite routing with a nondominance rank of 1, 2.

$$\Delta {{\tau }_{ij}}^{\mathrm{*}k}\left(t,t+1\right)=\left\{\begin{array}{l}(6-n)\ast C_P\ast C_N/n, n=\mathrm{1,2} \\ 0,other\end{array}\right.$$

(37)

(4) Max-min pheromone limits: Since there are various nondominance ranks, a large difference in pheromone concentration tends to occur between paths after the pheromone is updated. A too large difference in pheromone concentration can easily lead to premature stalling of the search, increasing susceptibility of the algorithm to falling into local optima. To avoid this, explicit limits are imposed on the minimum and maximum pheromone concentrations in the pheromone update strategy, as shown in Equation (38). After each iteration, it must be ensured that the pheromone concentration adheres to the limits.

$${\tau }_{ij}\left(t+1\right)=\left\{\begin{array}{c}{\log }_a\left({\displaystyle \frac{{\tau }_{ij}\left(t+1\right)}{2\sigma }}\right),{\tau }_{ij}\ge {\tau }_{min}\\ {\tau }_{min},other\\ \end{array}\right.$$

(38)

$${\tau }_{min}={\displaystyle \frac{1}{2\ast cusum\ast \left(1-\rho \right)}}$$

(39)

where $a$ represents the base of the logarithm, which is used to control the range of pheromone concentrations; $\sigma$ represents the limiting factor; and $cusum$ represents the number of nodes.

4.5. Local Search Strategy

The DDQN [56] is an off-policy DRL algorithm which is improved from the DQN. The DDQN decouples the two steps of selection of the target Q value action and computation of the target Q value to address the problem that the DQN potentially leads to maximization bias, which results in overestimation.

The decoupling process is as follows:

The DDQN constructs two networks: the evaluation network and the target network. The structure of the evaluation network and the target network are identical, but the weighting parameters are different.

First, the DDQN selects the action corresponding to the maximum Q value by the evaluation network, as shown in Equation (40):

$$a_{\mathrm{m}\mathrm{a}\mathrm{x}}=arg{max}_{a^{\prime }}Q\left(s_{t+1},a^{\prime }; {{\theta }^{\prime }}_t\right)$$

(40)

Then, the target network calculates the target Q value as shown in Equation (41):

$$y_t=r_t+\gamma Q(s_{t+1},a_{max}; {\theta }_t)$$

(41)

Combining the states, the target Q value of is shown in Equation (42):

$$y_t=\left\{\begin{array}{cc}\hfill r_t\hfill & if s^{\prime } is terminal\hfill \\ \hfill r_t+\gamma Q\left(s_{t+1},arg{max}_{a^{\prime }}Q\left(s_{t+1},a^{\prime }; {{\theta }^{\prime }}_t\right); {\theta }_t\right)\hfill & otherwise\hfill \end{array}\right.$$

(42)

As shown in the Equation (43), the DDQN adopts the mean square error (MSE) of the estimated Q value and the target Q value as the loss function, and updates the evaluation network by gradient descent backpropagation.

$$L\left({\theta }_t\right)={E\left[\right(y_t-Q(s_t,a; {\theta }_t))}^2]$$

(43)

The DDQN learns past experience offline by building a replay buffer. After $N^{-}$ rounds of the replay buffer sampling, the weight parameters of the evaluation network are copied to the target network so as to realize the model self-learning.

To solve the MOPs, the DDQN adopts the Pareto solution set to train the network and guide the learning direction (details are in Section 4.5.1 and Section 4.5.2). The pseudo-code is shown in the table below (Algorithm 2):

Algorithm 2: Double DQN in MOPs

Input:  $D:empty replay buffer; \theta : network parameters; {\theta }^{-}:copy of \theta$;  $N_r:replay buffer maximum size; N_b:training batch size$;$N^{-}:target network replacement freq$;  $G:the set of Pareto solutions that have been obtained$; $G^H$: the Pareto frontier in $G$ For $i$ from 1 to size $\left(G^H\right)$  for $j$ from 1 to size $\left(G_i^H\right)$   $\mathrm{Take} (s_j,a_j)$ $\mathrm{and} r_j$$. \mathrm{Do} \mathrm{a} \mathrm{gradient} \mathrm{descent} \mathrm{step} \mathrm{with} \mathrm{loss} {‖r_j(s_j,a_j)-Q(s_j,a_j; \theta )‖}^2$$\mathrm{to} \mathrm{train} \theta$.  end end For $episode e\in \{\mathrm{1,2},\dots M\}$ do  Initialize frame sequence x←()  for $t\in \{\mathrm{0,1},\dots \}$ do   $\mathrm{Set} \mathrm{state} \mathrm{s}\leftarrow \mathrm{x}, \mathrm{sample} \mathrm{action} a$$~{\pi }_B$   $\mathrm{Sample} \mathrm{next} \mathrm{frame} x^{\prime }$$\mathrm{from} \mathrm{environment} \epsilon$$\mathrm{given} (s,a)$ and receive $r$, and append $x^{\prime }$ to x   $\mathrm{Compare} \mathrm{the} \epsilon$$\mathrm{to} G$ to obtain the overall learning rate ${Alpha}_{target}$, set $r^{\prime }$ $\leftarrow r$$* Alph{a}_{target}$   if $\left|\mathit{x}\right|$> $N_f$ then delete oldest frame $x_{t_{min}}$ from x end    $\mathrm{Set} s^{\prime }$$\leftarrow \mathrm{x}, \mathrm{and} \mathrm{add} (s,a,r^{\prime },s^{\prime })$$\mathrm{to} D$,    Replacing the oldest tuple if $\left|\mathit{D}\right|$> $N_r$    $\mathrm{Sample} \mathrm{a} \mathrm{minibatch} \mathrm{of} N_b$$\mathrm{tuples} (s,a,r^{\prime },s^{\prime })$$~ \mathrm{Unif} (D$)    $\mathrm{Construct} \mathrm{target} \mathrm{values}, \mathrm{one} \mathrm{for} \mathrm{each} \mathrm{of} \mathrm{the} N_b$ tuples    $a^{max}\left(s^{\prime }; \theta \right)=arg{max}_{a^{\prime }}Q(s^{\prime },a^{\prime }; \theta )$;    $y_j=\left\{\begin{array}{cc}\hfill r^{\prime }\hfill & if s^{\prime } is terminal\hfill \\ \hfill r+\gamma Q\left(s^{\prime },a^{max}(s^{\prime }; \theta ); {\theta }^{-}\right),\hfill & otherwise.\hfill \end{array}\right.$;    $\mathrm{Do} \mathrm{a} \mathrm{gradient} \mathrm{descent} \mathrm{step} \mathrm{with} \mathrm{loss} {‖y_i-Q(s,a; \theta )‖}^2$    $\mathrm{Set} {\theta }^{-}$$\leftarrow \theta$$\mathrm{every} N^{-}$ steps   end  end end

4.5.1. Environment Setup

The evolutionary algorithm guides the reinforcement learning for searching. The following is the process of establishing the DDQN environment and directing its learning direction:

(1) Neural Network Construction

The input to the problem in this paper is a three-dimensional vector which has low dimensionality, so this paper constructs a function-fitting neural network which is structured as a fully connected network with three hidden layers. The input layer has three nodes. Each hidden layer has 40 nodes.

The activation functions are chosen to be linear transfer function (purelin), saturated linear transfer function (satlin), and positive linear transfer function (poslin). Multilayer networks use different activation functions for each layer to allow the neural network to approximate arbitrary functions.

The network chooses a multilayer neural network training function: the variable learning rate backpropagation algorithm (traingdx). It uses momentum optimization to accelerate convergence, and improves stability and accuracy through an adaptive learning rate to make the gradient descent smoother.

The network structure is shown in Figure 6.

After the initial construction of the neural network is completed, the initial parameter training of the neural network is required. In order to improve the effectiveness of parameter training and the rate of neural net approximation, the Pareto frontier is chosen as the input for initial training in this paper. The Pareto frontier, named $Hpareto$, is selected in the solution set G obtained by nondominated ordering; meanwhile the set with the largest nondominated rank, named $Lpareto$, is selected. The mean square error between the estimated Q value and the corresponding reward of $Hpareto$ is then used as the loss function to train the initial evaluation network, as shown in Equation (44).

$$\mathrm{L}\left({\mathsf{\theta }}_{\mathrm{t}}\right)={{\mathrm{E}}_{\left(\mathrm{s},\mathrm{a},\mathrm{r},{\mathrm{s}}^{\prime }\right)~\mathrm{U}\left(\mathrm{H}\mathrm{p}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{o}\right)}\left[\right(\mathrm{r}(\mathrm{s},\mathrm{a})-\mathrm{Q}(\mathrm{s},\mathrm{a}; {\mathsf{\theta }}_{\mathrm{t}}))}^2]$$

(44)

$r\left(s,a\right)$ represents the reward corresponding to Hpareto in the $(s,a)$ environment. $Q(s,a; {\theta }_t)$ represents the estimated Q value obtained by the evaluation network in the $(s,a)$ environment.

(2) Reward Return Design

This paper studies the multiobjective VRP problem where the reward value is related to the pheromone between the routes, the heuristic function, and the advance time. The rewards are shown in Equation (45):

$$r_t={\tau }_{ij}^{\alpha }(t)\times {\eta }_{ij}^{\beta }(t)\times (1/w_j(t_j){)}^{\delta }$$

(45)

This equation simultaneously takes full account of multiobjectives, so that when an agent completes an action choice that causes a change in the state, its corresponding reward also changes. This design allows for the effective mapping of the multiple objectives of the study to the rewards of deep reinforcement learning.

For the VRP problem, the optimal solution of the subpaths can easily cause the algorithm to fall into a local optimum, so this paper proposes an overall learning rate called $Alph{a}_{target}$. After the action selection is finished to obtain a complete path for all the workstation requirements, the IGD [57] metrics of this path with the two path sets of $Hpareto$ and $Lpareto$ are calculated to obtain $IGD\_Lpareto$ and $IGD\_Hpareto$.Then $Alph{a}_{target}$ is calculated according to Equation (46), and the $r_t$ is updated according to Equation (47). $Alph{a}_{target}$ quantifies the advantages and disadvantages of the obtained path as a ratio of distances from $Hpareto$ and $Lpareto$, in order to correct the rewards of the various subpaths of this path, so that paths with smaller IGDs have a higher advantage, whereas paths with larger IGDs have a larger disadvantage.

$$Alph{a}_{target}=IGD\_Lpareto/IGD\_Hpareto$$

(46)

$$r_t=Alph{a}_{target}\ast r_t$$

(47)

The structure of the DDQN algorithm is shown in Figure 7.

4.5.2. Output Designs

The reinforcement learning results are used to improve the optimization performance of the evolutionary algorithm.

Since the training of DDQN neural networks needs to rely on the replay buffer, and the solution sets obtained between adjacent iterations of the algorithm have a certain degree of approximation, this leads to the difficulty of having significant differences in the replay buffers of adjacent iterations, thus making it difficult for neural networks to perform effective parameter optimization. In order to improve the effectiveness of neural network optimization, this paper sets the neural network to be trained at a specific number of iterations. For the evolutionary algorithm, the greater the number of iterations, the better the results and the more iterations are needed for further optimization of the iteration results, so the neural network training is set at 50, 100, and 200 generations.

The trained network obtains 40 complete paths by action selection, which are recorded in the set $G\prime$, to participate in nondominated sorting in the next iteration. The pseudo-code is as follows (Algorithm 3):

Algorithm 3: Route Generation by DDQN

Input:  $Q\left(s,a; \theta \right):\mathrm{t}\mathrm{h}\mathrm{e} \mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d} \mathrm{n}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}$; epsilon: $parameter of greedy policy$; N:  $number of routes$;  $V:$$\mathrm{set} \mathrm{of} \mathrm{workstations}; cunsum:$$\mathrm{number} \mathrm{of} \mathrm{workstations}; s_0:the outset$ While $N\le 40$ do $\mathrm{Initialize} \mathrm{parameters}: s$$\leftarrow s_0$$; action$$\leftarrow \varnothing$;  While $size\left(action\right)\le cunsum$ do  $\mathrm{The} \mathrm{feasible} \mathrm{solutions} \mathrm{selected} \mathrm{from} \mathrm{the} \mathrm{unassigned} \mathrm{workstations} \mathrm{are} \mathrm{recorded} \mathrm{in} V_a$   if $V_a$ is not empty    if $\mathrm{rand} < epsilon$     $a^{\prime }=arg{max}_{a}Q\left(s,a; \theta \right); \left(a\in V_a\right).$$\mathrm{Set} s\leftarrow a^{\prime },$$\mathrm{append} a^{\prime }$$\mathrm{to} action$    else     $\mathrm{Take} \mathrm{a} \mathrm{random} \mathrm{integer} i$$\mathrm{from} 1 \mathrm{to} \mathrm{size} \left(V_a\right)$$, \mathrm{select} \mathrm{the} \mathrm{ith} \mathrm{action} a^i$$\mathrm{in} V_a$$, a^{\prime }\leftarrow a^i$;     $\mathrm{Set} s\leftarrow a^{\prime },$$\mathrm{append} a^{\prime }$$\mathrm{to} action$    end   else    $\mathrm{Set} s$$\leftarrow s_0$   end  $\mathrm{add} action$$\mathrm{to} G\prime$  end end Output:  $G\prime :\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{s}\mathrm{e}\mathrm{t}\mathrm{o}\mathrm{f}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{s}$;

In the probabilistic selection of nodes, the current workstation $i$ and the next workstation $j$ are used as inputs into the trained network to obtain the Q value between workstations to participate in the probabilistic selection as shown in Equation (48).

$$Q_{ij}=Q\left(s_i,j; {\theta }_t\right),(j\in E)$$

(48)

$s_i$ represents the state of the current node $i$; $j$ represents the next node; $E$ represents the set of feasible nodes.

4.6. Node Transfer Probability Rule

NSACOWDRL uses a roulette wheel for node selection. The probability formula for participation in the roulette wheel is chosen by Equation (49).

$$J=\left\{\begin{array}{ll}{p}_{ij}\left(t\right),& other\\ Q_{ij}(t),& iter\in U\end{array}\right.$$

(49)

$J$ represents the probabilistic selection method. $p_{ij}\left(t\right)$ represents the selection based on probability formula and $Q_{ij}\left(t\right)$ represents the selection based on Q table. $U$ represents the set of a specific number of iterations.

In the JIT-RMOVRPTW, the traditional time windows are eliminated, so the time window width is canceled. In the JIT-RMOVRPTW, the movement of ants is related to the pheromone concentration, path distance, waiting time, and the value of Q table. As shown in Equation (50), the probability formula of the JIT-RMOVRPTW is obtained based on the mathematical model of the problem.

$$p_{ij}^k\left(t\right)=\left\{\begin{array}{l}{\displaystyle \frac{{\tau }_{ij}^{\alpha }\left(t\right)\times {\eta }_{ij}^{\beta }\left(t\right)\times (1/w_j(t){)}^{\delta }\times Q_{ij}^{\gamma }(t)}{{\sum }_{s\in \left(J-B_k\right)}{\tau }_{is}^{\alpha }\left(t\right)\times {\eta }_{is}^{\beta }\left(t\right)\times (1/w_j(t){)}^{\delta }\times Q_{is}^{\left(SET\right)}\left(t\right)}}\\ 0,other\end{array}\right.$$

(50)

$$\gamma =iter-DDQ{N}_{iter}; \{DDQ{N}_{iter}\in U|{\displaystyle \frac{iter}{2}}\le DDQ{N}_{iter}<iter\}$$

(51)

$${\eta }_{ij}\left(t\right)={\displaystyle \frac{1}{d_{ij}}}$$

(52)

where $J$ represents the set of all neighboring nodes $j$ of node $i$; $s$ represents the node that ant $k$ is allowed to select next; ${\tau }_{ij}$ represents the pheromone concentration of each pathway; ${\eta }_{ij}$ is the heuristic function, as shown in Equation (52); $Q_{ij}$ is as shown in Equation (48), $\alpha$ is the pheromone concentration factor, $\beta$ is the heuristic function factor, $\gamma$ is the Q value factor, and $\delta$ is the waiting time factor.

4.7. Robust Multiobjective Optimization

The traditional robust selection strategy selects only the solution set with the highest feasibility. However, due to the conflicting nature among multiple objectives, it is difficult for the other objectives of the most feasible solution to be close to the optimum. For practical problems, appropriate delays are acceptable. So this paper proposes a robust selection method based on uniformly distributed weights in the solution space combined with Monte Carlo simulation. Since the solutions in the neighborhood centered at a certain point in the solution space have some degree of similarity, this robust optimization method splits the solution space by uniform weights, then converts the feasibility of solutions into Monte Carlo feasibility times. Finally, the optimal set of solutions in the solution space corresponding to each weight is found separately. This strategy obtains solutions that take into account robustness while guaranteeing diversity in multiple objectives.

Through this strategy, the more feasible solutions can be retained, and at the same time, based on the uniformly distributed weights, the solutions with lower feasibility but better other objectives are also retained, so as to provide the decision maker with more decision-making options in the face of different practical needs.

First, the solution space is partitioned by uniformly distributed weights, as shown in the Figure 8, with different colors representing different weights. Each weight represents a section of the solution space.

Then, each solution is associated with the weight which is closest to it. Following this, the Monte Carlo simulation is used to simulate each solution 1000 times to obtain the Monte Carlo feasible times for each solution, which is represented by $MC$; the pseudo-code is shown in Algorithm 4. Each solution can be represented as $(f_d,f_{alpha},f_{JIT},MC)$.

Algorithm 4: Monte Carlo Simulation

Input:  $NV$$: \mathrm{the} \mathrm{number} \mathrm{of} \mathrm{AGV}; route=\left\{route\left(1\right),route\left(2\right),\dots route\left(NV\right)\right\}$: a set of routes  $route\left(i\right)=\{node\left(1\right),node\left(2\right),\dots node(m\left)\right\}$$: \mathrm{a} \mathrm{route}; t=\{t_{\mathrm{1,2}},t_{\mathrm{1,3}},\dots t_{n,n-1}\}$: travel time  $\mathrm{s}=\{s_1$$, s_2,\dots s_n$$\}: \mathrm{serve} \mathrm{time}; \widetilde{u_{ij}}\overline{\zeta }$$: \mathrm{uncertain} \mathrm{time}; maxsim$: the times of Monte Carlo simulation  $MC:$ empty set of Monte Carlo feasible times for $i$$=1: length\left(route\right)$  for$j$$=1: length\left(route\right(i\left)\right)$   if $j$ = 1    $t_{route(i,j)}^{~}=t_{0,route(i,j)}+\widetilde{u_{ij}}\overline{\zeta }$;   else    $t_{route(i,j)}^{~}=t_{route\left(i,j-1\right)}^{~}+s_{route(i,j-1)}+t_{route\left(i,j-1\right),route(i,j)}+\widetilde{u_{ij}}\overline{\zeta }$;   end  end  for $times$$=1: maxsim$   if $route\left(i\right)$ satisfy constraints    $MC\left(i\right)$$=MC\left(i\right)+$1;   end  end end Output: $MC$: Monte Carlo feasible times for each solution

For each weight, if there exists at least one solution associated with it, the solution with the highest $MC$ is sought, the Monte Carlo times of this solution is represented as ${MC}_e$, and the solutions in the same weight for which $MC$ is less than ${MC}_e$ are deleted. The remaining solutions are the optimal set of solutions in the solution space represented by that weight. The final result is shown in Figure 9, which is the set of robust Pareto solutions that guarantees the diversity of the algorithm. The points in different colors represent the solutions corresponding to weights with the same color.

5. Experiments and Analyses

The comparison experiments to test the performance of NSACOWDRL is discussed in this section. In this section, we compare the performance of multiple multiobjective algorithms under different instances, NSACOWDRL under different JIT models, and NSACOWDRL under different network settings, respectively.

5.1. Experimental Setting

To test the performance of NSACOWDRL, four instances of Solomon’s VRPTW benchmark problems, named C202, C206, RC202, RC204, and one instance, named RL—according to the real case from a manufacturer in China—are adopted, in which each of the instances is tested with three different interference coefficients.

Based on the model and the experience of the manufacturer, the following modifications were made to the instances:

(1) The company divides the distribution time of one day into several cycle times. The different distribution tasks were performed at each cycle.

(2) All the workstations are available all day.

(3) The assembly workshop has standardized the delivery of materials. All the materials were delivered in standardized boxes, and all the materials required at the same station during a cycle were placed in the same carts.

(4) In this case, due to the constraints of the assembly workshop and the safety of the AGV operation, the maximum number of carts is nine.

Table 1. Material information dataset of the real case.

Material Number	X-Coordinate	Y-Coordinate	Number of Delivery Boxes	Left Time Window	Right Time Window	Service Time
0	0	11.2	0	0	0	0
1	6.9	30.4	1	792	1800	15
2	8.1	30.4	1	120	720	15
3	9.3	30.4	2	792	1800	30
4	9.3	30.4	1	72	720	15
5	12.9	30.4	1	240	1080	15
6	12.9	30.4	1	132	1080	15
7	15.3	30.4	1	240	1080	15
8	15.3	30.4	1	216	1080	15
9	20.1	30.4	1	288	1368	15
10	22.5	30.4	1	828	1800	15
11	24.9	30.4	2	1200	1800	30
12	24.9	30.4	2	450	1800	30
13	28.5	30.4	1	450	1800	15
14	28.5	30.4	1	540	1800	15
15	33.3	30.4	1	180	1080	15
16	6.9	25.3	1	1620	1800	15
17	10.5	25.3	5	900	1800	75
18	11.7	25.3	1	1200	1800	15
19	11.7	25.3	2	828	1800	30
20	21.3	25.3	3	115	720	45
21	22.5	25.3	1	1620	1800	15
22	27.3	25.3	5	600	1800	75
23	27.3	25.3	1	600	1800	15
24	28.5	25.3	1	540	1800	15

shows the dataset of RL. Figure 10 shows a simple illustrative scenario to visualize the layout and process of material delivery. The dotted lines indicate the routes on which the AGVs can run. The small rectangles joined together represent workstations. The black dots indicate the stations that need to be distributed.

The NSACOWDRL was executed for 250 generations. The number of ants was set to 40. $\alpha$, $\beta$, γ, and δ in the node transition probability formula were set as 1, 2, 2, and 2, respectively. The parameter $\rho$ in the pheromone update formula was set to 0.2. The parameter $H$ was set to 10. In the DDQN, the Buffer size was set to 60,000, the epsilon was set to 0.9, the Gama was set to 0.9, and the Target update interval was set to 1200.The parameters selected were not necessarily optimal for each model. This paper did not strictly select the optimal parameters.

5.2. Performance Metrics

In MOPs, the diversity and convergence of the algorithm are the main performance metrics that need to be evaluated. A single metric has difficulty comprehensively measuring the performance of the algorithm. In this paper, two performance metrics, HV [1] and IGD, are adopted to evaluate the performance of the algorithm. These metrics are described below:

As shown in Equation (53), the HV metric is the hypervolume metric, which is a measure of the diversity of an algorithm by calculating the volume of the target space formed by the set of nondominated solutions and reference points. The larger the HV is, the better the diversity of the algorithm.

$$HV=Leb({\bigcup }_{i=1}^{\left|S\right|}{v}_i)$$

(53)

$Leb$ represents the Lebesgue measure, which is used to measure the volume. $\left|S\right|$ represents the number of nondominated solution sets, and $v_i$ represents the hypervolume formed by the reference point and the ith solution in the solution set.

As shown in Equation (54), the IGD metric is an inverse generational distance metric: a comprehensive performance metric that evaluates the convergence and distribution performance of the algorithm by calculating the average minimum distance from each point on the true Pareto front (PF) to the set of individuals obtained by the algorithm. The smaller the IGD is, the better the convergence and distribution performance of the algorithm.

$$IGD\left(P,P^{\mathrm{*}}\right)={\displaystyle \frac{{\sum }_{x\in P^{\mathrm{*}}}{min}_{y\in P}dis\left(x,y\right)}{\left|P^{\mathrm{*}}\right|}}$$

(54)

$P$ is the solution set obtained by the algorithm, $P^{\ast }$ is a set of uniformly distributed reference points sampled from the Pareto front, and $dis\left(x,y\right)$ represents the Euclidean distance between the point $x$ in the reference set $P^{\ast }$ and the point $y$ in the solution set $P$.

Because it is difficult to find the true Pareto front in real scenarios, the PF is approximated by considering all nondominated solutions obtained by all algorithms.

In this experiment, the normalized metrics were used to unify the measures between the different objectives.

5.3. Experimental Results

Based on the dataset, to verify the effectiveness of the NSACOWDRL, experiments were carried out on the JIT-RMOVRPTW models successively. In the experiments, to demonstrate the superiority of NSACOWDRL for multiobjective problems, NSACOWDRL compares NSACO, NSGA-III, NSGA-II, and MOEA/D. Twenty independent runs were used to test the performance of the proposed algorithm in each experiment. The experiments presented the results of the proposed algorithm in terms of HV and IGD.

5.3.1. Results of the JIT-RMOVRPTW

(1) Analysis of the Performance Metric

Table 2. Mean and variance metric values of HV indicators.

Instance	$\overline{\mathit{\zeta }}$	NSACOWDRL	NSACO	NSGA-III	NSGA-II	MOEA/D
c202	Small	0.778107	0.659747	0.482335	0.620586	0.582376
		9.74E−05	0.000594	0.003465	0.002409	0.002838
	Middle	0.783101	0.685152	0.522198	0.622289	0.612993
		0.000137	0.000604	0.003457	0.004938	0.004171
	Large	0.708188	0.609913	0.404309	0.505273	0.508527
		0.000179	0.001238	0.004473	0.003483	0.005629
c206	Small	0.656716	0.557998	0.487161	0.652318	0.595443
		0.000298	0.000589	0.003799	0.007288	0.004559
	Middle	0.635402	0.514251	0.440545	0.561055	0.550337
		0.000254	0.000532	0.009169	0.003729	0.003936
	Large	0.5232	0.441354	0.345311	0.439341	0.478306
		0.000319	0.000555	0.005952	0.002454	0.003588
rc202	Small	0.635634	0.600488	0.560277	0.576173	0.582909
		0.000116	0.000439	0.001576	0.000463	0.001525
	Middle	0.591863	0.535674	0.495836	0.498852	0.567148
		0.000324	0.000513	0.000986	0.001867	0.002845
	Large	0.530264	0.474198	0.479061	0.556831	0.543224
		0.000897	0.000483	0.005019	0.001418	0.002381
rc204	Small	0.630401	0.597802	0.540024	0.471606	0.611194
		0.000183	0.000192	0.00278	0.003056	0.00235
	Middle	0.646317	0.617087	0.557666	0.571228	0.655838
		0.000236	0.00045	0.001948	0.002557	0.002609
	Large	0.607741	0.57318	0.55936	0.582608	0.559298
		0.001551	0.001147	0.003931	0.001271	0.008846
RL	Small	0.551697	0.515265	0.489368	0.436566	0.512563
		0.000503	0.0006	0.001642	0.000742	0.002456
	Middle	0.530147	0.507949	0.493525	0.383	0.417351
		0.000205	8.74E−05	0.00193	0.001007	0.00279
	Large	0.448719	0.427708	0.406559	0.311904	0.301467
		0.000595	0.000319	0.003167	0.001017	0.001066

and

Table 3. Mean and variance metric values of IGD indicators.

Instance	$\overline{\mathit{\zeta }}$	NSACOWDRL	NSACO	NSGA-III	NSGA-II	MOEA/D
c202	Small	0.067098	0.111056	0.221364	0.155425	0.149471
		9.94E−05	5.76E−05	0.001808	0.001132	0.000783
	Middle	0.070933	0.095232	0.195238	0.156117	0.142658
		0.000124	7.45E−05	0.000646	0.000893	0.000335
	Large	0.063498	0.096462	0.22384	0.179565	0.166596
		5.61E−05	5.11E−05	0.001706	0.001494	0.001324
c206	Small	0.07471	0.123859	0.214364	0.099614	0.157687
		0.000159	0.000116	0.001652	0.000463	0.000542
	Middle	0.089647	0.121177	0.205124	0.119925	0.15276
		6.96E−05	0.000127	0.001728	0.000238	0.001202
	Large	0.111621	0.14161	0.207618	0.117877	0.173173
		0.000449	0.000345	0.003593	0.000272	0.002181
rc202	Small	0.091929	0.095577	0.136415	0.12007	0.13582
		7.24E−05	4.42E−05	0.000804	0.000165	0.000713
	Middle	0.092914	0.107702	0.170686	0.158157	0.141599
		4.08E−05	5.77E−05	0.000682	0.000297	0.000587
	Large	0.143131	0.157795	0.211815	0.163551	0.184171
		0.000346	9.64E−05	0.000707	0.00036	0.000859
rc204	Small	0.099484	0.10002	0.155438	0.179327	0.124656
		0.000159	9.27E−05	0.001078	0.001097	0.000573
	Middle	0.107897	0.119688	0.217314	0.189579	0.22512
		9.31E−05	6.03E−05	0.001167	0.001371	0.001906
	Large	0.109758	0.110154	0.162679	0.119891	0.172319
		0.000275	0.00014	0.000765	0.000193	0.002228
RL	Small	0.081081	0.089546	0.114181	0.125006	0.085965
		1.93E−05	2.56E−05	0.000217	0.000377	0.000176
	Middle	0.07656	0.081223	0.12775	0.132086	0.107114
		1.51E−05	2.22E−05	0.000873	0.000183	0.000203
	Large	0.075806	0.075908	0.166999	0.113566	0.103795
		3.33E−05	1.65E−05	0.002057	7.47E−05	0.000376

present the mean and variance of the HV and IGD metrics for different algorithms on the JIT-RMOVRPTW dataset with three interference coefficients, respectively. The leftmost column represents the instance numbers. The second column represents three different interference coefficients, which are triangularly distributed. “Small”, ”Middle”, and “Large” represent interference coefficients from small to large. For the two rows that follow each disturbance coefficient, the top row represents the mean of the metrics obtained after the corresponding algorithm has been run 20 times independently with this disturbance coefficient, and the bottom row represents the variance of the metrics obtained.

As shown in Table 2 and Table 3, NSACOWDRL performs better with superior metrics in instances, and has more stable metrics. In comparison, in RL, rc204, and rc202, as the interference coefficient increases, the performance of NSGA-II starts to improve, and the performance of MOEA/D starts to decrease. However, in c202 and c206, as the interference coefficient increases, the performance of NSGA-II starts to decrease, and the performance of MOEA/D starts to improve. Meanwhile, NSGA-III performance is more stable. The performance of the algorithms is strongly influenced by the instances, where the performance of NSACO, NSGA-II, NSGA-III, and MOEA/D have advantages in different instances, whereas NSACOWDRL is always superior.

To determine the differences between the algorithms on different metrics, a two-by-two comparison p value test was carried out. As shown in Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15, the p value is represented by the heatmap. The X-axis is the interference coefficient, and the p values at different interference coefficients are shown on either side of the breakpoint in the X-axis. The upper triangle of each large square is the p value for HV and the lower triangle is the p value for IGD, where p values greater than 0.05 are marked in black.

The p-value plots show that the algorithms differ significantly between different instances and between different interference coefficients. NSACOWDRL is significantly better than the other algorithms, especially on the C202, C206, RL, whereas on RC202, and RC204, the performance of MOEA/D is close to that of NSACOWDRL in terms of the IGD metrics. Meanwhile, as the interference coefficient increases, the variation between algorithm performances begins to decrease on C202, C206, and RC204, whereas on RC202, RL, the variation fluctuates. Algorithm performance is affected by the combination of the instances and the interference coefficients; as the interference coefficients increase, the variation between algorithms is affected more in the RC-series instances than in the C-series.

As the result shows, the NSACOWDRL outperforms the other algorithms in terms of convergence and diversity when solving the JIT-RMOVRPTW. Furthermore, the NSACOWDRL has better generalization in solving problems for different scenarios.

5.3.2. Discussions

The results of the algorithms on the different instances show superiority of the NSACOWDRL on the proposed model. To evaluate the impact of multiple JIT strategies on the performance of the algorithm, the following three JIT strategies are compared:

Strategy 1: Consider transportation capacity and the slack time.

$${JIT}_i=b_i-\left(b_i-a_i\right)\ast p\ast \left(1-{\displaystyle \frac{q_i}{Q}}\right)$$

Strategy 2: Consider the slack time.

$${JIT}_i=b_i-\left(b_i-a_i\right)\ast p$$

Strategy 3: Schedule advance time for the time window.

$${JIT}_i=-advanced time$$

The results are shown in

Table 4. Mean and variance metric values of algorithms.

HV	Strategy	$\overline{\mathit{\zeta }}$	NSACOWDRL	NSACO	NSGA-III	NSGA-II	MOEA/D
	Strategy 1	Small	0.551697	0.515265	0.489368	0.436566	0.512563
			0.000503	0.0006	0.001642	0.000742	0.002456
		Middle	0.530147	0.507949	0.493525	0.383	0.417351
			0.000205	8.74E−05	0.00193	0.001007	0.00279
		Large	0.448719	0.427708	0.406559	0.311904	0.301467
			0.000595	0.000319	0.003167	0.001017	0.001066
		mean gap	18.67%	16.99%	17.62%	28.56%	41.18%
	Strategy 2	Small	0.548972	0.508876	0.469451	0.423446	0.45982
			0.00019059	0.00036	0.001223	0.001704	0.001119
		Middle	0.576724	0.558853	0.510205	0.437301	0.458765
			6.62792E−05	0.000253	0.002138	0.001383	0.001028
		Large	0.4549855	0.438979	0.430791	0.328832	0.333339
			9.52814E−05	0.000192	0.002057	0.001357	0.000807
		mean gap	21.11%	21.45%	15.57%	24.80%	27.51%
	Strategy 3	Small	0.607842	0.577377	0.531816	0.500355	0.537066
			0.000156	0.000275	0.001741	0.000921	0.002507
		Middle	0.548341	0.517545	0.481369	0.38915	0.397993
			0.000235	0.000321	0.005204	0.00121	0.001877
		Large	0.452389	0.43685	0.414419	0.325447	0.29446
			0.000199	0.000382	0.001707	0.00078	0.001737
		mean gap	25.57%	24.34%	22.07%	34.96%	45.17%
IGD	Strategy	$\overline{\zeta }$
	Strategy 1	Small	0.081081	0.089546	0.114181	0.125006	0.085965
			1.93E−05	2.56E−05	0.000217	0.000377	0.000176
		Middle	0.07656	0.081223	0.12775	0.132086	0.107114
			1.51E−05	2.22E−05	0.000873	0.000183	0.000203
		Large	0.075806	0.075908	0.166999	0.113566	0.103795
			3.33E−05	1.65E−05	0.002057	7.47E−05	0.000376
		mean gap	6.51%	15.23%	31.63%	14.02%	19.74%
	Strategy 2	Small	0.070188	0.078635	0.112759	0.120658	0.08723
			1.14E−05	1.22E−05	0.000214	0.000263	4.24E−05
		Middle	0.067106	0.069043	0.109341	0.110483	0.093927
			2.46E−05	1.16E−05	0.000552	0.000121	0.000114
		Large	0.064821	0.065644	0.135298	0.106542	0.091753
			1.53E−05	6.89E−06	0.001603	0.000113	0.000309
		mean gap	7.65%	16.52%	19.19%	11.70%	7.13%
	Strategy 3	Small	0.083499	0.085558	0.116708	0.126069	0.093247
			2.57E−05	4.2E−05	0.00021	0.000104	0.000116
		Middle	0.082514	0.085099	0.13417	0.127182	0.110645
			5.33E−05	2.18E−05	0.000563	0.000119	0.000152
		Large	0.074674	0.079224	0.13372	0.115114	0.113762
			1.47E−05	8.18E−06	0.000623	0.00011	0.000174
		mean gap	10.57%	7.40%	13.01%	9.49%	18.03%

. “mean gap” represents the gap between the best mean and the worst mean for different interference coefficients of the same strategy. The NSACOWDRL has better performance in both models, and the solutions obtained by the NSACOWDRL have more stable metrics. In comparison, in different models, as the interference coefficient increases, NSGA-III gradually approaches NSGA-II in terms of convergence, and finally its performance is weaker than that of NSGA-II. Meanwhile, the diversity of MOEA/D becomes progressively weaker than that of NSGA-III, and the convergence of MOEA/D begins to weaken to approach that of NSGA-II. Faced with different JIT strategies, NSGA-II, NSGA-III and MOEA/D have their own strengths and weaknesses in the performance, while NSACOWDRL always keeps the superiority.

Thus, NSGA-III, and MOEA/D are susceptible to interference coefficients and JIT strategies; however, the NSACOWDRL has better generalizability when faced with different models.

By analyzing the “mean gap”, as the interference coefficient increases, the HV and IGD metrics of NSACOWDRL of Strategy 1 have the smallest degradation, followed by Strategy 2, and the algorithm of Strategy 3 has the most degradation. Compared with other strategies, Strategy 1 can maintain the performance of the algorithm better, which verifies the superiority of Strategy 1.

5.4. Different Parameter Settings

In the previous experiments, this paper did not make any serious attempt to find the optimal parameter settings for NSACOWDRL. For the DDQN, the type of neural network affects the performance of the algorithm. In this section, based on the proposed model, the neural networks of the DDQN are set as the back propagation (BP), radial basis function (RBF), the general regression network (GRNN), and product-based network (PNN), respectively, so as to test the effects brought by different networks.

Table 5. Mean and variance metric values of HV and IGD for different networks.

HV	$\overline{\mathit{\zeta }}$	NSACOWDRL _BP	NSACOWDRL _RBF	NSACOWDRL _PNN	NSACOWDRL _GRNN
	Small	6.215E−01	6.208E−01	6.367E−01	6.427E−01
		8.768E−04	9.602E−04	1.183E−03	5.848E−04
	Middle	6.037E−01	5.901E−01	6.068E−01	6.026E−01
		2.763E−04	2.104E−04	1.639E−04	1.809E−04
	Large	4.874E−01	4.733E−01	4.763E−01	4.901E−01
		7.729E−04	2.365E−04	5.044E−04	4.079E−04
IGD	$\overline{\zeta }$
	Small	6.881E−02	7.188E−02	6.537E−02	6.816E−02
		3.990E−05	1.428E−05	2.453E−05	1.668E−05
	Middle	6.906E−02	6.994E−02	6.620E−02	6.826E−02
		2.326E−05	2.518E−05	8.431E−06	1.017E−05
	Large	7.867E−02	7.937E−02	7.560E−02	7.539E−02
		1.390E−05	9.746E−06	1.416E−05	1.164E−05

shows the performance of the algorithm with different networks. Figure 16 shows the p value of the metrics. By comparison, it can be seen that PNN has the best convergence and diversity, PNN has the closest performance to GRNN, and RBF has the worst performance. At the same time, as the interference coefficient increases, the variability between different networks begins to decrease.

For illustration, Figure 17 shows the set of Pareto solutions in the approximate PF obtained by each algorithm with different interference coefficients. The interference coefficient increases from left to right and then from top to bottom in the subplots of Figure 16. As the interference coefficient increases, each algorithm has more Pareto solutions which are more widely distributed in the solution space, and the variation in the distribution of solutions between algorithms begins to decrease. The figure shows that the solutions of NSACOWDRL _ PNN are distributed at all levels between (0, 1) on the $1/alpha$ axis in all subplots, while the solutions of NSACOWDRL _ RBF are concentrated in two blocks. In terms of solution space distribution, the solution distribution obtained by NSACOWDRL _PNN is comparable to NSACOWDRL _ GRNN which is better than the other algorithms.

From these results, it can be shown that when adjusting the parameters of the algorithm, choosing the appropriate networks can effectively improve the performance of the algorithm. The appropriate neural network is selected by the type of problem, the Convolutional Neural Network (CNN) can be used to deal with discrete action space problems, while GRNN can be used to deal with nonlinear problems, which requires different network architectures and tuning for different problems.

6. Conclusions

This paper constructs the multiobjective AGV routing planning mathematical model of the JIT-RMOVRPTW for assembly workshops which considers the uncertainty in the transportation time. Based on the constraints, the metrics alpha—which measures the feasibility of the solution under uncertainty—is introduced into the model. To solve the problem, the two-stage NSACOWDRL is proposed. In stage 1, the NSACOWDRL adopts the nondominated routing selection method through small habitat protection based on reference points to obtain the set of nondominated routes, which ensures the diversity of the NSACOWDRL. The nondominated rank is introduced into the pheromone update strategy, which adopts the elite routing update method to protect better routings and uses the maximum-minimum pheromone strategy to prevent the algorithm from easily falling into local optimality due to an overly large gap between pheromone concentrations. Then, the DDQN is introduced as a local search algorithm which trains the DDQN networks by using nondominated solution sets. The trained network is used to generate routes to participate in the next nondominated sorting; in addition, the network participates in the probability formula. The state transfer probability formula based on objectives, the DDQN, and constraints is proposed in the NSACOWDRL. In stage 2, the feasibility of the nondominated solutions obtained in stage 1 was quantified by Monte Carlo simulation as Monte Carlo times, which were then robustly selected for diversity based on uniform weights in the solution space to obtain the Pareto solution set. The JIT-RMOVRPTW deals with the conflict between punctuality and robustness in uncertain environments in a better way. Meanwhile, based on multidimensional space, the multiobjective problem is also solved better. The NSACOWDRL complements evolutionary algorithms with deep reinforcement learning algorithms to enhance the optimization capability.

The experimental results among the different algorithms under different disturbance coefficients verify the superiority of the NSACOWDRL in terms of diversity and convergence in the robust VRP problem. A comparison of the models for different JIT strategies also reveals the superiority of the proposed model. The experiments demonstrate that NSACOWDRL has better generalization in different scenarios.

The uncertainty of material requirements due to the volatility of processing times, equipment failures, and other issues will be further considered. Future research can further enhance the complexity of the model and further optimize the material distribution scheme. Through the comparison experiments of different networks, DRL with different network structures has a large impact on the model performance, therefore the research on DRL models needs to be further deepened.