Learning-Based Optimisation for Integrated Problems in Intermodal Freight Transport: Preliminaries, Strategies, and State of the Art

Featured Application

Synchromodal optimisation; decision support systems; dynamical transport optimisation.

Abstract

Intermodal freight transport (IFT) requires a large number of optimisation measures to ensure its attractiveness. This involves numerous control decisions on different time scales, making integrated optimisation with traditional methods almost unfeasible. Recently, a new trend in optimisation science has emerged: the application of Deep Learning (DL) to combinatorial problems. Neural combinatorial optimisation (NCO) enables real-time decision-making under uncertainties by considering rich context information—a crucial factor for seamless synchronisation, optimisation, and, consequently, for the competitiveness of IFT. The objective of this study is twofold. First, we systematically analyse and identify the key actors, operations, and optimisation problems in IFT and categorise them into six major classes. Second, we collect and structure the key methodological components of the NCO framework, including DL models, training algorithms, design strategies, and review the current State of the Art with a focus on NCO and hybrid DL models. Through this synthesis, we integrate the latest research efforts from three closely related fields: optimisation, transport planning, and NCO. Finally, we critically discuss and outline methodological design patterns and derive potential opportunities and obstacles for learning-based frameworks for integrated optimisation problems. Together, these efforts aim to enable a better integration of advanced DL techniques into transport logistics. We hope that this will help researchers and practitioners in related fields to expand their intuition and foster the development of intelligent decision-making systems and algorithms for tomorrow’s transport systems.

1. Introduction

The challenges faced by intermodal freight transport (IFT) encompass synchronisation processes between multiple decision-makers, external barriers, and the lack of integration between different modes [1]. The attractiveness and effectiveness of IFT are therefore directly affected by the degree of synchronisation and optimisation across the entire logistics chain. Improving the performance of intermodal transport requires fast optimisation techniques capable of finding solutions for multiple problems simultaneously at various planning levels, taking into account a highly dynamic and uncertain environment.

In general, transport planning problems have been a key area of interest in optimisation science for decades, resulting in a wide range of solution methods. The conventional optimisation techniques from the field of operational research (OR) are known to be computationally expensive, especially when applied in dynamic environments [2]. Furthermore, classical OR mainly focuses on the formulation and solution of isolated combinatorial problems, often abstracted from the complexities of real-world, integrated transport planning applications. For instance, the Job Shop Scheduling Problem (JSSP), a classical combinatorial problem in optimisation science, deals with the efficient allocation of resources to activities under a set of constraints. From the transport planning perspective, JSSP solution methods can be applied in terminals to assign cranes or vehicles to containers, but also in train routing, where a resource denotes a subset of infrastructure elements that are exclusively assigned to a single train at a given time [3]. Obviously, despite different transport planning contexts, the solution methods are often problem agnostic and transferable to different application scenarios. Furthermore, solution methods for operational problems, such as itinerary replanning, demand a combination of different techniques [4]. Today, these problems are mostly handled within the framework of the linear programming paradigm, often encapsulated, and focused on one-dimensional optimisation problems such as the JSSP or the Travelling Salesman Problem (TSP). To improve the overall performance of intermodal transport systems, future optimisation frameworks must be highly integrated, computationally efficient, consider rich context, integrate multiple objectives, and handle uncertainties. Currently, there is a growing effort among scholars to apply learning-based methods into, alongside, or as substitutes for the canonical linear programming paradigm, as these approaches are better suited to the needs posed by modern optimisation tasks [5,6,7,8,9]. In this context, this study aims to address three research questions:

• IFT is a complex, integrated system involving multiple decision-makers with different objectives that need to be optimised in parallel. What are the key operational, tactical, and strategic planning problems in IFT and how can they be mapped to mathematical problem formulations commonly used from an OR perspective? How can these problems be categorised in a structured and systematic manner?
• Learning-based algorithms for combinatorial optimisation problems represent a very promising area for seamless and fast decision-making under uncertainties—a crucial factor for enhancing the overall IFT performance. Over the past five years, research in the area of computer science has converged on methods referred to as Neural Combinatorial Optimisation (NCO), which apply Transformer architecture [10] to combinatorial problems. However, a unified and structured overview of key methods, strategies, and framework setups is lacking in the current State of the Art. This raises the question: what are the key methodological components, algorithm setups and solution strategies in NCO and how can they be structured and abstracted to assist future research in building decision-making tools for integrated transport planning problems?
• The current State of the Art in NCO is highly heterogeneous and primarily involves basic algorithm research with atomic applications to classical OR problems or representative use-cases. To derive insights for more general applications, the question arises: how can the current NCO research be systematised, and what insights can be deduced to support the development of more integrated, generalised NCO frameworks in the future?

To address these questions and explore new directions for advanced optimisation frameworks, this study synthesises recent research from three areas: transport planning in bimodal freight transport (rail and road, referred to as IFT throughout the text), operational research (OR), and Deep Learning (DL) for combinatorial optimisation (referred to as NCO throughout this study). Today, these research streams are mostly conducted in parallel, but are highly interdependent in practice. To the best of our knowledge, despite the rapid growth of NCO applications in computer science, a systematic review from a broad perspective (transport planning, optimisation and DL) involving practical aspects is still lacking.

In the following, the system boundaries together with the key requirements, characteristics, stakeholders, and general operations in IFT will be first defined in Section 2.1. In Section 2.2, we will present a classification of the key optimisation problems in bimodal settings, aiming to identify joint optimisation areas where advanced optimisation techniques may become a key driver in the near future. Section 3 will introduce the methodological preliminaries for the key components and methods and setup strategies needed to design an NCO framework for solving integrated optimisation problems. By organising the State of the Art with respect to the key optimisation domains and framework architectures defined in Section 2 and Section 3, we then extrapolate potential strategies, designs, and future perspectives for advanced NCO frameworks in Section 4. The main findings, challenges, and opportunities for designing NCO algorithms for integrated optimisation problems, together with a discussion on future research directions, will be presented in Section 5. Finally, in Section 6, we will conclude on the objectives and insights drawn from this study.

2. Transport Planning Problems in IFT

In general, each IFT service consists of various discrete transport, loading/unloading, and sorting processes, which occur at different levels and involve multiple decision-makers, sometimes even with conflicting objectives [11]. To establish the system boundaries for this research, this section defines the key decision-makers and their roles and systematically describes the main transport-related processes in bimodal freight transport, as well as the models and objectives, from both a transport planning and optimisation science perspective.

2.1. Intermodal Transport Operations

In the IFT context, the transport process generally involves loading a shipment into a container or transport unit, such as a trailer, which is subsequently transported by truck to a transfer hub connected by railway connections. IFT represents a complex logistics chain that requires careful planning and coordination among multiple stakeholders. Note that, in this study, we only focus on IFT involving two modes, with a freight forwarder acting as the central agent responsible for organising the door-to-door transport. After a shipper delegates freight transport to the freight forwarder, optimising the information and transport flows between multiple stakeholders (such as the road operator, rail operator and terminal operator) becomes necessary. In what follows, we set out a framework for the IFT service and outline the key decisions, operations, and optimisation problems for the transport-related decision entities.

The freight forwarder first selects modes and carriers by comparing intermodal transport services with unimodal road services and develops transport plans. Once the transport service is determined and the transport plan is scheduled, the freight forwarder is notified of any disruptions. A train delay may require the rebooking of road services and, thus, fast optimisation and the rescheduling of planned services. Moreover, the freight forwarder’s decisions on scheduling, modes, the number of containers, and train lengths are impacted by past decisions, which subsequently influence future decisions, resulting in complex and non-linear behaviour. The freight forwarder schedules the road services based on the rail service’s transit time and expected trans-shipment time at the terminals [1]. When scheduling road operations, road operators encounter a complex task of determining the optimal routes and optimising vehicle fleets, considering multiple constraints such as time windows and vehicle capacity, and setting the priority to empty or loaded containers.

The rail operator is responsible for planning decisions regarding the railway network, such as infrastructure, network design, optimal pricing, and train dispatching. The railway undertakings provide the rail haul services according to the schedule of the freight forwarder and the rail operators. The main challenge here is to optimally assign resources such as locomotives, train routes, trailers, and containers. Finally, the terminal operator is responsible for the trans-shipment operations at the terminals such as the classification, consolidation, sorting, and exchange of load units. Although the raw transport costs of multimodal freight transport are generally lower than those for road transport, the handling costs at trans-shipment points make road–rail transport unprofitable for many markets [12]. The coordination and optimisation among all these six stakeholders require simultaneous consideration of multiple objectives such as cost, environmental impact, transit time, service level, delays, and resource utilisation.

In general, planning decisions in IFT can be characterised as long-term (strategic), medium-term (tactical), and short-term (operational). But in practice, companies make decisions for two-time horizons: offline and online [13]. For example, in railway offline decisions involve routing trains in advance, determining the primary path for each train under normal conditions. These decisions are made based on demand, typically every three to six months. The routes and schedules are carefully planned manually, considering regulations, safety measures, demand forecasting, and demand requirements. Online decisions are required when disruptions occur during network operations. As road operations on the road constitute approximately 40% of the total IFT costs [14], the decisions of the road operators are much more critical. Bektas and Crainic [15] categorise the key planning processes in IFT from the operator’s perspective and summarise them under three umbrellas: system design, service network design, and operational planning. In the following, we will briefly describe these three areas and add terminal operations planning as a fourth major area in IFT.

System Design: This mainly encompasses the decisions of the shippers, receivers, freight forwarder, and the road operator regarding the infrastructure and asset planning. This includes network design, network flow planning, and service network design problems [16]. Stakeholders consider long-term planning decisions such as determining the number, capacity and location of terminals, long-term line planning, and assignment of supply and delivery regions. These decisions typically entail the procurements and allocation of physical resources and the selection of geographical locations, spanning a time horizon from several years to even decades [17]. System design models are used to evaluate and rank alternatives for such planning issues. Most of them are often aggregated, static, and deterministic, where the uncertainties are handled based on strategic forecasts within formulated scenarios [15,18,19]. The objectives in this phase are mostly generic, aiming to find the system optimum by minimising the total system cost.

Service Design: Based on the results of the system design phase (i.e., fixed sites, infrastructure, and selection of dispatch/delivery zones, i.e., origin-destination), the freight forwarder plans its transport operations within the available resources and in accordance with the services provided by the rail and road operators. Thus, the main optimisation concern in this phase is to assign and redistribute resources across or within the intermodal network. Specifically, tactical service network design involves determining routes and frequencies, finding the schedule and selecting the modality for each corridor, and assigning transport services given the capacity in that corridor [20,21]. IFT, which involves additional operations such as trans-shipment and consolidation, tends to be slower and less reliable than unimodal road transport, making scheduling with the objective of minimising delays crucial for intermodal networks [22,23]. To operate effectively, decision-makers in this phase are highly dependent on mid-term predictions, such as demand and available resources. Some recent studies introduce stochastic models for service network design problems in order to tackle uncertainties in an integrated manner [24,25].

Operational Planning: This domain includes the near- or real-time decisions of all IFT stakeholders, ensuring efficient use of resources at any given point in time [15]. On this level, the carriers are primarily focused on vehicle routing, fleet management, personnel scheduling, and resource allocation. The routine of the freight forwarders and rail operators consists of delay management and rescheduling the train operations, i.e., dispatching [22]. The road operator redistributes and moves vehicles within the road network. SteadieSeifi et al. [4] classify the operational planning problems in intermodal transport into Resource Management Problems and itinerary replanning. The former class deals with the distribution of the resources and includes problems such as empty loading unit repositioning and fleet management problems. The latter focuses on real-time optimisation of schedules, modal routes, and relevant responses to operational disturbances. Since uncertainties are frequent in operational planning, the solution models usually incorporate high stochasticity and the dynamics of the environment, considering detailed time frames.

Terminal Operations: Terminal operations are highly integrated within service network design and encompass decisions regarding the assignment of resources (e.g., cranes) and sorting (e.g., loading units) under a given capacity and time constraints [15,26]. The objective of the intermodal terminal is to provide a seamless trans-shipment by minimising handling and consolidation costs. Load consolidation at the nodes is particularly important when dealing with heterogeneous cargo and less-than-container loads such as general cargo. Implementing the optimal inventory policy may result in greater vehicle utilisation, impacting routing plans and fleet sizes, which together aim to minimise empty trips or costly empty vehicle repositioning [27]. Planning decisions by the terminal operator are made either offline, using historical information and in coordination with road and rail operators, or online, through operational planning within the terminal. Figure 1 illustrates the main strategic, tactical, and operational planning problems in bimodal transport, highlighting their integration and interaction between three transport decision-makers.

2.2. Optimisation Problems in Transport Planning

In this section, our objective is to establish the link between planning problems described in Section 2.1 and combinatorial optimisation problems from the field of OR where NCO techniques are applicable.

Below, we categorise the most relevant decision problems related to IFT into seven general problem classes. These optimisation problem areas include the following: (i) Network Design Problem [28,29], (ii) Resource Assignment/Scheduling/Dispatching [25,30,31], (iii) Resource Allocation/Rebalancing/Fleet Composition [32,33,34], (iv) Vehicle Routing Problem [24,35,36], (v) Batching/Sequencing Problem [37,38], (vi) Prediction/Forecasting [39,40], and (vii) Online Control Problems with joint decision-making such as inventory management, bidding, and rescheduling [41,42]. We primarily concentrate on the first six problem classes, excluding Online Control and Joint Decision-Making as it relates more to supply chain and logistics science involving negotiations, bidding, storage, ordering, and shipping policies among multiple stakeholders. However, we include the class Predictions and Forecasting as it is closely interconnected with other strategic and tactical transport planning problems, such as the delay and demand prediction for service network design, which are fundamental subproblems in optimisation frameworks for integrated decision-making.

The category Network Design Problem involves optimising the number and locations of various types of facilities (nodes), establishing main transport corridors, and defining routes (arcs). Resource Assignment/Scheduling/Dispatching entails the dynamic assignment, selection, scheduling, and dispatching of activities, jobs, or vehicles to resources. This encompasses activities such as scheduling vehicles, assigning trains to routes in the railway network together with precise arrival and departure times, mode selection, or dispatching vehicles to customers or terminals. Resource Allocation/Rebalancing/Fleet Composition addresses the optimal allocation of resources over time, including the distribution and composition of heterogeneous vehicle fleets and the redistribution of empty vehicles or containers across multiple facilities. Vehicle Routing Problem (VRP) aims to find the sequence of nodes/customers (offline) or predict the next node/customer (online) for vehicles under various constraints. As the VRP naturally incorporates fleet management aspects, such as defining the number and type of vehicles at the starting facility, this class may also cover the Vehicle Fleet Problem (VFP), but is only related to a single facility. Batching/Sequencing Problems encompass the operations concerning the storage, picking, sorting, and processing of a group of resources (such as containers, wagons in shunting yards, or orders) within storage or sorting facilities. The items within each batch are then collectively picked along a single pickup route. In fact, these operations have an impact on vehicle routing and on the entire service network. Prediction Problems involve forecasting future events (e.g., outages), timings (e.g., delays or estimated time of arrival), or future demand. This subclass is critical for adequate network and service design planning.

Table A1. Transport problems and their OR formulations in transport science categorised into six main problem classes related to IFT operations, defined in Section 2.2. We additionally present the most conventional solution techniques from the optimisation point of view associated with these problems.

Optimisation Problem Class	Transport Problems	Related Problems in OR and Computer Science	Solution Techniques
Network Optimisation	Network/infrastructure planning, facility location planning, strategical network flow planning, long-term rail-corridors and terminals, assignment of delivery zones or customers to facilities.	Linear Assignment Problem (LAP) and Quadratic Assignment Problem (QAP), Facility Location Problem (FLP), Location Routing Problem (LRP), Hub-Location, Arc-Routing Problem, Clustering, Hub location Problem, Covering, Centre and Median Problem.	Graph optimisation, MIP/MILP, heuristics and metaheuristics, econometric models, aggregated transport models, land-use models, Multi Criteria Decision Analysis (MDCA), analytical methods, Geographic Information System (GIS).
Resource Assignment/Scheduling/Dispatching	Capacity planning, mode selection, train routing, occupation time scheduling, timetabling (offline) and train dispatching (online), locomotives/vehicles/containers assignment, drayage scheduling and assignment of pick-up and delivery points.	Linear Assignment Problem (LAP), Dynamic Quadratic Assignment Problems (DQAP), Job-Shop Scheduling and Flexible Job-Shop Problems, Resource Assignment, Scheduling and Allocation Problems, Shipping Point Assignment, Network Flow Problem.	Time-space network, graph optimisation, analytical methods, MIP/MILP, heuristics and metaheuristics, approximate dynamic programming, discrete event simulation, simple dispatching rules (e.g., First in First Out), behavioural models, Multi-Agent Systems (MAS).
Resource Allocation/Redistribution and Fleet Composition	Empty/idle vehicle/container repositioning, vehicle-redistribution, vehicle fleet composition at multiple locations, vehicle fleet management	Linear Assignment Problem (LAP), Vehicle Fleet Optimisation, Vehicle Repositioning, Vehicle Fleet Composition.	Discrete event simulation, MIP/MILP, graph optimisation, heuristics and metaheuristics.
Batching and Sequencing	Consolidation/shunting/sorting or container/wagons, sorting and batching of orders in terminals, order consolidation, picker routing.	Fixed Time Window Batching (FTWB) and Variable Time Window Batching (VTWB), Bin Packing Problem, (container) Reordering, Resorting, Pre-marshalling Problem, Block(s) Relocation Problem, Load Planning Problem.	Simple sequencing and batching rules, heuristics and metaheuristics, discrete event simulations, MIP/MILP.
Vehicle Routing Problem	Scheduling, dispatching of vehicles and vehicle routing.	Shortest Path Problem, TSP, VRP, Heterogeneous Fleet VRP (HF-VRP), VRP with Time Windows, Multi-Depot VRP (VRP with TW), VRP with Pickup and Delivery (VRPPD), Capacitated VRP (CVRP), Two-echelon VRP.	Graph optimisation, MIP/MILP, heuristics and metaheuristics, behavioural models, MAS.
Prediction and Forecasting	Prediction of delays, demand, events, vehicles and resource occurrences.	Delay prediction and propagation, demand forecasting, traffic flow forecasting.	Analytical methods, classical and Bayesian statistics methods, regression, time-series analysis, ML and DL, behavioural models, simulation, System Dynamics, MAS.

provides an overview of the most common optimisation problems related to the IFT and the corresponding variants and formulations from the OR perspective.

3. Methodological Preliminaries

The main methodological components needed to design a universal NCO-based framework for generic application are identified and systematically outlined in this section. Through our study, we refer to NCO as an optimisation framework based on Deep Neural Networks (DNN) and Reinforcement Learning (RL) [43]. In general, two strategies can be derived from the current State of the Art to solve dynamic optimisation problems under uncertainties. One group of researchers focuses on model-based approaches, mathematically modelling stochastic problems, and developing approximation algorithms for dynamic decision-making [44,45,46,47,48]. In particular, Powell [46] advocates that all combinatorial problems can be modelled as sequential decision problems, gathering four ultimate frameworks for solving dynamic and stochastic problems under the umbrella of approximate stochastic optimisation. The second group of researchers in the field of DL develop data-driven methods, applying and tuning DNN models with the state-of-the-art RL algorithms. Based on the State of the Art, we note that the recent advances in NCO are largely attributed to four design patterns: (i) DNN model (i.e., agent policy), (ii) RL algorithm, (iii) agents’ setup, and (iv) simulator setup strategy. The model design is usually established as an encoder–decoder DNN, which generates probabilities over all actions at each step in the agent’s environment. The RL algorithm is then applied to train the encoder–decoder model based on reward, which is either computed directly from the RL environment [49] or is retrieved from an external simulator model. Through our study, we consider the simulator as an external algorithm or external Multi-Agent system plugged into the RL agent to reproduce actions, provide states, and/or calculate rewards. The agents’ setup specifies the number of agent policies and the settings in which RL agents are trained, whether in shared or separate environments. Finally, the simulator setup strategy determines how the RL agent exchanges information with the external simulator. In the following sections, we will examine these design patterns in-depth, providing a comprehensive overview of their setups and features and outline the application domains.

3.1. DNN Model

During the past decade, DL scholars have extensively experimented with various DNNs, including Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) networks, and Convolutional Neural Networks (CNN) to predict target actions in dynamic systems [50,51,52,53,54]. The breakthrough in this direction was achieved by Vaswani et al. [10], demonstrating that sophisticated encoder–decoder networks can essentially predict the next value based on all previous context information. Today, many state-of-the-art solutions in the field of combinatorial optimisation employ encoder–decoder architecture, which involves training an encoder to learn a meaningful representation of the input problem and a decoder to use this representation to infer a probability over the next actions.

The encoder’s role in the context of combinatorial optimisation is to embed the problem state, including its features (e.g., distance, customer demand, and time windows in context of the VRP), into a fixed vector representation. The encoder thus encodes the input vector $s_t\in {\mathbb{R}}^x$ into a new extended vector $em{b}_{s_t}\in {\mathbb{R}}^n$, where $x$ is the number of problem state features and $n$ is the embeddings size (usually $n=128$). Typically, the encoder is represented by multiple feedforward networks containing several Multi-Head Attention (MHA) layers, whose role is to learn a new, extended representation of the input data. The encoder’s output can subsequently be used by the decoder to create a probability distribution for all possible next nodes. Most state-of-the-art encoder–decoder models today rely on the attention or MHA mechanisms at their core [55,56,57]. In general, MHA layers quantify the relationships between two nodes in an embedded space by calculating a dot product. The elements (e.g., node embeddings $em{b}_{s_t}$) are each multiplied by three matrices: (i) Query $Q^i=W_{q}em{b}_{s_t}$, (ii) Key $K^i=W_{k}em{b}_{s_t}$, and (ii) Value $V^i=W_{v}em{b}_{s_t}$, where $W_q, W_k,W_v$ are trainable parameters, and $em{b}_{s_t}$ represents the embeddings of the problem state $s$ at timestep $t$. By adjusting the weights $W_q$, $W_k$, and $W_v$ during training, the importance of each element (e.g., embedded node) is calculated in relation to itself and to all other problem elements. Concretely, this means that the probability of selecting the next action $a_{t+1}$ is derived from the attention score produced by the decoder: $a_{t+1}=P\left(u_{dec}\right)$, where $u_{dec}$ is the attention score calculated according to [10] as

$$u_{dec}=sofmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{\dim K}}}\right)V$$

(1)

In Equation (1), $softmax$ represents the softmax function that output the probabilities over the dot product between queries, keys and values, divided by the scaling factor $\sqrt{\dim K}$, where $dim$ indicates the size of the key vector $K$. To learn additional information from the problem context, such as dynamic problem states and implicit information, the context vector can be additionally concatenated and projected in the decoder. In this context, the additional context information from the environment can be added to the vector outputted by the encoder as follows: $c_t=FF\left(em{b}_{s_t}\right|\left|c_0\right)$, where $FF$ represents a feedforward network (usually with a single hidden layer), $c_0$ defines the problem context added to the learned representation, and $\left(\right|\left|\right)$ denotes concatenation. In fact, by including the context vector $c_t$ we inject additional state information, enabling the encoder–decoder models to capture problem dynamics during the learning episode, such as the remaining vehicle capacity or the remaining number of empty vehicles in VRP.

To prevent the selection of inappropriate actions, practitioners apply masking, a technique which eliminates the probabilities for undesirable actions in the decoder [55,58,59]. In fact, this approach helps to deal with the problem constraints, which are fundamental in the context of combinatorial optimisation. In general, the attention-based encoder–decoder architecture is the State of the Art in many DL applications including prediction and combinatorial optimisation [55,60].

Given that many transport problems, such as network design, vehicle routing, dispatching, and assignment problems, often take the form of graphs, Graph Neural Networks (GNNs) can be applied in the encoder to effectively process input vectors of various sizes [61]. In the graph-based models, node embeddings (e.g., represented by a vector $emb\in {\mathbb{R}}^{B\times N\times 128},$ where $B$ denotes a batch size and $N$ is the number of features) undergo iterative message-passing with their neighbours, effectively capturing structural information about the local neighbourhood [8]. In a graph attention layer, Multi-Head Attention is used to calculate values between node $i$ and its neighbourhood in a parallel manner [62]. We particularly outline Graph Attention Transformers (GATs) as a state-of-the-art neural network for practical scenarios, such as forecasting [63] and spatio-temporal predictions [64,65]. For a comprehensive review of the latest GNN advancements, including algorithms, implementations, and taxonomy, refer to the extensive survey by Sun et al. [66]. A typical end-to-end NCO framework with a corresponding simulator for providing rewards to the RL agent is illustrated in Figure 2.

3.2. RL Training Algorithm

DRL is a machine learning (ML) paradigm for learning actions based on observations and feedback information in a dynamic environment. In the context of NCO, DRL is primarily applied to train encoder–decoder Neural Networks (NNs), including MHA layers—usually referred to as a policy network for action selection. The RL algorithms rely on a Markov Decision Process (MDP), representing the problem environment as a tuple of state $s_t$, action $a_t$, reward $r_t$, and transition $T_t$: $MDP=(s_t,a_t,r_t,T_t).$ The state $s_t$ at each timestep $t$ forms the input vector for the policy network. The role of the RL training algorithm is to calculate the loss L—the reward difference between the previous state $s_{t-1}$ and the current state $s_t$ after taking action $a_t$. The backpropagation of the loss $L$ enables the encoder–decoder networks to learn a behaviour that maximises the reward.

For example, in the classical VRP using RL, the reward is defined as a negative tour length and the parameters $\theta$ are adjusted during the training, where $\theta$ is a vector of all trainable parameters in embedding, encoder–decoder and attention weighting matrices (e.g., $W_q,$ $W_k,$ $W_v$).

$$J\left(\theta |s\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[r\left(a|s\right)\right]$$

(2)

$$\theta =arg \underset{\theta }{\max }J\left(\theta \right|s)$$

(3)

Equation (2) represents the objective function, where $\theta$ is the policy parameters and s defines the current state of the environment. The objective is to maximise the expected reward obtained by following policy ${\pi }_{\theta }$. The expected reward is denoted by $E$, and $r\left(a\right|s)$ is the reward obtained by executing policy ${\pi }_{\theta }$ and taking the action $a$ in state $s$. Equation (3) denotes the policy parameters optimised by RL that maximise objective function $J\left(\theta |s\right)$.

In fact, the design choice of DRL algorithm is very problem-specific. Generally, the first choice lies on whether we want to train the policy network directly (policy-based learning) or can evaluate and learn from every state–action tuple (value-based learning or Q-Learning). Policy-based RL is more suitable for problems with a large state and action spaces due to the fact, that it only updates parameters of the policy NN. In contrast, value-based RL learns values for each state or state–action pair, which can lead to insufficient convergence when dealing with vast state–action spaces. Although value-based algorithms traditionally found their application in control problems, such as inventory and supply chain management (see [67]), transport researchers have also applied Q-Learning in train scheduling and dispatching, as demonstrated by [68,69,70]. We note that the majority of the studies in the current State of the Art use hybrid approaches called Actor–Critic (e.g., Proximal Policy Optimisation (PPO), REINFORCE) where policy update (actor) and value-based learning (critic) are performed in a feedback loop. Thus, the goal of (i) an actor network is to estimate the probability of the next action in each state, and (ii) a critic is to predict the reward (value) from a given state. To reduce the variance during training, the researchers subtract a so-called baseline from the loss (i.e., the best value so far, produced by the previous policy [55,56]. For instance, Adi et al. [71] subtract $\widehat{Q}$, an additional policy network with less frequent updates, from the current value function $\mathrm{Q}$ to stabilise training. For a comprehensive overview of DRL application in the field of transport, we refer to [72,73,74,75].

3.3. Agents’ Setup

The central challenge in IFT optimisation is to design a model that integrates heterogeneous agent behaviour, capable of reflecting multi-level decision-making processes, on different time scales and with different objectives. For instance, after a shipper decides to send cargo to a receiver, the rail, road, and terminal agents need to schedule their operations to ensure seamless transport. In this context, hybrid approaches are required to model multiple agents interacting together. Generally, designing the operational, highly dynamic transport problem as Multi-Agent RL (MARL) results in a nonstationary environment, reducing convergence and requiring more training steps. From a modelling standpoint, designing such systems offers two choices: either implementing single-agent RL with a centrally trained policy (e.g., [76]) or Multi-Agent RL, where agents learn heterogeneous or shared policies (e.g., [77]). The former approach may result in a larger state–action space, but the training is less computationally demanding. In contrast, the latter approach involves only formulating state actions for each agent, but this also leads to additional dynamics and therefore hampers convergence [78]. The decision regarding the policy update, whether to update only one model with all observations from the environment or to update multiple encapsulated models independently with partial information, is problem-specific and needs to be evaluated at the beginning. Alternatively, agents can operate and gather information independently in the environment, using a shared policy. Centralised Training Decentralised Execution (CTDE) MARL demonstrates a good ability to address highly dynamic Multi-Agent environments, enabling agents to act heterogeneously based on the global problem context [79,80,81]. A further challenge in MARL is to assign rewards to individual actions among multiple heterogeneous agents (credit assignment problem). Due to the collective decision-making, it is often unclear how to design and distribute rewards among many independent agents. Consequently, sophisticated reward distribution techniques are needed. We refer to the following studies that highlight reward shaping and reward distribution as critical steps: [77,82,83].

More recent studies in this field explore Hierarchical Reinforcement Learning (HRL), where an agent (upper policy) at the strategic or tactical level learns to delegate tasks to lower-level agents (lower policies) at the operational decision level. HRL shows great potential, especially for integrated optimisation problems as it can naturally model the asynchronous decision-making between multiple agents [84]. Figure 3 illustrates three main strategies for the MARL setups, which can potentially model integrated decision-making in IFT.

3.4. Simulator Setup Strategies

Given the complexity of transport systems, simulators are widely applied. Based on [85], the following three strategies for combining NCO with simulators can be highlighted: (i) end-to-end, where the entire problem is built and solved by NCO independently; (ii) learning-to-configure, where the DRL algorithm parametrises the external model, i.e., computes parameter for an optimisation algorithm and ML alongside optimisation algorithms; or (iii) learn-to-improve, where an algorithm or heuristics iteratively calls the ML model to improve the outcomes.

End-to-end: The first strategy requires the model and the agent implemented in the same environment. Here, the solution search is completely controlled by the RL agent. The simulator, if needed, is used solely to execute actions and provide the reward. This setup strategy is well-suited for one-dimensional combinatorial problems or problems with a comparatively low degree of interdependencies between decision-makers (VRP, JSSP, Facility Location Problem (FLP), network optimisation, prediction), where the model and the RL algorithm can be formulated as a system in one software environment. The end-to-end approach benefits from improved efficiency by treating the entire system as a single entity, allowing for direct learning from input to output without intermediate steps. However, it requires more expert knowledge and a deep problem understanding due to the need to model the entire transport problem as a dynamic MDP, usually formulated in line with the principles outlined in [49].

Learn-to-configure: The second strategy focuses on leveraging DRL as an oracle in scenarios where the environment is already implemented and building a new environment is costly. Using DRL to parametrise the simulator or even to select heuristics can help to encapsulate the entire problem logic and yield an optimal partial solution for each timestep. In this case, the algorithm being optimised is treated as a black box. Here, the DRL model can provide the state (e.g., demand and customer locations at timestep t) and the action (e.g., the number of vehicles or the set of parameters for the simulator). Biedenkapp et al. [86] define this technique as a Dynamic Algorithm Configuration (DAC). When the simulation environment for a certain transport problem is used, the DAC can dynamically determine the set of input parameters (e.g., optimal vehicles, facilities, demand points) for each instance.

Learn-to-improve: The third strategy is well suited for large-scale, one-dimensional combinatorial problems where traditional heuristics are computationally expensive and solution reliability is crucial. By applying DRL, the solution search process conducted by an external algorithm can be enhanced. In this case, the simulator provides an initial solution to the DRL model, which then executes the search and applies search operators based on the rewards received. The learn-to-improve paradigm consumes more computational time but can obtain higher-quality solutions [87]. Figure 4 illustrates these three scenarios in which the simulator is deployed alongside the NCO.

4. Overview of the State of the Art in the Field of NCO

In the following, we provide an overview of the studies published between 2018 and 2023 related to the six problem classes introduced in Section 2.2 and NCO. We design the main search query for Google Scholar using the following syntax: PUBLICATION NAME (“Arxiv OR Transport OR Logistics”) AND TITLE_WORDS (“(reinforcement OR learn OR neural) AND (freight OR transport OR logistics)”) AND KEYWORDS (problem class name from Table A1). In the first step, we include publishers from the fields of logistics, transport or computer science. In the second step, we narrow the list by including publications that relate to both NCO and transport logistics. Afterwards, we use the problem field keyword from Section 2.2 to further narrow the search space and review selected studies associated with the methods outlined in Section 3. In addition, we also analyse studies from adjacent reviews such as [73] and include articles from the manual search. The reason for the manual search was the broad range of possible topics related to the NCO. For example, Kerkkamp et al. [88] propose RL optimisation of the sewer network. Although this study is not directly related to the transport field, we chose to include it due to its methodological transferability. We explicitly exclude studies having main focus on topics such as signal controlling, supply chain, robotics, navigation, public transport, maritime logistics, and Multi-Agent bidding. This literature review also does not cover the classical RL methods such as simple Q-Learning from the control theory (see, e.g., [70,89]) as our research aims to explore more advanced NCO methods, which usually rely on Transformer architecture. Below, the surveyed studies are organised according to the problem classes outlined in Section 2.2.

Strategic network optimisation in the fields of OR and computer science is traditionally approached through discrete graph optimisation methods. In the reviewed State of the Art, only two studies address network optimisation using NCO. Zhu et al. [90] apply hybrid end-to-end DRL, where an Actor–Critic RL with Graph Convolutional Networks (GCNs) policy first finds the maximum capacity for the web links, and then an Integer Linear Programming (ILP) is applied to improve the final network solution. The authors emphasise that RL-based optimisation offers a more natural solution approach than traditional local search heuristics, as it can generate decisions based on global information and consider delayed outcomes. Kerkkamp et al. [88] propose a DRL framework based on GCNs for dynamic pipe network optimisation, where a centralised agent learns a maintenance plan based on network features for a 100-year period.

Resource assignment, scheduling, and dispatching class covers a broad range of problems related to the railway network and terminal processes. Since this class of problems can be formulated as a sequential decision problem (e.g., Flexible JSSP), the encoder–decoder and end-to-end methods find a natural application here [76,91]. Ren et al. [92] and Han and Yang [93] employ an encoder with LSTM/RNN and a decoder with attention to produce sequence-to-sequence solutions. Oren et al. [94] develop an end-to-end approach that can operate on problems with varying state and action sizes. They first encode the trajectories as a graph (with states $s$ as nodes and actions $a$ as edges) and process it using a GNN. Subsequently, Q-Learning is employed to select actions (edges). The learning-based approach outperforms benchmark heuristics when applied to various combinatorial optimisation problems, such as Capacitated Vehicle Routing Problem (CVRP) and Job Scheduling. Ni et al. [87] encode a scheduling plan (Gantt chart) as a multi-graph and apply a learn-to-improve strategy, where PPO with a GNN learns search operators to update the MIP solution. A reward shaping technique is employed to avoid local minima and ensure fair reward distribution among selected actions. Trained and evaluated on industrial warehouse scheduling data, the model effectively assigns orders to machines during inventory management, producing high-quality solutions and demonstrating robust generalisation. Multi-Agent (MA) cooperative behaviour plays a crucial role in resource assignment, scheduling, and dispatching. Chen et al. [82] utilise Multi-Agent Reinforcement Learning (MARL) to dispatch couriers with stochastic tasks, assigning each free courier to a task and selecting service times to fulfil requests in a grid world. In order to facilitate the cooperation between agents, the authors additionally introduce reward shaping, i.e., assigning weights to courier’s reward function, determining the extent to which they prioritise their individual rewards versus the average reward of all couriers. Chen et al. [79] propose QMIX, a Multi-Agent model-free DRL framework for the integrated dispatching problem. QMIX is a CTDE approach for environments that require coordinated behaviour among multiple decentralised actors [95]. The dispatch policy determines the sequence of dispatch decisions for each truck, which forms the route network of the fleet.

Resource Allocation/Redistribution and Fleet Composition Problems arise in scenarios, where end customers are served by a fleet of vehicles, and in redistributing vehicles across the network to balance the overall vehicle capacity based on the predicted demand. Advanced end-to-end strategies, such as MARL and HRL receive considerable attention in this area [96]. Ahn and Park [97] develop a decentralised MARL approach to redistribute idle vehicles in a manufacturing system between defined zones, where each zone acts as an agent. The authors address cooperative learning by utilising GCNs. Pan et al. [98] use HRL to optimise pricing schemes for vehicle repositioning in a grid world environment. They decompose actions into temporal and spatial components, with sub-Q-agents assigned to grid cells. Xi et al. [99] design an adaptive HRL solution for the dynamic vehicle redistribution problem. They employ a three-level hierarchical RL approach, dividing the problem into a centralised manager and coordinators. Long short-term memory (LSTM) networks are integrated to predict future demand/supply, and a mixed shared reward is designed to promote cooperation.

In the intermodal context, batching and sequencing problems have a decisive impact on the routes and transport schedule of the road and rail operators. Cals et al. [100] design the order selection process (batch and picking) in a warehouse by treating it as a Semi-Markov Decision Process (SMDP) so that the next step occurs continuously, i.e., as soon as a new order enters the system. Beeks et al. [83] use a similar DRL architecture to find a trade-off between two objectives, time and costs, for an online batching assignment problem. Hottung et al. [101] propose a Deep Learning Heuristic Tree Search, a supervised learn-to-improve strategy for a Container Pre-Marshalling Problem, where the DNN generates probabilities to move containers.

The State of the Art on VRP is dominated by end-to-end NCO techniques. As the most fundamental, this domain has become a playground for many computer scientists experimenting with learning-based optimisation techniques. Vinyals et al. [57] introduce a Pointer Network, attempting to solve the Travelling Salesman Problem (TSP) through supervised learning. Nazari et al. [102] enhance Pointer Networks with embedding and RL, enabling VRP solutions. Khalil et al. [103] use graph embedding and Q-Learning for TSP tours, while Joshi et al. [104] employ a GCN by means of supervised learning to predict edges in TSP. Transformers, introduced by Vaswani et al. [10], replace RNNs in sequential problem solving, including the TSP and VRP. Kool et al. [55], Deudon et al. [56], Li et al. [58], and Li et al. [105] develop advanced models for solving VRPs incorporating encoder–decoder and RL. Falkner and Schmidt-Thieme [106] propose an advanced encoder–decoder model for Capacitated Vehicle Routing Problem with Time Windows (CVRP-TW), extending the encoder with rich context embedding. Foa et al. [107] argue that a Convolutional Network can be suitable in cases where the adjacency matrix of a graph is structured similarly to the pixel image. Li et al. [108] solve the Dynamic Pickup and Delivery Problem (DPDP) with online generated orders. The environment of the dynamic dispatching problem is abstracted as a graph, where each vehicle is a node. Then, the attention network is used to calculate the relationships between the vehicle and all the neighbouring vehicles considering spatial and vehicle features. In this study, the authors tackle the stochastic environment by designing a spatial–temporal network, which can predict the demand for each factory at timestep $t+1$. Ma et al. [109] tackle the stochasticity in the online Large-scale DPDP by utilising the order buffer, where the n-orders are first cashed and then provided to the policy network designed as HRL. Based on the upper-level policy, the agent decides on the orders to be served from the cash buffer. The lower-level agents aim to serve these orders by assigning them to the most appropriate vehicles and constructing routes similar to classical heuristics. These agents apply four swap actions (inner-exchange, inner-relocate, inter-exchange, and inter-relocate) on the graph nodes of the constructed routes by utilising REINFORCE and GNNs. Thus, we refer to this model as learning-to-configure. We note that this study is among the first to introduce the HRL solution for the real-world large-scale DPDP. Alternatively, Wu et al. [110] adopt a similar architecture to [55], but instead of selecting the next node, two nodes are permuted. To preserve positional information within the tour, authors propose sinusoidal positional encoding. Da Costa et al. [111] extend this approach with dual encoding and pointing attention, allowing faster convergence to a near-optimal solution in TSP and CVRP. Hottung and Tierney [112] combine Ruin-Recreate and neural optimisation for CVRP, improving solution search by leveraging local information. We refer to these studies as learn-to-improve. Li et al. [113] introduce a learn-to-delegate technique for large-scale VRP, integrating attention neural networks and a Multi-Head Attention and VRP solver. This technique demonstrates promising potential for nested problems such as the Location Routing Problem (LRP) or vehicle fleet optimisation with integrated routing problems.

There is a notable trend in the prediction and forecasting domain towards the utilisation of the GNN alongside attention mechanisms. Earlier studies attempted to combine LSTM and Convolutional Neural Network (CNN) techniques to capture temporal and spatial patterns [114]. Hassan et al. [115] use RL to adjust weights in different time-series models to forecast freight demand movements over short- to long-term horizons. More recently, researchers have successfully applied GCNs and attention mechanisms to predict the dynamics of spatial–temporal sequences [116,117]. For instance, Li et al. [23] develop a sophisticated Graph Attention Network (GAN) and supervised learning approach to predict train delays. Interestingly, the authors first formulate the dynamic railway system as a multigraph, where each timestep is a graph containing nodes representing running trains and terminals. Subsequently, they leverage a GNN to dynamically predict delay propagation in the railway network.

From the presented State of the Art, it is evident that classical combinatorial optimisation problems, such as VRP and scheduling problems, are primarily treated using end-to-end attention-based encoder–decoder architectures, which are then trained with Actor–Critic DRL. The most common strategies among researchers to avoid the curse of dimensionality and achieve stability during training include masking actions in the decoder, applying baseline policies, and pruning the state–action space. However, only a few studies explicitly employ learn-to-configure or learn-to-improve paradigms. Zhao et al. [118] use RL to create an initial solution for heuristics, while Ma et al. [109] generate initial solutions using an upper policy, which is then utilised by lower heuristic-based agents to dispatch vehicles in dynamic pickup and delivery problem with MARL. Applying hybrid strategies for complex nested problems appears promising for integrated problems, such as vehicle fleet optimisation with dispatching and integrated vehicle routing. Figure 5 and

Table A2. The reviewed NCO studies related to the integrated optimisation problems classified according to the six optimisation classes defined in Section 2.2.

Studies	Problem	Learning	Models	ma *	Solution Strategy
Network Optimisation
Kerkkamp [88]	Network planning	DQL	GCN		End-to-end
Zhu et al. [90]	Network planning	Actor–Critic	GCN		Hybrid end-to-end, where the DRL first prunes the search space and then the heuristics improve the solution
Resource Assignment/Scheduling/Dispatching
Agasucci et al. [13]	Train dispatching	Q-Learning	NN		End-to-end. Decentralised and Centralised Deep Q-Learning
Obara et al. [68]	Train dispatching	Q-Learning	MLP		End-to-end. Convert train schedule to graph and apply DQL.
Khadilkar et al. [69]	Train scheduling	Q-Learning	Tabular		End-to-end. Tabular Q-Learning with simulator
Adi et al. [71]	Trucks assignment and dispatching	Q-Learning	Double DQN		End-to-end. Centralised Learning for truck routing/dispatching in intermodal terminals
Chen et al. [79]	Dispatching and package matching	QMIX	DQN	X	QMIX (Multi-agent Q-Learning with centralised training and decentralised execution)
Chen et al. [82]	Dispatching (couriers are assigned to tasks)	PPO	MLP	X	End-to-end MARL with decentralised value function for each vehicle and common policy network with parameter sharing + parameter sharing
Ni et al. [87]	Generic scheduling problem	RL	GNN + GCN with attention pooling		Learn-to-improve with reward shaping
Popescu [89]	Train dispatching	cross-entropy	NN		RL-agent executes action in a simulator
Song et al. [91]	Generic scheduling problem	PPO	GNN + GAN		End-to-end
Ren et al. [92]	Generic scheduling problem	A3C with baseline	Encoder-decoder with Attention,		End-to-end
Han and Yang [93]	Generic scheduling problem	Model-free DRL	Encoder-decoder with RNN		End-to-end
Oren et al. [94]	Generic scheduling problem	Q-Learning	GNN		End-to-end. DQL for online and offline dispatching problem
Iklassov and Medvedev [119]	Generic assignment	Actor–Critic	GNN, GCN		End-to-end
Porras-Valenzuela [120]	Network flow problem	Q-Learning	GCN		End-to-end. Single agent assigns flow links to warehouses under uncertainties
Zhang et al. [121]	Synchromodal re-planning	Q-Learning	DQN		Learn-to-improve
Zou et al. [122]	Dispatching/assignment	Q-Learning	Double DQN		End-to-end with simulator (SUMO) as environment and single centralised agent
Resource Allocation/Balancing/Fleet Composition
Li et al. [77]	Resource balancing	DQL	MLP for each agent	X	End-to-end. Cooperative Multi-Agent RL (MARL) with shared state and rewards
Ahn and Park [97]	Assignment and rebalancing	Factorised Actor–Critic	GCN	X	End-to-end. Decentralised policy for each zone with parameters sharing
Pan et al. [98]	Vehicle repositioning	Actor–Critic	Q-Network for each agent	X	Hierarchical RL (HRL) where each sub-agent adopts LSTM
Xi et al. [99]	Vehicle repositioning	PPO	DQN and LSTM	X	3-Level HRL. DQN for workers and LSTM for prediction.
Zhang et al. [123]	Vehicle fleet reposition	Q-Learning	NN and tabular Q	X	End-to-end
Batching and Sequencing
Beeks et al. [83]	Order batching and sequence problem	PPO	DNN		End-to-end with reward shaping for two-objective problems
Cals et al. [100]	Batching and sequencing orders	PPO	DNN		Hybrid-method. Heuristics to take sequencing decisions and DRL to take batching decisions.
Hottung et al. [101]	Reordering of containers	Supervised	DNN		Learn-to-improve. Heuristics utilise RL for improvement
Vehicle Routing Problem
Kool et al. [55]	TSP	REINFORCE	Encoder-decoder with GNN and attention		End-to-end. Prediction of next nodes using GAN Attention Network, masking and RL-baseline.
Deudon et al. [56]	TSP	REINFORCE	Encoder-decoder with self-attention and Pointer Network		End-to-end
Li et al. [58]	CVRP with heterogeneous vehicle fleet	policy DRL with baseline.	Encoder-decoder Multi-Head Attention		End-to-end
Ren et al. [59]	VRP with TW	Policy gradient method	Encoder-decoder	X	Centralised training with shared parameters and observation
Vera and Abad [80]	CMVRP	Actor–Critic	Encoder-decoder	X	End-to-end MARL with centralised training and decentralised execution
Nazari et al. [102]	VRP	Policy-based RL	Encoder-decoder with RNN and attention		End-to-end
Joshi et al. [104]	TSP	Supervised	GNN		End-to-end
Li et al. [105]	Pick-up and delivery VRP	policy DRL with baseline.	Encoder-decoder with self-attention		End-to-end
Falkner and Schmidt-Thieme [106]	CVRP-TW	REINFORCE	Encoder-decoder with attention		End-to-end. Extended encoder-decoder to embed rich problem context
Foa et al. [107]	Assignment and routing	modified PPO	CNN	X	End-to-end MARL with two actor-networks for node selection and assignment
Li et al. [108]	Dynamic pick-up and delivery	Double Q-Learning	Graph spatial–temporal attention network		End-to-end. Predict demand and formulate environment as a graph. Attention net is used to calculate the relationships between vehicle and all the neighbouring vehicles
Ma et al. [109]	Dynamic pick-up and delivery	Q-Learning (upper policy) and REINFORCE (lower policy)	Upper-level agent: MLP, Lower level agent: GNN	X	Learn-to-configure. Multi-Agent HRL for assigning orders and delivery
Wu et al. [110]	TSP/CVRP	Actor–Critic	Encoder-decoder		Learn-to-improve. Heuristics utilise RL for improvement
Da Costa et al. [111]	TSP	Actor–Critic	Encoder-decoder with GCN and LSTM		Learn-to-improve. Heuristics utilise RL for improvement.
Hottung et al. [112]	CVRP	REINFORCE	Encoder-decoder with attention and FNN		Learn-to-improve. Heuristics utilise RL for improvement
Li et al. [113]	Large-scale CVRP	Supervised	Encoder-decoder with self-attention		Learn-to-improve. Clustering heuristics utilise ML for improvement
Zhao et al. [118]	VRP	Actor–Critic	Encoder-decoder with graph embedding and attention layer		Learn-to-configure. RL provides initial-solution to heuristic search
Lu et al. [124]	CVRP	REINFORCE	Encoder-decoder with self-attention		Learn-to-improve. Heuristics utilise RL for improvement
Kalakanti et al. [125]	VRP	Q-Learning	Clustering and tabular		Learn-to-improve. Two phase heuristics with Q-Learning: Clustering to approximate vehicles and tours and then apply Q-Learning to learn routes
James et al. [126]	Dynamic VRP	A3C	Encoder-decoder with graph embeddings and Pointer Network		End-to-end
Wang [127]	VRP	REINFORCE	Encoder-decoder with GNN and Graph Reasoning Network (GRN)		End-to-end. Decomposition and assembling of graphs with GNN and RL
Xing and Tu [128].	TSP	Monte Carlo Tree Search	GNN		End-to-end
Xu et al. [129]	VRP	policy-based DRL	Encoder-decoder with graph attention and MHA		End-to-end
Zhang et al. [130]	VRP with TW	policy-based DRL	attention-based encoder-decoder	X	End-to-end. Enc.-Dec. MARL with centralised learning.
Zhang et al. [131]	Dynamic TSP	policy gradient with rollout baseline	Encoder-decoder with MHA		End-to-end
Zong et al. [132]	Pick-up and delivery	Cooperative A2C	Encoder-decoder with MHA	X	End-to-end MARL. utilise cooperative multi-agent decoders to leverage the decision dependence among different vehicle agents
Prediction and Forecasting
Li et al. [23]	Delay prediction	Supervised	GNN		End-to-end. GNN applying on temporal multigraph (dynamic railway network) for dynamic spatial–temporal modelling
Yu et al. [114]	Spatial–temporal forecasting	Supervised	GCN		GCN for extracting spatial features and Gated CNN for temporal features
Hassan et al. [115]	Forecasting freight movement	RL	Not specified		Learn-to-configure. RL adjust weights for forecast models.
Guo et al. [116]	Spatial–temporal forecasting	Supervised	Attention-based GCN		Dynamic spatial–temporal forecasting with GCN and attention
Zhao et al. [117]	Spatial–temporal forecasting	Supervised	Attention-based GCN		Dynamic spatial–temporal modelling with GCN and attentions
Anguita and Olariaga [133]	Demand forecasting	Supervised	CNN + LSTM		Hybrid LSTM + CNN for dynamic spatial–temporal modelling to extract spatial (CNN) and temporal (RNN) context
Heglund et al. [134]	Delay prediction	Supervised	GCN		End-to-end. Formulate graph with delays on nodes and edges

* ma: Multi-Agent.

(Appendix B) provide an overview of the reviewed studies, classified according to six major problem classes and their architectural and methodological structures.

5. Discussion

As evident from the reviewed State of the Art, the adoption of encoder–decoder networks in combinatorial problem solving has already resulted in significant advancements in the field of optimisation. The ability of advanced encoder networks (e.g., GNNs and GANs) to process a rich problem context and the fact that they are no longer limited by the fixed input size, suggests that more general frameworks for different combinatorial problems are on the horizon. In fact, the first attempts to develop universal end-to-end NCO frameworks for different types of optimisation problems are already underway (see [135,136,137]).

Nevertheless, strategic system design problems, such as network planning, identification of the customer regions, and strategic mode choice, have not yet been extensively addressed by NCO research. Data-driven solutions, such as RL, have their roots in optimal control, and, as a result, the application of NCO on static, sparse networks appears to be over-sophisticated for the optimisation community. However, problems involving integrated system design and operational planning, such as the dynamic Location Routing Problem (LRP) (see, e.g., [138]), are well-suited candidates for learn-to-improve NCO or HRL. In the first case, simple optimisation heuristics could be used alongside the NCO model. At each step, the algorithm provides suitable terminal locations, while the NCO agent calculates transport costs for each partial solution (e.g., routes). For large LRP problems, replacing the optimisation heuristics with another, higher-level RL agent may be feasible. This would result in an HRL approach for an integrated system design problem. In fact, end-to-end strategies are more suitable for dynamic, online sequential problems (e.g., routing, sorting at terminals, scheduling, and resource assignment) because agent policy constructs decision sequences based on input within the same environment. In contrast, the hybrid learn-to-configure strategy indicates applicability in practical scenarios involving nested problems, such as vehicle fleet optimisation with an integrated assignment to trans-shipment terminals and vehicle routing, where complex sub-problems can be effectively encapsulated within an individual simulation environment. A learn-to-improve strategy is more suitable for strategic and static problems, or for problems requiring fast and precise solutions, where the solution search is guided by an exact method. Generating the initial solution with a simulator and applying NCO to accelerate local search yields more stable and reliable results.

The combination of service network (e.g., determine modes, vehicles or service times) and operational planning problems (e.g., routing), together with terminal-related batching and sequencing (e.g., assign cranes to containers), can be approached using both end-to-end and learn-to-improve strategies with either a single central agent or a Multi-Agent NCO setup. Scheduling, dispatching, vehicle routing, and sorting represent typical sequential decision problems in service and operational planning. If the resources are not redistributed (i.e., optimisation focuses on a single terminal/train/vehicle), a central NCO agent would be a good design choice. If resources are redistributed and optimisation occurs across multiple locations or vehicle fleets simultaneously, MARL approaches such as Qmix [79] or HRL [99] with a central upper policy and decentral lower agents can be employed. However, designing decentralised MARL with central policy poses particular challenges. The State of the Art contains limited insights regarding the established ways to rebalance rewards from multiple agents in order to achieve reasonable interaction in a shared environment. Another interesting practical use-case is optimising transport plans for a service network modelled as a sequential decision process on graphs, where each node represents either a train station, trans-shipment terminal, or port [70]. A synchromodal matching platform for intermodal networks demonstrates the most practical use-case for NCO today, allowing freight forwarders or fourth party logistics (4PL) providers to create sustainable and reliable transport services.

Future Perspectives on NCO

Prediction and forecasting: In future logistics services, prediction and forecasting will play a critical role, especially for designing integrated AI-based decision support systems and smart digital twins. We particularly highlight the ability of GNNs and GANs to embed, process, and predict spatial and temporal information (e.g., vehicles, demand, delays). Today, Graph Attention Transformers (GATs) are already widely used in various forecasting scenarios such as traffic flow predictions [63,64,116,139], stocks [140], and time-series [60,66]. For future optimisation frameworks with NCO, GATs can be employed either as an upper policy in HRL or in a learn-to-configure setup, where the central agent predicts demand and parametrises the simulator for route planning based on the forecasts.

Need for dynamic encoder–decoders: Current encoder–decoder architectures do not directly consider the transition function of the RL agents. In complex environments, each action in step t can implicitly modify other related state information in this environment. For example, after taking action $a$ in step $t$, the environment changes dynamically (e.g., the capacity, demand, and time windows of the customers reduce), but these dependencies are only reflected in the mask and additional context embeddings. Embedding the entire graph at the beginning of each episode may reduce solution quality [119,141]. Additionally, once the state information has implicitly changed in state t + 1, the decoder should not rely on embeddings from the previous state t. Boffa et al. [142] suggest that current decoders often overlook this dynamic context information and enhancing decoders will lead to better solution quality.

Accelerating training and increasing solution diversity: Another key factor for achieving better NCO performance, and hence contributing to the development of smart decision support systems, is the training efficiency of the agents’ policies [136]. Xin et al. [141] state that, for most combinatorial problems, the diversity of solutions is critical for the overall performance. While some authors rely on parallel computing and fast matrix multiplication [137,143], other scholars enhance NCO training by developing algorithms with a parallel local search that starts from multiple problem states simultaneously. Proximal Optimisation with Multiple Optima (POMO), proposed by Kwon et al. [144], enables policy training starting from multiple different nodes in parallel (learning on trajectories at each timestep), which effectively enhances the solution search (a Multi-Greedy decoding). However, this approach has only been validated on classical one-dimensional sequential decision problems, like sorting and routing problems, but it is potentially a suitable choice for advanced NCO frameworks with upper and lower policy training [136,145].

Lack of safety, transparency and explainability: Although the majority of the reviewed studies advocate for achieving near-optimal solutions while significantly reducing computational time, a notable drawback of learning-based algorithms is the lack of safety, transparency, and explainability with respect to the decisions (actions) taken. Although building real-world safety-critical systems in the RL domain receives much attention [146,147,148], these efforts mostly address the selection of actions from a defined set of safe actions. In general, evaluating and proving the optimality of the learned policy is challenging, making the practical application of NCO techniques currently questionable. Among the reviewed literature, only Beeks et al. [83] address this issue by proposing decision trees to trace and evaluate actions. To this end, the practical application of NCO paradigms in intermodal scenarios appears limited. Garmendia et al. [149] state that while NCO cannot currently compete with classical metaheuristics in terms of the solution quality and practicability, this is primarily linked to its early stage of development. To achieve further practical advancements, there is a strong need for interdisciplinary research and closer collaboration between the fields of transport and NCO.

6. Conclusions

The central objective of this study was to systematise and synthesise recent research from the fields of transport logistics, optimisation, and DL, focusing on the integrated problems commonly occurring in IFT. Currently, the majority of new DL and transport optimisation models are developed within their own disciplinary domains, resulting in a fragmented perspective and a lack of necessary synergies. Overcoming these limitations could enable future research to design high-performance, intelligent solutions, and smart decision systems that support sustainable freight transport.

The presented study first established the framework for the IFT system, systematically outlining its main operations, decision entities, and problems that need to be addressed in an integrated manner to foster IFT operations and services. From this analysis, we inferred six relevant problem classes and associated them with common OR problems, which are often addressed in isolation. Through the methodological preliminaries, the key principles, strategies, architectures, and algorithms for designing advanced learning-based algorithms were outlined. Finally, an in-depth review of the State of the Art, with a focus on transport-related NCOs and the subsequent discussion was conducted, aimed to derive insights and articulate key principles for designing advanced learning-based frameworks for transport systems in the 21st century.