A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems

Abstract

This paper presents a systematic review on reinforcement learning approaches for combinatorial optimization problems based on real-world industrial applications. While this topic is increasing in popularity, explicit implementation details are not always available in the literature. The main objective of this paper is characterizing the agent–environment interactions, namely, the state space representation, action space mapping and reward design. Also, the main limitations for practical implementation and the needed future developments are identified. The literature selected covers a wide range of industrial combinatorial optimization problems, found in the IEEE Xplore, Scopus and Web of Science databases. A total of 715 unique papers were extracted from the query. Then, out-of-scope applications, reviews, surveys and papers with insufficient implementation details were removed. This resulted in a total of 298 papers that align with the focus of the review with sufficient implementation details. The state space representation shows the most variety, while the reward design is based on combinations of different modules. The presented studies use a large variety of features and strategies. However, one of the main limitations is that even with state-of-the-art complex models the scalability issues of increasing problem complexity cannot be fully solved. No methods were used to assess risk of biases or automatically synthesize the results.

1. Introduction

In recent years, there has been an explosion in the development and application of machine learning (ML). As models increase in complexity and parameters, more ambitious tasks become doable. Performance is not only tied to computational power but also to large volumes of high-quality data. However, most industrial problems are commonly represented as mathematical formulations or simulations.

Reinforcement learning (RL) is a subset of ML methods that are not so data-dependent. At the core of the RL loop is the iterative interaction between RL agent and environment. Instead of learning by inferring patterns from large amounts of data, RL agents learn by interacting and exploring a simulated environment step-by-step, learning with each interaction. This makes RL approaches perfectly suited for simulations and formulations of real problems, relinquishing the need for large datasets.

1.1. Reinforcement Learning Foundations

RL is a decision making framework where an agent learns by interacting with an environment to maximize the total received rewards. Many RL approaches model their environment as a Markov Decision Process (MDP), which consists of specific states, actions that navigate those states, rewards associated with reaching each state and the transition probabilities for the action–state paths. As represented in Figure 1, the agent takes action on the environment, making it change. The current state at the next time step, as well as a reward signal, are returned to the agent. This is repeated, possibly until a terminal state is reached, creating the agent–environment loop.

The reward signal quantifies how desirable it is to reach a state, and since the goal of the agent is to maximize the cumulative sum of all expected future rewards this signal guides the agent towards the desired behavior. This can be represented by value functions, where the state-value function V estimates the future cumulative reward starting from a certain state and the action-value function Q estimates the reward of taking a certain action in a specific state. These functions can be updated recursively to better estimate the long-term expected rewards when selecting a state or a state–action pair.

The mapping between states and actions is the policy, and different algorithms are used to adjust it. The Q-value is the expected reward for a specific action at a certain state and then following the policy. One core challenge of RL implementations is the trade-off between exploration, trying new actions to find better rewards, and exploitation, choosing known high-reward actions. A common strategy is to use the $ϵ$-greedy algorithm, which selects exploitation most of the time but has a small probability to explore by selecting a random action.

To store the value function estimates, tabular representations can be used. These lookup tables store the Q-values for each state, in a column, or state–action pair, in a matrix. If the number is large or the state space continuous, function approximation techniques can be used. These include linear models and neural networks, which are parametrized functions that generalize across states but may introduce approximation errors. When deep learning is used for this purpose, it is commonly referred to as deep reinforcement learning (DRL).

1.2. Previous Reviews

Many reviews and surveys using RL for the optimization of industrial problems exist, as seen in

Table 1. Selected literature reviews and surveys from the query used in this paper.

Reference	Type	Methods	Area	Objective
[2]	Survey	Soft computing	Wireless sensor networks	Approach overview
[3]	Review	ML	Smart energy and electric power systems	Approach overview
[4]	Review	DRL and evolutionary	Job shop scheduling	Overview
[5]	Review	ML	5G wireless communications	Potential solutions for area
[6]	Vision article	RL and digital twins	Maintenance	Approach overview
[7]	Review	RL	Multiple topics	MDP, RL algorithms and theory
[8]	Survey	RL	Software-defined network routing	Identifying and analyzing recent studies
[9]	Survey	RL and DRL	IoT communication and networking	Application analysis
[10]	Survey	DRL	Traffic engineering, routing and congestion	Application and approach overview
[11]	Review	RL	Production planning and control	Characteristics, algorithms and tools
[12]	Review	DRL	Intelligent manufacture	DRL applicability versus alternatives
[13]	Review	RL and DRL	Maintenance planning and optimization	Application taxonomy

. These reviews are selected from the results of the query used for the literature review. Reviews commonly base their analysis on problems solved and algorithms used, either only briefly commenting on the agent–environment interactions or doing so for a very specific context. Practical implementation details, such as state encoding or reward design, often fall outside the scope of such analysis or have only a couple of lines dedicated to it. There are important and novel insights that can be borrowed from various areas but that do not easily proliferate across different application types. Thus, this review focuses heavily on the different approaches to the agent–environment interactions, independently of application or algorithms used.

1.3. Research Questions

A systematic literature review is conducted in this paper. The focus is on combinatorial optimization problems that represent real-world industrial problems using RL. This paper analyzes and compares different approaches to three important design decisions of any RL framework, the agent–environment interactions. The objectives of this paper can be described by the following research questions (RQ):

• RQ1: how to sufficiently encode the problem (state representation);
• RQ2: what control is necessary over the problem (action space);
• RQ3: how to guide the agent towards the desired behavior (reward design);
• RQ4: what the main limitations are for practical implementation;
• RQ5: which future developments are needed.

When relevant, model architecture and problem constraints are also discussed. For the targeted industrial problems, most approaches either do not mention transition probabilities or define them as deterministic. The exceptions are some maintenance and resource breakdown problems, which explicitly detail the transition probabilities. For many problems, the traditional MDP representation is not feasible, for example, if there are too many states. When transition probabilities are fixed at one, the MDP representation also becomes irrelevant. Thus, these are considered as out of scope for the work developed here.

The review methodology is presented in Section 2. Section 3 discusses state representation and format. The action space is described in Section 3 and reward design in Section 5. The research questions are discussed in Section 6. Finally, Section 7 presents future research avenues.

2. Review Methodology

The methodology used in this review follows the PRISMA guidelines [14]. For this study, the following platforms were selected: IEEE Xplore, Scopus and Web of Science (WOS). For the Scopus (https://dev.elsevier.com/, accessed on 13 March 2024) and WOS (https://developer.clarivate.com/apis/wos, accessed on 13 March 2024) databases, the official APIs were used to extract the results. For IEEE Xplore, the same keyword search was conducted in the browser advanced search tool (https://ieeexplore.ieee.org/search/advanced, accessed on 13 March 2024). The keyword search for each platform was as follows:

(reinforcement learning) AND (industrial OR industry) AND ((combinatorial optimization) OR (operations research) OR (job shop) OR scheduling OR routing OR (bin packing) OR knapsack)) AND (real-world OR application OR benchmark OR (case study) OR (framework))

2.1. Search Criteria

For each database, the title, abstract and keywords were searched on 13 March 2024. All journal papers, conference papers and book sections were considered without date restriction. All the steps, from database extraction to the final included papers, are represented in the PRISMA flow diagram in Figure 2.

A total of 1286 items were returned from the three databases: IEEE Xplore (347), Scopus (565) and WOS (374). Each database search matched the title, abstract and keywords with the query. First, all retrieved items without DOIs were considered ineligible and discarded (119). Then, from all three database queries, duplicated (459) items were removed, resulting in a total of 715 unique papers, as seen in Figure 2. No papers were excluded in the screening phase. Of these, 33 papers were not retrieved due to lack of access (29) or not being available in English (4).

2.2. Eligibility Criteria

For the collected unique papers, the exclusion criteria used to determine eligibility that resulted in 298 papers included in the review were as follows:

• Out of scope: irrelevant topics outside of industrial applications or not of combinatorial optimization (221);
• Insufficient details: irreproducible papers without all needed explicit descriptions of agent–environment interactions (99);
• Reviews and surveys (64).

To collect these data, one author manually checked for sufficient descriptions of agent–environment interactions. This information, together with the title and abstract, was also used to evaluate if papers were in the scope of this review.

This review focuses on industrial combinatorial optimization problems. Thus, we are mostly concerned with problems that deal with resource allocation, such as entity distribution or selection. Common examples include manufacturing, scheduling, and network allocation. Papers are considered out of scope if the application mainly deals with control or energy management systems. For these examples, the action is often a continuous control signal.

The resulting 298 papers contain verified and thorough implementations of RL for industrial combinatorial optimization problems. The state representation, action space and reward design are clearly explained in every paper selected, and the following sections group and categorize the different approaches to each of the three agent–environment interactions. The distribution of publications over time is presented in Figure 3. It can be seen that while little interest was given prior to 2015, in the last 10 years the number of publications has kept growing.

Besides manually describing and categorizing agent–environment interactions, algorithms used, etc., which were then used to populate the respective tables and images of this document, no other synthesis methods were used. No certainty assessment measures were used, no effect measure outcomes were used, and no missing results risk of bias or certainty assessment was performed.

3. State Representation

The state is the representation of the environment available to the agent. This section groups and details different state representations found in this review.

Table 2. Common state representation features.

Group	Features	References
Resource features	Utilization and efficiency	[16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42]
	Queues	[40,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68]
	Load	[17,25,63,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101]
	State and status	[17,48,60,68,69,89,90,93,96,100,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131]
Entity features	Completed	[16,27,35,41,94,96,113,131,132,133,134,135,136]
	Demand	[18,32,58,79,81,94,137,138,139,140,141,142]
	State and status	[26,27,59,73,86,93,96,110,117,120,124,132,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160]
	Location	[26,61,69,75,81,83,93,119,140,141,156,161,162,163,164,165,166,167,168,169]
Time-related	Lateness and due dates	[18,27,52,66,75,134,168,170,171]
	Earliness and slack time	[18,29,33,34,41,57,59,134,171,172]
	Idle and waiting time	[16,27,39,40,60,78,93,123,132,173,174]
Solution	Full	[79,81,153,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193]
	Objectives and other metrics	[22,30,35,41,47,49,59,76,80,170,172,184,192,194,195,196,197,198,199,200,201,202,203]
	Infeasibility	[29,33,110,140,204]
	Simulation and prediction	[69,79,92,96,129,172,205,206]
	Past info	[20,25,64,73,79,81,90,197,206,207,208]
Static parameters	Processing times	[22,34,41,49,57,69,78,93,119,131,135,146,160,192,193,209,210,211,212]
	Arrival times or rates	[32,51,65,66,75,98,119,165,166,167,210,212,213,214]
	Capacity and demand	[24,33,39,69,81,85,121,134,155,176,177,186,215]
	Details	[18,27,30,38,52,60,64,72,73,74,81,85,93,103,119,123,136,140,155,160,174,176,177,182,189,201,206,207,213,214,216,217,218,219,220,221,222,223]
Problem-specific	Energy	[51,88,98,100,101,105,132,164,168,198,201,203,213,214,224,225,226,227,228,229,230]
	Maintenance	[27,45,48,60,89,116,155,231]
	Bin packing	[176,179,186,190,195,221,222,232,233,234]
	Networks	[51,63,72,76,159,173,186,205,226,231,235,236,237,238]
Multi-agent		[20,21,39,49,54,55,56,62,67,69,78,82,87,89,100,107,109,116,119,131,138,141,167,172,184,206,207,215,217,225,231,239,240,241,242,243,244,245,246,247,248,249,250,251]
Hybrid strategies		[163,203,252,253,254,255,256,257,258,259,260]

categorizes the most common state features in the reviewed papers, which are then further detailed in the following sections.

The most common approach is to use environment variables as different state features. These variables, representing different model states, change over time as the environment reacts to the actions given. By tracking key variables and related features, many approaches manage to create a short yet representative state.

Some of the most popular state features are resource-related. This includes machines, workers, vehicles, channels, etc. Other popular features relate to system entities such as jobs, operations, tasks, orders, products, etc. Job and order are used interchangeably to refer to the highest-level entity. A job can be the manufacturing of an object. Operation, sometimes used interchangeably with task, is a specific step in a job’s production process. Each step requires different resources. However, tasks can also be used as smaller units of work within an operation [29].

When the current environment should be represented, time-related metrics or solution representations can be part of the state. Time-related metrics can be computed for resources or entities and track lateness, wait times, etc. When representing the solution or its encoding, the objective function values or certain infeasibilities can be informative to the agent.

Static problem parameters provide extra context to the agent. These are frequent when the same agent must solve different problem instances, as it needs to be aware of changes in available resources, number and types of jobs, processing times, etc.

Some state representations are very problem-specific, only making sense for certain topics. This is the case for networks and energy problems. When the agent has a hybrid role, the state representation is also problem-dependent.

Lastly, it is also common for approaches to normalize state features [41,57,90,132]. In [132], the state is a list of seven features, such as percentage of finished jobs or idle time since last operation. Each feature has a unique scaling factor to keep all state values between zero and one. In [57], min–max scaling is applied to each job feature of the state.

3.1. Resource Features

It is very common to represent resource efficiency by measuring its utilization [18,29,33,40,41,42]. Non-normalized utilization is occupation, the amount of time a resource is busy [17,26,38,78].

Many approaches take particular interest in measuring queues [44,48,55,66]. While the number of queued jobs [45,49,50,54,60] is the most popular metric, the utilization of queues [60,68] or the free spaces available [40] are also used.

Load and demand can also give important information. In [78,96], the number of jobs currently being processed by machines is measured. In [93], the state contains the resource load, and in [17], the state has the current allocated demand. When describing more than the resource time needed, for example, considering volume and complexity of tasks, the workload is used [82,83,84].

Categorical information can be used to translate the condition or situation of a resource at a certain time, that is, the state. One of the most common state examples is the resource working behavior [60,69,93,110,113], for example, idle, working, under maintenance, etc. This notion of resource state is sometimes used interchangeably with status, which more often gives the position or condition of a measurement in relation to something.

3.2. Entity Features

Many metrics evaluate job completeness. Examples include the percentage of operations completed, or completion rate [35,41,94,136], the number of operations left [96,113] and the overall job demand [79,81,137]. Instead of counting jobs, refs. [41,132,133] count the remaining processing time. The working status of a job, if it is waiting, completed, etc., is a common feature [26,96,132], either categorical or one-hot-encoded.

Some problems require monitoring resource or entity locations. In [75], each vehicle is described by its current zone, available seats, pick-up time and destination zone of each passenger. In [165], each of the four flight legs are described in terms of origin and arrival locations and departure and arrival times. For entities, tracking the current entity location [81,140,161,168] can be of interest.

3.3. Time-Related

When a job entity is completed after its due date, the deadline for completion, there is lateness, or tardiness [18,27]. Lateness and tardiness are usually used interchangeably, but lateness can also be used as tardiness of larger magnitude.

Earliness [18,57] measures the surplus time from job completion to due date. Slack time [29,34,41,172] is used to measure how much a task start can be delayed without impacting overall completion time. If a job has zero slack time, it is the bottleneck of the system [59].

To measure idle time, the average waiting time [16,40,60,78,93], or the expected value of the wait time [39], measures how long job entities remain idle.

3.4. Solution

A solution representation can be part of the state. Some examples include operations sorted by start time [179,192], vehicles by sequence of visited nodes [188] or bin packing insertion order [176,179]. The whole solution [79,153,176,184], a partial solution [81] or the initial solution [180] can be used.

The objective function value [30,41,184,192,197] or similar metrics are also common, such as remaining processing time [41,57,62] or number of tasks left [142]. Some approaches measure or detail solution infeasibility [29,110,140,204].

Simulation-related metrics, such as simulation time [69,79,92], or past information, such as previous actions [20,197,206], can be useful in the state.

3.5. Static Parameters

Static problem parameters provide extra problem context for the agent. These are frequent when the same agent must solve different problem instances, as it needs to be aware of changes in available resources, number and types of jobs, processing times, etc.

It is very common to include entity [34,41,160] and resource [93,160,211] properties, such as resource maximum capacity [33,69,85], job processing time [49,57,69,78,192], arrival rates [65] or due dates [66,213].

3.6. Problem-Specific

Certain measurements only make sense considering the problem tackled. Energy management problems care about energy demands [132,198,213,225], energy levels [100,214,229] and consumption [201,203].

For network problems, many specific metrics are used, for example, signal-to-noise ratio [46]; resource backhaul transmission volume [91]; and resource radio channel quality [68].

For maintenance-related problems, specific state features include machine maintenance duration [45,116] or resource degradation level [48,89].

3.7. Multi-Agent

For multi-agent problems, the state of a single agent often contains all agents’ features [206,246,251]. Listing the resource metrics is also viable [250], especially when paring agents and resources [241,244,249].

For example, the agent input in [244] consists of three different matrices: processing time of jobs, assigned jobs to each agent and completed jobs. In [249], the state also includes info on current jobs requesting a decision and number of jobs in the other resources, the conveyors. Alternatively, some approaches can have a unique state per agent [49].

3.8. Hybrid Strategies

For some approaches, the RL agent is not used as the optimizer or solution generator. Instead, it takes an auxiliary role to other optimization strategies, using information about the state of the optimization itself, not the solution. Large-scale and uncertain resource scheduling problems are solved in [261], with an agent making pre-selections and order solving to simplify the problem a linear programming model must solve next. In [254], an RL agent complements a constraint programming algorithm. The state representation includes information on the instance solving, the current model and statistics from the past solving operations.

The RL agent can often be paired with metaheuristics [253,262], such as ant colony optimization [263] or particle swarm optimization [259]. Agents can read population metrics [203,257,260] or directly use algorithm optimization parameters as the state [252]. In [253], where the agent is used to control the parameters of a Cuckoo search metaheuristic, the state describes the population diversity and the tracks if there is diversification of intensification of solutions. In [259], the state is the current population levels of a particle swarm optimization algorithm.

The set of optimization parameters, which are used to generate circuit designs, are the state of [252]. A fuzzy rules-based approach is used in [258], where the conditional part of the fuzzy rules is the state.

Some agents read population metrics [203,253,257,260], such as average fitness and diversity or population improvement [260]. In [253], where the agent is used to control the parameters of a Cuckoo search metaheuristic, the state describes the population diversity and the tracks if there is diversification of intensification of solutions. The authors of [254] complement the RL agent with a constraint programming algorithm. The state representation includes information on the instance solving, the current model and statistics from the past solving operations.

The set of optimization parameters, which are used to generate circuit designs, are the state of [252]. In [259], the state is the current population levels of a particle swarm optimization algorithm. The agent will only work on the number of levels. This level-based learning approach forces particles to learn from upper levels only. Lower-level particles will focus on exploration and higher-level particles on exploitation.

3.9. State Format

The reviewed state representations always include some of the features previously described. The most common state format that gathers these features is a fixed list of variables. However, there are other noteworthy formats or considerations, as illustrated in

Table 3. Categorization of state representation formats with representative examples.

Format	Content	References	Insights
Entity lists	Averages	[16,18,27,34,35,37,42,78,203,264]	Compact and simple, loses individual details
	Per resource	[20,35,37,74,95,101,108,122,127,169,205,226,247,265]	Granular details but less scalable
Spatial represen- tations	Matrices	[22,70,71,76,112,120,152,166,167,172,193,194,195,208,210,212,221,228,232,233,234,266,267]	For structured environments and spatial reasoning
	Heightmaps	[195,221,222,232,233,234]	Capture 3D variations in a 2D representation
	Convolutional approaches	[22,61,173,181,198,205,232,234,266]	Automatic feature extraction
Graph solutions	Undirected and directed	[19,21,142,219,220,228,246,254,268,269,270]	Symmetric and asymmetric relational dependencies
	Disjunctive graphs	[49,78,80,117,130,271,272,273,274,275,276,277,278]	Complex relationships and multi-entity interactions
	Graph node features	[81,219,229,246,261,268,278,279,280,281,282,283]	Per-entity attributes
	Graph edge features	[19,21,113,136,150,220,229,239,273,284]	Captures relationships and dependencies between entities
	Graph NN	[21,70,78,80,130,142,188,219,246,254,261,268,269,270,271,274,275,277,278,279,283,285,286,287,288]	Process graph states, improving generalization at computational cost
Variable-sized		[141,179,181,220]	Flexible, adapts to dynamic state spaces
	Recurrent neural networks	[20,47,65,104,115,136,156,178,184,198,238,243,271,288,289,290,291,292,293]	Capture temporal dependencies in sequential decisions
Fuzzy		[83,246,258]	Model uncertainty, useful for imprecise states

. It summarizes how space representations can be shaped by providing a structured overview of the different formats in an application and feature-agnostic categorization. Note that the accompanying references are non-exhaustive lists of representative examples.

Entity lists, for example, resources or jobs, often present one of two approaches. When the entity features are averaged over all resources [16,18,27,34], it results in a compact, low-dimensional representation that is easy to handle but might overlook some relevant information. Alternatively, state features can each directly relate to individual entities [20,35,247], resulting in a more granular representation. However, this option requires a longer state that must also be fixed-sized, hurting generality.

Spatial-based representations allow for the data position within the structure to also play a role. States represented as matrices [71,221,232,233] leverage structured environments, such as grids and images, where the spatial proximity and order are relevant. An example of a specific matrix representation is a heightmap [195,221,232,233], often used to encode three-dimensional space into a two-dimensional representation. It is also important to highlight applications using convolutional neural networks (CNNs) [61,205,232,266] that, while more cumbersome to train, can automatically extract spatial features.

Graph-based representations are useful for relational and entity-based problems. Directed and undirected graphs [142,219,246,270] capture relational dependencies, respectively, asymmetric and symmetric interactions. An example of a directed arc would be a precedence constraint between operations, while an undirected arc could connect all job nodes compatible with the same resource. Disjunctive graphs [272,274,277,278] use both types of arcs and can encode structures with complex relationships, which is especially useful for multi-entity interactions such as scheduling.

Graph node features [81,246,278,282] allow for considering multiple features per resource regardless of the problem size, while edge features [19,113,150,239] are more focused on relationships between entities. These are both features used in graph neural networks (GNNs) [188,246,271,278], which process graph-based states more generally at the cost of higher computational complexity.

GNNs have the advantage of being size-agnostic. Other approaches also allow for variable-sized state formats [141,181], which are applicable to complex problems where the number of features can change. An example is recurrent neural networks (RNNs) [47,178,184,238], useful for capturing temporal dependencies for sequential decision making problems. These bring great flexibility and scalability to the problem, again at the cost of extra computational effort. Lastly, fuzzy approaches [83,246] can be used to handle uncertainty and imprecise states.

4. Action Space

The following chapter details popular actions made available to the agent in different literature approaches.

Table 4. Popular agent action categories.

Group	Features	References
Q-value estimation		[20,23,30,45,73,78,79,83,84,91,107,110,138,149,150,153,156,160,164,165,178,200,206,207,208,211,263,280,290,294,295,296,297,298]
List selection	Resources	[20,38,49,64,74,81,86,91,105,111,112,121,122,124,126,131,149,150,153,157,160,164,168,173,204,230,237,241,242,288,296]
	Entity	[31,32,47,49,50,52,54,57,59,64,68,73,79,80,83,84,87,94,96,108,109,115,130,131,132,134,135,139,143,148,154,163,172,176,177,179,198,199,210,213,215,220,232,271,273,274,277,278,280,286,287,291]
	Locations, positions and nodes	[70,71,75,76,81,97,102,128,136,137,142,152,158,166,175,186,187,188,190,191,195,219,221,222,229,239,242,251,268,269,276,281,282,284,285,299,300,301]
	Network-specific	[20,23,24,38,63,82,91,103,104,121,122,123,127,147,149,153,173,194,201,211,241,243,245,250]
Sequential decisions	Resource reallocation	[17,40,46,49,55,69,80,89,113,114,125,129,146,164,167,170,246,249,266,302]
	Accept or reject entities	[49,55,80,89,161,217,226,267,303,304]
	State change	[17,39,40,43,49,55,59,61,69,80,89,90,113,114,164,167,168,223,246,249,266,303,304]
	Cardinal and ordinal directions	[118,156,206,240,305]
	Maintenance actions	[45,48,60,151,231,238]
Heuristic selection	Dispatching	[16,18,22,27,30,34,35,36,37,41,42,53,62,66,93,120,169,171,209,242,244,264,289,306,307,308]
	Local rules	[29,33,181,183,197,203,208,260,270]
Multiple selections	Single-action-encoded	[59,64,116,117,189,205,233,275,309]
	Multiple decisions	[25,114,123,127,147,168,193,196,201,210,228,243,245,289,307]
	Resources	[23,26,28,47,59,64,82,86,95,99,100,103,123,135,147,162,182,191,226,239,307]
Variable-sized output		[19,47,65,184,185,243,289,292]
Number estimation	Number output	[140,202,207,247,310]
	Continuous value	[23,58,77,92,95,98,101,105,106,164,199,214,224,225,227,228,236]
	Discretized continuous values	[88,127,141,193,311,312]
	Hybrid strategy parameters	[144,159,218,235,252,253,254,255,256,257,258,259,261,262,263,286,298,313]

groups all the cited papers in relevant categories. Two common strategies compatible with multiple action spaces are $ϵ$-greedy exploration and action masking.

The classic approach is to have the RL agent estimate the Q-value of a certain state or state–action pair. However, it has become more popular to have the agent directly select a candidate option from a list, such as the next entity to consider or the next resource to allocate.

Alternatively, the list of actions can be a smaller, repeatable set of decisions, for example, accepting or rejecting a certain entity into a resource. Some approaches conduct hyperheuristic selection, having the agent choose the most appropriate heuristic based on the current state. The agent can also make multiple selections at once, combining any examples from this section.

Depending on the architecture, the agent might output a full solution. This is typically from approaches that work with variable-sized outputs. Finally, the output can produce one or multiple number estimates. The use cases include simulation or metaheuristic parameters.

In $ϵ$-greedy approaches [45,175,225,250,310], $ϵ$ is a small number. With probability 1-$ϵ$, the action that will lead to the highest Q-value is selected. Otherwise, with $ϵ$, a random action will be selected. This strategy is often used to balance the exploration of new solutions versus exploiting currently promising solutions. This is a common alternative to always selecting the highest Q-value [267,280,289]. The $ϵ$-greedy threshold can also decrease over the training [95,260].

Action masking refers to only allowing the agent to select feasible actions [139,204,233,271], such as unavailable resources [84,121], entities [57,80,96,132,179] or locations [39]. In [24], all options are available, but if the candidate option returns an unfeasible solution, the agent does nothing instead. The authors of [170] include the feasible actions in the state.

4.1. Q-Value Estimation

Estimating the Q-value is of fundamental importance to RL approaches. Traditional methods estimate the Q-value of a state [150,207] or a state–action pair [45,165,294,296,298]. By comparing the Q-values of reachable states, the agent can select the action leading to the highest expected return.

Function approximation methods can use non-linear models, such as deep neural networks, to estimate the Q-values of each state [138,153,160,297] or state–action pair [84,91,149,164], even if the number of states is infinite. For example, to predict state–action pair values, ref. [290] uses a long-short term memory (LSTM) model and ref. [160] uses a deep neural network with an attention mechanism.

Value-based approaches, which compute Q-values for states or state–action pairs, can apply a softmax layer to the value outputs and use the result as an expected return probability distribution for each action [30,78]. This works even for tabular approaches [73]. Instead of searching for the maximum Q-value, the agent can directly select a candidate option form a list. Policy gradient models’ policy outputs the action selection probability for the said list [20,79,178,206,208], also using a softmax layer.

4.2. List Selection

Besides the possible states or state–action combos, the agent can be used to select a candidate option from a list. This is a very common option, and this selection can be conducted with or without repetition of the elements.

The action space can be a list of resources [38,49,122,160,296], jobs [49,94,198,213,232] or operations [31,115,271,278]. Locations [136,239,299] can also be selected by the agents.

4.3. Sequential Decisions

Agent decisions can be used to change the resource allocated to entities [40,113,167,246] or to move entities in and out of buffers [49,217,304]. It can also be used to change the resource state [40,113,167,246,304].

When the agent can move in a grid, it is common for the available decisions to be the cardinal directions [240,305] and in some cases also the ordinal directions [156,206]. Not moving or halting movement [206,240] is also common.

Multiple output approaches also use agent decisions [123,210,243,307], but often in only one of the outputs, having some other type of agent action in the other output.

4.4. Heuristic Selection

Heuristic selection strategies are both list selection approaches, picking a single heuristic from a list, but also hybrid strategies. Furthermore, it is an extremely popular approach. In particular, dispatching rule selection is widely used in the literature. By using simple strategies such as Shortest Processing Time (SPT) or Earliest Due Data (EDD), an agent can sequentially add all actions to a schedule [18,22,35,41,307].

Instead of dispatching rules, local search methods can also be the heuristics selected. These simple operations search for the neighboring solutions in search of an improvement [29,208,260,314].

4.5. Multiple Selections

To make multiple selections with a single action, approaches provide all possible combinations of resources as individual actions to the agents [116,275,309]. For scheduling, this can be resource and job pairings [59,64], or job and factory [275]. More commonly, the agent outputs multiple decisions in as many output values, often pairs, or with a list output with as many elements as resources [23,47,103].

While some bin packing approaches also use every combination of item, location and orientation [233], others have each decision as a separate output [189]. For graph-based problems, it can also select nodes [81] or links [242].

4.6. Variable-Sized Output

While not common, some approaches present variable-sized action outputs to solve different-sized problems with the same model. Some approaches give the Q-value of each resource [65] or a new full solution [184,292]. This is achieved with specific RNN architectures [47,243,289], for example, using an LSTM model [65,184,292].

4.7. Number Estimation

The output of an agent can be a number estimate [140,247], such as the number of local updates [310] or the number of packets to forward [207]. For these examples, the agent action is an integer number.

Continuous values can be used directly from the agent output [21,67], which is very common for energy management systems [98,105,214,227,236]. Alternatively, discretized continuous value ranges are sometimes used [88,141,311,312].

Hybrid strategies with RL agents and other optimization strategies working together are often used to estimate certain variables. One approach is to output parameters for an optimization algorithm [252,253,254,257,261,262].

Alternatively to parameters, the agent in [263] learns the value of different cities to complement an ant colony optimization algorithm. In [313], the agent outputs three weights, which are used to select the next node. In other approaches, the agent is used to change resource limits, such as inventory [261] or reconfigurations [159].

5. Reward Design

The following section summarizes the types of reward functions used. Different approaches use different combinations of the categories mentioned below, as seen on

Table 5. Common reward designs.

Group	Features	References
Mirror objective function		[19,21,22,28,33,37,51,65,72,79,93,107,114,119,131,138,149,152,163,164,204,206,214,222,223,226,230,247,264,267,279,286,307,308]
	Negative	[49,50,80,81,175,199,237,295]
	Weighted	[46,59,73,89,113,159,228,249,284]
	Inverse	[16,55,87,247,280]
	Multiple objectives in one	[23,28,46,59,89,103,109,113,134,136,141,201,232,241,249,264,285,289,296]
Relative rewards	Previous solution increment	[26,29,35,49,56,62,77,78,80,82,84,98,109,112,117,122,130,139,142,144,189,190,198,204,208,211,212,220,232,259,260,266,268,269,273,274,275,277,278,308]
	Initial solution	[109,178,179,181,195]
	Global best	[29,188,244,259,263,286]
	Other baselines	[32,95,138,148,190,211,212,233]
Extra terms	Scheduling	[75,110,154,194,195,212,245,246,247,280,301]
	Lateness	[71,92,105,140,153,173,184,191,208,215]
	Cost	[101,108,115,150,151,215,225,227,267,280,307]
	Network-related	[46,47,128,162,187,205,213,236,246]
	Constraint as penalty	[43,44,131,155,199,213,235,239,251,258,261,267,283,292,301]
Conditional rewards		[24,36,40,42,58,76,90,106,120,156,161,165,169,185,193,217,219,249,250,284,308,309]
	Events and states	[26,40,57,68,69,71,86,88,97,116,118,124,129,145,166,168,206,219,231,243,249,250,279,281,282,298,306,315]
	Action selection or infeasibility	[17,24,53,74,76,108,123,125,132,133,136,146,156,163,165,168,185,189,200,206,224,241,242,262,263,294,299,303,305]
Conditional rewards	Metric threshold	[18,48,60,70,71,111,116,132,133,159,167,168,183,207,218,238,279,291,311]
	Solution evolution	[92,96,160,192,196,197,203,257,270,311,313]
	Terminal	[24,34,35,69,76,92,136,143,172,176,177,182,234,282,287,288,302]
Problem-specific	Continuous production systems	[49,59,80,100,173]
	Due dates	[50,65,87,102,119,170,295,307,308]
	Profit and costs	[46,79,89,102,105,134,135,137,162,208,227,304]
	Energy	[72,88,100,128,194,225,289,312]
	Routing	[81,175,206,249]
	Non-linear objective function	[18,27,67,85,110,153,184,210,215,290]
	Multiple and many objective	[18,23,32,63,87,104,132,193,196,226,257,280]
Multi-agent		[20,78,164,172,206,287]
Alternative goals	Unavailable objective function	[22,31,38,39,42,54,55,61,64,66,91,99,115,161,174,186,202,221,247,304,307,308]
	Ratios	[38,39,55,71,126,158,171,185,221,256,304,308]
	Estimation	[22,38,66,91]
	Hybrid strategies	[235,252,253,254,255,263,300]

. Some approaches normalize the rewards [95,152,163], for example, using upper bound [17], min–max [207,249], z-score [296] or average expected time [170].

One of the most direct yet simple approaches is to mirror the objective function as the iteration reward or penalty. The reward value can also reflect the relative improvement of the solution over successive iterations. Both these approaches are often complemented with extra terms that help the agent better achieve the original objective.

Some rewards are the result of multiple reward functions, where certain environment conditions decide which function is used in each iteration. Based on the problem to solve, specific objectives might also be applicable. Multi-agent approaches can use multiple reward functions per agent or have specific rewards based on local and global events. Finally, when it is not possible or convenient to measure the objective function, alternative goals can be used.

5.1. Mirror Objective Function

One of the simplest and yet most common approaches is to reward based on the current objective function value [49,79,81,175]. The current value, the difference between iterations or some linear transformation of either case can also be used. The values measured can be summations, averages and standard deviations. Makespan, or completion time, is very common in scheduling problems [113,279,286,307].

Non-linear transformations to the objective function are also used. Using the max operator [110,153,184] breaks linearity but forces measurements to only take meaningful numbers. For example, negative lateness is not desired [18,27]. Ref. [290] punishes delay increments more severely when the total delay is small than when it is large.

This approach is very popular due to its simplicity and ease of implementation, since it directly optimizes the objective function. However, it can also be too sensitive to problem scale and magnitude differences in returned reward values.

5.2. Relative Rewards

It is common to reward the difference to the previous value [29,122,198,232,278], i.e., the temporal difference (TD). Comparison can also be made with other solutions such as initial [178,179,181,195] or best [29,188,244,259,263,286] or from other baselines [211,212]. This relative fitness can be either the ratio of both values [190,233,269] or the absolute difference [49,208,274].

The makespan is often used as a temporal difference measurement [49,84,142,188,278], but other measures such as profit [98,208,266] or resource utilization [109,178] are sometimes used.

Relative rewards shift the focus to improvement over time instead of immediate absolute improvements. They often have some type of normalization built in, which makes states across different problems more comparable. However, if wrong reference points are used or if noise is too impactful, the agent can converge into suboptimal solutions.

5.3. Extra Terms

When composing the reward, certain metrics can complement the objective function [75,184,195,246,247]. This can be performed to influence the agent behavior to account for costs [115,225,267] or lateness [71,173,208], for example.

Problem constraints can also be used as extra penalty terms [267,292,301], allowing infeasible solutions to be explored while also increasing the state space, for example, exceeding resource limits [258,261] or missing entity due dates [44,213].

Extra terms can make approaches more general with the flexibility they bring, allowing penalties and secondary objectives to further guide the agent. However, balancing multiple terms becomes of extra importance, as extra terms can dominate the primary reward function and introduce biases.

5.4. Conditional Rewards

Instead of having a single reward function, some authors prefer if–else statements to alternate between different reward functions [40,165,217,250,284]. Many approaches provide a zero reward [40,165,249,250] or a small negative reward [76,156,284,308] every step no key event is achieved, falling under the else or the otherwise condition. Alternatively, some approaches provide a fixed positive reward [24,185,219] on key events and zero or negative values otherwise.

Certain events can trigger conditional terms, often from entity or resource behavior. For example, in [309], the default reward is a small penalty, but if traffic flow is successively received, the reward is zero instead. On the other hand, ref. [106] constantly provides a small fixed reward, but if a open edge is selected, then a larger reward is returned.

The selected action can also decide the conditional reward terms [123,132,168,224], for example, if the candidate action is not valid [76,136,189,206] or if the action is successful [165,303]. Highly desirable actions can have greater reward magnitudes [146]. This can also happen upon visiting certain states [116,219] or state transitions [298,315], such as if undesirable states are reached [69,243].

Specific metrics can be used as threshold values to switch between terms [18,48,167,238,291]. The evolution of the solution can be used to select the appropriate reward equation [160,203,257,270,311].

A reward returned only at the end of the episode is called terminal or episodic. This can mean replacing the reward function with a specific terminal reward [69,136,282], with a fixed positive [76,282] or fixed negative [24,136] value. Extra penalty terms can also punish unfinished tasks [92] or training length [234]. Alternatively, agents might only be rewarded at the terminal state [35,176,177,234,302].

Conditional rewards also offer flexibility to the reward function by switching between functions that better adapt to ongoing events. However, these event-based triggers must be carefully designed so that the abrupt reward transitions do not lead to unstable learning. Thus, they require careful tuning.

5.5. Problem-Specific Objective

Certain metrics, either as extra terms or objective functions, only make sense for specific problems. This is valid for thematic areas, such as energy and routing problems, but also for different formulation types, such as non-linear and multi-objective approaches.

Some approaches consider multiple objectives, yet many combine all into a single objective function. In this scenario, it is common to represent each in different weighted terms of the reward function [113,141,161,193,249,289,296]. True multi-objective approaches [18,132,196,257] consider each objective variable separately. Multi-objective goals are adversary: improving one will deteriorate the others.

Domain-specific requirements might call for problem-specific rewards, which provide more meaningful performance evaluations by considering the context. This, however, hurts the generality of the approaches.

5.6. Multi-Agent

Multi-agent approaches can distinguish between individual agent rewards, local rewards, and rewards shared by all agents, global rewards. In [172,206,287], an additional complement is given to all agents when completing the schedule [172,287] or all routes [206]. During the episode, a smaller reward can be given based on the percentage of completed tasks [287] or upon job completion [172].

Some multi-agent approaches return different rewards for each agent. Ref. [164] minimizes routing costs using three different cooperative agents. While one agent rewards utilization and daily rewards, the others share the reward function, including extra penalty costs. In [78], one agent balances the load among factories, while another wants to minimize factory makespan. The former is the difference between maximum and minimum factory total processing time, while the latter reflects the makespan increment with each step.

Specific multi-agent rewards allow for coordination and competition between agents, creating scalable solutions for decentralized systems. However, the complexity increases greatly with the number of agents, and considering global and local rewards requires careful balancing.

5.7. Alternative Goals

Sometimes, the objective is not available during the episode, such as when minimizing total tardiness [66,174] or final makespan [22,38,54,247,307], forcing the approaches to utilize alternative or estimated metrics for the reward calculation. Ref. [22] demonstrate that, for their formulation, the makespan and the resource utilization are equivalent and use the resource utilization increments as reward. Some estimation examples use tardiness [66] or worst possible task completion [38,91].

When the agent’s role is not to optimize, but to assist other optimization strategies in hybrid approaches, the reward might not be related to the overall objective function. In [252], the agent output updates the optimization parameters for a simulation software. The resulting design-performance value is used to update the critic network using mean squared error. For the constraint programming approach in [254], the agent is penalized at every step, encouraging it to reach feasible solutions faster. In the bee colony optimization of [263], where metaheuristic parameters are given by the agent, the reward depends only on the metaheuristic finding a better solution. Lastly, in [276], no reward is used since the authors use imitation learning.

Learning through proxy metrics can be the only viable choice when the direct objective value is not available. This also enables hybrid approaches to optimize for their particular task and not for the overall problem. However, it may lead to unintended behaviors if the alternative metrics are not fully representative of the true goal.

6. Discussion

From the reviewed papers, it is clear that successful applications rely on good problem encoding and manipulation. The algorithm choice and the complexity need to fit the problem, and the reward signal must successfully guide the agent. Depending on how well the problem encoding leverages the state representation and the action space, simpler approaches are often as effective as complex state-of-the-art methods.

RL agents are used for many different purposes.

Table 6. Common RL agent roles and highlighted examples.

RL Role	Advantages	Disadvantages	Examples
Tabular methods	Simple implementation. Explainable results.	Limited state representation. Must sufficiently explore all states.	[45,166,175,294,296]
Iterative list selection	Very flexible. Widespread literature adoption.	Single-use actions. Often requires action masking.	[31,40,49,217,232]
Hybrid approaches	Simplify agents’ decisions. Enhance other methods with RL decision making.	RL only optimizes a subset. External methods or tools might be insufficient to optimize.	[18,123,184,254,258]
Specific neural network models	Variable-sized inputs and outputs. Relative positions of pixels, nodes and tokens provide extra context.	Frameworks are harder to train. Scalability issues.	[141,188,232,238,271]
Multi-agent	Agents can make simpler decisions. Specialized agents for smaller tasks.	More complex frameworks. Decentralization requires communication protocols.	[49,116,217,246,251]

summarizes the different roles RL agents take, showing only selected citations instead of the exhaustive approach from previous tables.

6.1. Research Questions

6.1.1. RQ1—State Representation

The state space representation is crucial to supply the agent with sufficient information so that it can make an informed decision. Of the three agent–environment interactions, the state representation shows the most format variety, as summarized in Table 3. Variables, lists, matrices and even 3D matrices can be used for very different applications. Lists of environment variables are the most popular approach. The metrics included vary greatly but often include information about the resources and entities. While a list of metrics is a simple representation, it has been shown multiple times that it is sufficient for many applications.

Two specific model architectures show great success on multiple examples: recurrent and graph approaches. Both strategies create a state that aggregates information from individually considering all its parts, while still being compatible with problems of any size. Normalizing the state also seems to be beneficial.

6.1.2. RQ2—Action Space

It is important for the action space to allow the agent enough freedom to explore the solution space thoroughly. If the number of actions is finite, ultimately the agent is selecting one option from a list. For approaches that use this strategy, using some form of action masking seems to always be beneficial.

When the model used outputs multiple Q-values or selection probabilities, the agent must take an extra step to choose one of the actions. For exploration considerations, this might not always be the best value. Naturally, one common use for agents is to have them select from a predetermined list. This is often paired with action masking, only allowing feasible actions to be selected, which seems to be fundamental for faster complex problem convergence. Selecting resources and entities is very popular, either as a list item or a graph node.

6.1.3. RQ3—Reward Design

The reward design heavily influences whether the agent will behave as expected. It will not only guide the agent throughout the optimization, but it will also influence the speed at which it converges to the desired behavior. There is no clear preference regarding mixing penalties and rewards or simply using one of them. However, a careful balance between the magnitude of all rewards returned during an episode seems to be useful.

Interestingly, the reward design can be seen as the most modular approach of the three agent–environment interactions. There are many approaches to the design of the reward function, yet these can be broken down into a relatively small set of concepts, such as mirroring the objective function as the reward. Reward functions can be highly customizable by simply picking and choosing a number of options from this set.

Most approaches tend to mirror the objective function change between different episode iterations. Moreover, it is very common to have a conditional reward function. In either case, the studied approaches often include extra penalties based on lateness and formulation constrains. Similarly, rewards can be associated with successful entity or resource events.

6.1.4. RQ4—Limitations

There is a huge variety of states, actions and rewards. Since they are often very problem-dependent, there is a lack of consensus on what the best approaches are for each agent–environment interaction.

Exploration is a key part of RL approaches. They require extensive interactions with the environment, which can be computationally expensive. As a result, RL training is likely slower than other end-to-end deep learning approaches, such as supervised learning, that train on large, curated datasets. However, dataset generation can be cumbersome and expensive. Thus, the development of more data-based approaches can also be considered slow. Since RL approaches can instead leverage existing mathematical formulations, they effectively skip this step. Moreover, various methods such as curriculum learning and offline RL help reduce inefficiencies by improving both training speed and sample efficiency.

While RL training can be slow, a deployed agent is fast when compared to traditional exact or search-based methods used for combinatorial problems. For example, linear programming models and metaheuristic methods can take significantly longer to reach good solutions. Also, the iterative nature of RL easily allows for hybrid approaches.

There are multiple examples where very simple representations and simple approaches are enough to solve small instances of complex problems. However, these smaller state and action spaces are often not enough to represent and interact with the real problem. Industrial problems can be represented in many ways, as seen in Table 2. However, a very complex and thorough state might still not solve the problem. Adding unnecessary or redundant information increases the complexity of the approaches, often increasing the number of parameters greatly, without being matched by the same increase in performance. A careful balance is needed between state and action space length versus their inherent complexity increase.

When more complex approaches are needed, such as GNNs or RNNs, these require some expertise to train efficiently. They are often accompanied by comparatively longer training computational times, which can prevent these models from converging and seeing competitive results. Furthermore, while these bring some advantages compared to traditional models, such as variable-sized inputs or outputs, they do not solve the scalability issues of these representations. Graphs’ edge numbers grow exponentially with number of nodes. Recurrent approaches are forced to use extra mechanisms, such as attention, to not lose important information.

6.1.5. RQ5—Future Developments

It is a recurring issue from the reviewed papers that adding more constraints to the problem is desired, for example, adding due dates to consider lateness or adding resource breakdowns to consider a dynamic scenario. This further exacerbates the scalability issues, which require an urgent solution.

GNN and RNN approaches cannot yet fully solve this problem. Graph-based approaches have trouble scaling past a certain point since increasing the number of nodes might exponentially increase the number of edges. Recurrent-based approaches need more and more strategies to solve problems, such as using attention weights or the latest model architectures, further increasing required computational power.

It is important to use simpler architectures and problem encodings whenever feasible. However, more efficient agent–environment interactions, training frameworks or algorithms are needed to satisfy the desire to further constrain the already-complex problems tackled.

Traditional optimization methods still outperform RL for combinatorial optimization problems. However, RL offers an alternative approach better suited for dynamic environments and changing problems without customized problem tuning.

Various hybrid approaches where RL methods complement classic optimization methods have already been discussed in this document. However, there is still potential for RL to further save computational effort and increase efficiency by reducing the search space, providing metaheuristics parameters and warm-starting both solvers and metaheuristics, for example.

6.2. Popular Algorithms

There are many options available for RL algorithms, as shown in Figure 4. There is some overlap between policy gradient and actor–critic methods. For clarity, priority is given to algorithm naming conventions. It is common for papers to simply refer to their approach as DRL. These cases are categorized in the “undisclosed” label of the overall figure.

Q-learning is a common choice, allowing the agent to estimate Q-values for given state–action pairs [73,165,175,207,296]. Other algorithms used in the same context include REINFORCE [178,216,288], value iteration [92], TD-learning [166] and Bayesian RL [110].

Many approaches use deep Q-learning (DQN) [38,52,54,121,160]. Some common variations include double Q-learning [70,80,91,149,301] and dueling double DQN [168].

Actor–critic methods [63,105] are also used for option selection. Many variations are used, including advantage actor–critic (A2C) [20,79,122], soft actor–critic (SAC) [100,103] and asynchronous advantage actor–critic (A3C) [173]. Proximal policy optimization algorithms (PPO) [104,132,197,277,285] and deep deterministic policy gradient (DDPG) [68,227,316,317] are also very common.

6.3. The Limitations of This Review

Regarding what is included in this review, there are multiple noteworthy considerations. While the review was aimed at industrial applications with RL, by also restricting to combinatorial problems, some important industrial sectors can be under-represented, such as robotics and control applications. Similarly, by requiring the term industry or industrial, some services might be less represented, such as transportation problems. Excluding papers without sufficient, reproducible details regarding all agent–environment interactions might also discard more theoretical papers with valuable insights. Finally, our search was only conducted in English-language databases, possibly overlooking other high-quality papers.

Regarding the review process used, despite our best efforts to manually check more than 700 papers, this labor-intensive undertaking is subject to human error and bias. Also, there are diverse definitions of states, actions and rewards, and their categorization is inherently subjective. Differences in terminology and framework definition can also influence the classifications. It is also inevitable that for some agent–environment interactions their categorization is oversimplified, ignoring some category overlaps. Also, by not considering papers without practical implementation details, some valuable theoretical contributions might be missed.

Considering the results presented here, key categories and examples of actions, states and rewards are provided for future practitioners for RL-based industrial applications. These findings can be used to guide future research towards the most popular approaches, since their popularity proves their efficacy. At the same time, the identified less-explored approaches can lead to untapped opportunities.

Note that due to the heavy manual data processing conducted in this review, no assessment or reporting of risk of biases, study heterogeneity, robustness or confidence in the results was performed. No statistical synthesis and no assessment of certainty to the body of evidence were conducted. The present review was not registered, and no review protocol was prepared.

7. Conclusions

This review explores the RL state of the art for the combinatorial optimization of industrial problems on the following three databases: IEEE Xplore, Scopus and WOS. The main focus of the analysis was the agent–environment interactions, namely, the state representation, the action mapping and the reward design. Also, the current limitations and needed future developments are explored. A thorough categorization of each approach regarding all three agent–environment interactions is proposed. This review analyzes 298 studies out of the retrieved 715. The most common example is resource scheduling, either for manufacturing or wireless network problems. This includes partial resource allocation to different tasks, such as in network resource sharing.

The analysis of the literature shows that many studies are not always clear on practical implementation details, which hurts both readability and reproducibility. As an extreme example, some studies do not clearly state the algorithm used. As research in this area grows, it is important to make papers informative and unambiguous. In general, authors tend to focus on two main topics for future work: adding more constraints to the problem and using more advanced models and algorithms.

The desired extra constraints intend to approximate the formulation to the real problem. Two common examples are considering due dates and making the resource availability dynamic. However, these increases in complexity must be well managed since heavily constrained problems translate into agents harder to train.

Many approaches want to complement state-of-the-art RL methods with complex ML models. RNNs, CNNs and GNNs are all increasing in popularity. While the results are promising, the computational effort to train these models increases greatly for small performance gains. It is important to keep in mind that simpler RL approaches can also provide good results, as long as the agent–environment interactions are well designed.

Since most studies reflect a future desire to increase the complexity of either the problem or the approach, the authors suggest that future research should focus on problem representation and training frameworks that can handle this inevitable scaling in complexity.

RL still struggles with sample efficiency, interpretability and generalization of certain encodings. However, by combining RL with established optimization techniques, the combinatorial optimization community can develop efficient, scalable and adaptive solvers that leverage the strengths of both areas.