Artificial Intelligence to Solve Production Scheduling Problems in Real Industrial Settings: Systematic Literature Review

Abstract

This literature review examines the increasing use of artificial intelligence (AI) in manufacturing systems, in line with the principles of Industry 4.0 and the growth of smart factories. AI is essential for managing the complexities in modern manufacturing, including machine failures, variable orders, and unpredictable work arrivals. This study, conducted using Scopus and Web of Science databases and bibliometric tools, has two main objectives. First, it identifies trends in AI-based scheduling solutions and the most common AI techniques. Second, it assesses the real impact of AI on production scheduling in real industrial settings. This study shows that particle swarm optimization, neural networks, and reinforcement learning are the most widely used techniques to solve scheduling problems. AI solutions have reduced production costs, increased energy efficiency, and improved scheduling in practical applications. AI is increasingly critical in addressing the evolving challenges in contemporary manufacturing environments.

1. Introduction

The adoption of Industry 4.0 principles, the advancement of smarter factories, and the integration of intelligent sensors and interconnectivity across various organizational components have contributed to the expanding volume of literature regarding the application of artificial intelligence (AI) in manufacturing systems. This field of study has seen rapid growth in recent years. Contemporary manufacturing settings are affected by numerous factors that impact the production process [1], including machine failures [2,3], order fluctuations, and unpredictable job arrivals. To be competitive in the actual context it is important to be flexible and to be able to respond faster to variations in production planning [4]. Currently, production procedures are dynamically changed to actively satisfy consumer wants and create a wide range of products. To this purpose, the manufacturing ecosystem of today is distinguished by a reduced product life cycle, a high level of product variability, and an escalating level of international competition [5]. AI is an important instrument in the context of manufacturing systems, in order to respond quickly and predict future anomalies in the production plan; the AI instrument can be used as support for the decision-making process. In the literature, there are a lot of contributions about the use of AI instruments to realize dynamic scheduling [6,7] algorithms, or articles dealing with scheduling issues in the context of Industry 4.0.

The development of a dynamic scheduling program based on AI is the major objective of the European AIDEAS project’s “Fabrication Optimizer” tool, which was born in this context. Consequently, examining how other authors have addressed similar challenges was essential.

In [8], a systematic literature review on new job scheduling methods was carried out. In that analysis, the focus was on the methodology used to solve scheduling problems in the context of Industry 4.0, but without focusing on the AI techniques used and what they are used for. Furthermore, an overview of the use of machine learning (ML) algorithms to solve scheduling problems was presented in [9]. Here, the authors focused only on the use of ML techniques (supervised, unsupervised, and reinforcement learning). Excluded from the study are all the AI techniques that are not classified as ML (e.g., evolutionary algorithms, expert systems, etc.), and the impact these applications have on a real case study is not highlighted.

The study by [10] offers a systematic examination of the academic literature, aiming to establish a connection between Industry 4.0, scheduling, digital twin, and zero-defect manufacturing concepts.

The objective of this review is to focus more on the use of AI for solving industrial scheduling problems and, especially, to understand the benefits it generates for companies. The purpose of this document is twofold:

-

To understand what the trends are in solving scheduling problems through the use of AI and what AI techniques are most widely used in the literature;

-

To analyze how other authors solve production scheduling problems in real industrial settings and see what advantages have been achieved for the companies where the solutions have been implemented.

Thus, a systematic literature review was conducted using the Scopus and Web of Science (WoS) databases and bibliometric tools, such as VOSviewer [11].

The paper we propose is an extended version of the literature review presented at the XXVIII Summer School ‘Francesco Turco’.

The scheduling problem is a classic NP-hard problem [12] and is also one of the key links for the efficient operation of an intelligent production system because dynamic scheduling can optimize several KPIs in the production space; for example, it can reduce the tardiness [13], the cost of storage [14], minimize the makespan [15], reduce the travelling time [16], and benefit other KPIs that change from company to company. Intelligent production has numerous advantages in terms of flexibility, maintainability, and cost. AI is not only used for dynamic scheduling, but is used in production plans to help the decision-making process. However, it is important to emphasize that production scheduling problems are classified into several subsets. The main scheduling problems are:

-

The single machine scheduling problem (SMSP) [17]. The SMSP concerns the allocation of a set of tasks using a single machine in order to optimize an objective function.

-

The flow-shop scheduling problem (FSSP) [18]. In an FSSP, there are a set of tasks that must be scheduled using a set of machines. In this type of problem, the items to be produced must follow a precise sequence of tasks, so each task will have a precedence constraint concerning other tasks. All the items to be scheduled must follow the same manufacturing sequence, so the flow of material and information in this type of problem is unidirectional.

-

The job-shop scheduling problem (JSSP) [19]. A JSSP is similar to an FSSP, in that there will be a set of items that will have to be processed on a set of machines. However, unlike an FSSP, here the items do not necessarily have the same manufacturing sequence, so the flow of materials will be multi-directional.

-

The open-shop scheduling problem (OSSP) [20]. Also, in an OSSP, there will be a set of elements that must be processed on a set of machines, but in this case, there are no precedence constraints between the activities to be performed.

-

The parallel machine scheduling problem (PMSP) [21]. A PMSP involves scheduling a set of jobs to be processed on multiple machines simultaneously, or in parallel. The primary objective is to determine how to allocate the jobs to the machines and in what order. If all the machines have the same processing speed and capabilities it is called an identical PMSP; if the machines are grouped into classes, and the machines within the same class have the same processing speed, it is called a uniform PMSP. Meanwhile, if each machine has a unique processing speed it is called an unrelated PMSP.

These are, in short, the main scheduling problems; we will discuss a flexible JSSP (FSSP) [22] or flexible FSSP (FFSSP) [23], when the scheduling problem combines one of the aforementioned problems with a PMSP.

The dynamic nature of manufacturing systems implies the necessary adoption of a dynamic scheduling paradigm to deal with unforeseen events that disrupt the execution of a schedule, as the assigned apparitions can be immediately redirected to other machines. According to Elbasheer et al. [24], there are three major manifestations of dynamic scheduling in the AI literature: task re-scheduling concerns the reprogramming of a specific activity within the production process as a reaction to an interruption in the original program; resource allocation, especially in flexible shop floors, where the use of AI should improve the ability to allocate resources to deal with plan disruptions and line balancing after any interruptions in the production process.

In this paper, Section 2 explains how the research was conducted and which tools were used to study the publication trends, and which AI techniques are most prevalent in the literature. In Section 3, relevant contributions found in the literature are reported. Section 4 provides a discussion. Section 5 reports on future developments and includes the conclusion.

2. Methods and Data

The authors conduct a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) statement. In Appendix A (Figure A1) is present the PRISMA flow chart.

The literature review was conducted in order to respond to the two main areas of focus in this study: to analyze the trends in the use of AI to solve scheduling problems and to understand what techniques are the most widely used and to study how authors have solved problems in industrial settings and what benefits they have brought to the companies.

Figure 1 shows the steps taken to carry out the literature review. In the following sections, all these steps are explained in detail.

2.1. Preliminary Research

The literature search is set using Scopus and the WoS as databases. Web of Science and Scopus, renowned for their broad coverage, publication quality, advanced search tools, and citation search capabilities, enable a comprehensive literature review on scheduling and AI in production, providing access to relevant and influential sources for quality research.

A preliminary search was conducted for the keywords “Scheduling” AND “Artificial Intelligence” in titles, abstracts, and keywords in Scopus and in all the fields for the WoS. AI scheduling is mainly applicable in the context of a smart factory or Industry 4.0. In order to realize accurate and dynamic algorithms, it is necessary to have large amounts of data, and to connect information from different areas, etc. For this reason, the research was conducted by considering publications from 2011, the year of the advent of Industry 4.0, and October 2023, the period in which the analysis was conducted. A total of 5481 papers were found in Scopus and 4355 in the WoS, with 1545 papers in common.

Scheduling problems managed using AI is an attractive topic for scientific debate in various fields, with a high number of publications on the matter, as shown in Figure 2.

However, this research focuses on the application of AI for solving industrial scheduling problems. From this assumption, filters were applied to exclude out-of-context topic areas. Specifically on Scopus, only publications in the area of engineering (2095 papers) were considered. On the WoS, publications on the areas of multidisciplinary engineering, manufacturing engineering, mechanical engineering, and industrial engineering were considered, with a total of 533 publications. Then, bibliometric analysis was conducted on these articles.

2.2. Bibliometric Analysis

One of the research questions was to identify which AI techniques are most commonly used to solve scheduling problems and for what purpose. To conduct a specific literature review, it is important to know the specific keywords to search for in order to quickly identify useful papers in the Scopus database. For this purpose, VOSviewer 1.6.20, a bibliometric software capable of finding the most frequent occurrences in a large database of publications, was used.

Thus, it has been possible to find common correlations among several keywords. In particular, only keywords capable of leading to AI techniques were selected. A minimum threshold on keyword occurrences of 10 has been set to exclude less frequent keywords.

In Figure 3, it is possible to see the main AI techniques used to solve scheduling problems like, for example, neural networks (NNs), deep neural networks (DNNs), reinforcement learning (RL), swarm intelligence (SI), particle swarm optimization (PSO), ant colony optimization (ACO), decision trees (DTs), and support vector machines (SVMs).

2.3. Specific Search

From the results of the bibliometric analysis, a more specific search on Scopus and the WoS was conducted, searching scheduling and AI techniques as keywords in order to see the number of publications.

Table 1. Keywords searched in Scopus and WoS, with the relative number of publications.

	No. of Publications
Keywords	Scopus	WoS
“Scheduling” AND “Particle Swarm Optimization”	2843	788
“Scheduling” AND “Neural Network”	2372	296
“Scheduling” AND “Reinforcement Learning”	2095	294
“Scheduling” AND “Ant Colony Optimization”	960	274
“Scheduling” AND “Decision trees”	323	13
“Scheduling” AND “Support Vector machine”	278	32
“Scheduling” AND “Swarm Intelligence”	263	51

shows the number of publications for each couple of keywords searched, in order to see what AI techniques are the most widespread. This research shows that the largest contributions in the literature concern the use of PSO, NNs, and RL for solving scheduling problems.

The next stage in the research is to analyze how the authors solve production scheduling problems in real industrial settings through the use of PSO, NN, and/or RL algorithms. For this reason, specific searches on the Scopus and WoS databases were conducted, searching for “Scheduling” AND (“Particle Swarm Optimization” OR “Neural Network” OR “Reinforcement learning”) AND “case study”.

This choice was made because this analysis aims to analyze what benefits and advantages companies have obtained from using such algorithms; thus, excluding all the articles that illustrate an algorithm without real case applications.

Only articles and conference papers written in English and published from 2011 to 2023, in the field of engineering on Scopus, and in the fields of multidisciplinary engineering, manufacturing engineering, mechanical engineering, and industrial engineering on the WoS, were considered. As shown in Figure 4, a total of 425 publications were found from reading the titles and abstracts in Scopus and 89 papers in the WoS, with 62 publications in common. The publications that do not concern the scheduling of production orders within an industrial context (e.g., energy storage and distribution, urban transport planning etc.) were excluded. The publications that illustrate the algorithm and test it on a simulation plant without reporting the benefits obtained from the application of the AI techniques are also excluded [25]. A total of 31 papers were found that applied NNs, PSO, or RL to solve scheduling problems in industrial case studies.

3. Literature Review of Relevant Papers

This section reports on the contributions by other authors to solve scheduling problems within production sites using PSO, NNs, and/or RL. This research focuses on understanding how the authors used these techniques and what benefits they brought to the companies.

3.1. Particle Swarm Optimization

PSO is an optimization technique inspired by the social behavior of birds and fish. In the context of production scheduling problems, PSO algorithms mimic the collective intelligence and cooperation observed in these natural systems to find optimal solutions. PSO offers a dynamic approach to tackling complex scheduling challenges, with the aim of improving efficiency and minimizing production costs.

In PSO, a population of particles explores the solution space iteratively, adjusting their positions and velocities based on their own and their neighbors’ experiences. This cooperative search mechanism allows the PSO to efficiently navigate the vast solution space of scheduling problems.

Table 2. Relevant papers on PSO utilization to solve different scheduling problems.

References	AI Techniques	Type of Problem
		SMSP	FSSP	JSSP	PMSP	FFSSP	FJSSP
[26,27]	PSO + Genetic Algorithm						x
[28]	PSO + Variable Neighborhood Search				x
[29]	PSO + Variable Neighborhood Search					x
[30]	PSO + Artificial Immune System			x
[31,32,33]	PSO		x
[34]	PSO					x
[35]	PSO and ε-constraint method						x
[36]	PSO	x
[37]	PSO + GA + K-means						x
[38]	PSO			x

shows contributions illustrating the application of the PSO algorithm to solve scheduling problems in real industrial settings.

It can be seen that PSO has a high degree of flexibility in solving different scheduling problems, even when applied to different contexts (JSSP, FSSP, etc.). Hecker et al. [31] designed two algorithms to solve a scheduling problem in the bakery industry: the first one uses PSO, while the second uses the ACO algorithm. A comparison of the two algorithms was conducted and showed that the PSO was faster (39 s when optimizing the makespan and 15 s when optimizing the total machine idle time within the average calculation time) than the ACO and returned better results in both optimization problems. A different approach to solving an FSSP was used by Huang et al. [32]. In that case, a re-entrant two-stage FSSP, where all the jobs must occur twice in the sequence of the production process, was solved using a farness PSO (FPSO). An FPSO differs from a traditional PSO because the swarm behavior learns from experience and improves the solution through the self-owned and distant population. The method was tested in a real case and the results were compared with the results from a traditional PSO and ACO. The FPSO outperformed the PSO and ACO in both approaches, providing an average improvement in effectiveness of 39.47% and 42.99% compared to the PSO and ACO, for small-scale problems.

Instead, Ramezanian et al. [34] applied PSO to solve a classic four-stage FFSSP. In that case, the algorithm solved the lot size and scheduling problems in the tile industry, where the objective function was to find the minimum cost of production, inventory, and external acquisition (a multi-objective function). The proposed algorithm provided a scheduling program and lot size in 479 s; an acceptable time for the company to solve a large optimization problem.

A bi-objective PSO algorithm to solve an SMSP that minimizes energy consumption and total tardiness was presented in [36]. The algorithm was applied on a CNC machine and returned multiple solutions with different values on energy consumption or tardiness that supported the process planner’s choices. The high performance and flexibility of the PSO was also demonstrated by Zhang H, et al. [38]. They proposed an improved multi-phase PSO algorithm to solve a dynamic JSSP and the solution was tested in a real industrial setting. The proposed method was more efficient in terms of the processing time than the other algorithms tested, especially for large problems. A multi-objective production schedule and maintenance plan that considered the energy cost, machine production efficiency, and production target, were developed by Sun et al. [33]. The PSO model presented was tested by one company and involved the implementation of joint energy and maintenance management. The implementation generated a reduction in the production costs compared to the previously used approach.

Table 2 highlights another important aspect to consider concerning PSO. Due to its high flexibility, even in the programming phase, PSO allows easy collaboration with other AI techniques or optimization algorithms in order to create hybrid algorithms that increase the overall performance of the solution.

To solve a particular scheduling problem, Wang et al. [26] develop a two-stage optimization method to improve the energy efficiency of an FJSSP. The first phase involved the use of a GA to optimize the selection of the machine tools for the production process. The second phase combined PSO with GA to improve the sequence of the operations. In this combined approach, the GA helps to improve the global exploration capability to avoid early convergence problems in the PSO. The proposed algorithm was evaluated in a practical case study, achieving an 8.5% reduction in production costs and a 10.2% reduction in energy consumption compared to the scheduling programs previously employed by the company tested.

A different hybrid approach was developed by Chen et al. [28], who realized an algorithm combining variable neighborhood search (VNS) and PSO to solve a PMSP in the solar cell industry. In the proposed case study, VNS is used to decide in which order the tasks are to be performed and PSO is used to decide the assignment of the machines for all the production orders. The proposed solution performed better than the traditional PSO and the heuristic algorithm used by the company under investigation and achieved the solution of the scheduling problem in 43.16 s, faster than the other two solutions. While in [29], a combination of PSO and VNS is used to solve an FFSSP. In this case, VNS is used to search for an optimal solution in order to reduce the computation time taken by the hybrid algorithm. A multi-objective problem was proposed in which the makespan and total flow time must be optimized in a case study involving an automotive manufacturing unit. Instead, Du et al. [30] propose a combination of PSO with an artificial immune system to solve an assembly JSSP, to minimize the completion time. The algorithm was tested in a real case study and was able to find an optimal solution to the problem in only 106 s, faster than the previous approach used by the company. A different use of PSO is provided by Mohammadi et al. [35], who propose a combination of PSO and the ε-constraint method, a multi-objective decision-making method. It considered a ‘make-to-order’ production system, responsible for the production and transportation of customer orders, and the described problem is a combination of an FJSSP and a vehicle routing problem. The proposed scheduling algorithm is a bi-objective mixed-integer model that can find a solution that minimizes the production and transport costs and the weighted sum of the delivery earliness and tardiness. Li et al. [27] also developed a bi-objective algorithm to solve an FJSP with uncertain processing times. They realized a combination of GA and a binary PSO in order to minimize the makespan and the value of deviation from the expected makespan. The proposed method was tested in nine case studies and performed better in terms of robustness than the stochastic method and a conventional method, such as a hybrid GA. A different use of PSO was provided by Yin L, et al. [37]; they realized a combination of PSO and a K-means algorithm to predict machine health. In addition, a GA was applied to generate a scheduling solution based on the machine’s health. They realized a dynamic method to solve the FJSSP in a metalworking workshop, where the health of the machine was evaluated according to the quality of the parts and the results of the evaluation were applied for dynamic scheduling. The proposed method allowed the impact of machine failures on the production process to be reduced and preventive maintenance to be carried out in advance.

Researchers and practitioners have successfully applied PSO to a variety of scheduling problems, including job sequencing, resource allocation, and execution time minimization. By taking advantage of PSO’s adaptability and flexibility, manufacturing companies can achieve greater scheduling accuracy and better operational performance, reducing costs and increasing productivity. PSO’s ability to handle both single-objective and multi-objective scheduling problems makes it a valuable tool for the manufacturing industry.

3.2. Neural Networks

NNs used in production scheduling exploit artificial NNs, a subset of machine learning, to improve the planning and optimization of production processes. These networks are trained using historical data and are designed to predict and optimize various elements of scheduling, such as resource allocation, job sequencing, and production timing. A NN demonstrates its adaptability in dealing with complex and dynamic production environments, increasing the accuracy and efficiency of scheduling.

Table 3. Relevant papers on NN utilization to solve different scheduling problems.

References	AI Techniques	Type of Problem
		SMSP	FSSP	JSSP	PMSP	FFSSP	FJSSP
[39,40]	NN	x
[41]	NN + MARL				x
[42]	NN		x
[43]	NN						x
[44]	NN + other techniques	x
[45]	NN + other techniques						x

shows publications on the realization of NN algorithms to solve or improve a scheduling problem in a real industrial setting.

Unlike the articles analyzed on PSO, NNs are not only used for sequencing problems, but they are used to find correlations or hard-to-find features, and use this information to create a more accurate scheduling plan. One case is that of Wang et al. [39], who realized an artificial NN algorithm to track the energy consumption of CNC machines.

The proposed algorithm is combined with a multi-objective optimization model for the production re-scheduling process, which minimizes energy consumption, makespan, and balances machine utilization levels. The proposed algorithm was validated on several industrial trials and achieved a 30% improvement in energy consumption and a 50% improvement in productivity. A similar case, but with different objectives, can be found in [40]. Here, ANNs are used to schedule the workload of a CNC-milling machine by keeping the tool load constant. This strategy resulted in a reduction of almost 50% in the machining time.

A different use is present in [44]. In that case, an ANN to optimize the milling process parameters (energy consumption and surface roughness) for producing one single part was designed. ANNs are employed to model intricate non-linear connections between essential process variables and the recorded data on both energy usage and surface quality. Based on the optimized parameters, several intelligent methods, such as pattern search, GA, and simulated annealing are applied to find optimal sequencing, set up, and scheduling for multiple machines. In the case study, the simulated annealing algorithm was used in two forms. The first model aimed to optimize energy consumption and makespan, while the second only optimized energy consumption. With the second approach, there was a reduction in energy consumption of 2795 kJ, but an increase in the makespan of about 23 min compared to the first model. Azab et al. [42] developed a framework that combines commercial software tools for scheduling with a machine learning approach to predict machine failure in scheduling programs. The proposed approach was tested in a pharmaceutical company and different AI techniques were tested; the results show that the best performance was achieved due to the use of the decision forest algorithm, but the NN algorithm achieved better results in predicting the machine failure time.

Another interesting approach to a scheduling problem was realized by Zhou et al. [41]. In this case, there is a smart factory with four equal workstations; every single station has its schedule program that runs on a distributed computer and the scheduling is realized using a metaheuristic method and the computer has its own NN algorithm that learns from the workstations. The learned knowledge is shared with a centralized computer system, where there is a scheduling system based on multi-agent RL (MARL) logic (DQN method) that learns from the four workstations and shares this knowledge with the four workstations. The proposed solution reduced the lead time by 11.9% compared with the use of the DQN algorithm only.

A model that uses artificial NNs to schedule the workforce of a company was designed by Simeunović et al. [43]. The proposed algorithm aimed to predict the number of employees for the following days based on various factors, such as customer requests and the number of working hours, etc. Thanks to this contribution, the waiting time of the company’s employees was reduced to 2.3 min, leading to an increase in the company’s productivity and a higher degree of customer satisfaction.

Stanković et. al. [45] propose a novel methodology to solve an FJSSP. Three algorithms for solving the scheduling problem were implemented, the first using GA, the second using PSO, and the third using the artificial bee colony (ABC). They propose a combination of ANNs and fuzzy logic that selects the most appropriate algorithm to solve the scheduling problem according to the results in terms of the calculation time and makespan. The methodology was applied to a footwear company case in which, according to the conditions of the problem under consideration, the framework recommended the use of a GA to solve the problem.

3.3. Reinforcement Learning

In the field of production scheduling, RL involves the use of AI algorithms to make the best possible choices in regard to production operations. RL agents acquire knowledge through practical experience (a try and error approach) and engagement in the production environment, with the aim of optimizing efficiency, reducing expenses, and improving scheduling results. This methodology offers versatility and adaptability in dealing with the intricate and ever-changing scheduling dilemmas encountered in manufacturing. RL techniques are used in production planning and control, but are mainly used to solve production scheduling problems [46,47].

Table 4. Relevant papers on RL utilization to solve different scheduling problems.

References	AI Techniques	Type of Problem
		SMSP	FSSP	JSSP	PMSP	FFSSP	FJSSP
[48]	MARL–QMIX					x
[49]	MARL–DQN	x
[50]	RL–Q-learning		x
[51]	RL–AC algorithm			x
[52]	RL–Q-learning				x
[53]	RL–Q-learning + CTPNs			x
[54]	RL–Q-learning			x
[55]	MARL–Deep RL					x
[56]	MARL–Deep RL						x
[57]	MARL–SARSA					x
[58]	RL–DQN	x

highlights the high level of flexibility of RL in solving different production scheduling problems with different approaches. The second column shows the different RL algorithms and methodologies found, such as multi-agent RL (MARL) or single-agent RL (SARL).

Wang X, et al. [48] propose a MARL approach to solve an FFSSP, with the aim of minimizing the makespan value. The problem involved assigning workloads to 18 robot stations, in parallel with different processing times. A QMIX algorithm was used to learn in the environment and the proposed algorithm outperformed the other classic heuristic approaches in terms of the computational time, as well as a distributed agent scheduling architecture (DASA), another RL method. These two approaches differ, because in MARL the reward function is shared among all the agents, while in DASA each agent aims to maximize its own reward function. The effectiveness of MARL is also confirmed by the study in paper [49], which used several SARL algorithms for solving a scheduling problem in a human–robot context, in the case of an SMSP. Again, MARL (here, however, a DQN is used as an algorithm) outperforms other RL algorithms in terms of the calculation time, training speed, and goodness of the solution. A DQN algorithm was used in [58] to optimize the planning in a robotic disassembly sequence for a workstation. The authors tested the solution in a real industrial case, and the algorithm turned out to provide a lower disassembly time than other algorithms, such as a GA, an improved discrete bee algorithm, and a dueling DQN.

Liu Y, et al. [55] propose a different approach, in which two multi-agent Deep RL (DRL) algorithms are implemented. A specialized DRL model is proposed to handle sub-decisions associated with tasks, such as the sequencing of work and the selection of machines. Simultaneously, an attention-based network is proposed to handle the sub-decisions related to worker assignment. This approach optimizes the extraction of relevant features and facilitates more efficient decision-making processes. The work uses DRL for intelligent data-driven decision making and integrates DRL with multi-agent systems to solve the dynamic FFSSP. The presented solution was tested within a pharmaceutical facility, with three distinct processing steps. In this evaluation, the method demonstrated its superiority to the established dispatching rules and pre-existing DRL techniques in terms of the mean and variance of the total tardiness obtained. The importance and the flexibility of the DRL-based multi-agent method are also highlighted by Huang J, et al. [56]. They solve a distributed JSSP (DJSSP) for an automotive engine manufacturing company. The DJSSP involves two coupled decision-making processes, the assignment of the job to one of a set of factories and the sequencing of the jobs. In this case, the proposed solution outperformed the other approaches like PSO and, for the case study, the results were provided in 0.35598 s.

A different approach was used by Vijayan S, et al. [50]. They tested an RL method exploiting a Q-learning algorithm to solve an FSSP. The first instance involved the case of a plastic toy factory, where the algorithm was compared with other metaheuristic approaches and the results were better, in particular, there was a decrease in the computation time of up to 18%, which even outperformed the PSO. The second instance involved stator core manufacturing; here too, there were low computation times and improvements in the makespan. In comparison, Ghaleb M, et al. [52] used the Q-learning algorithm to solve a scheduling problem involving three parallel machines (PMSP). These machines were subject to planned and unplanned outages that have a major impact on the scheduling plan. The solution proposed by the authors is an example of a multi-objective scheduling problem; the company wants to maximize the throughput, minimize the mean cycle time, and minimize the number of tardy orders. The agent follows a set of rules, which include obtaining the current status of the production unit, computing the reward from the previous action, choosing the subsequent action, transmitting the newly chosen action to the shop floor, and revising the state–action table with the recently acquired system status. The solution by the proposed approach outperformed the previous EDD rule scheduling method used by the company in terms of the total weighted tardiness, throughput, and mean cycle time. Also, Said N, et al. [54] introduced an RL model that utilizes a Q-learning algorithm. The problem presented is a dynamic JSSP in a real-world scenario involving a pharmaceutical factory, with 18 machines and 22 different products. The algorithm suggested in the study demonstrates its ability to attain efficient scheduling within a brief production cycle, requiring minimal time and without relying on prior scheduling knowledge. This leads to an enhancement in the overall productivity of the factory. The proposed approach reduced the makespan value by 20–40% (depending on the size of the problem) with respect to an FIFO strategy. The efficiency of MARL systems is demonstrated also in [57], where a scheduling algorithm was implemented to create an efficient and flexible scheduling plan in a multisite enterprise. In the proposed case study, the SARSA (state–action–reward–state–action) algorithm was implemented to train the MARL model. The model was tested on a real case (FFSSP) and the results, compared with a GA model and a mixed-integer linear program, provided improvements in terms of the calculation time.

A different type of problem and a different algorithm were presented by Elsayed, E.K, et al. [51], who adopted the actor-critic (AC) network’s training algorithm-based RL for achieving the optimal policy for the JSSP. The algorithm was tested in a real case, where scheduling was previously conducted following FIFO logic; the proposed algorithm achieved better results in terms of the makespan, by going from 97 UT to 60 UT.

Drakaki M, et al. [53] present a combination of Timed Colored Petri Nets (CTPNs) and RL to solve a scheduling problem in a manufacturing plant. The authors propose a CTPNs model to solve the scheduling problem and a Q-learning RL algorithm is used as a guide to improve the solution and reduce the computational time for large-scale problems. The method was tested in a case study to solve a warehouse order-picking schedule and also applied to known JSSP benchmark examples and compared with other approaches in order to validate the solution.

From the articles studied, RL is a technique for solving scheduling problems in real-world contexts using different strategies (e.g., MARL, SARL), but also using different learning algorithms. This great flexibility, combined with its high level of performance, especially in terms of the calculation time, makes RL a technique that is also highly appreciated in an industrial context.

4. Discussion

This section will discuss the results of this research. Using the Scopus and WoS databases and bibliometric analysis, it was possible to answer the first research question: to identify the trends and major AI techniques for solving scheduling problems. The use of AI to solve scheduling problems has been a growing topic over the years, attracting the attention of researchers worldwide. PSO, NNs, and RL are the most widely used approaches in the literature, so the focus was on analyzing how these techniques are applied in real industrial settings and assessing the benefits brought to companies. One aspect noted during the analysis is that although the number of publications on the use of AI associated with scheduling is high, the number of publications reporting a real-life application case in a manufacturing industry is not very high. This may depend on various aspects, such as the difficulty for companies to access certain data, or the inability of the authors to publish confidential information.

Table 5. Benefits from the analyzed papers.

Benefits for the Company	PSO	NN	RL
Production costs	[26,33,34]	-	-
Production efficiency	[29,31]	[39,40,43]	[49,52,53,58]
Calculation time	[27,28,30,32,38]	[45]	[48,49,50,53,56,57]
Makespan	[29,30,35]	[41,44,45]	[50,51,52,54]
Tardiness	[34]	-	[52,55]
Transport cost	[35]	-	-
Energy consumption	[26,36]	[39,44]	-
Machine failure	[37]	[42]	-
Employee waiting time	-	[43]	-

brings together the 31 contributions analyzed in the previous section and illustrates the business benefits found.

The previous section pointed out that PSO, NNs, and RL are techniques used to solve different types of scheduling problems (FSSP, JSSP, SMSP, etc.) in different manufacturing sectors. Industrial problems are often very complex and require the use of multi-objective algorithms. PSO, NNs, and RL have made possible the realization of algorithms that optimize more than one criterion (makespan, EDD, production cost, etc.). This is very important in manufacturing realities because it allows for production that takes several aspects into account. All three techniques also proved to be suitable for the realization of single-objective and multi-objective algorithms. As far as business benefits are concerned, here too they differ. As with any AI or data-driven solution, the results will depend very much on the quality and availability of data on the part of companies. However, in the documents analyzed, there is a great diversity in the benefits, such as a reduction in the makespan or delays, which are often critical for companies. As mentioned previously, an important aspect that emerged from the analysis concerns the small number of publications, compared to the large number of papers found, that explicitly report on the company benefits. Certainly, this aspect is strongly influenced by the difficulty in accessing and sharing company data, but it is also linked to the fact that some results obtainable from the application of a new scheduling plan cannot be seen in the short term. One aspect that can be analyzed immediately is certainly the reduction in the calculation time, which in fact is a parameter reported in 36% of the cases analyzed. Reducing the calculation time is also an important aspect in an industrial context, because it allows for greater flexibility and the possibility of being able to schedule several times a day and, thus, be able to respond to any abnormal events. Having the possibility to modify the scheduling plan in real or near real-time is a difficult challenge, often in complex contexts, with different production constraints. The more complex the problem, the more critical the calculation time will be; in fact, from the RL articles analyzed, it was seen that MARL solutions provide better execution times and flexibility than single-agent RL solutions and other algorithms like PSO, GA, etc. Furthermore, with RL, the authors were able to obtain scheduling algorithms that provide solutions in real or near real-time. This is of enormous benefit to companies because it allows them to always have an up-to-date scheduling plan based on the working conditions for a given time point. In this regard, it can be noted that articles using RL have a very recent publication date (81.8% of the analyzed articles were published from 2021 onwards). This might suggest that research is moving towards the use of RL-based solutions that guarantee excellent results in terms of quality and also in terms of calculation time. Another aspect that emerged from the study concerns the use of NNs, which unlike RL and PSO, are not used only for sequencing problems. In fact, in the papers analyzed, NNs are mainly used to find difficult correlations or features that are then used to improve the scheduling program and make it more reliable. This aspect is important in unstable production environments, because having a more accurate scheduling plan that takes into account the various production disruptors makes it possible to better balance the workload and meet the scheduling dates.

5. Conclusions

The following paper was written with the aim of clarifying two objectives:

-

To understand what the trends are in solving scheduling problems through the use of AI and which AI techniques are most widely used in the literature;

-

To analyze how other authors solve production scheduling problems in real cases and to see what advantages they have achieved.

Thanks to bibliometric analysis and the Scopus and WoS databases, it was possible to answer the first question and see that the trend of using AI to solve scheduling problems in engineering is growing year after year, with the use of PSO, NNs, and/or RL being the most widely used approaches in the literature. From this point, a more specific literature review was conducted to see how the authors solve production scheduling problems in real industrial settings through the use of PSO, NNs, and RL. The three AI techniques present different contributions regarding which algorithms are used to solve different types of scheduling problems, classic NP-hard problems, like single-objective or multi-objective problems in several scenarios like job-shop and flow-shop problems. This study showed how, using AI, the companies concerned obtained benefits that can be of different types depending on the internal problems.

Future steps will concern the realization of an algorithm for the optimization of production scheduling for the pilots by the European AIDEAS project, in order to enrich the contribution to the literature.