Low-Carbon Flexible Job Shop Scheduling Problem Based on Deep Reinforcement Learning

Abstract

As the focus on environmental sustainability sharpens, the significance of low-carbon manufacturing and energy conservation continues to rise. While traditional flexible job shop scheduling strategies are primarily concerned with minimizing completion times, they often overlook the energy consumption of machines. To address this gap, this paper introduces a novel solution utilizing deep reinforcement learning. The study begins by defining the Low-carbon Flexible Job Shop Scheduling problem (LC-FJSP) and constructing a disjunctive graph model. A sophisticated representation, based on the Markov Decision Process (MDP), incorporates a low-carbon graph attention network featuring multi-head attention modules and graph pooling techniques, aimed at boosting the model’s generalization capabilities. Additionally, Bayesian optimization is employed to enhance the solution refinement process, and the method is benchmarked against conventional models. The empirical results indicate that our algorithm markedly enhances scheduling efficiency by 5% to 12% and reduces carbon emissions by 3% to 8%. This work not only contributes new insights and methods to the realm of low-carbon manufacturing and green production but also underscores its considerable theoretical and practical implications.

1. Introduction

With carbon dioxide concentrations in the Earth’s atmosphere reaching the highest levels in at least 2 million years [1], the imperative to peak carbon emissions and achieve carbon neutrality [2] underscores the critical importance of advancing low-carbon manufacturing, green production, and carbon emission reduction efforts. Leveraging deep reinforcement learning algorithms for production scheduling optimization holds promise in significantly reducing resource consumption and environmental pollution associated with manufacturing processes.

The Job shop Scheduling Problem (JSP) stands as one of the most prevalent scheduling challenges. Johnson’s seminal work in 1954 [3] on scheduling for two machines laid the foundation for various scheduling problems, including JSP. Central to JSP is the allocation of each job operation to designated machines and the scheduling of processing sequences and start times under specific constraints to optimize performance metrics. The Flexible Job shop Scheduling Problem (FJSP), an extension of traditional JSP, was introduced by Brucker in 1990 [4]. As a complex NP-Hard problem [5], FJSP not only entails operation sequencing on machines but also necessitates machine assignment for each operation prior to sequencing, aligning more closely with real-world production scheduling environments.

In recent years, there has been a growing focus on environmental considerations in production scheduling. Jiang [6] and others proposed a multi-objective flexible job shop scheduling model incorporating energy consumption, maximum completion time, processing cost, and cost-weighted processing quality, alongside an enhanced non-dominated sorting genetic algorithm for solution. Zhang [7] and team established a low-carbon scheduling model for FJSP, considering factors like maximum completion time, machine load, and carbon emissions, and proposed a mixed NSGA-II algorithm for resolution. Jiang [8] developed a low-carbon flexible job shop scheduling model aiming to minimize the weighted sum of energy consumption costs and completion time costs, along with a tailored grey wolf algorithm for solutions. Luo [9] optimized for minimal maximum completion time and total energy consumption using an enhanced multi-objective grey wolf algorithm. Lu [10] introduced an effective multi-objective discrete virus optimization algorithm for solving the flexible job shop scheduling problem with variable processing times.

Despite their prevalence, the aforementioned problem is often addressed using heuristic algorithms lacking in generalization capability. Deep Reinforcement Learning (DRL)-based production scheduling has emerged as a potential solution, capable of learning and generalizing knowledge from training data to address new challenges. Notable studies have applied DRL to scheduling problems of various scales [11,12]. For instance, Naimi et al. [11] proposed and trained a multi-objective Q-learning algorithm for optimal rescheduling to minimize completion time and energy consumption changes. Proximal Policy Optimization (PPO), a mainstream DRL algorithm, has been employed [13,14] to solve FJSP. However, effective application of DRL algorithms to learn scheduling strategies necessitates neural network architectures capable of processing varying state graph sizes due to the diverse FJSP instance sizes. While Graph Neural Networks (GNNs) have shown promise [15,16] in handling variable-size instances, they are limited to isomorphic graph problems. Song [17] addressed this limitation by employing HGNN with a two-stage embedding process for heterogeneous graph problems, along with a decision model for simultaneous operation and machine selection. Graph Attention Networks (GATs) [18] have demonstrated efficacy in processing machine and operation features by leveraging attention mechanisms [19] to learn node importance, aiding in identifying high-priority operations and suitable operation–machine arc combinations.

The success of DRL in scheduling underscores its potential in addressing flexible job shop scheduling problems. However, existing DRL-based research tends to focus on single-objective scheduling problems, with limited attention paid to multi-objective FJSPs, and even fewer studies considering environmental impacts. Additionally, DRL-based solutions face challenges in improving model performance and narrowing the gap with exact solutions. Existing methods often fail to adequately model machine relationships, which can be viewed as competing for remaining unscheduled operations. Yang [20] proposed a novel approach to solving multi-objective FJSP problems by leveraging deep reinforcement learning-trained hand controllers to manipulate objects, combined with Bayesian optimization methods.

In summary, there has been a growing concern about carbon emissions and their impact on the environment. Industries worldwide are seeking ways to reduce their carbon footprint while maintaining productivity and profitability. One area that requires attention is flexible job shop scheduling, a critical process in various sectors such as manufacturing, logistics, and services.

The existing research in flexible job shop scheduling mainly focuses on optimizing economic indicators such as production efficiency, makespan, and resource utilization. However, with the increasing emphasis on sustainability, it is crucial to consider environmental factors, particularly carbon emissions, in the scheduling process. This study addresses carbon emission concerns in flexible job shop scheduling, developing a multi-objective model and proposing a deep reinforcement learning-based algorithm for LC-FJSP resolution. The aim is to optimize both economic and environmental indicators.

The proposed model has several practical implications for real-world applications. By simultaneously considering economic and environmental objectives, it provides decision-makers with a comprehensive solution that balances production efficiency and carbon emission reduction. This approach can help industries achieve sustainable and efficient scheduling, contributing to their environmental targets while maintaining operational performance. The primary contributions of this research are outlined as follows:

• This research broadens the scope of the classic Flexible Job shop Scheduling Problem (FJSP) by integrating considerations of both operating and idle energy consumption into the machine parameters, formulating a mathematical model for the Low-carbon Flexible Job shop Scheduling Problem (LC-FJSP). The model’s primary goal is to minimize the weighted sum of total completion times and carbon emissions.
• The study introduces a sophisticated graph attention network that accounts for machine energy consumption in its feature extraction process, deploying an adaptive graph attention mechanism to accurately discern the relationships and priorities between each node and its adjacent nodes, thereby elucidating the intricate dependencies among machines and operations.
• It establishes an innovative end-to-end training framework known as the Low-carbon Graph Reinforcement Learning (LCGRL) framework. This framework incorporates a Bayesian optimization module designed to fine-tune weight coefficients for multi-objective challenges, significantly enhancing the training convergence and delivering a Deep Reinforcement Learning (DRL) model with exemplary composite performance.

Our proposed approach demonstrates significant novelty:

• Firstly, this extension of the traditional Flexible Job shop Scheduling Problem (FJSP) provides a novel perspective on environmental sustainability issues.
• Secondly, the method employs a graph attention mechanism that considers machine energy consumption, accurately identifying the complex dependencies between machines and operations, which is innovative in this field.
• Finally, our end-to-end training method not only enhances training convergence but also achieves exceptional overall performance. This framework represents a pioneering application in addressing similar multi-objective challenges.

The structure of the remainder of this paper is as follows. Section 2 reviews recent scheduling methods; Section 3 explores the Flexible Job shop Scheduling (FJSP) concept and details its mathematical modeling; Section 4 elaborates on the algorithm’s design; Section 5 presents experimental setups and comparative results. Section 6 summarizes the study and suggests avenues for future research.

2. Related Works

2.1. Traditional Optimization Methods

In traditional optimization methods for the flexible job shop scheduling problem, many studies focus on improving existing genetic algorithms and other heuristic approaches to address multi-objective optimization challenges. These methods often introduce new neighborhood search strategies, adaptive operations, and fuzzy logic to enhance algorithm performance, achieving significant progress in optimizing production efficiency and reducing energy consumption.

Zhao et al. [21] proposed an Enhanced Non-dominated Sorting Genetic Algorithm II (ENSGA-II) to address the Multi-Objective Energy-Saving Flexible Job Shop Scheduling Problem (MOEFJSP). Extensive experiments show that the ENSGA-II is effective in optimizing production criteria and reducing energy consumption. Li et al.’s [22] study investigates a Bi-Population Balancing Multi-Objective Evolutionary Algorithm (MOEA) for solving the Distributed Flexible Job Shop Scheduling Problem (DFJSP) in a steelmaking system. The algorithm simultaneously considers fuzzy processing times and crane transportation processes, aiming to minimize both the maximum fuzzy completion time and the energy consumption during machine processing and crane transportation. Zhang et al. [23] discuss the flexible job shop scheduling problem in the context of remanufacturing, proposing an optimized approach based on an improved genetic algorithm. The method incorporates several enhancements, such as adaptive mutation and dynamic crossover strategies, to accelerate convergence and improve solution quality in complex scheduling scenarios.

2.2. Quantum Computing

Traditional optimization methods often struggle to balance solution quality and computation time, highlighting the need for more efficient algorithms. Recent advancements in quantum computing have shown promising results in this area. Schworm et al. [24] have proposed a Quantum Annealing-based approach (QASA) to tackle the Flexible Job Shop Scheduling Problem (FJSSP), achieving superior solution quality over classical methods like tabu search and simulated annealing. This approach effectively balances multiple objectives, including makespan, total workload, and job priority, by leveraging the strengths of Quantum Annealing combined with classical optimization techniques. Additionally, other studies have demonstrated the potential of quantum annealing in solving complex scheduling problems, suggesting it as a viable alternative or complement to traditional methods. Denkena et al. [25] utilized a hybrid Quantum Annealing and digital annealing approach to solve larger instances of FJSSP, showing significant improvements in computational efficiency and solution quality compared to traditional methods. Their work suggests that Quantum Annealing can be particularly beneficial for industrial-scale scheduling problems where quick rescheduling is needed to adapt to dynamic changes such as machine failures or supply chain disruptions.

Additional studies have benchmarked Quantum Annealing against state-of-the-art classical algorithms [26], indicating that it not only achieves comparable or better solution quality but also significantly reduces computation times. This makes Quantum Annealing a viable alternative or complement to traditional methods for complex scheduling problems.

2.3. Deep Reinforcement Learning

Li et al. [27] addressed the Energy-Aware Distributed Heterogeneous Flexible Job Shop Scheduling (DHFJS) problem, which extends traditional flexible job shop scheduling with added complexity. Their goal was to minimize total energy consumption (TEC) and makespan. They proposed a Deep Q-networks-based Co-Evolution algorithm (DQCE) to solve this NP-hard problem. Huang et al. [28] address the Distributed Job Shop Scheduling Problem (DJSP) using an end-to-end deep reinforcement learning (DRL) approach based on Graph Neural Networks (GNNs). Traditional methods separate job selection and factory assignment, but this method combines them, treating the problem as a Markov Decision Process (MDP). A specially designed disjunctive graph represents the problem, and a GNN extracts state features. The Proximal Policy Optimization (PPO) algorithm trains the policy, leading to improved decision making. Chen et al. [29] proposed a Q-learning-based multi-objective immune algorithm (Q-MOIA) to address the flexible job shop scheduling problem (FJSSP) with fuzzy processing times and dynamic disruptions. Their approach integrates a predictive–reactive dynamic/static rescheduling model, utilizing a mixed integer linear programming (MILP) model for static scenarios and rescheduling heuristics for dynamic disruptions like new job arrivals and machine breakdowns. The Q-MOIA algorithm enhances initial solutions with active decoding and improves exploration and exploitation capabilities through clonal selection and Q-learning mechanisms. In recent advancements, Ding et al. [30] proposed a multi-policy deep reinforcement learning framework to tackle the Multi-Objective Multiplicity Flexible Job Shop Scheduling Problem (MOMFJSP), which aims to minimize makespan and total tardiness. The framework utilizes a multi-policy proximal policy optimization algorithm (MPPPO), treating the scheduling problem as a Markov decision process. This approach integrates multiple policy networks with different objective weight vectors, a fluid model for state feature extraction, and a multi-policy co-evolution mechanism (MPCEM) to enhance interaction among policies. Their results demonstrate that this method significantly improves decision accuracy and schedule optimization compared to traditional dispatching rules and other scheduling methods.

Quantum computing methods, such as quantum annealing and digital annealing, have shown promising results in solving complex optimization problems. These methods can potentially explore vast solution spaces more efficiently than classical algorithms due to their ability to simultaneously evaluate multiple solutions. Specifically, quantum annealing has been successfully applied to various scheduling problems, demonstrating its capability to handle the combinatorial nature of these tasks.

Despite the potential advantages of quantum computing, we chose DRL and LCGAN for several reasons:

• DRL, especially when combined with advanced neural network architectures like LCGAN, provides a flexible framework that can adapt to dynamic changes in the scheduling environment. This adaptability is crucial for real-time applications where production schedules may need frequent adjustments due to unexpected disruptions.
• The use of DRL allows for a more nuanced representation of the scheduling problem through state–action pairs, enabling the learning of complex policies that can optimize multiple objectives simultaneously. LCGAN further enhances this by incorporating environmental considerations, making it suitable for low-carbon manufacturing processes.
• Current quantum computing technologies are still in their nascent stages, with practical implementations being limited by hardware constraints and error rates. In contrast, DRL and neural networks are well-established, with extensive libraries and frameworks available, facilitating easier implementation and experimentation.

3. Problem Description and Model Construction

3.1. Problem Description

Assume the workshop and n jobs that need to be processed, denoted as the job set $J=\left\{J_1,J_2,\cdots ,J_n\right\}$. Each job $J_i$ consists of $n_i$ operations, and each operation $O_{ij}$ can be processed on any machine from a subset of capable machines $M_{ij}\subseteq M$. Each operation is assigned to only one machine, and the processing time of each operation varies depending on the machine used, with different operations also resulting in different amounts of carbon emissions. Typically, the performance of workshop scheduling is evaluated based on the maximum completion time of all jobs within the workshop. However, it is imperative to also consider the energy consumption costs within the workshop. This paper primarily focuses on two types of carbon emissions associated with workshop machines; one is the carbon emission produced during the machining process, and the other is the carbon emission resulting from electrical energy consumption when the machine is in an idle state. Addressing the Low-Carbon Flexible Job Shop Scheduling Problem (LC-FJSP) proposed in this study, the primary task involves assigning appropriate machines for operations and reasonably arranging the processing sequence of different job operations on the machines to minimize the idle carbon emissions of the machines. Thus, the LC-FJSP considered in this paper, similar to the general FJSP, encompasses two sub-problems: machine assignment and operation sequencing.

For the problem in question, the following assumptions are made:

1.

Both the jobs and machines are available at the initial starting time of zero;

2.

Each machine can process only one operation at a time, and each operation can only be processed by one available machine;

3.

Once the processing of a job begins, it cannot be interrupted;

4.

Different jobs are independent of each other, and there are precedence constraints between different operations within the same job;

5.

The preparation time of machines before processing operations is not considered.

3.2. LC-FJSP Math Model

3.2.1. Symbol Definition

In our study, the following symbol definitions are used, as shown in

Table 1. Symbol definitions of LC-FJSP Math Model.

Symbol	Description
n	Number of jobs
m	Number of machines
$J_i$	The i-th job
$M_k$	The k-th machine
$n_i$	Total number of operations in job $J_i$
$O_{ij}$	The j-th operation of job $J_i$
$M_{ij}$	Set of optional machines for the j-th operation of job $J_i$
$t_{ijk}$	Processing time of operation $O_{ij}$ on machine $M_k$
$S_{ij}$	Start time of operation $O_{ij}$
$C_{ij}$	Completion time of operation $O_{ij}$
$C_i$	Completion time of job $J_i$
$C_{max}$	Maximum completion time
$S{T}_k$	Start time of machine $M_k$
$C{T}_k$	Completion time of machine $M_k$
$p_{ijk}$	Average energy consumption of operation $O_{ij}$ on machine $M_k$
$p_{ek}$	Average energy consumption of machine $M_k$ under idle condition
${\alpha }_e$	Carbon emission coefficient of energy consumption
$TCE$	Total carbon emission
$x_{ijk}=$	$\left\{\begin{array}{c}1,\mathrm{if}\mathrm{operation}{O}_{ij}\mathrm{is}\mathrm{processed}\mathrm{on}\mathrm{machine}{M}_k\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$
$y_{ij{i}^{\prime }{j}^{\prime },k}=$	$\left\{\begin{array}{c}1,\mathrm{if}\mathrm{operation}{O}_{ij}\mathrm{precedes}\mathrm{operation}{O}_{i^{\prime }{j}^{\prime }}\mathrm{on}\mathrm{machine}{M}_k\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$

.

3.2.2. Carbon Emissions

Under processing conditions, the carbon emissions are calculated as follows:

$$E_1={\alpha }_e·W_1$$

(1)

$$W_1=\sum _{k=1}^m\sum _{i=1}^n\sum _{j=1}^{n_i}{p}_{ijk}{t}_{ijk}{x}_{ijk}$$

(2)

Under idle conditions, the carbon emissions are calculated as follows:

$$E_2={\alpha }_e·W_2$$

(3)

$$W_2=\sum _{k=1}^m{p}_{ek}\left(C{T}_k-S{T}_k-\sum _{i=1}^n\sum _{j=1}^{n_i}{t}_{ijk}{x}_{ijk}\right)$$

(4)

Total carbon emissions are calculated as follows:

$$TCE=E_1+E_2$$

(5)

3.2.3. LC-FJSP Mathematical Programming Model

This article aims to minimize the weighted sum of the maximum completion time and carbon emissions as optimization objectives. The constructed LC-FJSP model is as follows:

$$minf=min\left\{\alpha C_{max}+\beta TCE\right\}$$

(6)

$s.t.\{\begin{array}{}\mathrm{(7)}& C_{max}\ge C_i,\forall i\hfill \mathrm{(8)}& {\displaystyle \sum _{M_k\in M_{ij}}}{x}_{ijk}=1,\forall i,j,k\hfill \mathrm{(9)}& S_{i,j+1}\ge C_{ij},\forall i,j\hfill \mathrm{(10)}& C_{ij}-S_{ij}={\displaystyle \sum _{k=1}^m}{x}_{ijk}{t}_{ijk},\forall i,j\hfill \mathrm{(11)}& S_{i^{\prime }{j}^{\prime }{j}^{\prime }}+L·(1-y_{i{j}^{\prime }{\prime }^{\prime }{j}^{\prime },k})\ge S_{ij}+t_{ijk}=C_{ij},\forall i,j\hfill \mathrm{(12)}& S_{ij}+L·y_{ij{j}^{\prime }{j}^{\prime },k}\ge C_{i^{\prime }{j}^{\prime }},\forall i,j,i^{\prime },j^{\prime },k\hfill \end{array}$

The mathematical programming model for the Low-carbon Flexible Job shop Scheduling Problem (LC-FJSP) is formulated as follows. Equation (6) defines the objective to minimize the weighted sum of the maximum completion time and carbon emissions, with $\alpha$ and $\beta$ serving as weight coefficients. These coefficients are optimally adjusted via Bayesian optimization to adapt to varying conditions. Equation (7) ensures that the completion time for each job does not exceed the established maximum completion time. Equation (8) mandates that each operation be exclusively assigned to a single machine for processing. Equation (9) establishes a hierarchy of priorities among operations within the same job. Equation (10) stipulates that operations, once commenced, must proceed uninterrupted. Equations (11) and (12) specify that a machine can only handle one operation at a time, with L denoted as a sufficiently large positive constant.

3.3. Instances of LC-FJSP and Disjunctive Graph Heterogeneity

LC-FJSP allows a job to be processed on multiple machines, and different machines may have different processing times, as shown in

Table 2. An instance of LC-FJSP.

Job	Opreation	Optional Machines and Processing Time
		${\mathit{M}}_{\mathbf{1}}$	${\mathit{M}}_{\mathbf{2}}$	${\mathit{M}}_{\mathbf{3}}$
$J_1$	$O_{11}$	5	-	-
	$O_{12}$	-	5	2
	$O_{13}$	6	-	7
$J_2$	$O_{21}$	3	-	5
	$O_{22}$	-	4	-
$J_3$	$O_{31}$	-	-	3
	$O_{32}$	3	6	-
	$O_{33}$	-	2	-

, which presents an instance of LC-FJSP.

In the provided table, $J_i$ denotes the i-th job, $O_{ij}$ denotes the j-th operation within job $J_i$, and $M_k$ denotes the k-th machine. The Low-carbon Flexible Job shop Scheduling Problem (LC-FJSP) is effectively modeled using a disjunctive graph.

We introduce a disjunctive graph $G=(O,C,D)$, where $O=\left\{O_{ij}|\forall i,j\right\}\cup \left\{S,E\right\}$ comprises all nodes, including operational nodes, as well as virtual start node S and end node E. Set C comprises directed arcs that enforce precedence constraints among consecutive operations within the same job. Set D includes undirected disjunctive arcs linking operations that may be processed on a common machine. For instance, operations $O_{11}$, $O_{13}$, $O_{21}$, and $O_{32}$ in Figure 1 are interconnected by red disjunctive arcs, signifying that they are all eligible for processing on machine $M_1$.

Introduce machine nodes $M_1, M_2, M_3$ and establish connections between operations capable of being processed on these machines ($M_i$, where $i=1, 2, 3$) and their respective machines using undirected arcs, thus forming a heterogeneous graph model, as depicted in Figure 2.

4. Design of Solution Algorithms

The LC-FJSP studied in this paper is an extension of the traditional FJSP, introducing carbon emission constraints, thereby increasing the complexity of the problem. With the increase in problem size, the search space for feasible solutions grows exponentially, categorizing it as an NP-Hard problem. This implies a challenge in finding polynomial-time algorithms, necessitating further research and exploration. Consequently, this paper devises a deep reinforcement learning algorithm to efficiently solve this problem. In this section, the LC-FJSP is regarded as a sequential decision-making problem, iteratively scheduling job operations, assigning operations to compatible machines at each state until all operations are scheduled. The overall framework of the algorithm is illustrated in Figure 3.

4.1. Markov Decision Process (MDP)

To address the LC-FJSP using deep reinforcement learning, we initially define the states, actions, state transitions, and rewards, transforming the problem into a Markov Decision Process (MDP). A DRL-based decision framework is then established, which treats the selection of operations and machines integrally and outputs a probability distribution for decision making. A greedy strategy is employed, focusing on selecting operation–machine pairs with the highest scores. Lastly, we explain the training methodology for the proposed model.

The scheduling process in FJSP is conceptualized as assigning a ready operation to a suitable idle machine. The procedure is as follows. At each decision point t (either at the start or upon the completion of an operation), the agent assesses the current state $s_t$ and selects an action $a_t$, specifically assigning an unplanned operation to an available machine, beginning its execution from time $T\left(t\right)$. Subsequently, the system transitions to the next state at step $t+1$. This sequence continues until all operations are scheduled. The MDP framework is defined as follows:

State: The state representation captures the primary attributes and dynamics of the scheduling environment, considering both processes and machines as the composite state. The collective state of all processes and machines at any decision step t forms state $s_t$, starting from the initial FJSP instance denoted as $s_0$.

Action: The paper integrates process selection and machine assignment into a unified action choice, defining all feasible process–machine pairs as the action space. As scheduling progresses, the action space naturally diminishes as more operations are allocated.

State Transition: At each decision step t, from state $s_t$, the agent selects an action from the available space, performing action $a_t$, This leads to an environmental shift to the subsequent state $s_{t+1}$.

Reward: The purpose of designing the reward function is to guide the agent to select actions that minimize the maximum completion time and total carbon emissions of all operations. The reward function $r_t$ at time step t is defined as $f\left(s_t\right)-f\left(s_{t+1}\right)$, where f represents the value of $\alpha C_{max}\left(s_t\right)+\beta TCE\left(s_t\right)$ in the current state $s_t$. When the discount factor $\gamma =1$, the accumulation of rewards at each step yields ${\sum }_{t=0}^{|\mathcal{O}-1|}{r}_t=f\left(s_0\right)-f\left(s_t\right)$. In a specific problem instance, $f\left(s_0\right)$ is a constant, implying that minimizing f and maximizing the cumulative reward are equivalent.

Policy: We adopt a stochastic policy $\pi \left(a_t\right|s_t)$, which defines a probability distribution over the action set $A_t$ for each state $s_t$. The distribution of this policy is generated by a deep reinforcement learning algorithm, optimizing specific parameters during training to maximize the cumulative reward.

For example, consider a simple scenario where there are two jobs, $J_1$ and $J_2$, each with one operation, $O_1$ and $O_2$, respectively, and three machines, $M_1$, $M_2$, and $M_3$. At a decision point t, both $O_1$ and $O_2$ are ready to be processed.

At time t, the current state $s_t$ includes the status of all jobs and machines. For instance, $J_1$’s operation $O_1$ is pending assignment, $J_2$’s operation $O_2$ is pending assignment, and all machines $M_1$, $M_2$, and $M_3$ are idle.

The action space includes all possible job–machine assignments. In this scenario, the possible actions are:

1.

Assign $O_1$ to $M_1$;

2.

Assign $O_1$ to $M_2$;

3.

Assign $O_1$ to $M_3$;

4.

Assign $O_2$ to $M_1$;

5.

Assign $O_2$ to $M_2$;

6.

Assign $O_2$ to $M_3$.

Suppose the agent selects the action to assign $O_1$ to $M_1$.The system transitions to the next state $s_{t+1}$ where $O_1$ is being processed on $M_1$. For example, the new state could be as follows: $O_1$ is running on $M_1$ with an expected completion time of 5 units, $O_2$ is still pending assignment, $M_2$ and $M_3$ remain idle.

The reward for this action is calculated based on the reduction in the combined metric of maximum completion time ($C_{max}$) and total carbon emissions ($TCE$). Suppose at state $s_t$ that $C_{max}$ is 20 and $TCE$ is 30. After transitioning to state $s_{t+1}$, $C_{max}$ reduces to 19 and $TCE$ reduces to 28. The reward function $r_t$ is defined as $f\left(s_t\right)-f\left(s_{t+1}\right)$, where f represents the weighted sum of $C_{max}$ and $TCE$. Assuming equal weights, $f\left(s_t\right)=20+30=50$ and $f\left(s_{t+1}\right)=19+28=47$; hence, $r_t=50-47=3$.

By considering both process selection and machine assignment in the actions, the agent effectively learns to balance the workload and optimize the overall performance metrics in LC-FJSP.

4.2. Low-Carbon Graph Attention Network (LCGAN)

To address the Low-carbon Flexible Job shop Scheduling (LC-FJSP) challenge, characteristics such as processing time on operation-to-machine (O-M) arcs are critical. Based on these times, we can calculate the carbon emissions $E_1$ during processing on different machines; likewise, from the machine’s idle times, we can infer the carbon emissions $E_2$ when the machines are not active. This study takes advantage of the unique features and benefits of the heterogeneous graph structure by introducing a tailored LCGAN network architecture specifically designed for LC-FJSP. By enhancing two attention modules, as cited in [31], this framework skillfully captures feature representations of process and operation nodes. Energy consumption features are added to the O-M arcs within the machine feature attention module to aid in the amalgamation and filtration of process features. To address the relationship between time and carbon emissions in subsequent calculations, we employ Bayesian optimization to adjust the weights for maximum completion time and carbon emissions, aiming to identify the optimal solution. The input feature dimensions for the operation and machine are $d_o$ and $d_m$, respectively. Figure 4 illustrates the architecture of LCGAN.

4.2.1. Operation Feature Attention Module

The operation feature attention module aims to connect the operations within the same workpiece by finding the most important operations through their inherent attributes. For each input operation feature $h_{O_{ij}}\in {\mathbb{R}}^{d_O^{\prime }}$ of $O_{ij}\in O_u$, this module establishes relationships between $O_{ij}$, its predecessor $O_{i,j-1}$, and successor $O_{i,j+1}$ by calculating their attention coefficients as follows:

$$e_{i,j,p}=LeakyReLU\left({\overrightarrow{\mathbf{a}}}^{\top }\left[\left(\mathbf{W}{h}_{O_{ij}}\right)\parallel \left(\mathbf{W}{h}_{O_{ip}}\right)\right]\right)$$

(13)

where ${\overrightarrow{\mathbf{a}}}^{\top }\in {\mathbb{R}}^{2d_O^{\prime }}$ and $\mathbf{W}\in {\mathbb{R}}^{d_O^{\prime }\times d_O}$ are linear transformations, for all $|p-j|\le 1$.

We chose the LeakyReLU activation function over the standard ReLU for several reasons. Firstly, LeakyReLU helps mitigate the “dying ReLU” problem, ensuring that neurons remain active and gradients flow during training, which is crucial for our operation feature attention module. Secondly, our preliminary experiments indicated the presence of noisy features and outliers in the dataset. LeakyReLU’s small negative slope allows for non-zero gradients when units are inactive, helping the model handle noise and outliers more robustly. Lastly, LeakyReLU’s ability to provide gradients for negative inputs aids in better gradient propagation, particularly beneficial for deep networks.

These calculations are similar to those in GAT but narrowed in scope. Since the predecessors (or successors) of some operations may not exist or may be removed at some step, dynamic masking is applied to the attention coefficients of these predecessors and successors. The softmax function normalizes all $e_{i,j,p}$ to obtain the normalized attention coefficients ${\alpha }_{i,j,p}$. Finally, by weighted linear combination of the transformed input features $\mathbf{W}{h}_{O_{i,j-1}}$, $\mathbf{W}{h}_{O_{i,j}}$, and $\mathbf{W}{h}_{O_{i,j+1}}$, followed by a nonlinear activation function $\sigma$, the output feature vector $h_{O_{ij}}^{\prime }\in {\mathbb{R}}^{d_O^{\prime }}$ is obtained:

$$h_{O_{ij}}^{\prime }=\sigma \left(\sum _{p=j-1}^{j+1}{\alpha }_{i,j,p}\mathbf{W}{h}_{O_{ip}}\right).$$

(14)

By sequentially connecting multiple operation feature attention modules, the message of $O_{ij}$ can be propagated to all operations in $J_i$.

4.2.2. Machine Feature Attention Module

Each operation to be processed can ultimately be completed on only one machine; hence, there exists a competitive relationship between different machines, involving the same operations to be processed. This competitive relationship may dynamically evolve as the production process progresses. We define $C_{kq}$ as the set of operations competing between machines $M_k$ and $M_q$, and $E_{p_{ijk}}$ and $E_{p_{ijq}},$ respectively, represent the energy consumption of operation $O_{ij}$ on machine $M_k$ and machine $M_q$. Furthermore, we use $c_{kq}={\sum }_{O_{ij}\in C_{kq}}(E_{p_{ijk}}+E_{p_{ijq}})h_{O_{ij}}$ to measure the intensity of competition between $M_k$ and $M_q$, where more intense competition indicates that the candidate operations are more important. The machine feature attention module uses $c_{kq}$ to calculate the attention coefficient $v_{kq}$. For each $M_k\in {\mathcal{M}}_u$ with input feature $h_{M_k}\in {\mathbb{R}}^{d_m}$, the attention coefficients $v_{kq}$ for all $M_k$ competing are calculated as follows:

$$v_{kq}=LeakyReLU({\overrightarrow{\mathbf{b}}}^{\top }[\left({\mathbf{V}}^1{h}_{M_k}\right)\left|\right|\left({\mathbf{V}}^1{h}_{M_q}\right)\left|\right|\left({\mathbf{V}}^2{c}_{kq}\right)\left]\right)$$

(15)

where ${\mathbf{V}}^1\in {\mathbb{R}}^{d_m^{\prime }\times d_m}$ and ${\mathbf{V}}^2\in {\mathbb{R}}^{d_m^{\prime }\times d_o}$ are weight matrices, and ${\overrightarrow{\mathbf{b}}}^{\top }\in {\mathbb{R}}^{3d_m^{\prime }}$ is a linear transformation.

$C_{kk}$ represents the set of unscheduled operations that $M_k$ can process. $c_{kk}$ can be considered a measure of $M_k$’s processing capability, and we similarly apply the above formula to calculate $v_{kk}$. Then, normalized attention coefficients are obtained using softmax, and the transformed input features are combined and activated with ELU to obtain the machine output feature $h_{M_k}^{\prime }\in {\mathbb{R}}^{d_m^{\prime }}$.

4.2.3. Multi-Head Attention Module

We utilize multiple attention heads to process the aforementioned modules, aiming to learn the diverse relationships between entities. Let H denote the number of attention heads in the attention layer; we apply $2H$ attention mechanism modules, each containing different parameters. Firstly, parallel computations are performed to derive attention coefficients and combinations. Secondly, their outputs are integrated through an aggregation operator. We adopt concat as the aggregation operator, and an average operator is used in the last layer. Finally, an activation function $\sigma$ is applied to obtain the output of the layer.

4.2.4. Graph Pooling

When we use graph neural networks (GNNs) to process graph-structured data, the input graphs may have a varying number of nodes and different edge connections. This diversity can make the model very sensitive to changes in the input graphs, making it difficult to generalize well to new graph data. Graph pooling operations can help solve this problem by aggregating nodes or subgraphs in the graph to obtain a higher-level representation. This higher-level representation is equivalent to a summary or abstraction of the entire graph, containing important information and key features of the graph. The original features of the operation $O_{ij}$ and the machine $M_k$ are denoted as $h^{\left(0\right)}{O}_{ij}$ and $h^{\left(0\right)}{M}_k$, respectively. After processing by L layers of a GNN, the features aggregated with attention weights, $h^{\prime \left(L\right)}{O}_{ij}$ and $h^{\prime \left(L\right)}{M}_k$, are used for subsequent decision tasks. Following the method in reference [19], we first average pool the features of the operations and machines, respectively, and then concatenate their results to form the global feature of the FJSP instance, as shown below:

$$h_G^{\prime \left(L\right)}=\left[\left({\displaystyle \frac{1}{|{\mathcal{O}}_u|}}\sum _{O_{ij}\in {\mathcal{O}}_u}{h}_{O_{ij}}^{\prime \left(L\right)}\right)\parallel \left({\displaystyle \frac{1}{|{\mathcal{M}}_u|}}\sum _{M_k\in {\mathcal{M}}_u}{h}_{M_k}^{\prime \left(L\right)}\right)\right].$$

(16)

4.3. Making Decision

We designed a decision network based on the actor–critic reinforcement learning framework, where both the actor and critic utilize Multilayer Perceptrons (MLPs), with parameters denoted by $\theta$ and $\varphi$, respectively. The actor network first generates a scalar for each and then uses the softmax function to output the desired distribution, resulting in a stochastic policy that guides the agent’s behavior in the environment. We concatenate all information related to $a_t$ into a single vector and feed it into $ML{P}_{\theta }$:

$$\mu \left(a_t\right|s_t)={MLP}_{\theta }\left[h_{O_{ij}}^{\prime \left(L\right)}\left|\right|h_{M_k}^{\prime \left(L\right)}\left|\right|h_G^{\prime \left(L\right)}\left|\right|h_{(O_{ij},M_k)}\right].$$

(17)

The probability of choosing action $a_t$ is given by

$${\pi }_{\theta }\left(a_t\right|s_t)=\frac{exp\left(\mu \left(a_t\right|s_t)\right)}{{\sum }_{b_t\in A\left(t\right)}exp\left(\mu \left(b_t\right|s_t)\right)}.$$

(18)

The critic, represented by the value function network, evaluates the value of taking a certain action in different states, i.e., predicts the cumulative reward under a given policy. It takes the global feature $h_G^{\prime \left(L\right)}$ as input and produces a scalar $v\left(s_t\right)$ as the estimate of the state value. The objective of the critic network is to estimate the state value as accurately as possible, thereby providing better feedback and guidance for the actor network to improve its strategy and obtain higher rewards.

4.4. Bayesian Optimization

Reference [32]: We employ Bayesian optimization methods to determine the weights of the reward function. This method selects appropriate sampling points in the search space and adjusts the positions of sampling points based on observation results, gradually approaching the optimal solution. Our objective is to optimize the black-box function $f(C,E)=\alpha C+\beta E$, where $\alpha$ and $\beta$ are the coefficients to be optimized. We utilize the results of the decision network to obtain some sample points $(C_i,E_i)$ and their corresponding function values $f_i=f(C_i,E_i)$. According to the Bayesian optimization formula, we establish a Gaussian process model to describe $f(C,E)$. To select the optimal sampling point, we need to find a point $\left(C_{t+1,E_{t+1}}\right)$ under the current Gaussian process model that minimizes the objective function:

$$\left(C_{t+1},E_{t+1}\right)={argmin}_{(C,E)\in X}\mathbb{E}[f(C,E)\mid X,y].$$

(19)

To update the Gaussian process model, we need to observe the function value $f_{t+1}=f(C_{t+1},E_{t+1})$ at the point $(C_{t+1},E_{t+1})$, and then add $(C_{t+1},E_{t+1})$, $f_{t+1}$ into the sample points and function values. Next, using Bayesian theorem and Gaussian process regression methods, the mean vector and covariance matrix of the Gaussian process model are updated.

Repeat the above steps until convergence is reached or a preset number of iterations is achieved. The final mean vector $\mu \left(X\right)$ can be used to estimate the values of $\alpha$ and $\beta$. Specifically, this is represented as:

$$\alpha =\frac{{\mu }_C}{\mu },\beta =\frac{{\mu }_E}{\mu }.$$

(20)

where ${\mu }_C$ and ${\mu }_E$ represent the mean values of C and E in the input space, respectively, and $\mu$ represents the mean value of the function values at all points in the input space.

4.5. LCGRL Training Framework

In this study, our training framework leverages the Proximal Policy Optimization (PPO) algorithm [33] for the effective training of our scheduling model, utilizing a framework of 1000 episodes. Throughout each episode, the agent concurrently interacts with a batch of Low-carbon Flexible Job shop Scheduling (LC-FJSP) environments, denoted as $X_{tr}$, gathering state transition data to facilitate the iterative refinement of model parameters. To ensure variability, the environments undergo a resampling process every $N_{tr}$ episodes. Additionally, for verification purposes, our policy undergoes evaluation at intervals of $N_{val}$ episodes on a predefined set of validation datasets $E_{val}$, alongside the execution of Bayesian optimization for hyperparameter tuning within the reward function. Our experimental design includes two distinct action selection strategies: a greedy strategy, favoring actions with the highest likelihood $\mu \left(a_t\right|s_t)$ for outcome validation, and a random sampling strategy, derived from the policy distribution ${\pi }_{\theta }$, to enhance exploratory behavior during training. The methodology for the training process Algorithm 1 is summarized as follows:

Algorithm 1 Training LCGRL for LC-FJSP

1: Input: LCGAN network, policy network, and critic network with initial trainable parameters $\theta$, $\omega$, and $\varphi$, time weighting parameters $\alpha$, energy parameters $\beta$, pre-sampled training data $X_{tr}$, and fixed validation data $X_{val}$ 2: for$iter=1,2$ to I do 3:    for $i=1,2$ to $|X_{tr}|$ do 4:      Initialize $s_{i,t}$ based on instance i 5:      Update reward parameters $\alpha$ and $\beta$ 6:      for $t=1,2$ to T do 7:         Extract embeddings using LCGAN 8:         Sample $a_{i,t}\sim {\pi }_{\theta }(·|s_{i,t})$ 9:         Receive reward $r_{i,t}$ and next state $s_{i,t+1}$ 10:         Collect the transition $(s_{i,t},a_{i,t},r_{i,t},s_{i,t+1})$ and reward 11:         Update $s_{i,t}\leftarrow s_{i,t+1}$ 12:      end for 13:      Compute the generalized advantage estimates ${\widehat{A}}_{i,t}$ for each step 14:    end for 15:    Compute the total PPO loss ℓ, and optimize the parameters $\theta$, $\omega$, and $\varphi$ for $\kappa$ epochs 16:    if $iter$ mod $N_{tr}==0$ then 17:      Resample $X_{tr}$ instances to from the training data 18:    end if 19:    if $iter$ mod $N_{val}==0$ then 20:      Validate ${\pi }_{\theta }$ on $X_{val}$ 21:     Compute the parameters $\alpha$ and $\beta$ of the reward function using Bayesian optimization 22:    end if 23: end for

5. Experiments

This section presents the comparative experimental outcomes of our newly developed framework, LCGRL, against the DRL framework cited in [17], across both synthetic and public traditional FJSP instances. The objective is to validate the enhanced performance of our training framework on conventional FJSP problems. Furthermore, we assess our trained model against a fixed-weight counterpart on the LC-FJSP challenges, factoring in machine energy consumption, focusing on the differences in reward function outputs and carbon emissions. This comparison is intended to underscore the strengths of our model in effectively addressing the LC-FJSP.

5.1. Experimental Settings

5.1.1. Datasets

Consistent with the practices of the majority of related studies, this study generates synthetic FJSP instances for both training and testing purposes.

Given the scarcity of public benchmarks, the datasets currently available are inadequate for training deep reinforcement learning models. Consequently, this study employs the format of existing public datasets to randomly generate new instances that adhere to these standards, creating 100 instances for each specified scale.

Table 3. Instance generation distributions.

Size ($\mathit{n} \times \mathit{m}$)	${{\mathit{n}}_{\mathit{i}}}^1$	${|{\mathit{M}}_{\mathit{ij}}|}^2$	${{\mathit{t}}_{\mathit{ij}}}^3$
$10\times 5$	U$(4,6)$	U$(1,5)$	U$(1,20)$
$20\times 5$	U$(4,6)$	U$(1,5)$	U$(1,20)$
$15\times 10$	U$(4,6)$	U$(1,10)$	U$(1,20)$
$20\times 10$	U$(4,6)$	U$(1,10)$	U$(1,20)$
$20\times 10$	U$(4,6)$	U$(1,10)$	U$(1,20)$
$40\times 10$	U$(4,6)$	U$(1,10)$	U$(1,20)$

¹ Represents the number of operations in job $J_i$. ² Represents the number of machines that operation $O_{ij}$ can select. ³ Represents the average processing time of operation $O_{ij}$.

details the data ranges for these instances. Additionally, the second dataset extension expands the processing time range for each task from (1, 20) to (1, 100).

5.1.2. Configuration

In this study, the model training is set with I = 1000 iterations, $N_{tr}$ = 20, $N_{val}$ = 10. LCGAN employs L = 2 attention layers, and each module consists of H = 4 attention heads, with ELU as the activation function $\sigma$. The output dimensions of each attention head in the first layer are $d_o^{\left(1\right)}=d_m^{\left(1\right)}=32$, while in the second layer, they are $d_o^{\left(2\right)}=d_m^{\left(2\right)}=8$. Both $ML{P}_{\theta }$ and $ML{P}_{\varphi }$ have two hidden layers with 64 dimensions, and tanh serves as the activation function.

Regarding the PPO parameters, the coefficients in the loss function for policy, value, and entropy are set to 1, 0.5, and 0.01, respectively. The clipping parameter $\epsilon$, GAE parameter $\lambda$, and discount factor $\gamma$ are set to 0.2, 0.98, and 1, respectively. During training, the Adam optimizer [34] is utilized with a learning rate of $lr=3\times {10}^{-4}$, and the policy network is updated four times per episode.

The hardware specifications for our experiments include a machine with an Intel Xeon Gold 6152 CPU, a single NVIDIA Tesla T4 GPU, and Ubuntu 16.04 64-bit operating system. The software environment includes Python 3.8, PyTorch 1.8.1 for deep learning, OpenAI Gym 0.18.0 for reinforcement learning, and Scikit-Optimize 0.8.1 for optimization. Additional libraries used are NumPy 1.21.6, Pandas 1.3.5, and Matplotlib 3.3.4.

5.1.3. Baselines

This research contrasts the LCGRL scheduling strategy, developed through comprehensive training, against the DRL scheduling strategy [17], and four heuristic priority scheduling rules that are extensively applied in real-world scenarios, as documented in [35], namely FIFO, MOR, SPT, and MWKR. To guarantee a fair evaluation, these heuristic rules are executed within the same computational setting as our proposed approach, confirming the superiority of our learned strategy over traditional manual methods. Additionally, our findings are benchmarked against the Google OR-Tools solver [36]. In the context of public benchmark scenarios, this study further evaluates the efficacy of a two-stage genetic algorithm, as detailed in [37].

5.2. Results on Synthetic Datasets

The data presented in

Table 4. Results on synthetic datasets with two small to medium training scales.

Size			PDRs				Greedy		Sample		OR-Tools
			FIFO	MOR	SPT	MWKR	DRLG	LCGRLG	DRLS	LCGRLS
Range A 1	$10\times 5$	Obj	119.13	114.87	130.23	113.69	111.67	96.39	105.59	94.51	96.32
		Gap	23.74%	19.32%	35.21%	18.03%	16.03%	0.03%	9.66%	−1.87%
		Time	0.04	0.04	0.04	0.04	0.45	0.29	1.11	0.99
	$15\times 10$	Obj	182.82	171.42	195.28	168.74	166.92	153.22	160.86	141.20	143.53
		Gap	27.45%	19.47%	36.01%	17.59%	16.33%	4.56%	12.13%	−1.57%
		Time	0.13	0.13	0.12	0.13	1.43	0.97	3.98	5.25
	$20\times 5$	Obj	230.73	231.37	236.52	236.86	211.22	189.49	207.53	180.27	188.15
		Gap	22.64%	22.96%	25.74%	25.79%	12.27%	0.69%	10.31%	−4.17%
		Time	0.16	0.16	0.16	0.16	0.90	0.60	2.36	2.64
	$20\times 10$	Obj	244.33	231.34	266.08	234.46	215.78	197.32	214.81	185.82	195.98
		Gap	24.73%	18.07%	35.69%	19.7%	10.15%	0.63%	9.64%	−5.13%
		Time	0.36	0.36	0.35	0.36	1.91	1.34	6.23	8.67
Range B 2	$10\times 5$	Obj	561.56	550.92	507.52	536.53	553.61	400.69	483.9	331.49	326.24
		Gap	73.78%	70.59%	55.8%	66.06%	71.42%	23.1%	49.71%	1.78%
		Time	0.04	0.04	0.04	0.04	0.46	0.25	1.11	0.72
	$15\times 10$	Obj	883.53	855.44	728.92	843.85	807.47	582.21	756.07	473.51	377.17
		Gap	135.57%	128.09%	93.69%	124.94%	115.26%	54.7%	101.52%	25.77%
		Time	0.12	0.12	0.12	0.12	0.88	0.75	4.16	5.20
	$20\times 5$	Obj	1074.52	1067.48	842.22	1080.28	1059.04	677.57	962.9	573.76	602.04
		Gap	79.5%	78.17%	40.01%	80.67%	76.79%	12.65%	60.7%	−4.68%
		Time	0.16	0.16	0.16	0.16	0.61	0.48	2.37	2.71
	$20\times 10$	Obj	1148.42	1115.98	870.06	1100.89	1045.82	613.11	990.37	506.01	464.16
		Gap	148.27%	141.42%	87.64%	137.97%	126.12%	32.25%	114.15%	9.09%
		Time	0.35	0.35	0.36	0.35	1.95	1.33	6.03	8.54

¹ The processing time for each operation ranges from 1 to 20. ² The processing time for each operation ranges from 1 to 100.

clearly demonstrate that, for small to medium instances, our LCGRL model significantly surpasses the DRL model in terms of scheduling outcomes, regardless of whether actions are selected through greedy or random sampling strategies. In small to medium instances, our LCGRLG exhibits faster resolution times than DRLG. On extended datasets, both models perform comparably, with the increased variability in processing times having a minimal effect on the efficacy of greedy solutions. Additionally, the performance of the four PDRs on these extended sets highlights that, with broader processing time ranges, the SPT method significantly outperforms other PDRs, consistent with its inherent properties. Under random sampling conditions, our model not only shows considerable improvements but also surpasses the OR-tools solver on the standard dataset. Despite DRLS having quicker resolution times under random sampling strategies, the integration of a low-carbon graph neural network and Bayesian optimization modules does introduce some time costs. Nonetheless, the significant enhancements in solution quality achieved by LCGRLS are substantial and indicate potential for future advancements. To validate the generalization capabilities of the model, we documented its performance on large-scale instances after training on small to medium-sized datasets.

As illustrated in

Table 5. Results on synthetic datasets with two big training scales.

Size			PDRs		Greedy				Sample				OR-Tools
			MWKR	SPT	DRL	DRL	LCGRL	LCGRL	DRL	DRL	LCGRL	LCGRL
					$\mathbf{10}\times \mathbf{5}$	$\mathbf{20}\times \mathbf{10}$	$\mathbf{10}\times \mathbf{5}$	$\mathbf{20}\times \mathbf{10}$	$\mathbf{10}\times \mathbf{5}$	$\mathbf{20}\times \mathbf{10}$	$\mathbf{10}\times \mathbf{5}$	$\mathbf{20}\times \mathbf{10}$
Range A 1	$30\times 10$	Obj	312.93	350.07	314.71	313.04	291.21	291.52	308.55	312.59	289.59	290.56	274.67
		Gap	13.96%	27.47%	14.61%	14.01%	6.04%	6.16%	12.36%	13.83%	5.45%	5.81%
		Time	0.56	0.56	2.84	2.86	2.06	2.07	12.79	12.80	17.44	17.27
	$40\times 10$	Obj	414.82	445.17	417.87	416.18	383.30	382.95	410.76	415.25	384.28	384.86	365.96
		Gap	13.37%	21.66%	14.21%	13.75%	4.75%	4.67%	12.26%	13.49%	5.02%	5.18%
		Time	0.80	0.80	3.82	3.81	3.05	3.01	24.54	24.40	29.20	27.92
Range B 2	$30\times 10$	Obj	1539.666	1105.99	1564.57	1543.69	911.18	923.37	1486.56	1461.16	884.32	883.34	692.26
		Gap	122.89%	59.74%	126.55%	123.57%	31.75%	33.56%	115.21%	111.51%	27.86%	27.72%
		Time	0.56	0.55	2.93	2.93	2.13	2.13	12.88	12.75	18.32	18.38
	$40\times 10$	Obj	2037.652	1357.16	2048.96	2032.54	1112.51	1135.62	1976.25	1945.53	1111.92	1107.50	274.67
		Gap	108.66%	38.74%	109.87%	108.12%	13.76%	16.18%	102.45%	99.26%	13.76%	13.3%
		Time	0.79	0.77	3.87	3.93	3.01	2.96	24.55	24.50	30.76	28.72

¹ The processing time for each operation ranges from 1 to 20. ² The processing time for each operation ranges from 1 to 100.

, the LCGRL model consistently outperforms both the DRL model and heuristic algorithms across various scenarios. In Range A with a 30 × 10 size, LCGRL (10 × 5) achieved a lower objective value and gap compared to DRL (10 × 5) and the heuristic MWKR. For a 40 × 10 size in Range A, LCGRL (10 × 5) again outperformed DRL (10 × 5) and MWKR, demonstrating better solution quality. In Range B with a 30 × 10 size, LCGRL (10 × 5) achieved significantly better results than DRL (10 × 5) and the heuristic SPT, with a notably lower objective value and gap. For a 40 × 10 size in Range B, LCGRL (10 × 5) maintained its superior performance over DRL (10 × 5) and SPT. These results highlight the robustness and efficiency of the LCGRL model in handling different problem scales and processing times, consistently delivering superior performance compared to both DRL and heuristic algorithms.

5.3. Results on Public Datasets

In this subsection, the scheduling strategies discussed previously are applied directly across four distinct public datasets of varying sizes to further assess their generalization capabilities on two standard public benchmarks frequently utilized in conventional studies. The results also include those from the DRL method [17] and the two-stage genetic algorithm [37]. The Gap is calculated based on the optimal solutions provided in [38].

Given that real-world scheduling challenges feature non-static data distributions, we delved into the LCGRL model’s efficacy across four distinct public benchmarks, divergent from the training instance distributions. Each benchmark comprises instances varying in size. For instance, the mk benchmark involves job and machine counts ranging from 10 to 20 and 5 to 15, respectively. We tested models trained on unexpanded datasets, which demonstrated the most favorable outcomes on these benchmarks. Results presented in

Table 6. Results on public datasets.

Method	mk			la (rdata)			la (edata)			la (vdata)
	Obj	Gap	Time	Obj	Gap	Time	Obj	Gap	Time	Obj	Gap	Time
DRLG10 × 5	201.40	28.52%	1.25	1030.83	11.15%	1.40	1187.48	15.53%	1.40	955.90	4.25%	1.37
DRLG15 × 10	198.50	26.77%	1.25	1030.38	11.14%	1.40	1182.08	15.03%	1.40	954.33	4.02%	1.37
LCGRLG10 × 5	184.90	14.61%	0.81	1026.60	10.64%	0.89	1181.65	15.14%	0.90	943.08	3.01%	0.90
LCGRLG15 × 10	186.30	15.43%	0.82	1029.98	11.13%	0.90	1179.58	15.01%	0.90	945.13	3.22%	0.89
DRLS10 × 5	190.30	18.56%	4.12	985.30	5.57%	4.81	1116.68	8.17%	4.90	930.80	1.32%	4.71
DRLS15 × 10	190.60	19.0%	4.13	988.38	5.95%	4.81	1119.43	8.69%	4.87	931.33	1.34%	4.72
LCGRLS10 × 5	180.40	8.87%	5.65	980.68	5.21%	6.74	1119.13	8.73%	6.56	924.43	0.57%	6.86
LCGRLS15×10	180.40	8.87%	5.75	981.75	5.39%	6.85	1119.37	8.81%	6.78	924.20	0.56%	6.81
MOR	200.36	28.08%	0.11	1066.73	15.07%	0.12	1227.07	19.24%	0.12	966.01	5.68%	0.12
SPT	237.52	44.88%	0.11	1200.41	29.47%	0.12	1312.84	26.79%	0.12	1082.88	18.2%	0.12
FIFO	205.56	31.82%	0.11	1087.12	17.25%	0.12	1244.92	20.83%	0.12	982.89	7.58%	0.12
MWKR	201.74	28.91%	0.11	1053.10	13.86%	0.12	1219.01	18.6%	0.12	952.01	4.22%	0.12
2SGA	175.20	3.17%	57.60	-	-	-	-	-	-	922.20	0.39%	51.43
OR-tools	174.20	1.5%	1447.1	935.80	0.11%	1397.4	1028.93	−0.03%	899.60	919.60	−0.01%	639.17

indicate that MWKR outperforms the other three PDRs. According to the performance of models trained under OR-Tools, the 2SGA baseline, and DRL with greedy and random sampling strategies, while OR-Tools and 2SGA generally lead, they require longer computational times. In contrast, the two deep reinforcement learning approaches not only deliver commendable performances, surpassing the best PDR, but also maintain reasonable runtimes. LCGRL consistently exceeds DRL in most tests, with slight variations noted in tests utilizing sampling strategies on la (edata). Moreover, in mk instances, LCGRL substantially excels over DRL under both action selection strategies. These findings underscore LCGRL’s capability to authentically discern the inherent structural nuances of FJSP, focusing on distinguishing compatible machine–operation pairs rather than simply mastering the regularities underlying specific data distributions.

5.4. t-Tests and Confidence Intervals for Synthetic and Public Data Results

The objective of this section is to perform a statistical comparison of different methods on synthetic and public datasets. By leveraging t-tests and confidence intervals, we aim to evaluate the performance and effectiveness of these methods under various conditions.

Table 7. t-Tests for synthetic and public data results.

Comparison	Synthetic Data			Public Data
	t-Statistic	p-Value	Significant	t-Statistic	p-Value	Significant
FIFO vs. MOR	3.29	0.0133	Yes	4.47	0.0209	Yes
FIFO vs. SPT	1.95	0.0928	No	−4.32	0.0229	Yes
FIFO vs. MWKR	2.31	0.0539	No	3.47	0.0404	Yes
FIFO vs. DRLG	2.73	0.0292	Yes	2.83	0.0662	No
FIFO vs. LCGRLG	2.76	0.0283	Yes	4.63	0.0190	Yes
FIFO vs. DRLS	3.54	0.0095	Yes	2.95	0.0602	No
FIFO vs. LCGRLS	2.85	0.0247	Yes	3.46	0.0407	Yes
FIFO vs. OR-tools	2.76	0.0281	Yes	2.75	0.0708	No
MOR vs. SPT	1.74	0.1261	No	−4.40	0.0217	Yes
MOR vs. MWKR	0.82	0.4369	No	2.39	0.0966	No
MOR vs. DRLG	2.34	0.0520	No	2.14	0.1219	No
MOR vs. LCGRLG	2.67	0.0320	Yes	4.39	0.0219	Yes
MOR vs. DRLS	3.37	0.0119	Yes	2.63	0.0785	No
MOR vs. LCGRLS	2.79	0.0271	Yes	3.17	0.0503	No
MOR vs. OR-tools	2.71	0.0303	Yes	2.53	0.0855	No
SPT vs. MWKR	−1.68	0.1366	No	4.12	0.0259	Yes
SPT vs. DRLG	−1.39	0.2085	No	4.08	0.0266	Yes
SPT vs. LCGRLG	3.96	0.0055	Yes	4.85	0.0167	Yes
SPT vs. DRLS	−0.53	0.6112	No	4.07	0.0268	Yes
SPT vs. LCGRLS	3.70	0.0077	Yes	4.41	0.0216	Yes
SPT vs. OR-tools	3.33	0.0126	Yes	3.81	0.0319	Yes
MWKR vs. DRLG	2.26	0.0580	No	1.47	0.2374	No
MWKR vs. LCGRLG	2.66	0.0325	Yes	3.65	0.0356	Yes
MWKR vs. DRLS	3.38	0.0118	Yes	2.40	0.0962	No
MWKR vs. LCGRLS	2.78	0.0273	Yes	2.96	0.0598	No
MWKR vs. OR-tools	2.71	0.0301	Yes	2.37	0.0982	No
DRLG vs. LCGRLG	2.58	0.0365	Yes	3.40	0.0426	Yes
DRLG vs. DRLS	2.79	0.0270	Yes	2.94	0.0606	No
DRLG vs. LCGRLS	2.72	0.0297	Yes	4.09	0.0264	Yes
DRLG vs. OR-tools	2.66	0.0324	Yes	2.61	0.0799	No
LCGRLG vs. DRLS	−2.40	0.0475	Yes	1.82	0.1667	No
LCGRLG vs. LCGRLS	3.06	0.0184	Yes	2.51	0.0867	No
LCGRLG vs. OR-tools	2.35	0.0511	No	2.11	0.1250	No
DRLS vs. LCGRLS	2.62	0.0343	Yes	1.78	0.1740	No
DRLS vs. OR-tools	2.54	0.0385	Yes	2.32	0.1029	No
LCGRLS vs. OR-tools	0.83	0.4334	No	1.81	0.1676	No

presents the t-test results for synthetic and public datasets. In the synthetic data, several methods, such as FIFO vs. MOR, FIFO vs. DRLG, and SPT vs. LCGRLG, show significant differences, with p-values less than 0.05. This indicates that these methods have statistically different performances in synthetic environments.

In the public dataset, significant differences are less frequent but still present in some cases. For example, the SPT method shows significant differences with methods like FIFO, MWKR, and DRLG, suggesting distinct performance profiles in real-world settings.

Specifically, our trained LCGRLG model shows significant differences in several comparisons. On synthetic data, LCGRLG significantly differs from FIFO, MOR, MWKR, DRLG, and SPT methods, with all p-values less than 0.05; on public data, LCGRLG shows significant differences with DRLG and SPT methods.

Table 8. Confidence intervals for synthetic and public data results.

Method	Synthetic Data			Public Data
	Mean	Lower CI	Upper CI	Mean	Lower CI	Upper CI
FIFO	555.38	327.42	783.34	880.12	144.30	1615.95
MOR	541.48	318.98	763.98	865.04	139.48	1590.60
SPT	471.61	305.62	637.60	958.41	179.22	1737.60
MWKR	538.91	330.77	747.05	856.46	140.18	1572.75
DRLG	558.94	338.91	779.97	842.61	502.84	1182.38
LCGRLG	388.75	264.24	513.26	834.65	491.28	1178.03
DRLS	484.39	306.11	662.67	806.60	482.93	1130.28
LCGRLS	300.34	193.78	406.90	801.30	474.71	1127.88
OR-tools	299.77	171.53	428.01	764.63	133.62	1395.65

displays the confidence intervals for synthetic and public data. For synthetic data, the LCGRLG method has a mean performance of 388.75 with a confidence interval from 264.24 to 513.26, indicating some variability in performance. On public data, LCGRLG’s mean performance is 834.65 with a confidence interval from 491.28 to 1178.03, showing high and stable performance in real-world scenarios.

Overall, the t-test and confidence interval analyses indicate that the LCGRLG method performs well across different datasets, particularly showing high performance and significant statistical differences on public data. This suggests that LCGRLG has strong competitiveness and applicability in practical scenarios.

5.5. Results Considering Machine Energy Consumption

In the referenced experiments, the machine energy consumption coefficient was assumed to be zero, making the objective equivalent to makespan. To explore further, we incorporated machine energy consumption into our experiments and trained a variant of our model, designated LCGRLF (Low-Carbon Graph Reinforcement Learning Fixed), without the Bayesian optimization module. This allowed us to evaluate the benefits of including this module. We utilized synthetic datasets of sizes 10 × 5 and 20 × 10, with respective machine energy coefficients of [0.8, 0.9, 1.0, 1.1, 1.2] and [0.8, 0.8, 0.9, 0.9, 1.0, 1.0, 1.1, 1.1, 1.2, 1.2], maintaining fixed weights of 0.5 for both Cmax and TCE. Throughout the training process, we monitored the reward function values at each iteration and carbon emissions every ten samples, facilitating a comparative analysis with our Bayesian-enhanced LCGRL model, as depicted in Figure 5 and Figure 6.

Figure 5’s comparative results distinctly illustrate that, following stabilization, the carbon emissions of the LCGRL with the Bayesian optimization module are substantially lower than those of LCGRLF. This demonstrates that incorporating Bayesian optimization effectively supports our low-carbon objectives in scheduling tasks that account for machine energy consumption. The reward function comparison in Figure 6 reveals that the model enhanced with the Bayesian module converges more swiftly and maintains a lower variance in reward values post-convergence compared to the model without the module, indicating superior stability in managing the variability of multi-objective reward functions.

6. Conclusions

In this study, we introduce an innovative end-to-end deep reinforcement learning framework, LCGRL, marking the first integration of Bayesian optimization with deep reinforcement learning to tackle the flexible job shop scheduling problem. Our approach is designed to derive decision-making rules tailored for the Low-carbon Flexible Job shop Scheduling Problem and to train robust reinforcement learning models. Utilizing a heterogeneous graph model to delineate the scheduling state, our framework employs the Low-carbon Graph Attention Network to process the input of decision-related operations and raw machine features, achieving feature representation across heterogeneous graph nodes. We developed an actor–critic architecture embedded with Bayesian optimization to dynamically adjust the weight parameters of various objectives during training, facilitating the generation of low-carbon decisions that address both operation sequencing and machine allocation. The decision network is trained using the PPO algorithm. Empirical results illustrate that our trained decision rules significantly surpass heuristic scheduling rules, exhibiting superior generalization across large and public datasets. Our method outperforms conventional reinforcement learning techniques, particularly by incorporating a Bayesian optimization module that enhances the speed of reward function convergence and aligns outcomes more closely with low-carbon objectives.

While our method has demonstrated significant advantages, it is not without limitations. The primary limitation is the increased computational complexity introduced by the incorporation of multi-head attention modules and Bayesian optimization. This complexity could potentially hinder the implementation and scalability of our approach in real-world manufacturing environments, especially those with limited computational resources. Additionally, the current model may face challenges when dealing with real-time scheduling tasks due to the computational overhead.

To mitigate these limitations, future work could focus on optimizing the computational efficiency of the model and exploring hardware acceleration techniques. Another limitation is the assumption of static machine and job parameters, which might not hold true in highly dynamic manufacturing settings. Future research could address this by integrating adaptive mechanisms to handle such dynamic uncertainties.

The complexity introduced by multi-head attention modules and Bayesian optimization indeed poses challenges. However, these can be managed through various strategies. For instance, adopting high-performance computing resources and distributed computing techniques can help manage the increased computational demands. Furthermore, modularizing the framework allows for incremental upgrades and scalability across different manufacturing environments. Our empirical results have shown good generalization across various datasets, indicating potential applicability in diverse manufacturing contexts. Nevertheless, extensive real-world validation is necessary to fully establish the generalizability of our approach across different industries.

Our optimization approach primarily aims at minimizing carbon emissions while maintaining operational efficiency. However, there could be potential trade-offs such as increased production costs due to the need for advanced computational resources and potential delays in real-time decision making. Additionally, focusing heavily on low-carbon objectives might impact other sustainability metrics like resource utilization and overall production throughput.

In the future, our methodology could be adapted to address multi-objective flexible job shop scheduling issues characterized by dynamic uncertainties, such as the sudden insertion of new jobs and the unpredictability of deadlines. Additionally, future research might explore integrating additional optimization objectives, accounting for various sources of carbon emissions, and employing more sophisticated exploration techniques to develop practical scheduling solutions for complex manufacturing environments.