An Enhanced Multi-Objective Evolutionary Algorithm with Reinforcement Learning for Energy-Efficient Scheduling in the Flexible Job Shop

Abstract

The study of the flexible job shop scheduling problem (FJSP) is of great importance in the context of green manufacturing. In this paper, with the optimization objectives of minimizing the maximum completion time and the total machine energy consumption, an improved multi-objective evolutionary algorithm with decomposition (MOEA/D) based on reinforcement learning is proposed. Firstly, three initialization strategies are used to generate the initial population in a certain ratio, and four variable neighborhood search strategies are combined to increase the local search capability of the algorithm. Second, a parameter adaptation strategy based on Q-learning is proposed to guide the population to select the optimal parameters to increase diversity. Finally, the performance of the proposed algorithm is analyzed and evaluated by comparing Q-MOEA/D with IMOEA/D and NSGA-II through different sizes of Kacem and BRdata benchmark cases and production examples of automotive engine cooling system manufacturing. The results show that the Q-MOEA/D algorithm outperforms the other two algorithms in solving the energy-efficient scheduling problem for flexible job shops.

1. Introduction

The job shop scheduling problem (JSP) is the core part of the production management of the job shop system, which plays an important role in processing and manufacturing and making production decisions, and the efficient scheduling program helps the enterprise to improve production efficiency. The multi-objective flexible job shop scheduling problem (MO-FJSP) enables the simultaneous optimization of multiple indicators in the intelligent manufacturing process. By considering issues such as machine energy consumption in solving the problem of process sequencing, it aligns better with the practical production requirements. As a result, numerous scholars both domestically and internationally have conducted extensive research on this subject.

The MO-FJSP plays a key role in active production scheduling and distributed scheduling. With the development of artificial intelligence and swarm intelligence optimization algorithms, all kinds of new heuristic algorithms have been widely applied to solve the MO-FJSP, such as the grey-wolf algorithm, NSGA-III, and so on. Wei et al. [1] proposed a multi-objective hybrid evolutionary algorithm and designed a multi-dimensional encoding and decoding scheme to represent the feasible solution of the MO-FJSP. Chen et al. [2] proposed a hybrid adaptive differential evolutionary algorithm that combines an elite selection strategy using Pareto dominance relation and a simulated annealing algorithm to improve the performance of the algorithm. In addressing the impact of processing auxiliary tasks on traditional low-carbon scheduling in flexible workshops, Jin et al. [3] proposed an improved NSGA-II algorithm based on dominance strength. The algorithm aims to optimize the maximum completion time, carbon emissions, and machine energy consumption as objectives while solving the model. Pei et al. [4] proposed an improvement on the shortcomings of the backtracking search algorithm to prevent the population from falling into a local optimum during the iterative process by combining the dynamic control of the variation operation with changing the individual search magnitude factor.

In recent years, research on reinforcement learning has gained attention among scholars. By incorporating reinforcement learning mechanisms into various optimization algorithms, agents interact with the environment iteratively, allowing the algorithms to autonomously adapt to different solving environments. Du et al. [5] used deep reinforcement learning to learn and select heuristic local search operators, significantly enhancing the performance of distributed estimation algorithms. Zeng et al. [6] proposed a self-learning taboo search algorithm (DSLTS) based on deep reinforcement learning to address the dynamic adjustment of key parameters in flexible job shop scheduling algorithms. Lin et al. [7] employed the DQN algorithm to autonomously adjust the parameters of the grey-wolf optimization algorithm, accelerating its convergence. Lu et al. [8] addressed the flexible job shop scheduling problem in the context of green manufacturing. They established a multi-objective integer programming model aimed at minimizing the completion time, machine energy consumption, and workshop energy consumption. Furthermore, they proposed an improved, fast, non-dominated sorting genetic algorithm based on Q-learning to solve the problem. Pan et al. [9] introduced a feedback mechanism to adjust the population sizes of dual populations based on evolutionary performance, enhancing the convergence of the dominant population.

Integrating RL into the MOEA/D enables the algorithm to adaptively select the optimal neighborhood search strategies or adjust the algorithm parameters. For instance, RL can learn to choose different weight vectors at various stages of the search or dynamically adjust the scope of neighborhood searches during the process to enhance the algorithm’s global search capability and local search ability. By combining the MOEA/D with RL, intelligent scheduling strategies can be developed that optimize multiple scheduling objectives simultaneously. These strategies not only improve production efficiency but also significantly reduce energy consumption, leading to greener and more sustainable manufacturing practices.

In light of this, this study proposes an improved multi-objective evolutionary algorithm with decomposition based on Q-learning (Q-MOEA/D), aimed at solving the energy-saving scheduling problem in flexible job shops. By incorporating the strategy optimization of reinforcement learning algorithms, not only is the search efficiency of the algorithm enhanced, but its adaptability and robustness in volatile environments are also strengthened. Moreover, by considering energy saving as one of the key objectives of scheduling optimization, this study not only focuses on improving production efficiency but also commits to achieving green manufacturing, responding to the global call for energy conservation, emission reduction, and sustainable development.

The main contributions of this paper are as follows: (1) A combination of three initialization strategies is designed to generate the initial population in a certain proportion, which helps to accelerate the population convergence and improve the accuracy of the final solution. (2) A self-learning tuning strategy for parameters based on reinforcement learning is proposed to help MOEA/D select appropriate T and improve the diversity of optimal solutions. (3) A variable neighborhood search combining four local search strategies is designed. (4) The proposed algorithm is applied to a production example of an engine plant, which is used to verify the effectiveness of the algorithm.

2. Basic Theory

2.1. The MOFJSP Model

2.1.1. Description of the Problem

The MOFJSP needs to solve individual workpieces $\left\{J_{1 },J_2,J_3\cdots ,J_n\right\}$ and the problem of the machining sequence $\left\{M_{1 },M_2,M_3\cdots ,M_k\right\}\left(k\in \left\{1,2,3,\cdots m\right\}\right).$ In this case, each workpiece $J_i(i\in \left\{1,2,\cdot \cdot \cdot ,n\right\}$ has a $p_i$ process, in addition to the process $O_{ij}$ $\left(j\in \left\{1,2,\cdot \cdot \cdot ,p_{\dot{i}}\right\}\right).$ The machining time is determined by the performance of the selected machine. The goal of scheduling is to optimise the desired multiple performance metrics while satisfying the machining constraints. In addition, the following constraints need to be satisfied during machining:

• At any given time, a machine can only process one job.
• At any given time, a machine can only process one operation of the same job.
• Once an operation of a job begins, it cannot be interrupted.
• All jobs have equal priority.
• There are no precedence constraints between operations of different jobs, but there are precedence constraints between operations of the same job.
• All jobs can be processed at time zero.

2.1.2. Mathematical Modelling

This paper focuses on the simultaneous optimization of two evaluation metrics, namely the maximum completion time $C_m$ and minimum total machine energy consumption $W_t$. These two objectives are commonly used in practical production settings. The maximum completion time is crucial to the production cycle of each job, while the machine energy consumption relates to the overall utilization rate of machines in the workshop.

Minimizing the maximum completion time: The completion time refers to the time when the last operation of each job is finished. Among these completion times, the maximum is known as the maximum completion time. It is the fundamental metric for evaluating scheduling plans, reflecting the production efficiency of the workshop. It is also one of the most widely used performance indicators in MOFJSP research.

$${\mathit{minC}}_{\mathrm{m}}=\underset{1\le i\le n}{\mathit{max}}\left(\underset{i\le j\le p_i}{\mathit{min}}{C}_{\mathit{ij}}\right)$$

(1)

Minimizing total machine energy consumption: Variations in the processing times across different machines result in a diverse energy consumption profile, which is contingent upon the scheduling strategy employed. Specifically, when the maximum completion time is held constant, prioritizing the reduction in total energy consumption becomes a strategic objective. This approach not only aligns with sustainability goals but also contributes to cost efficiency within manufacturing operations. Thus, minimizing the aggregate energy consumption of all machines emerges is a critical component in the multi-objective optimization framework, particularly in scenarios where energy conservation is a key performance criterion.

$${\mathit{minW}}_{\mathrm{t}}={\sum }_{i=1}^n{\sum }_{j=1}^{p_i}{\sum }_{k=1}^m{T}_{\mathit{ij}}^k$$

(2)

$$S_{ij}+T_{ij}^k=C_{ij},\forall i,j,k$$

(3)

$$C_{ij}⩽S_{i_{\left(j+1\right)}}, \forall i , j$$

(4)

$$S_{tf}+T_{tf}^k\ll S_{hg}+L\left(1-{\beta }_{tjhg}^k\right),\forall \left(i,h\right),\left(j,g\right),k$$

(5)

$${\sum }_{k=1}^m{\alpha }_{\mathit{ij}}^k=1,\forall i.j$$

(6)

$$S_{ij}⩾0,C_{ij}⩾0$$

(7)

In Equation (1), $p_i$ represents the number of operations for job $i$, and $C_{ij}$ represents the completion time of operation $O_{ij}$. In Equation (2), $T_{ij}^k$ represents the processing time of operation $O_{ij}$ on machine $k$. Equation (3) indicates that the processing of operations cannot be interrupted. $S_{ij}$ represents the start time of operation $O_{ij}$. Equation (4) represents the constraint on the sequence of operations. Equation (5) imposes a restriction on the simultaneous processing of multiple operations on a machine. Here, ${\beta }_{tjhg}^k$ is a decision variable. If operation $O_{ij}$ precedes operation $O_{hg}$ in processing on machine $k$, then ${\beta }_{tjhg}^k$ = 1. Otherwise ${\beta }_{tjhg}^k$ = 0. L is a sufficiently large positive number. Equation (6) indicates that an operation can only be processed on one machine. In this equation, ${\alpha }_{ij}^k$ is a decision variable. If operation $O_{ij}$ is processed on machine $k$, then ${\alpha }_{ij}^k$ = 1. Otherwise, ${\alpha }_{ij}^k$ = 0. Equation (7) states that all parameter variables must be non-negative.

2.2. The MOEA/D

The MOEA/D is a decomposition-based multi-objective evolutionary algorithm proposed by Zhang [10] in 2007. It is one of the effective methods for solving multi-objective optimization problems, especially when facing complex problems with multiple conflicting objectives.

By decomposing a multi-objective optimization problem into a series of single-objective sub-problems and using the neighborhood structure information, these sub-problems are optimized in parallel by evolutionary algorithmic operations such as selection, crossover, and mutation. Algorithmic features include efficient parallel processing capabilities, flexibility in choosing the appropriate decomposition strategy based on problem characteristics, and broad applicability. The MOEA/D ensures that the optimal solution to the sub-problems is iteratively improved in each generation by updating the strategy, but its performance can be challenged by parameter settings and large-scale problems. This algorithm is remarkably effective and adaptable in solving complex multi-objective problems in a variety of domains.

2.3. Reinforcement Learning

Reinforcement learning (RL) is a machine learning method that mimics the cognitive and trial-and-error processes of human learning [11]. Unlike supervised and unsupervised learning, RL possesses the ability to learn independently, starting from scratch. This approach is particularly suitable for solving sequential decision-making problems, where traditional dynamic programming methods often struggle when faced with large state spaces or unknown state transition matrices or reward functions [12]. In contrast, RL does not require prior knowledge of the state transition probability matrix nor exhaustively searching the state space, making it effective in handling complex sequential decision-making problems.

The core idea of RL is to learn how to make optimal decisions through the interaction between an agent and an environment [13,14]. In RL, the agent takes actions in the environment, and the environment provides feedback in the form of a reward signal. The agent’s objective is to explore different strategies and find an optimal policy that maximizes the cumulative reward obtained over a series of actions. The interaction process between the RL agent and the environment is illustrated in Figure 1.

When solving a problem with reinforcement learning, it is necessary to describe the problem as a Markov Decision Process (MDP), characterized by a conditional probability distribution of future states that depends only on the current state, given the present state and all past states. It is common to use the quintuple $\left\{S,A,R,P,\gamma \right\}$ to describe the process, with $S$ denotes the state space;$A$ denotes the action space;$R$ is the reward obtained for choosing action $a$ in the current state $s$ to transfer to the next state $s^{\prime }$.

$$R\left(s,a\right)=E\left[R_{t+1}|S_t=s,A_t=a\right]$$

(8)

$P$ is the state transfer probability,

$$P_{s{s}^{\prime }}^a=P\left[S_{t+1}=s^{\prime }|S_t=s,A_t=a\right]$$

(9)

$\gamma$ is the attenuation factor that $\gamma \in \left(0,1\right)$.

3. The Q-MOEA/D Algorithm

3.1. Coding and Decoding Mechanisms

Encoding refers to representing the solution of a problem with a code, aligning the problem’s state space with the encoding space of an algorithm. The purpose of encoding is to enable genetic operations, such as crossover and mutation, similar to those seen in the biological realm. The most common encoding method for the MO-FJSP is MSOS encoding, which consists of two parts: the machine selection (MS) and the operations sequencing (OS). It is shown in Figure 2:

In the OS segment, each gene is directly encoded with a job number. The order in which the job numbers appear indicates the sequential processing order between operations of the respective job. For example, when compiling the chromosome from left to right, the $i$-th occurrence of job number $j$ represents the $i$-th operation of job $j$, and the total number of occurrences of each job number is equal to the total number of operations for that particular job. In the MS segment, the job numbers and their corresponding operations are arranged sequentially. Each integer represents the order of the currently selected machine for the respective operation from the set of available machines.

Decoding involves traversing the OS vector from left to right to obtain the processing sequence. Subsequently, the MS vector is traversed from left to right to determine the machine selection for each operation. Each operation is scheduled and processed on the corresponding machine. The start time and completion time of each operation are calculated, along with the waiting time at each processing position. This ensures that the start times adhere to the constraints of the MO-FJSP model. Finally, the total machine energy consumption and maximum completion time can be obtained from the current solution.

3.2. Population Initialization

A well-initialized population can reduce computational efforts by narrowing down the search space and accelerating the attainment of optimal solutions. To achieve a high-quality initialization population, this study proposes a hybrid initialization strategy consisting of three heuristic initialization rules:

Rule 1: Minimum Processing Time: Randomly generate the OS segment and select the machine with the minimum processing time for each operation. Choosing the minimum processing time allows for proximity to the lower bound of $C_m$.

Rule 2: Minimum Machine Energy Consumption: Randomly generate the OS segment and select the machine with the minimum energy consumption from the current scheduling candidate machines for each operation. If two machines have the same energy consumption, select the one with the shorter processing time.

Rule 3: Random Initialization: Randomly generate the OS segment and choose a random machine from the candidate machines for each operation.

The specific steps of the hybrid population initialization strategy proposed in this paper are as follows:

Step 1: Generate a subpopulation $P1$, comprising 1/10 of the total population, using Rule 1.

Step 2: Generate another subpopulation $P2$, comprising 1/10 of the total population, using Rule 2.

Step 3: Generate a subpopulation $P3$, comprising 8/10 of the total population, using Rule 3.

Step 4: Merge the populations $P=P1 \cup P2 \cup P3$ to get the final population. This approach combines the fast construction of well-converged subpopulations through heuristic rules with the random initialization to ensure population diversity. It effectively balances convergence and diversity, resulting in a high-quality initialization population.

3.3. Population Renewal Strategies

In the crossover operation strategy, two improved crossover operators are utilized: Improved Process-based Order Crossover (IPOX) and Machine-based Multi-Point Reserve Crossover (MPX).

IPOX (Improved Process-based Order Crossover): As shown in Figure 3, consider a case with 5 jobs and 12 operations. Let the parent chromosomes be denoted as $P1$ and $P2$, and the offspring chromosomes generated through crossover as $C1$ and $C2$. Firstly, randomly partition all jobs into two non-empty sets, $A$ and $B$. For example, let $A=\left\{J_1,J_3\right\},B=\left\{J_2,J_4, J_5\right\}$. Then, copy the corresponding jobs from $P1$ to $C1$ that match the positions of $A$, while maintaining the original order. Similarly, copy the corresponding jobs from $P2$ to $C2$ that match the positions of $B$, also preserving their original order. Next, copy the corresponding jobs from $P2$ to $C1$ that match the positions of $B$, and copy the corresponding jobs from $P1$ to $C2$ that match the positions of $A$, while preserving the order of the copied jobs.

MPX (Machine-based Multi-Point Reserve Crossover): As depicted in Figure 4, consider parent machine chromosomes $M1$ and $M2$. After the crossover, the offspring chromosomes $C3$ and $C4$ are generated. Firstly, generate an array $C$ with the same length as the parent machine chromosomes, filled randomly with 0 and 1. Traverse the $C$ array and record the positions where 1 appears. Copy the corresponding jobs from $M1$ to $C4$ at the positions where 1 appears, while copying the remaining jobs from $M2$ at the corresponding positions. Similarly, copy the corresponding jobs from $M2$ to $C3$ at the positions where 1 appears, and copy the remaining jobs from $M1$ at the corresponding positions.

OS mutation operator: In this paper, we use a random two-point exchange operator to perform mutation operations on OS vectors. Firstly, two positions on the OS vector are randomly selected and two genes are exchanged. Finally, replace the original solution with the mutated one. The mutation probability is $P_m$.

MS mutation operator: a random replacement machine strategy is applied to the MS vector. Several processes are randomly selected to be processed by another machine different from the current one, within the range of their optional machines. The mutation probability is $P_m$.

3.4. Neighborhood Search

The MOEA/D algorithm exhibits excellent global search capability, enabling it to quickly obtain optimized solutions. However, the neighborhood search strategy focuses on optimizing local solutions, thereby enhancing the algorithm’s local search ability and preventing it from getting trapped in local optima. Nonetheless, excessive local search can decrease the algorithm’s search efficiency [15]. In this study, four types of neighborhood actions are utilized.

For the job-based neighborhood structure:

Neighborhood Action 1: Randomly select two adjacent positions on the chromosome, exchange the corresponding sections of job chromosomes, and then randomly insert them into a chosen position on the chromosome.

Neighborhood Action 2: Randomly remove several jobs from the chromosome, and then sequentially insert them into the job encoding segment based on the order of removal.

For the machine-based neighborhood structure:

Neighborhood Action 3: For any machine encoding position in the chromosome, cyclically select a different machine from the feasible machines for the current position’s job. If the newly generated solution improves upon the original one (dominates the original solution or is mutually non-dominated), it is retained.

Neighborhood Action 4: For any machine encoding position in the chromosome, select the machine with the lowest energy consumption among the feasible machines for the current position’s job.

3.5. Self-Learning MOEA/D-Based Q-Learning

The proposed method leverages reinforcement learning (RL), specifically the Q-learning algorithm, to enhance the decision-making process within the multi-objective evolutionary algorithm based on decomposition (MOEA/D). This integration is crucial for optimizing energy consumption within the flexible job shop scheduling problem (FJSSP). The primary goal of applying RL in this context is to dynamically adjust the algorithm’s parameters to better achieve energy-saving objectives.

3.5.1. Introduction to the Q-Learning Algorithm

The Q-leaning algorithm, on the other hand, is a specific algorithm in reinforcement learning for solving Markov Decision Processes (MDPs) in discrete state and action spaces. The core idea of the Q-leaning algorithm is to guide the agent’s decision-making by learning an action-value function (Q-value function). This Q-value represents the expected value of the long-term cumulative reward that can be obtained by performing an action in a given state.

In the Q-leaning algorithm, the Q-value (also known as the action value function) is an expected value that represents the long-term cumulative reward for taking an action in a given state. Specifically, the Q-value is represented by a function $Q\left(s, a\right)$, where $s$ denotes the current state and $a$ denotes the action taken. The definition of the Q-value reflects the agent’s estimation of the expected value of the long-term reward that the agent expects to obtain after executing action $a$ in a given state $s$. The Q-value includes the immediate reward of the current action and the expected value of the long-term reward of action $a$. The Q-value is also known as the action-value function. This expectation of a long-term reward includes the immediate reward of the current action as well as possible future rewards.

The update rule of Q-learning is to gradually update the Q-value by continuously interacting with the environment in order to eventually find the optimal action value function. The update rule of Q-learning is usually based on Bellman’s equation with the following expression:

$$Q\left(S_t,A_t\right)=Q\left(S_t,A_t\right)+\alpha \left[R_t+\gamma \max \left(Q\left(S_{t+1},A_t\right)-Q\left(S_t,A_t\right)\right)\right]$$

(10)

The core idea of Q-learning is to continuously approximate the optimal action value function through such an iterative updating process, enabling the agent to learn an optimal strategy in its interaction with the environment. This updating rule enables the Q-learning algorithm to find the optimal strategy in an unknown environment and maximize the cumulative reward.

3.5.2. Action Definitions

In multi-objective optimization algorithms, the Pareto front is the set of optimal solutions calculated by the MOEA/D, and it reflects the algorithm’s capability. To enhance the diversity of the Pareto front, Q-learning is used to guide the population in selecting the optimal parameter T. Increasing the diversity of the entire population can improve the diversity of the genetic factors. Therefore, each iteration’s genetic factor is abstracted as an agent, reflecting the success of parameter selection. The four candidate values T = 5, 10, 15, and 20 are defined as actions [16].

3.5.3. Status Definitions

The change in the agent state gives Q-learning feedback to judge whether the action performed this time improves the diversity of the PF. To represent the agent state, two metrics, IGD and DV, are used to define convergence and diversity:

$$\mathit{IGD}({\mathit{PF}}_{\mathit{true}},{\mathit{PF}}_{\mathit{known}})=\frac{{\sum }_{i=1}^n\left|d_i\right|}{n}$$

(11)

$$\mathit{DV}=\frac{{\sum }_{i=1}^{N-1}|d_i-\overline{d}|}{(N-1)\overline{d}}$$

(12)

$$\mathsf{\Delta }IGD=IG{D}_{i-1}-IG{D}_i$$

(13)

$$\mathsf{\Delta }DV=D{V}_{i-1}-D{V}_i$$

(14)

IGD (Inverted Generational Distance) reacts to the convergence of the solution, which is obtained by shooting a reference point from the real Pareto front $P{F}_{true}$ to the algorithm’s solution $P{F}_{known}$. Points are taken uniformly from the real Pareto front, and for each point on the real front, the closest point on the known Pareto front is found, and the distances between these points are summed and averaged. In Equation (11), where $n$ denotes the number of points in the $P{F}_{true}$, $d_i$ denotes the nearest Euclidean distance from the known frontier for each point on the true frontier in the target space. The smaller this value is, the better the comprehensive performance of the algorithm.

DV (Diversity) reflects the distribution of solutions in the solution set; a good solution set should not only cover all possible optimal solutions, but also be evenly distributed among the solutions, which can provide the decision-maker with a wider range of choices. In Equation (12), $d_i$ denotes $P{F}_{known}$, the Euclidean distance between two adjacent points; $\overline{d}$ denotes $d_i$, the mean value of the solution. The larger this value is, the better the distribution of the solution.

During the solution processes, $\mathsf{\Delta }IGD$ and $\mathsf{\Delta }DV$, there are four combinations, or four states of agent: state I: $\mathsf{\Delta }IGD>0 and \mathsf{\Delta }DV>0$; state II: $\mathsf{\Delta }IGD>0 and \mathsf{\Delta }DV\le 0$; state III: $\mathsf{\Delta }IGD\le 0 and \mathsf{\Delta }DV>0$; and state IV: $\mathsf{\Delta }IGD\le 0 and \mathsf{\Delta }DV\le 0$.

3.5.4. Definition of Incentives

Reward definition: In order to guide the algorithm to choose parameters to enhance the distributivity of the frontier surface, when the PF is updated, if the $\mathsf{\Delta }DV>0$, this means that the DV index becomes larger then the distributivity of the neighboring solutions is more uniform. As the distributivity is better, a reward is given. $\mathsf{\Delta }IGD>0$ indicates that the convergence and diversity of the algorithm is better; the following reward values are set for comprehensive consideration:

$$R=\left\{\begin{array}{c}10 \Delta IGD>0 and \Delta DV>0\\ 10 \Delta IGD\le 0 and \Delta DV>0\\ 2 \Delta IGD>0 and \Delta DV\le 0\\ 0 \Delta IGD\le 0 and \Delta DV\le 0\end{array}\right.$$

(15)

3.6. Algorithmic Process

The improved MOEA/D based on Q-learning (Q-MOEA/D) solves the MO-FJSP; the pseudo-code framework of the algorithm is shown in Algorithm 1.

The specific steps of the algorithm are as follows:

Step 1: Input the information regarding the workpieces and equipment. Initialize the algorithm parameters: the proportion of individuals with different generation methods in the initial population, maximum number of iterations, population size, initial crossover probability, mutation probability, learning rate, discount factor, ε-greedy algorithm parameters, local search probability, and MOEA/D neighborhood range. Initialize weight vectors, neighborhoods, and the reinforcement-learning Q-table.

Step 2: Generate the initial population. The individual encoding includes two parts: job scheduling and machine selection. Generate the corresponding machining time for each machine separately. Calculate the objective function values for the initial population. Initialize the reference points, where the reference points are formed by taking the minimum values of each sub-objective function across all individuals.

Step 3: Initialize the $IGD$ (inverted generational distance) and $DV$ (diversity value). Determine the current state based on $\mathsf{\Delta }IGD$ and $\mathsf{\Delta }DV$. Use an ε-greedy strategy to select an action within the MOEA/D neighborhood range.

Step 4: For each individual in the current population $Pt$, perform crossover and mutation operations to generate an offspring population $Ct$. Calculate the objective function values for the offspring population $Ct$ and update the neighborhoods. Merge the parent population $Pt$ and the offspring population $Ct$ to form the new-generation population. Extract the non-dominant solutions from the current population.

Step 5: Calculate the convergence and diversity indicators ($\mathsf{\Delta }IGD$ and $\mathsf{\Delta }DV$) for the current population solutions. Determine the next state and select the action with the maximum reward. Update the Q-table according to the formula (10). Check if the maximum number of iterations is reached. If yes, end the iteration; otherwise, go back to Step 3.

Algorithm 1 Q-MOEA/D

Input: population number $pop-num$,learning rate $\alpha$,exploration rate $ϵ$, discount factor $\gamma$, number of iterations N Output: Nondominated solutions(PF) 1: Initialize $Qtable$,population,weights,neighbor 2: for$i\leftarrow 1$Ndo 3:    $IG{D}_i=IG{D}_{i+1}$,$D{V}_i=D{V}_{i+1}$ 4:    $S\leftarrow (\Delta IGD,\Delta DV)$ 5:    if random number $<ϵ$ then 6:      Select a random action A for scheduling 7:    else 8:      Select action A with maximum Q value for state S 9:    end if 10:    for $j\leftarrow 1$ to $pop-num$ do 11:      Select mating pool from the neighborhood of $x_i$ 12:      Apply crossover with probability $P_c$ 13:      Apply mutation with probability $P_m$ 14:      Evaluate offspring solutions using decomposition 15:      Update neighborhood with better solutions 16:    end for 17:    $P{F}_{known}\leftarrow P{F}_i$ 18:    $IG{D}_{i+1}\leftarrow IGD\left(P{F}_{known}\right),D{V}_{i+1}\leftarrow DV\left(P{F}_{known}\right)$ 19:    $\Delta IGD=IG{D}_i-IG{D}_{i+1}$,$\Delta DV=D{V}_i-D{V}_{i+1}$ 20:    $S^{\prime }\leftarrow (\Delta IGD,\Delta DV)$ 21:    Update Q value for $(S,A)$ using Q-learning update rule: 22:       $Q(S,A)\leftarrow Q(S,A)+\alpha [R+\gamma {max}_{T^{\prime }}Q(S^{\prime },A^{\prime })-Q(S,A)]$ 23: end for

4. Simulation Experiment Analysis

4.1. Experimental Design

In order to test the performance of this paper’s algorithm, the Q-MOEA/D, in solving the MO-FJSP considering energy consumption, this paper uses two sets of benchmark algorithms with different characteristics, namely the Kacem algorithms [17] (Kacem01–Kacem04) and BRdata algorithms [18] (mk01–mk10). A total of 14 examples are used in this paper. In the given sets of test cases, the Kacem dataset represents a scenario of complete flexibility in scheduling, while the BRdata dataset represents a scenario of partial flexibility in scheduling. These two sets of test cases include different scales of production environments to evaluate the performance of the algorithm in handling small-, medium-, and large-scale instances. The test cases contain information about the machines available for each operation and their respective processing times ($\min$). The energy consumption information is randomly generated, where the machine’s unit processing energy consumption ${\displaystyle \frac{\mathrm{KJ}}{\min }}$s randomly generated within the range of $\left[0.5,2\right]$, and the idle energy consumption ${\displaystyle \frac{\mathrm{KJ}}{\min }}$s randomly generated within the range of $\left[0.1,0.3\right]$.

The experiment was conducted on a 64-bit Windows 11 Home Edition operating system with an AMD Ryzen 7–5800H processor and Radeon Graphics, running at a frequency of 3.20 GHz and 16 GB of memory. The experimental environment used was MATLAB 2023a. To ensure the validity of the comparisons, all algorithms were independently run 20 times in each instance. In order to evaluate the convergence and diversity of the proposed algorithm, the average results were collected and compared for performance after running independently 20 times.

4.2. Experimental Parameters

The Q-MOEA/D algorithm contains four important parameters: population size pop-num, discount factor γ, learning rate α, and greed factor ε. In order to evaluate the effect of each parameter on the algorithm to obtain the optimal parameter combination, this paper uses Taguchi’s test based on the mk01 algorithm. The settings of each parameter are shown in

Table 1. Parameterization.

Parameters	Rank
	1	2	3	4
pop-num	70	80	90	100
γ	0.8	0.85	0.9	0.95
α	0.1	0.15	0.2	0.25
ε	0.8	0.85	0.9	0.95

.

According to the number of parameters and factor levels, an orthogonal experiment of size L₁₆ (4⁴) is chosen. That is, the number of trials is 16, the parameters are 4, and the factor level is 4. Setting the maximum number of iteration generations as 100, the algorithm is run independently 20 times under each parameter combination, and the HV mean of the 20 results is calculated; the test results are shown in

Table 2. Orthogonal table and HV mean value.

Parameter Combinations	Rank				HV Mean Value
	Pop-Num	γ	α	ε
1	70	0.8	0.1	0.8	1.2063
2	70	0.85	0.15	0.85	1.2083
3	70	0.9	0.2	0.9	1.2063
4	70	0.95	0.25	0.95	1.2044
5	80	0.8	0.15	0.9	1.2056
6	80	0.85	0.1	0.95	1.2083
7	80	0.9	0.25	0.8	1.2058
8	80	0.95	0.2	0.85	1.2038
9	90	0.8	0.2	0.95	1.2071
10	90	0.85	0.25	0.9	1.2071
11	90	0.9	0.1	0.85	1.2069
12	90	0.95	0.15	0.8	1.2075
13	100	0.8	0.25	0.85	1.2077
14	100	0.85	0.2	0.8	1.2078
15	100	0.9	0.15	0.95	1.2067
16	100	0.95	0.1	0.9	1.2061

. From comprehensive observation, it can be seen the optimal configuration for setting parameter values is pop-num = 100, γ = 0.8, α = 0.1, ε = 0.9.

4.3. Algorithm Comparison Test

To validate the effectiveness of the proposed algorithm, the comparative algorithms chosen are the improved decomposed-based multi-objective evolutionary algorithm IMOEA/D [19] and the dominance-based multi-objective optimization algorithm NSGA-II [20]. In order to analyze the distribution and convergence of the solution sets obtained by the proposed algorithm and the comparative algorithms, the hypervolume metric (HV) formula [21], the Spread, and the inverted generational distance (IGD) were determined. The IGD metric has been explained in Section 3.5.3 of this paper. HV computes the volume of the hypercube constituted between the solution set and the reference point with the following formula:

$$\mathit{HV}(X,P)=\bigcup _{x\in X}^{X}v(x,P)$$

(16)

where X is the set of non-dominated solutions obtained by the algorithm; $P$ is the reference point of the solution set, setting the coordinates of $P$ to (1.1,1.1); x is the normalized solution; and v denotes the volume of the hypercube surrounded by all x in the solution set and the reference point P.

$$\begin{array}{c}\mathit{Spread}(P,P^{*})=\frac{{\sum }_{i=1}^{\left|P^{*}\right|}d(P,P_i^{*})+{\sum }_{X\in P}|d(X,P)-\overline{d}|}{{\sum }_{i=1}^{\left|P^{*}\right|}d(P,P_i^{*})+(|P|-|P^{*}|)\overline{d}},\hfill \\ d(X,P)=\underset{Y\in P,Y\ne X}{\mathit{min}}\Vert F(X|)-F(Y)\Vert ,\hfill \\ \overline{d}=\frac{1}{|P|}{\displaystyle \sum _{X\in P}}d(X,P),\hfill \end{array}$$

(17)

where P* denotes the true frontier; since the true frontier cannot be obtained in advance in the shop floor scheduling problem, P* is replaced by the set of non-dominated solutions obtained by all the comparison algorithms and focused on the non-dominated solution set. The Euclidean distance is denoted by d. The smaller the value of the Spread, the more uniformly the algorithms obtain the distribution of the frontiers, which indicates greater distributivity.

The HV, Spread, and IGD values for the algorithm comparison are presented in

Table 3. HV, Spread, and IGD results for comparison with other algorithms.

Instance	Q-MOEA/D			IMOEA/D			NSGA-II
	HV	IGD	Spread	HV	IGD	Spread	HV	IGD	Spread
Kacem01	0.8851	0.01055	0.9125	0.82209	0.093395	0.9435	0.88631	0.057375	1.0145
Kacem02	1.1744	0	0.9356	0.51563	0.57449	1.02	0.53425	0.51616	0.9527
Kacem03	1.0893	0.00592	0.9611	0.65483	0.41631	0.9478	0.43792	0.50712	0.9763
Kacem04	1.1759	0	0.9136	0.91573	0.24258	0.9235	0.051462	1.0485	0.9488
mk01	0.95688	0.0056	1.0017	0.42235	0.45995	0.9853	0.88897	0.066392	0.9784
mk02	1.032	0	0.9978	0.21576	0.67148	0.9874	0.55505	0.3377	1.0026
mk03	1.1986	0	1.0048	0.11574	1.1134	1.0112	0.23589	0.87935	1.0245
mk04	1.0254	0	0.9564	0.78535	0.20557	0.9744	0.69446	0.23298	0.9845
mk05	1.1317	0	0.9741	0.27731	0.63545	0.9878	0.38293	0.56656	1.0000
mk06	1.1963	0	0.9023	0.11091	0.97606	0.9456	0.22955	0.74107	0.9231
mk07	1.004	0	0.9954	0.52767	0.29211	0.9648	0.58824	0.2422	0.9784
mk08	1.0741	0.12495	0.9969	0.18562	1.0.02	0.9799	1.21	0	1.0374
mk09	1.1185	0	1.0016	0.045981	1.0155	0.9956	0.42816	0.43823	1.0212
mk10	1.1847	0	0.9786	0.040788	1.0989	0.9836	0.48734	0.50705	1.0001

. The bold text indicates the optimal average HV, average Spread, and average IGD values for the current algorithm, which correspond to the largest HV value, the smallest IGD value, and the smallest Spread value. According to the analysis of the results in Table 3, the Q-MOEA/D algorithm outperforms the other algorithms in most of the arithmetic cases. Figure 5 shows the comparison of PF solutions with three different sizes of Kacem04, mk02, and mk05 selected for the arithmetic cases. Due to the Q-MOEA/D algorithm being combined with reinforcement learning, the obtained solutions have good convergence and distribution, which are significantly better than the other two algorithms. Figure 6 shows three Gantt charts representing the total completion time and total energy consumption for the Kacem04, mk02, and mk05 arithmetic cases, respectively.

Based on the analysis presented in Table 3, it is evident that the Q-MOEA/D algorithm has achieved superior results in terms of the average hypervolume (HV) and inverted generational distance (IGD) of the solution sets across various standard test cases. It fell short of achieving optimality only in the Kacem01 and mk08 test cases. Overall, the Q-MOEA/D algorithm has exhibited commendable diversity and distribution within its solution sets. Further examination of Figure 5 and Figure 6 reveals that the Q-MOEA/D algorithm has demonstrated optimal convergence of the non-dominated solution set and exhibited a well-distributed presence within the solution space.

5. Engineering Case

DF Automotive’s engine plant produces four parts, namely the adapter plate bracket, fan bracket, flange, and adapter flange, for the engine cooling system, and a schematic diagram of the engine cooling system is shown in Figure 7 [22]. The parts in the boxes are the production parts of the company.

There are eight machines in the factory shop for machining these parts, i.e., two milling machines (M1, M2), two drilling machines (M3, M4), one machining center (M5), one manual lathe (M6), and two computer numerical control (CNC) lathes (M7, M8), with the specific parameters of the machining process as shown in

Table 4. Parameters required for processing.

Job	Operation	Processing Time (min)/Energy Consumption (KJ)
		Miller 1	Miller 2	Drilling Machine 1	Drilling Machine 2	Machining Centre 1	Manual Lathe 1	CNC1	CNC2
Adapter plate bracket	1–1	8/1.20	8/1.2	–	–	6/1.20	–	–	–
	1–2	12/1.80	12/1.80	–	–	9/1.80	–	–	–
	1–3	6.0/0.90	6.0/0.90	–	–	7.0/1.40	–	–	–
	1–4	9.0/1.35	9.0/1.35	–	–	4.0/0.80	–	–	–
	1–5	3.0/0.45	3.0/0.45	–	–	2.7/0.54	–	–	–
	1–6	–	–	4.0/1.00	4.0/1.00	3.0/0.60	–	–	–
	1–7	–	–	3.0/0.75	3.0/0.75	2.0/0.40	–	–	–
	1–8	–	–	3.0/0.75	3.0/0.75	2.0/0.40	–	–	–
	1–9	–	–	5.0/1.25	5.0/1.25	3.8/0.76	–	–	–
	1–10	–	–	4.0/1.00	4.0/1.00	2.2/0.44	–	–	–
	1–11	–	–	6.0/1.50	6.0/1.50	4.3/0.86	–	–	–
	1–12	4.2/0.63	4.2/0.63	–	–	3.7/0.74	–	–	–
	1–13	2.0/03	2.0/0.30	–	–	1.8/0.36	–	–	–
Fan bracket	2–1	3.0/0.45	3.0/0.45	–	–	2.0/0.40	–	–	–
	2–2	7.0/1.05	7.0/1.05	–	–	1.8/0.36	–	–	–
	2–3	5.0/0.75	5.0/0.75	–	–	2.5/0.50	–	–	–
	2–4	3.0/0.45	3.0/0.45	–	–	1.8/0.36	–	–	–
	2–5	3.0/0.45	3.0/0.45	–	–	1.8/0.36	–	–	–
	2–6	–	–	3.0/0.75	3.0/0.75	1.8/0.36	–	–	–
	2–7	–	–	2.0/0.50	2.0/0.50	1.0/0.20	–	–	–
	2–8	–	–	2.0/0.50	2.0/0.50	1.0/0.20	–	–	–
	2–9	–	–	–	–	5.0/1.00	–	2.0/0.67	2.0/0.67
	2–10	–	–	–	–	3.2/0.64	–	1.0/0.33	1.0/0.33
	2–11	–	–	–	–	2.0/0.40	–	1.0/0.33	1.0/0.33
	2–12	–	–	3.0/0.75	3.0/0.75	2.2/0.44	–	–	–
	2–13	–	–	3.0/0.75	3.0/0.75	2.0/0.40	–	–	–
	2–14	5.0/0.75	5.0/0.75	–	–	1.8/0.36	–	–	–
	2–15	3.0/0.45	3.0/0.45	–	–	1.5/0.30	–	–	–
Fan hub	3–1	–	–	–	–	–	1.0/0.30	–	–
	3–2	–	–	–	–	–	0.5/0.15	–	–
	3–3	–	–	–	–	–	2.5/0.75	–	–
	3–4	–	–	–	–	–	0.5/0.15	–	–
	3–5	–	–	–	–	–	–	2.0/0.67	2.0/0.67
	3–6	–	–	–	–	–	–	1.0/0.33	1.0/0.33
	3–7	–	–	–	–	–	–	1.0/0.33	1.0/0.33
	3–8	–	–	–	–	–	–	1.5/0.50	1.5/0.50
	3–9	–	–	–	–	–	–	1.0/0.33	1.0/0.33
	3–10	–	–	–	–	–	–	3.0/1.00	3.0/1.00
	3–11	–	–	–	–	–	–	2.0/0.67	2.0/0.67
	3–12	–	–	–	–	–	–	2.0/0.67	2.0/0.67
	3–13	–	–	–	–	–	–	2.0/0.67	2.0/0.67
	3–14	–	–	–	–	–	–	1.2/0.40	1.2/0.40
	3–15	–	–	–	1.0/0.25	1.0/0.2	–	–	–
Adapter flange	4–1	–	–	–	–	–	1.5/0.45	–	–
	4–2	–	–	–	–	–	3.5/1.05	–	–
	4–3	–	–	–	–	–	–	3.0/1.00	3.0/1.00
	4–4	–	–	–	–	–	–	2.0/0.67	2.0/0.67
	4–5	–	–	–	–	–	–	0.8/0.27	0.8/0.27
	4–6	–	–	–	–	–	–	0.5/0.17	0.5/0.17
	4–7	–	–	–	–	–	–	0.5/0.17	0.5/0.17
	4–8	–	–	–	–	–	–	0.5/0.17	0.5/0.17
	4–9	–	–	–	–	–	–	0.5/0.17	0.5/0.17
	4–10	–	–	–	–	–	–	0.5/0.17	0.5/0.17
	4–11	–	–	–	–	–	–	2.2/0.73	2.2/0.73
	4–12	–	–	–	–	–	–	1.5/0.50	1.5/0.50
	4–13	–	–	2.0/0.50	2.0/0.50	1.5/0.30	–	–	–
	4–14	–	–	1.0/0.25	1.0/0.25	1.0/0.20	–	–	–
Machine no-load energy consumption (KJ/min)		0.1	0.1	0.15	0.15	0.13	0.21	0.23	0.23

.

Using the same experimental environment and algorithm parameter settings as in Section 4 of this paper, the Q-MOEA/D is used with the IMOEA/D and NSGA-II to solve the production example, and each algorithm is run 20 times to obtain the average value; the results of the production example are shown in

Table 5. Production example solution results.

Algorithm	Maximum Completion Time (min)	Total Energy Consumption (KJ)
Q-MOEA/D	51.5	3.8621
IMOEA/D	52.2	4.144
NSGA-II	53	4.1572

.

As can be seen from Table 5, when the Q-MOEA/D is applied to the production case, the solution results of the two indicators are better than those of several other algorithms. Since both the Q-MOEA/D and IMOEA/D adopt a mixed initialization population with multi-heuristic strategies, they can reasonably balance the machine energy consumption, improve the machine utilization rate, and reduce the waste of ineffective energy consumption. The Gantt chart of the scheduling results of the Q-MOEA/D solving this production instance is plotted, and the scheduling Gantt chart of the production instance is shown in Figure 8.

As can be seen in Figure 8, drilling machine 2 (M4) is not used throughout the whole process, while the machining center (M5) is occupied almost all the time, which reflects the special characteristics of the actual production environment; the machining center (M5) is preferred many times because of its versatility and high machining efficiency, and drilling machine 2 (M4) does not have any machining advantages compared with drilling machine 1 (M3), and it is therefore discarded in the current round of scheduling. In summary, the Q-MOEA/D can be validated in real machining environments, balancing production capacity while optimizing productivity, and achieving energy-efficient scheduling for flexible job machining.

6. Summary

This study presents an enhanced multi-objective evolutionary algorithm with decomposition (MOEA/D) integrated with reinforcement learning (RL) for energy-efficient scheduling in flexible job shops. The proposed method has demonstrated promising results in balancing production efficiency and energy consumption. In this section, we discuss the feasibility of our approach in real-time scheduling scenarios and potential areas for future enhancement.

6.1. Feasibility for Real-Time Scheduling Scenarios

The integration of the MOEA/D with RL has shown potential for real-time scheduling applications. The adaptive nature of RL allows the algorithm to respond quickly to changes in job shop conditions, such as machine breakdowns or sudden changes in job orders. However, the computational complexity of the algorithm could pose challenges in real-time environments where rapid decision-making is crucial.

To address this, we propose further research into optimizing the computational efficiency of the algorithm. This could involve developing more-efficient state representation schemes, exploring parallel computing techniques, or implementing incremental learning methods that require less-frequent updates to the model. Additionally, the development of a hybrid system that combines the strengths of our approach with other real-time scheduling methods could be beneficial.

6.2. Potential Areas for Further Enhancement

While our method has shown effectiveness in optimizing energy consumption and production efficiency, there are several areas where further enhancement could be pursued:

Dynamic Scheduling Conditions: Future work could focus on enhancing the algorithm’s capability to handle dynamic scheduling conditions, such as real-time job arrivals and machine availability changes.

Integration with IoT and Big Data: With the advent of Industry 4.0, integrating our method with Internet of Things (IoT) devices and leveraging big data analytics could provide more accurate and real-time data for decision-making.

Multi-Criteria Decision-Making: Expanding the algorithm to include additional objectives, such as cost minimization, quality optimization, or risk management, could provide a more comprehensive optimization solution.

In conclusion, the proposed MOEA/D-RL approach has demonstrated significant potential for energy-efficient scheduling in flexible job shops. While the method shows promise, continuous research and development are necessary to enhance its capabilities and expand its applicability in real-time and dynamic scheduling scenarios.