**2. Problem Description**

The dynamic scheduling problem of the hybrid flow shop can be described as follows: *n* dynamically arriving jobs *Ji*(*i* = 1, 2, . . . , *n*) are processed continuously in *K*(*K* ≥ 2) stages, stage *k* has *mk*(*mk* ≥ 1; *k* = 1, 2, . . . , *K*) machines, and there exists at least one stage with multiple machines. Flexible preventive maintenance of the machine is considered

**Citation:** Zhang, J.; Chen, Y.; Mumtaz, J.; Zhou, S. Neuro-Evolution of Augmenting Topologies for Dynamic Scheduling of Hybrid Flow Shop Problem. *Eng. Proc.* **2023**, *45*, 25. https://doi.org/ 10.3390/engproc2023045025

Academic Editors: Mohammad Javed Hyder, Muhammad Mahabat Khan, Muhammad Irfan and Manzar Masud

Published: 11 September 2023

School of Mechanical and Electrical Engineering, Wenzhou University, Wenzhou 325035, China; zhangjunjie6232@163.com (J.Z.); zhousw\_zsw@126.com (S.Z.)

in accordance with production reality, i.e., the machine cannot be continuously used for more than the maintenance threshold *UT*, and the maintenance time is *t <sup>m</sup>*. The scheduling objective is to minimize the makespan. The decision moment is triggered by the job being processed at a certain stage or by the arrival of a new job, which is then scheduled to the machine to optimize the objective value.

The problem studied in this paper considering the following assumptions: (1) Each job can be processed on any one of the machines at stage *k*. (2) Job processing time is the same for all machines at the same stage. (3) A machine can only process one job at a time. (4) All jobs are processed according to the same process sequence. (5) After determining the sequence of processing of the jobs in the first stage, the other stages are processed in this order.

#### **3. Dynamic Scheduling Method for Hybrid Flow Shop Based on NEAT Algorithm**

The interaction flow of the hybrid flow shop dynamic scheduling system based on reinforcement learning is shown in Figure 1.

**Figure 1.** Interaction flow of hybrid flow shop system based on reinforcement learning.

### *3.1. State Space*

Setting a suitable state space can reflect the change in the scheduling system in real time. For the characteristics of the dynamic scheduling problem of the hybrid flow shop, the environment is described in three dimensions (jobs, machines and stages) and the state space vector *S* = *pik*, *T<sup>r</sup> <sup>i</sup>* , *Tkc*, *<sup>T</sup><sup>a</sup> kc*, *<sup>U</sup><sup>r</sup> kc* is defined. *pik* is the processing time corresponding to job *Ji* at stage *k*. *T<sup>r</sup> <sup>i</sup>* is the time interval between the current moment and the arrival time of job *Ji*. *Tkc* is the processing time of the job being processed by machine *Mkc*(*c* = 1, 2, . . . , *mk*) at stage *k*, increasing the maintenance time *t <sup>m</sup>* if maintenance is required. *T<sup>a</sup> kc* is the processing time of the job being processed by machine *Mkc* at stage *k*. *U<sup>r</sup> kc* is the maintenance threshold remaining at the current moment for machine *Mkc* at stage *k*.

### *3.2. Action Space*

In the shop scheduling model, an action *at* ∈ *At* is taken to determine a machine *Mkc* for processing job *Ji* at decision time *t* in state *St*.Based on the characteristics and objectives of the dynamic scheduling problem of a hybrid flow shop considering flexible preventive maintenance, the following three actions are designed.

Action 1: Selecting a job based on the SPT rule. The job *Ji* with the shortest processing time is selected from the waiting queue based on the SPT rule. One of the jobs *Ji* is selected at random if more than one job can be selected.

Action 2: Selecting a job based on FIFO rule. The job *Ji* with the earliest arrival time is selected from the waiting queue based on the FIFO rule. One of the jobs *Ji* is selected at random if more than one job can be selected.

Action 3: Waiting. No job is selected for machining.

The following constraints are set for several special cases: (1) When there are no jobs in the waiting queue, the agent can only select action 3. (2) When a new job arrives but there are no idle machines, action 3 is selected. (3) When there are jobs in the waiting queue and the machines are all idle, action 3 cannot be selected.

#### *3.3. Fitness Function*

The NEAT algorithm searches the optimal scheduling strategy through the optimization of the fitness function, so the fitness function of NEAT is designed to be *G* = <sup>1</sup> *<sup>C</sup>*max . Maximizing the fitness function is equal to minimizing the makespan. *C*max can be described using Equation (1):

$$\mathbb{C}\_{\text{max}} = \max\_{i} \mathbb{C}\_{i} \tag{1}$$

where *Ci* is the completion time of all stages of processing of job *Ji*.

#### **4. Numerical Example and Analysis**

#### *4.1. Parameter Settings*

According to the pre-experimentation, we set the evolutionary generations in the NEAT algorithm to 10 and the population size to 100. The job processing time *pik*, the arrival time *ri* and the number of machines *mk* at each stage were generated based on a uniform distribution. The problem was divided into three instances according to the number of stages *K* and the number of jobs n. Each type of instance was run three times for the experiments, and the parameters of the training set are shown in Table 1.


**Table 1.** The range of parameter values for the training set problem.

#### *4.2. Results Analysis*

In this paper, the performance of the NEAT scheduling method and the two scheduling rules are evaluated based on the makespan *C*max, running time *T<sup>R</sup> <sup>t</sup>* and response time *T<sup>R</sup> s* . The experimental results of the different algorithms are shown in Table 2.

**Table 2.** Experimental results of performance indicators for different scheduling methods.


It can be seen from Table 2 that the NEAT-based approach outperforms the scheduling rules for all three instances with the objective *C*max. For the computation time, the SPT and FIFO rules have less time than the NEAT-based approach, because the NEAT reinforcement learning algorithm requires decisions to be made through interaction with the environment, while it is also able to respond the dynamic events quickly.

#### **5. Conclusions**

A NEAT-based approach is proposed for a dynamic scheduling problem of hybrid flow shops with machine preventive maintenance and the dynamic arrival of jobs to minimize maximum completion time. Our experimental results show that the NEAT algorithm is able to obtain better objective values while responding quickly. In future research, the proposed NEAT method can be used to solve dynamic scheduling problems considering some more realistic emergency events such as the insertion and withdrawal of orders.

**Author Contributions:** Conceptualization, J.Z. and Y.C.; methodology, J.Z. and S.Z.; software, J.Z. and J.M.; validation, J.Z., and J.M.; formal analysis, J.Z. and Y.C.; writing—review and editing, Y.C. and S.Z.; visualization, S.Z.; supervision, J.Z.; funding acquisition, Y.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China, grant number [No.51705370].

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data that support the findings of this study are available from the corresponding author upon reasonable request.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

