Dynamic Scheduling Method for Job-Shop Manufacturing Systems by Deep Reinforcement Learning with Proximal Policy Optimization

Abstract

With the rapid development of Industrial 4.0, the modern manufacturing system has been experiencing profoundly digital transformation. The development of new technologies helps to improve the efficiency of production and the quality of products. However, for the increasingly complex production systems, operational decision making encounters more challenges in terms of having sustainable manufacturing to satisfy customers and markets’ rapidly changing demands. Nowadays, rule-based heuristic approaches are widely used for scheduling management in production systems, which, however, significantly depends on the expert domain knowledge. In this way, the efficiency of decision making could not be guaranteed nor meet the dynamic scheduling requirement in the job-shop manufacturing environment. In this study, we propose using deep reinforcement learning (DRL) methods to tackle the dynamic scheduling problem in the job-shop manufacturing system with unexpected machine failure. The proximal policy optimization (PPO) algorithm was used in the DRL framework to accelerate the learning process and improve performance. The proposed method was testified within a real-world dynamic production environment, and it performs better compared with the state-of-the-art methods.

1. Introduction

With the rapid development of automation and digitization, Industry 4.0 signifies a remarkable shift in industrial revolution [1,2]. As a practical application of manufacturing in the Industry 4.0 era, smart factories are presented through the leverage of new technologies, including the Internet of Things (IoTs) and artificial intelligence (AI) [3,4]. Through the full interconnection between the digital and the physical world, smart factories can realize sustainable development that significantly improves productivity and quality [5]. Sustainability has been proposed as the economic indicator to evaluate sustainable practices in the manufacturing sector [6,7,8]. In today’s globalized market, manufacturing industries need to deploy strategy, demanding that it could continuously maintain high efficiency, competitiveness, and sustainability [9]. There are several factors that could affect the sustainability of manufactured products, such as the supplement of raw materials, stability and efficiency of production processes, and distribution capability [10].

As the smart production system becomes increasingly complicated, operation decision making has become more and more important for keeping sustainable working and production efficiency [11,12]. Many factors can seriously affect the utilization and sustainability of production lines. For instance, the rapidly changing market needs and unexpected machine failures can significantly reduce the efficiency and performance of the production system [13]. For the complex job-shop manufacturing system, the job-shop scheduling problem becomes one of the most important production scheduling problems in smart manufacturing areas. The scheduling optimization approaches can improve the sustainable production of manufacturing enterprises as well as their market competitiveness [14,15].

The job-shop scheduling problem is a combinatorial optimization problem related to scheduling and production control, which aims to identify the optimal sequential and process order of the jobs in the manufacturing industry [16,17]. It can be analyzed as a nondeterministic polynomial hard (NP-hard) problem when there are two or more machines involved [18]. Generally, heuristic-based methods are the main way to solve job-shop scheduling, including linear programming approaches [19], simulated annealing algorithm [20], genetic algorithm [21], particle swarm optimization [22], and teaching–learning-based optimization [23]. These approaches can obtain the optimal solution under certain conditions. However, they do not have the ability to deal with the dynamic job-shop environment with uncertainties [24]. Combining advanced AI and data-driven approaches, the dynamic job-shop scheduling methods are proposed to handle the uncertain factors and dynamic environments of smart manufacturing, which allow both real-time production scheduling and dynamic adaptive production scheduling [25,26,27]. Nowadays, more and more data-driven AI methods have been proposed to tackle the dynamic scheduling problem for job-shop manufacturing systems [28,29,30].

The dynamic job-shop scheduling problem is difficult to solve by the typical heuristics method due to the existing dynamic events such as random ordering arrivals and unexpected machine breakdowns. To deal with this issue, the deep reinforcement learning (DRL) method was proposed to combine deep learning and reinforcement learning methods [31,32,33,34]. DRL can formulate scheduling strategies based on real-time production data and optimize strategies according to the dynamic changes of the system states. In the actual scenarios of smart manufacturing, the job-shop environment can apply sensory technologies to detect and collect large amounts of production process data. Through processing and training the real-time data of the job-shop environment states, valuable scheduling information and policy can be learned for dynamic optimizing the scheduling. Zhao et al. proposed the deep Q-network (DQN) to improve the performance of the adaptive scheduling algorithm in dynamic smart manufacturing [35]. Wang et al. [36] and Zeng et al. [37] proposed the dual Q-learning (D-Q) method as the solution of the dynamic job-shop scheduling problem. Luo et al. [38] proposed a two-hierarchy deep Q-network to deal with flexible job-shop scheduling problems with the disturbance of new jobs. The DRL method has become one of the dominant methods in dealing with the dynamic job-shop scheduling problem. However, all the mentioned DRL methods are constructed based on the designed rules, which means the learned agent is used to select the optimal scheduling rule, and they did not involve the machine failure situation. There are very few studies that could directly control the raw actions of agents to schedule the dynamic job-shop manufacturing system with unexpected machine failure.

To tackle these challenges, we propose a deep reinforcement learning framework under the proximal policy optimization (PPO) algorithm for addressing the dynamic scheduling problem of a designed job-shop manufacturing system with unexpected machine failure. Different from previous works, our method continuously outputs the raw actions that keep the production system sustainable, working and producing the products with high efficiency even under the unexpected machine failure situation. Compared with different policy gradient methods, we select the PPO algorithm that has a stable learning process and fast optimization speed. Different reward functions were designed and tested to guide the dispatcher agent in learning varying policies.

Preliminary knowledge of RL is detailed in Section 2. The proposed method and detailed specifications are described in Section 3. Section 4 contains the detailed experiment design as well as result analysis. Finally, the conclusions are given in Section 5.

2. Preliminary

In this section, we briefly describe the fundamental theory of policy gradient-based reinforcement learning, which we proposed to use for the dynamic job-shop scheduling problem.

2.1. Markov Decision Process

The Markov decision process (MDP) is the idealized mathematical formulation of reinforcement learning [39] with the control problem, which consists of a state space S, an action space A, a state transition probability distribution $p\left(s_{t+1}\right|s_t,a_t)$, and a reward function $r:S\times A\to R$. The RL aims at training an agent with policy to act in a certain environment and maximize the cumulative rewards coming from the selected actions across the sequence of time. The actions are determined based on the policy ${\pi }_{\theta }:S\to \mathcal{P}\left(A\right)$ in each time step, and a trajectory of states, actions, and rewards, $Tra{j}_{1:T}:\{(s_1,a_1,r_1),(s_2,a_2,r_2),\dots ,(s_T,a_T,r_T)\}$ from $S\times A\times R$ can be obtained by the agent using its policy to interact with the environment. The purpose of the agent is to learn a policy that can maximize the cumulative discounted rewards from the state–action pair:

$$J\left({\pi }_{\theta }\right)=\mathbb{E}\left[r_t\right|{\pi }_{\theta }],$$

(1)

where the return is the total discounted reward $r_t={\sum }_{i=t}^T{\gamma }^{i-t}r(s_i,a_i)$, and $\gamma$ is the discount factor.

2.2. Policy Gradient Theorem

For the continuous control problem of the reinforcement learning, we usually use the policy gradient method to maximize the expected total reward. The fundamental idea of this algorithm is to train the policy with the parameter $\theta$ by utilizing the stochastic gradient descent algorithm with the objective function:

$${\nabla }_{\theta }J\left({\pi }_{\theta }\right)=\mathbb{E}\left[{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t){\mathsf{\Psi }}_t\right],$$

(2)

where $\mathbb{E}$ represents the integration of the variables; the states $s_t$ and actions $a_t$ are sampled from the dynamics model $P\left(s_{t+1}\right|s_t,a_t)$ and policy $\pi \left(a_t\right|s_t)$, as is explained in the MDP; and ${\mathsf{\Psi }}_t$ denotes the sum of expected rewards which is the value function in practice. So, we can consider that the Equation (2) consists of the policy gradient and the cumulative rewards, and the direction of the gradient is significantly related to ${\mathsf{\Psi }}_t$. Previous studies [40] have been conducted to explore the approaches of approximating the total expected rewards ${\mathsf{\Psi }}_t$, and there are several formulations in the discounted way for variance reduction:

$$V^{\pi ,\gamma }\left(s_t\right)={\mathbb{E}}_{(s_{t+1:\infty },a_{t:\infty })}\left[\sum _{l=0}^{\infty }{\gamma }^l{r}_{t+l}\right],$$

(3)

$$Q^{\pi ,\gamma }(s_t,a_t)={\mathbb{E}}_{(s_{t+1:\infty },a_{t:\infty })}\left[\sum _{l=0}^{\infty }{\gamma }^l{r}_{t+l}\right],$$

(4)

$$A^{\pi ,\gamma }(s_t,a_t)=Q^{\pi ,\gamma }(s_t,a_t)-V^{\pi ,\gamma }\left(s_t\right).$$

(5)

The advantage function, $A^{\pi ,\gamma }(s_t,a_t)=Q^{\pi ,\gamma }(s_t,a_t)-V^{\pi ,\gamma }\left(s_t\right)$, is used for assessing whether the action is better or worse than the policy’s default behavior. Since the advantage function has the lowest variance in the process of policy promotion, ${\mathsf{\Psi }}_t$ should choose the advantage function $A^{\pi ,\gamma }(s_t,a_t)$, so that the objective function can provide the direction of advance ${\pi }_{\theta }(s_t,a_t)$ only if $A^{\pi ,\gamma }(s_t,a_t)>0$.

3. Proposed Methods

Figure 1 shows an overview of the proposed dynamic job-shop scheduling approach. This problem is a typical NP-hard problem and its computational complexity significantly increases with the number of elements in the environment [18,19]. In the production environment, the transport agents take the jobs or orders to the machines, then the production is generated from the machine in order to be transported to the sink waiting for delivery to the next processing station. The transport agents should determine the next actions based on the current state of the environment. The multiple objective functions should be considered to increase production efficiency, such as increasing the utilization of machines and reducing order waiting time and their combinations. The dynamic environment can be affected by certain random events, such as machine breakdown, resource cut off, etc. The dynamic scheduling should maintain the production system operating at high-efficiency conditions so that the stakeholders are able to save more costs when unexpected random events happen.

3.1. Dynamic Simulation of Production Environment

The simulation environment of job-shop manufacturing production constitutes the foundation of the proposed method. The agent is trained with a large amount of historical data, which are the outcome of interacting with the dynamic simulation [34]. The intelligent scheduling strategy was explored during the interaction procedure and recorded in the agent model.

As shown in Figure 1, the input of the simulated production system is ordered, and the entry resource is called Source. Machine is responsible for processing orders, then the disposed orders are transported to the resource called Sink. The critical resource called Dispatcher is going to transport the order between Source, Machine, and Sink. Therefore, there are three states for each order, including waiting, intransport, and inprocess. When the order is in the waiting state, it should be put in one of the buffers. There are entrance and exit buffers for each machine, which are used to receive undisposed orders and store the disposed of orders, respectively.

The constraints of the real-world production system are considered in this dynamic simulation environment, making it similar to the real-world application.

• Machines that have similar disposal ability of orders are put into the same group.
• Orders are released by the sources and they are performed according to specific probability.
• The processing time of each order is decided by the predefined probability distribution.
• Machine failures are considered in this environment, which can result in the breakdown of all of the machines. The failure events are random triggered based on the mean time between failure (MTBF) and mean time off-line (MTOL).

3.1.1. Action Module

The transport agent is the most important in the dynamic environment because it dispatches all orders from the source to machine and machine to sink. It is called the dispatcher in this work. Accordingly, there are three types of actions, i.e., $A_{waiting}$, $A_{S\to M}$, and $A_{M\to S}$, as shown in

Table 1. Action Type.

Action Type	Description
$A_{waiting}$	Dispatcher waits at its current position.
$A_{S\to M}$	Dispatcher takes an undisposed order from a source to a machine.
$A_{M\to S}$	Dispatcher takes a disposed order from a machine to a sink.

.

The actions of $A_{S\to M}$ and $S_{M\to S}$ belong to executable actions $A_{exec}$. At every time step, the dispatcher selects an action. However, not every $A_{exec}$ could be performed. It would be relying on the states of current production. For instance, an action $a_{S_i\to M_j}^t$ could be valid only if the source $S_i$ has a new order while the machine $M_j$ has a free buffer and they are working at normal function and capable of processing it. Therefore, the action could be determined as either valid $A_{valid}^t$ or invalid $A_{invalid}^t$ at each time step. If the transport agent successively selects the invalid action when it is repeated up to a maximum recursion count, the $A_{waiting}$ is performed.

3.1.2. States

The states are the current observations from the production environment, which are the outcome of previous actions. The agent can determine the next action based on the historical and current states. Ideally, the state contains all the information both related and unrelated to the production process. However, an excess of unrelated states can increase the dimension of solution space, which leads to a significant decline in the performance. Therefore, the states in this simulation environment are carefully designed and could synchronize with the real-world system. Each element in the states can be calculated at the current time t, and we neglect the t for better readability.

• Firstly, the state of action $S_{a{s}_i}$ shows whether the current action is valid or not, and it is defined as:

  $$S_{a{s}_i}=\left\{\begin{array}{cc}\hfill 1& a_i\in A_{valid}\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (6)
• The machine breakdown was designed in the simulation, and the state of failure for each machine $S_{m{f}_i}$ is also considered, which is defined as:

  $$S_{m{f}_i}=\left\{\begin{array}{cc}\hfill 1& if{M}_{i}hasafailure\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (7)
• The remaining processing time of each machine $M_i$ is defined as:

  $$S_{rp{t}_i}=\frac{T_{rp{t}_i}}{T_{ap{t}_i}}$$

  (8)

  where $T_{rp{t}_i}$ is the remaining processing time, and $T_{ap{t}_i}$ is the average processing time at $M_i$.
• $S_{be{n}_i}$ indicates the state information of remaining free buffer spaces of each machine $M_i$ in its entry buffer:

  $$S_{be{n}_i}=1-\frac{N_{oc{c}_i^{en}}}{N_{ca{p}_i^{en}}}$$

  (9)

  where $N_{oc{c}_i^{en}}$ is the number of occupied buffers, and $N_{ca{p}_i^{en}}$ is the capacity of the entry buffer.
• $S_{be{x}_i}$ indicates the remaining free buffer space in the exit buffer for each machine $M_i$:

  $$S_{be{x}_i}=1-\frac{N_{oc{c}_i^{ex}}}{N_{ca{p}_i^{ex}}}$$

  (10)

  where $N_{oc{c}_i^{ex}}$ is the number of occupied buffers, and $N_{ca{p}_i^{ex}}$ is the capacity of the exit buffer.
• $S_{w{t}_i}$ indicates the waiting times of orders waiting for transport:

  $$S_{w{t}_i}=\frac{T_{w{t}_i^{max}}-T_{w{t}_i^{mean}}}{T_{w{t}_i^{std}}}$$

  (11)

  where $T_{w{t}_i^{max}}$ is the longest waiting time, and $T_{w{t}_i^{mean}}$ and $T_{w{t}_i^{std}}$ are the average and standard variation of waiting time of orders.

3.2. Deep Reinforcement Learning for Dynamic Scheduling

The framework of deep reinforcement learning for the dynamic scheduling problem is shown in Figure 2. The initial dispatcher agent interacts with the simulation of the production environment randomly. Then, the inside policy of the agent is optimized using certain periods of trajectories including the state, action, and the reward of each time t. The intelligent policy is explored at the interaction procedure, and the deep neural network records and transfers them as the parameters. Following the foundation of MDP, the DRL method aims at obtaining a long period of rewards. It is obvious that the proposed method should be more suitable for the dynamic scheduling problem than the general method, such as FIFO and NJF [41], which make decisions depending on the current states.

3.2.1. Optimization Objectives

As shown in Figure 2, to continuously perform the expected actions that meet the designed requirements, the DRL agent need to receive the rewards at each time step. Following the MDP, it is obvious that the DRL agent not only considers the best action at present but also aims at acquiring good performance in the long-term period. As the multiple objective optimization, there are two objectives in the proposed production environment, including average utilization of the machines and average waiting time of orders.

• The constant reward $R_{const}$ rewards the valid action with value ${\omega }_1$ for $A_{S\to M}$, and ${\omega }_2$ for $A_{M\to S}$ is defined as:

  $$R_{const}(S_t,A_t)=\left\{\begin{array}{cc}\hfill {\omega }_1& A_t\in A_{S\to M}\hfill \\ \hfill {\omega }_2& A_t\in A_{M\to S}\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (12)
• To promote the average utilization U, $R_{uti}$ was designed with exponential function when the agent provides a valid action. The purpose of this reward function is to maximize utilization, and it is defined as:

  $$R_{uti}(S_t,A_t)=\left\{\begin{array}{cc}\hfill {exp}^{\frac{U}{1.5}}-1& A_t\in A_{valid}\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (13)
• To shorten the waiting time $WT$ of orders, $R_{wt}$ is designed to award the valid action determined by the agent. The reward function also follows the exponential function to accelerate the order leaving the system, which is defined as:

  $$R_{wt}(S_t,A_t)=\left\{\begin{array}{cc}\hfill {exp}^{-0.1WT}-0.5& A_t\in A_{valid}\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (14)
• Combining $R_{const}$ with $R_{uti}$ and $R_{wt}$, two complex reward functions are designed as follows:

  $$R_{\omega -uti}(S_t,A_t)=\left\{\begin{array}{cc}\hfill {\omega }_1{R}_{uti}(S_t,A_t)& A_t\in A_{S\to M}\hfill \\ \hfill {\omega }_2{R}_{uti}(S_t,A_t)& A_t\in A_{M\to S}\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (15)

  $$R_{\omega -wt}(S_t,A_t)=\left\{\begin{array}{cc}\hfill {\omega }_1{R}_{wt}(S_t,A_t)& A_t\in A_{S\to M}\hfill \\ \hfill {\omega }_2{R}_{wt}(S_t,A_t)& A_t\in A_{M\to S}\hfill \\ \hfill 0& else\hfill \end{array}\right.$$

  (16)
• For implementing the multiple-objective optimization, the hybrid reward function with $R_{uti}$ and $R_{wt}$ is defined as:

  $$R_{hybird}(S_t,A_t)=w_1{R}_{uti}+w_2{R}_{wt}$$

  (17)

3.2.2. Proximal Policy Optimization

We take advantage of the proximal policy optimization (PPO) algorithm [42] to implement the proposed deep reinforcement learning framework to tackle the dynamic scheduling problem in the production process system. The loss objective of the PPO algorithm is the surrogate item with a little change to the typical policy gradient (GP) algorithm. The implementation of the objective function is to construct the loss $L_t^{CLIP}$ to substitute $L^{PG}$, and then perform multiple steps of stochastic gradient descent on this objective. In this work, we adopt the clipped version of objective $L^{CLIP}$ which performs best in the comparison results. The objective is expressed as follows:

$$L^{CLIP}\left(\theta \right)=\widehat{\mathbb{E}}\left[min(r_t\left(\theta \right){\widehat{A}}_t,clip\left(r_t\left(\theta \right)\right),1-ϵ,1+ϵ){\widehat{A}}_t\right],$$

(18)

where $r_t\left(\theta \right)$ denotes the probability ratio $r_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{old}}\left(a_t\right|s_t)}$ and $ϵ$ is a hyperparameter. With the clipping constraint, the policy promotes or declines into a certain boundary, this can make the optimization of policy performance more stable and reliable.

There is a critical part in the policy gradient method, which is to estimate the variance-reduced advantage function $\widehat{A}$ by utilizing the learned state-value function $V\left(s\right)$. The generalized advantage estimation (GAE) algorithm [40] is one of the most effective approaches for policy gradient implementation, living on the policy running through T time steps and updating with the collected trajectory. The generalized advantage estimation function is defined as:

$${\widehat{A}}_t={\delta }_t+\left(\gamma \lambda \right){\delta }_{t+1}+\cdots +{\left(\gamma \lambda \right)}^{T-t+1}{\delta }_{T-1},$$

(19)

where ${\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)$ is the temporal-difference error.

The final objective of the PPO algorithm with fixed-length trajectory segments combines the surrogate policy loss item, the value function error item, and an entropy item for sufficient exploration, and it can be expressed as follows:

$$L_t^{CLIP+VF+S}={\widehat{\mathbb{E}}}_t[L_t^{CLIP}\left(\theta \right)-c_1{L}_t^{VF}\left(\phi \right)+c_{2}S\left[{\pi }_{\theta }\right]\left(s_t\right)],$$

(20)

where $L_t^{CLIP+VF+S}$ denotes the surrogate policy loss; $c_1$, $c_2$ are coefficients; S denotes an entropy for exploration; and $L_t^{VF}$ is a squared-error loss ${(V_{\phi }\left(s_t\right)-V^{targ}\left(s_t\right))}^2$. The pseudocode of the proposal method is summarized in Algorithm 1.

Algorithm 1 DRL with PPO for dynamic scheduling

1: Initialize the actor ${\pi }_{\theta }$ and the critic $V_{\phi }$ in global network 2: Initialize the dynamic production environment 3: for$episode=1,2,\dots$do 4:  for $t=1,2,\dots ,T$ do 5:   Determine action $a_t$ based on the current policy ${\pi }_{\theta }$ with state $s_t$ as input 6:   Obtain the target $V^{targ}\left(s_t\right)$ from the critic $V_{\phi }$ with state $s_t$ as input 7:   Act $a_t$ in environment to obtain the current reward $r_t$ and next state $s_{t+1}$ 8:   Save $(s_t,a_t,r_t,V^{targ}\left(s_t\right))$ in trajectory storage ${\mathcal{B}}_{\pi }$ 9:  end for 10:  Compute the TD error ${\delta }_t$ 11:  Compute the GAE advantage $\widehat{A}$ based on ${\delta }_t$ and save it in trajectory storage ${\mathcal{B}}_{\pi }$$(s,a,r,V^{targ},\widehat{A})$ 12:  for $k=1,\dots ,K$ do 13:   Sample mini-batch ${\left\{{(s_i,a_i,r_i,V_i^{targ},A_i)}^{\pi }\right\}}_{i=1}^m$ from ${\mathcal{B}}_{\pi }$ 14:   Update the actor parameters $\theta \leftarrow \theta -\alpha {\nabla }_{\theta }{L}_t^{CLIP+VF+S}$ 15:   Update the critic parameters $\phi \leftarrow \phi -\alpha {\nabla }_{\theta }{L}_t^{CLIP+VF+S}$ 16:  end for 17: end for

4. Experiments

4.1. Case Description

The simulation case is built based on a real-world wafer front-end fab [43], which is composed of three source entries ($S{E}_1,S{E}_2,$ and $S{E}_3$), eight machines ($M_1,\dots ,M_8$), and three sinks ($S_1,S_2,$ and $S_3$). The layout of the simulation environment is shown in Figure 3. There are three working areas including $W_1:\{S{E}_1,M_1,M_2,S_1\}$, $W_2:\{S{E}_2,M_3,M_4,M_5,S_2\}$, and $W_3:\{S{E}_3,M_6,M_7,M_8,S_3\}$. One dispatcher is used to transport the material or product from source to machine and from machine to sink, as described in Section 3.1.

4.2. Implementation Details

In this work, we proposed using the PPO algorithm as the default method for the DRL framework. The neural network with two hidden layers of 64 nodes and $tanh$ activation function can serve as the dispatcher, and the Adam optimizer is selected as the default optimizer for training the deep learning model. The parameters of PPO algorithm are shown in

Table 2. Parameter configuration of PPO algorithm.

Parameter	Value
Learning rate	0.001
Batch size	128
Epoch number	5
Gamma $\gamma$	0.9
Lamda $\lambda$	0.95
Clipping $ϵ$	0.01

.

In this work, we set up three scenarios for testing the proposed method and comparing it with other alternative methods. The setup parameters of the dynamic simulation environment are shown in

Table 3. Experiment setup parameters.

Parameter	Default Scenario	Scenario 1	Scenario 2
Dispatcher speed factor	1	0.3	1
Machine buffer factor	6	0.5	1
MTBF $\beta$	1000	1000	1000
MTOL $\beta$	200	200	200
${\omega }_1$	0.5	0.5	0.5
${\omega }_2$	0.5	0.5	0.5

. The changes are focused on the dispatcher speed factor and machine buffer factor. The mean time between failure (MTBF) and mean time off-line (MTOP) subject to exponential distribution which is defined as:

$$f(x;\frac{1}{\beta })=\frac{1}{\beta }ex{p}^{\left(-\frac{x}{\beta }\right)}$$

(21)

where $\beta$ is the expected value. The two parameters ${\omega }_1$ and ${\omega }_2$ of reward function are set to $0.5$ for the three scenarios shown in Table 3. Finally, the different combinations of ${\omega }_1$ and ${\omega }_2$ are tested to analyze their effect.

4.3. Results and Analysis

In order to testify the performance of the PPO algorithm [42], we compare it with two different methods including PG [44] and TRPO [45] under the default scenario. In this task, we select high dispatcher speed and very large machine buffer size so that the utilization of machines could be as high as possible. The overall learning process is shown in Figure 4. The results show that the DRL framework has the best performance by applying the PPO algorithm. It only takes $0.08\times {10}^6$ simulation steps to converge, and the average rewards could be over $0.9$. Meanwhile, the PG algorithm needs at least $0.2\times {10}^6$ steps and the TRPO needs $2\times {10}^6$. All average rewards are under 0.85. Although the learning speed of PG is reasonable, the learning process of PG is seriously fluctuated. It is difficult to determine the optimal policy in this way. The TRPO algorithm has a stable learning process, however, it takes too many steps to determine the optimal solution. It is 20 times slower than the PPO algorithm.

The parameters of the PPO algorithm are determined following the original research [42]. The clipping $ϵ$ is the most sensitive parameter because it controls the range of policy progress for each update. As shown in Figure 5, the $ϵ$ equal to $0.01$ obtains the best performance. The $ϵ=0.1$ and $ϵ=1.0$ results show they have a faster learning speed, but the average rewards could not reach the best average rewards. Meanwhile, the $ϵ=0.001$ result shows that the best performance could be reached with longer learning steps.

In order to demonstrate the performance and effectiveness of the proposed method, we test the optimal policy of the dispatcher under the same condition and compare it with a random policy. The Gantt chart of operation state is shown in Figure 6 and Figure 7. The dark blue represents the machines are working and the orange represents the dispatcher. The simulation environment is set to run 12 h with random policy and trained policy, respectively. Due to the existence of MTBF and MTOL, each machine breaks down, then needs a certain time period to recover. Therefore, one of the most important goals is to take full advantage of the machines when they become available. From the results, it is obvious that the learned policy could keep the machines continuously working, while it can be difficult to implement this goal with the random policy.

To further analyze the proposed method, two scenarios have been set up for deploying quantitative comparison. In the first scenario, the dispatcher has a relatively slow speed ($factor=0.3$) and the buffers of machines are relatively small ($factor=0.5$). On the contrary, the speed of the dispatcher is faster ($factor=1.0$) and the machines have larger buffers ($factor=1.0$). Meanwhile, we compare these with different rule-based policies including Random, FIFO, and NJF [41]. Different reward functions are carried out for deep analysis. The average utilization of machines U, the average waiting time of orders ($WT$), and the alpha value ($\alpha$) [46,47] are selected based on the evaluation criterion.

As shown in

Table 4. Results for different rule-based heuristic dispatching approaches in both production scenarios.

Heuristic	Scenario 1
	$U(\%)$	$WT\left(s\right)$	$\alpha$
Random	$38.93\pm 8.28$	$203.76\pm 54.76$	$5.21\pm 3.40$
FIFO	$46.15\pm 3.68$	$182.58\pm 17.44$	$2.94\pm 0.78$
NJF	$50.84\pm 5.29$	$196.48\pm 19.07$	$2.57\pm 0.87$
Heuristic	Scenario 2
	$U(\%)$	$WT\left(s\right)$	$\alpha$
Random	$54.86\pm 10.71$	$138.79\pm 57.54$	$1.57\pm 1.10$
FIFO	$70.72\pm 6.82$	$125.18\pm 22.51$	$0.48\pm 0.16$
NJF	$72.99\pm 7.35$	$125.68\pm 23.57$	$0.38\pm 0.11$

, the rule-based dispatching methods are compared with the random policy. Compared with random policy, both FIFO and NJF performed well. The NJF method is inclined to the utilization, while the FIFO method contributes more to waiting time. The utilization is affected by the transport speed of the dispatcher and the waiting time is affected by the size of the machine buffer. Therefore, compared with the first scenario, the U is higher and $WT$ is fewer under the second scenario. The alpha value $\alpha$ is a composite index combining multiple indicators, where a small value represents good performance.

According to the proposed DRL framework with the PPO algorithm, we test the performance in both production scenarios with different reward functions. In this experiment, we set two parameters ${\omega }_1$ and ${\omega }_2$ of all the reward functions as $0.5$. The results are shown in

Table 5. Results for PPO dispatching approaches under different reward function in both production scenarios.

PPO	Scenario 1
	$U(\%)$	$WT\left(s\right)$	$\alpha$
$R_{const}$	$43.20\pm 3.72$	$119.30\pm 11.04$	$2.30\pm 0.63$
$R_{\omega -uti}$	$44.21\pm 3.60$	$130.65\pm 11.51$	$2.37\pm 0.59$
$R_{\omega -wt}$	$43.68\pm 4.11$	$126.61\pm 12.02$	$2.38\pm 0.71$
$R_{hybird}$	$43.35\pm 3.67$	$124.53\pm 19.15$	$2.32\pm 0.62$
PPO	Scenario 2
	$U(\%)$	$WT\left(s\right)$	$\alpha$
$R_{const}$	$62.29\pm 5.02$	$80.79\pm 14.87$	$0.56\pm 0.15$
$R_{\omega -uti}$	$66.31\pm 7.09$	$99.87\pm 20.55$	$0.54\pm 0.18$
$R_{\omega -wt}$	$62.03\pm 5.98$	$80.10\pm 15.63$	$0.57\pm 0.18$
$R_{hybird}$	$62.75\pm 6.99$	$80.56\pm 17.12$	$0.54\pm 0.19$

. It is obvious that the performance of dispatchers under different reward functions reached a great level based on the comparison of results in Table 4. From the results, $R_{\omega -uti}$ leads to a higher machine utilization and $R_{\omega -wt}$ results in less order waiting time in both scenarios. The constant reward $R_{const}$ is also a good objective function but less flexible. The $R_{hybird}$ is to identify the optimal solution which can balance the utilization and waiting time. Just by adjusting different combinations of ${\omega }_1$ and ${\omega }_2$, the dispatcher could learn different policies to meet the varying requirements. As shown in

Table 6. Results for different combination of parameters ${\omega }_1$ and ${\omega }_1$ under reword function $R_{hybrid}$ in production scenario 2.

	$\mathit{U}(\%)$	$\mathit{W}\mathit{T}\left(\mathit{s}\right)$	$\mathit{\alpha }$
${\omega }_1=0.1$, ${\omega }_2=0.9$	$61.89\pm 5.81$	$80.99\pm 16.14$	$0.57\pm 0.16$
${\omega }_1=0.25$, ${\omega }_2=0.75$	$62.30\pm 6.08$	$80.35\pm 14.69$	$0.56\pm 0.17$
${\omega }_1=0.5$, ${\omega }_2=0.5$	$62.75\pm 6.99$	$80.56\pm 17.12$	$0.54\pm 0.19$
${\omega }_1=0.75$, ${\omega }_2=0.25$	$68.46\pm 7.02$	$106.22\pm 19.30$	$0.48\pm 0.16$
${\omega }_1=0.9$, ${\omega }_2=0.1$	$69.79\pm 7.16$	$104.88\pm 20.29$	$0.44\pm 0.16$

, the results show that utilization increase with the ${\omega }_1$ tend to $1.0$, while the waiting time is controlled by ${\omega }_2$. All in all, the results of different reward functions indicate that the proposed DRL framework can be more flexible than the general rule-based method. It could be implemented for different purposes by simply changing the reward shapes or their parameters. Meanwhile, the PPO algorithm guarantees the convergence efficiency of the optimal policy.

5. Discussion and Conclusions

In this paper, we proposed a deep reinforcement learning framework with the PPO algorithm to address the dynamic scheduling problem of the job-shop manufacturing system. The new method is designed to improve learning efficiency and performance. The dynamic simulation of a real-world production environment was used for testifying the proposed method. The results demonstrate that the DRL with the PPO algorithm performs well, and it could obtain the fastest converge speed and the best rewards compared with the other state-of-the-art algorithms. The different reward functions on behalf of the multiple objectives were tested and compared with the rule-based heuristics approaches. The quantified results indicate that the proposed framework is more flexible and could perform effectively as well as the carefully designed rule-based method.

The proposed framework was testified and analyzed based on a real-world job-shop manufacturing system. However, there are other complicated industrial applications that need more comprehensive algorithms and solutions. Future extensions that may be beneficial to the proposed framework include the following:

• The optimal policy is only learned from the massive interaction data with the production environment. Expert knowledge would be considered as a support of further enhancement of efficiency and performance.
• In the current simulation, only one dispatcher is used as the transport agent. However, a dynamic simulation environment with multiple transport agents should be developed in future studies. The proposed deep reinforcement learning framework needs to be improved in multiagent situations.
• Toward the dynamic job-shop scheduling problem, other well-known algorithms, such as GA, PSO, and TLBO, will be implemented and compared with the deep reinforcement learning framework. With the corresponding benchmark problems developed, we will validate all the algorithms within the dynamic environment.