Multi-Task Multi-Agent Reinforcement Learning for Real-Time Scheduling of a Dual-Resource Flexible Job Shop with Robots
Round 1
Reviewer 1 Report
The work is of importance and significance. The paper can be considered as a publication after the following concerns are well addressed.
1. In Introduction, the main contributions of the paper are not well concluded. The authors should revise this part to clarify which are the new merits of this paper.
2. Literature review is incomplete; some relevant research on dual-resource constrained FJSP should be reviewed. For example, An Improved African Vulture Optimization Algorithm for Dual Resource Constrained Multi-Objective Flexible Job Shop Scheduling Problems, Sensors, 2023, 23, 90, DOI:10.3390/s23010090 has presented and analyzed related problems. But such related approaches were not introduced and analyzed in the Introduction.
3. Simulation results of the developed approach should be compared with at least 1 to 3 similar references or recent methods.
4. The drawbacks of the developed approach should be discussed for potential readers.
Author Response
Response to Reviewer 1 Comments
Manuscript ID: processes-2159013Type of manuscript: ArticleTitle: Multi-task multi-agent reinforcement learning for real-time scheduling of a dual-resource flexible job shop
Dear expert:
We are writing the letter to convey my thanks and my major revisions of your comments. We are honored to be reviewed by your comments. Those comments, which make up for our shortcomings of considering less, are very important for enhancing our paper. All authors have read and approved the manuscript. We have carefully taken the comments into account and responded to each of the points raised by you. Some necessary corrections have been made, and all the altered passages have been highlighted in light yellow. We hope that our improvements can meet your approval.
Point 1: In Introduction, the main contributions of the paper are not well concluded. The authors should revise this part to clarify which are the new merits of this paper.
Response 1:
Thank expert for pointing out our deficiencies.
We have revised the introduction, and concluded the main contributions of the paper.
The main contributions have been added in the paper (Lines 231~242).
Point 2: Literature review is incomplete; some relevant research on dual-resource constrained FJSP should be reviewed. For example, An Improved African Vulture Optimization Algorithm for Dual Resource Constrained Multi-Objective Flexible Job Shop Scheduling Problems, Sensors, 2023, 23, 90, DOI:10.3390/s23010090 has presented and analyzed related problems. But such related approaches were not introduced and analyzed in the Introduction.
Response 2:
Thank expert for pointing out our deficiencies.
We have supplemented some relevant research on dual-resource constrained FJSP. In the introduction (Lines96-106), we cited relevant literature, including the paper you suggested.
Point 3: Simulation results of the developed approach should be compared with at least 1 to 3 similar references or recent methods.
Response 3:
Thanks for the expert advice.
This paper aims to solve the common flexible job shop scheduling problem involving robot loading and unloading, as well as flexible process planning in the actual production workshop. The main work focuses on how to establish a mathematical model for this type of scheduling problem, and propose an efficient real-time approach to solve the scheduling problem. The feasibility and effectiveness of the proposed approach are verified by the example experiment. The work of comparing with other methods is the focus of our future research. We are also paying attention to whether other reinforcement learning or machine learning methods have better mechanisms to improve the performance of our proposed method. This part of the work will be presented in future papers.
Point 4: The drawbacks of the developed approach should be discussed for potential readers.
Response 4:
Thanks for the expert advice.
In the MTMARL framework, the feature of the gate layer neural network for multi-task coordination is designed subjectively. This may lead to insufficient discrimination of the specificity for multi-task. To make up for this shortcoming, an automatic feature extraction method can be added to the framework to improve the overall performance of the algorithm. In addition, this framework is based on the mode of centralized train-ing and decentralized execution. Although it effectively solves the real-time schedul-ing of FJSP-DP, we think that it may not be a multi-tasking system in the strict sense. This shortcoming can be addressed by combining the latest multi-task reinforcement learning, such as IMPALA (Importance-Weighted Actor-Learner Architectures).
These contents have been added in the paper (Lines 942~951).
The above-mentioned major revisions are responses to your comments. Once again, thank you very much for what you have done. Please accept my sincere thanks. Wish you all the best.
Yours truly,
Xiaofei Zhu, Jiazhong Xu, Jianghua Ge *, Yangping Wang, Zhiqiang Xie.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Manuscript Number: processes-2159013-peer-review-v1
Title: Multi-task multi-agent reinforcement learning for real-time scheduling of a dual-resource flexible job shop
The focus of the study is on a real-time scheduling problem of a dual-resource flexible job shop with robots. A mixed integer programming model is established, which considers the scheduling problems of jobs and machines in the work cells. Then, a framework of reinforcement learning based on centralized training and decentralized execution is proposed to make real-time schedules. Each robot interacts with the environment and completes job sequencing, machine selection, and process planning.
There is novelty in this paper. However, it is not precise enough to have a clear explanation of robotic work cells. It requires a revision to be accepted. Please see my comments below.
- My main issue is about the model in Section 2. Section 2 describes the FJSP-DP and establishes its mixed integer programming mathematical model. Please look at objective function 3. It is nonlinear as I can observe. So, my question is whether the MIP is nonlinear? How does it affect the problem complexity? Does the search tree grow fast preventing the algorithm to find a high-quality solution in a reasonable time?
If so, it requires further explanation.
- There is a constraint between Constraints 9 and 10. is it a constraint? Why does not it have a number? Is it a linear or nonlinear constraint?
- There are many summations and subscript in constraints. For example, there are 5 summation in Constraint 6. How does it affect the complexity of the problem when solve it?
- The title of the paper is general. Specifically, I wonder if “multi-agent” can be replaced with “multi-robot”? This is because the paper is specifically about having resources like robot in a cell. I think the word “robot” should be in title to let readers know about content of the paper by looking at its title.
- In addition, there is no discussion about the type of robot applied for cells in this paper? Are they mobile robots or the study is limited to robotic arms? If they are only robotic arms, please let’s know the type of robotic arm used. They can be single-gripper or dual gripper robots, and each of them has different capacity regarding material handling. Accordingly, I believe this information should be added to the first section of the paper.
- Some equations in Figure 4 have low resolution. Please increase the resolution of this figure.
- Authors have developed Python code for the problem in this paper. If authors share it with readers, they can increase the visibility and impact of the code they develop during the research. Please share the code as supplementary material.
- Figure 9 shows a Gantt chart. If you ask me, I believe that authors have not mentioned what the meaning of colors is. Are colors showing the set of jobs. Please say it in the body of the paper to make it clear for readers.
- Literature review of robotic job shops is immature. I suggest citing the followings for scheduling of a job shop with robot: [a] A branch and bound algorithm for the cyclic job-shop problem with transportation, Computers & Operations Research, vol.39, pp. 3200-3214 [b] Scheduling of multi-robot job shop systems in dynamic environments: mixed-integer linear programming and constraint programming approaches, Omega, vol. 115, pp. 102770
- Never use etc. at the end of a series that begins with for example, e.g., including, such as, and the like, because these terms make etc. redundant: they already imply that the writer could offer other examples.
Page 2: such as evolutionary algorithm, ant colony algorithm, particle swarm algorithm, etc.
After resolving the comments above, the review paper will be in a good shape for publication.
Author Response
Response to Reviewer 2 Comments
Manuscript ID: processes-2159013Type of manuscript: ArticleTitle: Multi-task multi-agent reinforcement learning for real-time scheduling of a dual-resource flexible job shop
Dear expert:
We are writing the letter to convey my thanks and my major revisions of your comments. We are honored to be reviewed by your comments. Those comments, which make up for our shortcomings of considering less, are very important for enhancing our paper. All authors have read and approved the manuscript. We have carefully taken the comments into account and responded to each of the points raised by you. Some necessary corrections have been made, and all the altered passages have been highlighted in green. We hope that our improvements can meet your approval.
Point 1: My main issue is about the model in Section 2. Section 2 describes the FJSP-DP and establishes its mixed integer programming mathematical model. Please look at objective function 3. It is nonlinear as I can observe. So, my question is whether the MIP is nonlinear? How does it affect the problem complexity? Does the search tree grow fast preventing the algorithm to find a high-quality solution in a reasonable time? If so, it requires further explanation.
Response 1:
In this model, the objective function 3 is nonlinear, and all constraints are linear, so the model is a mixed integer non-linear programming or a mixed integer quadratic programming. If objective function 3 is removed, the model becomes a mixed integer linear programming. This is a multi-objective optimization problem, and objective function 3 aims to keep the workload balance among each robot. When evaluating solutions, the impact of objective function 3 needs to be taken into account, thus increasing the complexity of the problem. The existence of objective function 3 does not affect the constraints, this will not cause the search tree to grow rapidly.
Point 2: There is a constraint between Constraints 9 and 10. is it a constraint? Why does not it have a number? Is it a linear or nonlinear constraint?
Response 2:
Constraints 9 and 10 are inequality constraints. The number 0 can be obtained by moving the term on the right side of the inequality to the left. Constraints 9 and 10 are linear.
Point 3: There are many summations and subscript in constraints. For example, there are 5 summation in Constraint 6. How does it affect the complexity of the problem when solve it?
Response 3:
In the FJSP-DP model, each operation is scheduled considering the flexibility of the process planning, the flexibility of the machine, which work cell to choose, and the loading and unloading activities. More variables are required in the constraints to define the solution space, resulting in many subscripts. When searching for a feasible scheme of scheduling, many summations need to act simultaneously to satisfy these constraints. For example, in constraint 6, any activity of any operation is only assigned to one work cell (or robot) and one machine, and the sequence of activities of different operations must be considered. We need 5 summations to satisfy this constraint, that is, selected cells, selected machines, operations to be processed, activities that may be performed, and machines used by adjacent operations. The combined effects of these summations and subscripts increase the complexity of the scheduling solution.
Point 4: The title of the paper is general. Specifically, I wonder if “multi-agent” can be replaced with “multi-robot”? This is because the paper is specifically about having resources like robot in a cell. I think the word “robot” should be in title to let readers know about content of the paper by looking at its title.
Response 4:
Thanks for the expert advice.
We have carefully considered your question. We believe that "multi-agent" can not seem to be replaced by "multi-robot". Because the multi-agent is the subject of scheduling decisions, it can be a software system. Manufacturing resources (including robots) or computers can be the carrier of the agent. The control system of the robot can be installed with a multi-agent decision module so that the robot has scheduling decision ability. The robot is the executor of the agent's decision results, it cannot be completely replaced. These contents have been added in the paper (Lines 531~535).
However, considering your suggestions, we added “robot” to the title and revised the title to “Multi-task multi-agent reinforcement learning for real-time scheduling of a dual-resource flexible job shop with robots”.
Point 5: In addition, there is no discussion about the type of robot applied for cells in this paper? Are they mobile robots or the study is limited to robotic arms? If they are only robotic arms, please let’s know the type of robotic arm used. They can be single-gripper or dual gripper robots, and each of them has different capacity regarding material handling. Accordingly, I believe this information should be added to the first section of the paper.
Response 5:
The type of robot we assume is a robotic arm with a single gripper. Each of them has the same capacity regarding material handling(i.e., the loading and unloading). The definition of robot type has been added in the paper (Lines 250~252).
Point 6: Some equations in Figure 4 have low resolution. Please increase the resolution of this figure.
Response 6:
Thank expert for pointing out our deficiencies.
We have modified the formula part in Figure 4 to increase the resolution of this figure.
Point 7: Authors have developed Python code for the problem in this paper. If authors share it with readers, they can increase the visibility and impact of the code they develop during the research. Please share the code as supplementary material.
Response 7:
We are very happy to share our research results and communicate with other researchers or teams. We will apply for patents and computer software copyrights of developed Python codes. After authorization, we will gradually release the source code. We will also upload these codes to GitHub in due course. Readers can contact the author by email.
Point 8: Figure 9 shows a Gantt chart. If you ask me, I believe that authors have not mentioned what the meaning of colors is. Are colors showing the set of jobs. Please say it in the body of the paper to make it clear for readers.
Response 8:
Thank expert for pointing out our deficiencies.
We have explained Figure 9 in detail, including the meaning of the colors. These contents have been added in the paper (Lines 858~866).
Point 9: Literature review of robotic job shops is immature. I suggest citing the followings for scheduling of a job shop with robot: [a] A branch and bound algorithm for the cyclic job-shop problem with transportation, Computers & Operations Research, vol.39, pp. 3200-3214 [b] Scheduling of multi-robot job shop systems in dynamic environments: mixed-integer linear programming and constraint programming approaches, Omega, vol. 115, pp. 102770.
Response 9:
Thank expert for pointing out our deficiencies.
We have supplemented the literature review of robotic job shops. In the introduction (Lines77-88), we cited relevant literature, including the two you suggested.
Point 10: Never use etc. at the end of a series that begins with for example, e.g., including, such as, and the like, because these terms make etc. redundant: they already imply that the writer could offer other examples. Page 2: such as evolutionary algorithm, ant colony algorithm, particle swarm algorithm, etc.
Response 10:
Thank expert for pointing out our error.
We rechecked the full text carefully and corrected this error.
The above-mentioned major revisions are responses of your comments. Once again, thank you very much for what you have done. Please accept my sincere thanks. Wish you all the best.
Yours truly,
Xiaofei Zhu, Jiazhong Xu, Jianghua Ge *, Yangping Wang, Zhiqiang Xie.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
I have read the manuscript once more to check its quality. I can observe a significant improvement from title to conclusion section of the manuscript. The reinforcement learning (RL) side of the study is in a better shape. The automation side and flexible job shop exploration are stronger, and finally typo errors are corrected as the green highlights show it.
I believe that the manuscript can be accepted as it is.
