Multi-Task Learning Using Gradient Balance and Clipping with an Application in Joint Disparity Estimation and Semantic Segmentation
Round 1
Reviewer 1 Report
The manuscript is in general a very well-constructed one and has merits, but I would like to bring forward certain points for the authors to tackle before publishing:
- First point I would like to rise is the missing comparison with other MTL models. In section 2 the authors present different solution but in Section 4 only the proposed solutions are taken in consideration. Please add other MTL models performance on Cityscape into account, like [1]–[3].
- Small typo in Figure 4, please correct
- Please add the Accuracy and loss train/validate graph for the learning process. It is a useful graph to represent the evolution of the training process.
- A solution to visually represent the performance of MTL network of this type was proposed in [4]. Maybe the authors can inspire from it.
- I proposed the authors to create a section 5-Discussions in which to deep dive in the different effects of proposed solution and benefits it offers compared to solutions found in literature.
[1] Chennupati, Sumanth, et al. "Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.
[2] Liu, Shikun, Edward Johns, and Andrew J. Davison. "End-to-end multi-task learning with attention." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[3] Jha, Ankit, et al. "Adamt-net: An adaptive weight learning based multi-task learning model for scene understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.
[4] Sener, Ozan, and Vladlen Koltun. "Multi-task learning as multi-objective optimization." Advances in neural information processing systems 31 (2018).
Author Response
This manuscript has been submitted to electronics with electronics-1642181. We thank the associate editor and reviewers for their invaluable comments to improve the quality of our manuscript. We have revised this manuscript carefully based on the comments. The major changes of this revised manuscript include:
- Some comparison with other MTL models are added. Besides, a mIou-loss graph for the learning process also supplemented to the revised manuscript.
- We revised the manuscript to support the topic on joint disparity estimation and semantic segmentation. In addition, some related works are supplemented to the revised manuscript.
- We have checked the symbols and formulas used in the manuscript carefully. Some equations are further defined.
- A more detailed description is provided for the causes of the failures causes. Moreover, we all also offer the advantages of our multiple-task network
- We have proofread the manuscript carefully to eliminate grammatical errors and typos.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This paper presents a multi-task learning strategy from gradient optimization.
1. Since this paper targets image segmentation and depth estimation, it would be good to include these in the paper title, such as "Multi-task learning using gradient balance and clipping with an application in joint disparity estimation and semantic segmentation." Authors feel free to consider other ways to revise the paper title to be more precise about the paper's content.
2. Section 2, "muti-task learning", it would be good to include related works on joint disparity estimation and semantic segmentation.
3. Figure 1: Is this visualization a demonstration of a concept or calculated based on an actual example? In addition, it would be good to change the three arrows to dash lines (not the current solid lines), as the gradient contours are depicted using solid lines.
4. Figure 2: It is not clear what the causes of this failure are. It is not convincing to simply say it is due to the usage of SGD.
5. Figure 3: Is there any way to define certain indices or statistics to summarize these distributions? Currently, we have to manually look at these distributions to explain their differences.
6. Equations (6) and (7) are not defined in the main text.
7. Equation (8): Please clearly provide the formula for the proposed gradient update (I think it is (8)) and that used in the traditional SGD.
8. Table 1: From the experimental results shown in Table 1, the proposed result (joint + MTSGD) is worse than the individual "segmentation + SGD" or "disparity + SGD". In this case, what is the purpose of performing multiple tasks? Can we perform tasks individually one by one so that we achieve the best performance for each task?
Author Response
This manuscript has been submitted to electronics with electronics-1642181. We thank the associate editor and reviewers for their invaluable comments to improve the quality of our manuscript. We have revised this manuscript carefully based on the comments. The major changes of this revised manuscript include:
- Some comparison with other MTL models are added. Besides, a mIou-loss graph for the learning process also supplemented to the revised manuscript.
- We revised the manuscript to support the topic on joint disparity estimation and semantic segmentation. In addition, some related works are supplemented to the revised manuscript.
- We have checked the symbols and formulas used in the manuscript carefully. Some equations are further defined.
- A more detailed description is provided for the causes of the failures causes. Moreover, we all also offer the advantages of our multiple-task network
- We have proofread the manuscript carefully to eliminate grammatical errors and typos.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
This revision is fine by me.
Reviewer 2 Report
The revision is fine for me.
