The Gap between the Admitted and the Measured Technical Debt: An Empirical Study
Round 1
Reviewer 1 Report
1. More information needs to be provided about the second stage of the research project: How were the projects selected? Was it based on availability or project similarity?
2. Table 2 provides the list of projects. It appears to me that some of these projects started before the Covid crisis, others during the Covid crisis. Does this create a problem in terms of the comparability of the projects? Are the efforts comparable? What is the impact of the remote working environment on the efficiency and measurability of these efforts?
3. The projects in Table 2 have different efforts ranging from 200 to 1350 hr. What are the implications of these differences?
4. Is it possible to hire an unbiased third party to measure technical debt?
5. I find RQ1b very important in terms of making this research relevant and valuable for users. The authors’ answer to this question between lines 603 and 605 makes me question what we learned from this research. The authors ask the readers to be careful while comparing technical debt values gained from different tools, which is common sense. I would expect this research to give a conclusive result to users.
6. This whole research is based on 14 projects. How do we know that their results are special cases related to their samples? The authors need to convince us that this sample size is enough and we can generalize the results they find.
7. The authors do not offer any solution to the problem they analyze. They conclude that the technical debt has deficiencies. The self-admitted debt is not reliable either since they would report different debt amounts. What exactly they recommend to IT developers is not clear to me.
Author Response
Please see the attachment.
Author Response File: 
Author Response.docx
Reviewer 2 Report
Please see the attached document
Comments for author File: Comments.docx
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 3 Report
1. The objectives of the research are clear and very focused.
2. The research methodology used are clearly explained and well elaborated. It is interesting to observe how the SLR and empirical methods been used.
3. The results of the analysis are very well discussed and presented.
4. Just the concern on the last phase of cause analysis in section 5.2. The details of the process is given. However, there is no mention of specific technique or method of cause analysis that have been used.
A diagram to summarize the analysis result that has been made will be more readable.
5. Overall , the paper is very elaborative, clear and well presented.
6. The discussion should relate back to the current state of the discussed elements which are the technical debt. This paper should mention the contribution of this study.
7. The possible limitations and future work of this study will be beneficial to be discussed.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
