Machine Learning-Based Code Auto-Completion Implementation for Firmware Developers
Round 1
Reviewer 1 Report
The paper presents an approach to apply machine-learning techniques to develop code auto-completion features in IDEs used by software engineers. The paper focuses on firmware development, but the approach seems to be general. A Generative Pre-trained Transformer (GPT) model is used as baseline machine-learning technique. Open source code is used for training. Code snippets are then used to validate the performance of two different designs.
Overall, the paper is informative. There are however some gaps that the authors may want to address:
- The authors should include a background section illustrating the main aspects of GPT techniques.
This will make the paper self-contained and easier to follow.
- The results presented in Section 4.2 discuss only one example and a simple table indicating the overall prediction score obtained for the two alternative designs considered in the paper. One of the designs seems to obtain a score of 100% accuracy? The authors may want to clarify.
- Additional information should be presented in Section 4.2 for the evaluation, e.g., number of test executed, a diagram plotting the accuracy of the results for the different test cases, an identification and a discussion of outliers / corner cases (if present). As it is at the moment, the authors are cherry-picking a single test case.
- It is unclear if the validation presented in Section 4.2 is actually covering the challenge defined in the use case presented at the beginning of the paper. The validation checks if the auto-completion engine suggests correct code snippets based on previous input. The use case, on the other hand, describes a need to have an auto-completion engine that facilitates the creation of similar code. Perhaps I am misunderstanding what is being said.
- In the concluding remarks, the authors argue that the computing power needed by GPT techniques may exceed the capabilities of a personal computer, and therefore a cloud service may be necessary to implement the auto-completion engine. This seems to be in disagreement with what the authors
indicated earlier on in the paper, in Section 2.2 (research gap), where they argue that companies may not want to use cloud services because of security concerns.
Author Response
Dear Reviewer 1,
Thank you so much for your efforts and time in reviewing this manuscript. We really appreciate all your thoughtful comments. The attached cover letter includes point-by-point details of the revisions and the revised manuscript. The format of the letter attached is as follows:
1) Comment #: We copied and pasted your comment.
2) Response to the comment #: We tried to answer your comment and provided details when necessary.
3) Citing the line number: We indicated the line number for you to easily find the changes from the revised manuscript.
Thanks again for your time. Please let us know if you have any questions.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The article is considered relevant from a technical point of view, however, it is considered that it must improve in the following aspects.
- References should be reviewed according to the format required in the journal especially those available online.
- In the related work paragraph it is considered that it should include a comparative table of the related work included and a conclusion of the same.
- To include more examples of code element tested by the framework developed by this research.
Author Response
Dear Reviewer 2,
Thank you so much for your efforts and time in reviewing this manuscript. We really appreciate all your thoughtful comments. The attached cover letter includes point-by-point details of the revisions and the revised manuscript. The format of the letter attached is as follows:
1) Comment #: We copied and pasted your comment.
2) Response to the comment #: We tried to answer your comment and provided details when necessary.
3) Citing the line number: We indicated the line number for you to easily find the changes from the revised manuscript.
Thanks again for your time. Please let us know if you have any questions.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
page 3, ln 53-54: "it can be hypothesized that the pre-trained GPT-2 models are fine-tuned with SSD firmware source codes; therefore, the fine-tuned model predicts..."
--> "... it can be hypothesized that if the pre-trained GPT-2 models are fine-tuned with SSD firmware source codes, the fine-tuned model predicts..."
Figures 17--19: The code snippets presented in the figures indicate that the probabilistic design is 100% correct. This is misleading -- what actually happens is that the code is 100% correct *after* the user selects the correct option, among various presented by the autocompletion engine. The authors somehow indicate this in the text (ln 266-267), but the figures need to be amended to reflect this reality. An option could be to augment the code snippets shown in figures 17--19 to indicate, next to each line, the probability of correct prediction. The authors should keep in mind that the order in which the autocompletion engine suggests code snippets to the user might be important, as the user may select (intentionally or by mistake) the first entry, which may not always be the correct one. A controlled user study or statistics based on actual use of the autocompletion engine would be necessary to understand the impact on use errors of different ordering, this might be possible future research.
Figures 14--19: The authors should avoid the use of colors as main (and only) element to distinguish correct code from incorrect code. I would recommend to underscore incorrect code with wavy lines, in a way similar to how code editors highlight errors in source code.
Author Response
Thank you so much for all your efforts and time in reviewing this manuscript. We really appreciate all your comments. Please see the attachment for more details about a point-by-point response.
Author Response File: Author Response.pdf
Reviewer 2 Report
The article is considered relevant from a technical point of view.
All comments included in the first review have been resolved.
From my point of view the paper has been improved and I have no more comments
It is considered suitable for publication
Author Response
Thank you so much for all your efforts and time in reviewing this manuscript. We really appreciate all your comments. Based on your comments (i.e., "I have no more comments), we did not prepare a cover letter for a point-by-point response. Thanks for your kind words.
