Model-Based Test Suite Generation Using Mutation Analysis for Fault Localization
Round 1
Reviewer 1 Report
This paper proposes a methodology for generating test data for fault localization which explores the use of genetic algorithms and mutation analysis. This is an interesting topic of research and interesting results are reported.
My main issue with this work is that it lacks an adequate Literature Review, which is able to adequately contextualize this research and clarify its impact and novelty. Even though related work is often cited throughout the paper, it is not discussed or compared to the proposed approach. I suggest the introduction of a Literature Review section which can fill this gap.
The article also lacks focus and, in my opinion, does not succeed in adequately introducing the most significant concepts of the research and the necessary background. The authors often digress, repeat concepts (sometimes out of context) and, as such, I believe that significant work must take place in order to have the manuscript convey the research in a simpler and more direct fashion – particularly in terms of the writing style and overall organization.
In spite of the verbose writing style, some aspects would benefit from further clarification and illustration (e.g., e.g., in Figure 3, example alleles). The experimental setup, and in particular all the Genetic Algorithms parameters (number runs, number of generations, population size, selection methods, the genetic operators’ probabilities, …), should have been described, and a rationale for the selection of the subject of the experiments should have been provided (are those benchmark case studies?). Finally, the results should have been compared to those yielded by alternatives described in literature. The quality of the images must also be improved, as the text is often imperceptible (e.g., Figures 10 to 13).
Author Response
Dear Reviewer 1
Thank you for reviewing our manuscript in detail.
Your review was very valuable for improving our manuscript.
First, we agree with the suggestion to add a literature review section. To improve this, we decided to add a literature review part. In addition, we have found areas that are out of context. In particular, we think that the parts of referenced papers referred to during the conceptual explanations is distracting to the context (The first paragraph in section 3.2 Liu et al. referencing part, the last paragraph in section 3.3.4 Yoo et al. referencing part). So, we moved paper referencing parts in Section 3 to the literature review part in Section 2 (paragraph 7, 8) and changed it to explain it in more detail. Also, we've removed the repeated concepts (The first paragraph in section 3.2: A TP is a continuous~, the first paragraph in section 3.3.3: the goal was to use~) and moved digress from context (Paragraph 3 in section 3.3.3: Therefore, test suites~) to Section 2. And some sentences removed because of digress from context (Paragraph 3 in Section 1: As the conditions~, Paragraph 2 in Section 4.1: Faulty statements ~).
Second, we agree with the suggestion that we should add genetic parameters to the experimental part. So, we added a table (table 3 in revised manuscript) to that in Section 4. Figures 10 ~ 13 (Figure 11 ~ 14 after revision) have been improved to make the text look better. We also added a few sentences before each table to provide a basis for selecting experimental data tables.
However, we thought that the proposed method was difficult to compare with the alternatives because it was applied in the test generation phase unlike the existing technology. We have added a description in section 2 to help readers understand what is different from existing technologies.
Third, manuscript has been checked by a professional English service to improve the readability of manuscript. (http://www.textcheck.com)
We thank for your detailed and specific comments and believe that our revised manuscripts have been improved due to your comments.
Yours sincerely
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper is proposing a method for fault localization based on a model-based approach for test selection and model mutation analysis. Some experimental results are presented in which the method is applied on four models.
----------- Detailed evaluation -----------
The idea of improving fault localization using a model-based testing approach is interesting. However the paper spends a bit too much time on the background and for this reason I find the ideas not sufficiently fleshed out. The paper could be improved by showing a concrete example of how the steps in the method work, especially how the mutation testing is used.
The paper is overall well written, and the method is nicely described. Overall sound and practically promising approach for a very specific (but important) problem of model-based fault localization.
It would be nice if the presentation of the technique was a little more formally and tangibly presented and exemplified rather than the narrative description in the paper.
Not clear if you check if a mutation is killed by a previously generated test (this could reduce the cost of test generation and make smaller test suites).
How about the case in which the mutation is equivalent and the approach says that there is no test that can kill that mutation?
The experimental study relies only on some rather small example programs and models. Also, using just a subset of all mutated models in these experiments is not adequate. Hence, whether the results can be generalized remains open to some extent. An extension to include more programs and models of larger size would be highly recommended.
The approach generates test cases from the presumed correct specification whereas it should generate it from the mutant as this corresponds to the situation where the designer/developer has made an error that is resident in the code now to be tested!
It is quite unclear how the faulty versions of the state charts are inserted. What mutation operators have been used? I am expecting more details on how the mutation generation is performed. More information on how the mutants were created and how equivalent mutants were handled would be valuable.
Faults considered are limited. Would be interesting to see mutation on the underlying code. Also, this approach is not considering requirements or functionality based defects, only logic defects.
In some situations, there are sentences that need to be supported by references. Please rework and revise the paper based on this issue. You can find at least two places like that on page 2.
The structure of the results section (Section 4) could benefit if it is explicitly aligned with the structure proposed in the guidelines for conducting case study research in software engineering by Runeson et al. who considers for instance sections on the data collection procedure and the data analysis procedure (which are implicitly there in the paper).
Pros:
(+) Targeted problem has practical value
(+) The paper is well structured
(+) The model-based test generation approach is well presented
Cons:
(-) Mutant generation could be improved
(-) Conclusions of the paper are overgeneralized
(-) The idea is interesting but it is not well motivated in the abstract.
(-) Relevance of used metrics not clear
(-) Unsupported claims in the experimentation and results section
(-) This paper has grammar and language issues, which need to be addressed. Check the use of definite and indefinite articles throughout the text.
Author Response
Dear Reviewer 2
Thank you for reviewing our manuscript in detail.
Your review was very valuable for improving our manuscript.
1. We agree that concrete example could improve the manuscript. So, we added example of how the steps in the method work in section 2 (figure 1)
2. We have read Section 3 again from the reader's perspective to clarify the technical expression. And we thought that the parts of the description of the referenced paper in the technical expression might seem a bit distracting. To improve this, we moved the sentences to Section 2 and deleted redundant or unnecessary sentences.
3. We clarified this by adding the phrase "include previously generated test suites" in the second paragraph of section 3.3.4 to improve the fact that the previously generated test is not clear whether it kills mutants.
4. This method does not remove equivalent mutants that perform the same behavior as the original software. Equivalent mutants unnecessarily increase the number of executions between mutants and test suites, so it is better to remove them. We have added this to the second paragraph of section 3.3.4.
5. We would like to answer that the experimental study depends on a small example program. Our experiment is to verify the validity of the proposed method. The size of the model is only related to the time it takes to generate the test, rather than the effectiveness of the proposed method. So we focused on the form rather than the size of the model.
6. We would like to explain that this approach generates a test case based on a precise specification. The test scope of the proposed method is based on the assumption that the specification is made correctly according to the requirements. One test method can’t take all cases into consideration. The proposed technique can be applied at the stage of verifying that the software has been developed in accordance with specification, assuming that the specification is correct. Also, we would like to answer to your question that the proposed method considers only logical defects. Because, the proposed method is to test the matching of code and model. And, due to the characteristic of mutation testing, it seems more appropriate to test whether the behavior is the same as the model specification, rather than testing for requirements and functionality-based defects.
7. To clarify the part of how the faulty version of the statecharts are inserted, we have added a sentence describing it in the second paragraph of section 4.1. We also mention mutation operators used in the next sentence. And the effect of the equivalent mutant on the proposed method and whether or not it should be eliminated is mentioned in the second paragraph of section 3.3.4.
8. We added a reference to a few sentences that are supposed to be supported by a reference.
9. Thank you for recommending a good book to improve the experiment section. However, the contents of this book are rather vast, so I did not have enough time to familiarize myself with them and to modify them in accordance with the guidelines. However, we have tried to improve the experimental section. We have added a few sentences to help readers understand why the experiment data was presented.
10. In response to the opinion that the abstract of manuscript is not well motivated, the abstract has been modified to show the contribution of the paper.
11. Manuscript has been checked by a professional English service to improve the readability of manuscript. (http://www.textcheck.com)
We thank for your detailed and specific comments and believe that our revised manuscripts have been improved due to your comments.
Yours sincerely
Author Response File: Author Response.pdf
Reviewer 3 Report
The paper “Model‐based Test Suite Generation using Mutation Analysis for Fault Localization” proposes the application of the fault localization technique in the test generation phase in order to aid in the process of creating test suites. The manuscript is well written, the experimental validation is wellchosen and the proposed approach is scientific relevant Thus, the reviewer suggests to accept this paper.
Author Response
Dear Reviewer 3
Thank you for giving us a good evaluation of our manuscript. If you have a new opinion, I will gladly accept it.
Again, thanks for the good evaluation.
Yours sincerely
Author Response File: Author Response.pdf
