Searching Questions and Learning Problems in Large Problem Banks: Constructing Tests and Assignments on the Fly

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article ‘Searching for Learning Questions and Problems in Large Problem Banks: Constructing Tests and Assignments on the Fly’ details an algorithm-based application for evaluating large question banks for educational purposes.

The introduction makes clear the problem to be addressed, the motivations, as well as the objectives of the work and its contribution. It is a particularly clear and straightforward section in this respect.

This is followed by a section on the state of the art, which is complete and based on relevant contributions. It allows the basis of prior knowledge on which the work will be based to be established.

I think the section detailing the algorithm is very interesting. The algorithm is precisely and transparently defined, connecting clearly with previous studies and the objectives of the work. The contribution of the specific question request is remarkable. 

As required, the article has a section in which the results generated by the algorithm are evaluated and its use is validated. The evaluation criteria are appropriate and a simple study that uses them effectively is presented. When it is necessary to obtain the difficulty of one of the questions in the bank, the author presents the limitations of the algorithm in a transparent way.

The discussion of results could be more explicitly connected to the state of the art. I think it would be advisable to improve this section to clarify the contribution made and connect it to previous studies and developments. 

Finally, the conclusions are satisfactory and relevant. The article is well written, adequately defends its proposal and deserves to be published.

Author Response

Thank you for your kind and careful review. Here are the response to your comments that asked for improving the paper.

Reviewer: "The discussion of results could be more explicitly connected to the state of the art. I think it would be advisable to improve this section to clarify the contribution made and connect it to previous studies and developments.'

I improved the discussion according to your suggestions (see lines 608--632).

Reviewer 2 Report

Comments and Suggestions for Authors

For the experiment's replicability, it is recommended that the author include GitHub access for easy access to the necessary resources. Additionally, the authors should thoroughly review the works of Chen, G., Yang, J., Hauff, C., & Houben, G.-J. (2018) on "LearningQ: A Large-Scale Dataset for Educational Question Generation" and Hwang, G., Lin, B.M., & Lin, T. (2006) on "An Effective Approach for Test-sheet Composition with Large-Scale Item Banks".

Furthermore, it is of utmost importance for the authors to clearly explain the contribution of their own research in a concise and comprehensible manner. This will help the readers understand the significance of your work and its potential impact on the field of educational question generation and test-sheet composition.

Comments on the Quality of English Language

Some minor errors in English.

Author Response

Thank you for your kind and useful review.

Reviewer "For the experiment's replicability, it is recommended that the author include GitHub access for easy access to the necessary resources.  "

I included the code into the supplementary materials.

Reviewer " Additionally, the authors should thoroughly review the works of Chen, G., Yang, J., Hauff, C., & Houben, G.-J. (2018) on "LearningQ: A Large-Scale Dataset for Educational Question Generation" and Hwang, G., Lin, B.M., & Lin, T. (2006) on "An Effective Approach for Test-sheet Composition with Large-Scale Item Banks". "

Thank you very much for your suggestion. I added the review of those and similar works to the Related Works section. (see lines 160-164 and lines 278-299).

Reviewer "Furthermore, it is of utmost importance for the authors to clearly explain the contribution of their own research in a concise and comprehensible manner. This will help the readers understand the significance of your work and its potential impact on the field of educational question generation and test-sheet composition."

I improved the description of the article contribution in the Introduction section (see lines 124-131)  and in the Discussion section (see lines 610-634).

Reviewer 3 Report

Comments and Suggestions for Authors

In summary, this study aims to develop an algorithm for constructing exercises from questions and learning problems stored in large problem banks. The paper is well-structured, and the methodology used is adequate and well-explained. The introduction and conclusion sections are clear and effectively present the problem and the achievements of the study. This study has a strong background to support the results. The findings align with the results obtained.

 

In general, this study is very interesting. The solution proposed by the authors is novel and represents a significant contribution to the field.

Author Response

Thank you for your find review.