Predicting Missing Values in Survey Data Using Prompt Engineering for Addressing Item Non-Response
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThis paper presents a method to utilize LLM to mitigate the missing fields in the survey. The proposed column selection / question sampling method effectively leverages the in-context learning of LLMs. Given the main body and the appendix, the method are described with sufficient details for reproduction. The ablation study is also helpful for the readers to understand the effect on top K and the number of n-shot.
One minor thing to improve is that the performance gap of the proposed method is yet to be further investigated and discussed. Is the current method generating biased data, or is the dataset itself is noisy?
Author Response
Thank you deeply for your appreciation and feedback on our contribution. We have carefully considered the points you raised and prepared our responses. Please review our answers below.
Comments 1: One minor thing to improve is that the performance gap of the proposed method is yet to be further investigated and discussed. Is the current method generating biased data, or is the dataset itself is noisy?
Response 1: To prevent the model’s performance from being artificially biased by predicting only the most frequent answer, we have already balanced the evaluation data evenly across all answer options. With an equal number of samples for each option, if the model becomes biased toward predicting a single answer, the F1 score will not exceed 50% in cases with two answer options.
Additionally, to eliminate the influence of noisy data on the LLM’s predictions, we have already performed preprocessing to exclude any unanswered or noisy data. These data were excluded from both the examples generated during row selection and the evaluation data used for assessment.
However, we did not mention all the preprocessing steps we performed. Therefore, we have added the all the preprocessing steps to the manuscript. - page 9, line 361-362 / page 10, line 374-382
"The dataset consists of approximately 150 questions and responses from 5,098 participants."
"Finally, we selected 100 participants for the evaluation data and used the remaining participants for the row selection data. To avoid the risk of the model predicting the most frequent answer option and thus achieving a biased F1 score, we ensured that the responses for each question were evenly balanced. This approach allows us to rigorously evaluate the model's performance across a variety of response types, ensuring that the model is not biased toward the most frequent answers, but instead tested on a balanced dataset. Additionally, any unanswered or noisy data, such as Don't Know (DK) and Refused (RF) answers, were left out of the examples used for both row and column selection, as well as the evaluation data."
Reviewer 2 Report
Comments and Suggestions for AuthorsThe manuscript is well-structured and formatted according to the standard norms. To enhance the manuscript, the following recommendations are suggested:
Formatting Issues - The document’s formatting shows inconsistencies in indentation and spacing between text and references. For examples:
Inconsistencies in lines 23, 26, 35, 38, 39, as well as in many other situations throughout the document.
Equations and Symbols:
Equation numbers need to be formatted, put in parentheses on the right margin, aligned with the last line of each equation.
The manuscript should also reference equations in the text using the format Eq. (1) and Eq. (2).
Conclusions section - It is important to address any limitations of the study and discuss their potential impact on the conclusions. Additionally, the conclusions section appears too short. The conclusion section should be expanded to align with the logical research progression, covering research questions, methods, conclusions, and associated values. The discussion should emphasize the practical implications and application value of the research.
Author Response
We deeply appreciate your insightful feedback and thoughts on our contribution. We have carefully considered each of the points you raised and prepared our responses. Please review our answers below.
Comments 1: The document’s formatting shows inconsistencies in indentation and spacing between text and references.
Response 1: We have addressed the formatting inconsistencies you highlighted, including issues with indentation and spacing between text and references. - page 1, line 23, 30-31, 35-36, 38/ page 2, line 39, 58, 61-62 / page 3, line 109-110 / page 9, line 352 / page 10, line 384, 389, 392 / page 11, line 394, 401, 413
Comments 2: Equation numbers need to be formatted, put in parentheses on the right margin, aligned with the last line of each equation.
Response 2: All equations have been reformatted to include equation numbers enclosed in parentheses, aligned to the right margin, as per standard formatting guidelines. We have also added a reference to clarify which part of the equation is being used in the method section. - page 5, line 190, 201 / page 7, line 265, 267 / page 8, line 316, 318
Comments 3: It is important to address any limitations of the study and discuss their potential impact on the conclusions.
Response 3: We have added a new section titled ”Limitations and Future Work” to discuss the limitations of our study and their potential impact on our conclusions. We also outline potential future research directions to address these limitations. - page 14-15, line 538-585
Comments 4: Additionally, the conclusion section appears too short. The conclusion section should be expanded to align with the logical research progression, covering research questions, methods, conclusions, and associated values. The discussion should emphasize the practical implications and application value of the research.
Response 4: We have expanded the Conclusion section to better align with the logical progression of our research. The revised conclusion now thoroughly covers the research questions, methodologies employed, key findings, and the associated values of our study. - page 15, line 587-596
Reviewer 3 Report
Comments and Suggestions for AuthorsThis is a very interesting article and well-structured to express the strength of the proposed method.
One thing that I could not understand is:
In the column selection process in Section 3.3.2, you generate questions with similar content but different expressions (Step 1, lines 292-298), instead of selecting k-questions that are similar to the target question from among the actual questions. Is this right? I cannot understand why you needed to do that. And why can you use question-answer pairs in Step 4 (line 320) although there should be no answers?
Typos:
1. line 173: each each ...
2. line 346: Dataset Experimental Setting ..... ablation study
Author Response
Thank you deeply for your attention and feedback on our contribution. In particular, we are very grateful for your effort in identifying typos. We have carefully considered your comments and prepared our responses. Please review our answers below.
Comments 1: In the column selection process in Section 3.3.2, you generate questions with similar content but different expressions, instead of selecting k-questions that are similar to the target question form among the actual questions. Is this right?
Response 1: In Step 1 of Section 3.3.2, we generate a set of Related Questions using an LLM. The purpose of generating these related questions is not to include them directly in the context or prompt but to use the as refined queries to retrieve actual questions from the dataset that are semantically related to the target question.
However, due to the lack of clarity in our previous explanation, we have revised the description in Step 1 to emphasize that the generated related questions serve as queries for retrieval, thereby expanding the search space and improving the quality of the retrieved context. - page 8, line 293-302
Comments 2: Why can you use question-answer pairs in Step 4 although there should be no answers?
Response 2: In Step 4, we utilize the retrieved question-answer pairs in constructing the prompt. Since the retrieved questions are actual questions from the dataset, their corresponding answers exist for the respondents. Therefore, we can use these question-answer pairs in the context provided to the LLM. - page 8, line 293-302
Comments 3: Typos in line 173 and line 346.
Response 3: We have thoroughly reviewed and corrected the typos, thanks to your dedicated efforts. - page 5, line 173 / page 9, line 350-351
