Improving the Effectiveness and Efficiency of Web-Based Search Tasks for Policy Workers
Round 1
Reviewer 1 Report
The paper presents how to combine both generic task models and domain-specific search engines, by adapting the generic model for the purpose of developing specific applications. The topic is interesting and the paper well corresponds with the journal’s aim and scope.
However, there are shortcomings in this paper. I suggest the Authors start the abstract using for example the aim of the paper. The 3 sentences (lines 1-3) are good, but not in this place.
The Section 1 Introduction should provide an introduction to the considered problem. The Authors stated the research questions – it is ok, but there should be an introduction - it is missing. After the previous remark, I suggest using part of the abstract in the Introduction part.
The aim of the paper and the Authors’ contribution should be emphasized.
Please also add the limitations in the Conclusions section.
Overall, the paper looks good, but some changes are required.
Minor typos:
Figures 1, 2, 4, and 5 should be provided in better quality and higher resolution.
Author Response
Dear editor(s) and reviewers,
Thank you for your consideration, and for your improvements to the manuscript. All comments were incorporated in this new version, which resulted in the following changes:
* Added small intro in Section 7, before 7.1
* Suggested textual corrections (typo's, a duplicate sentence) were applied
* The Figures were re-added in 300dpi
* Tables were reformatted according to suggestions (to be horizontal)
* Added clarification to the specific points that raised questions for the reviewers
* An introduction/motivation for the paper was added (first paragraph)
* Research contributions were made more explicit (line 52)
* Added limitations and future work paragraph in the conclusion (second to last)
Please let us know if these changes sufficiently address your comments
Best regards,
Thomas Schoegje
Reviewer 2 Report
The paper is well-written and scientifically sound, even performing significance testing involving humans that is a plus.
Please find my minor remarks below:
L. 46-47 there is a duplicate sentence.
L. 250 There with capital T
L. 269 Further study required sentence should be rephrased.
L. 490 ANOVA is mentioned here first, but the acronym is in L. 530. Please change.
Figure 8. cannot differentiate the colors, create a colored diagram
Minor typos.
Author Response
Dear editor(s) and reviewers,
Thank you for your consideration, and for your improvements to the manuscript. All comments were incorporated in this new version, which resulted in the following changes:
* Added small intro in Section 7, before 7.1
* Suggested textual corrections (typo's, a duplicate sentence) were applied
* The Figures were re-added in 300dpi
* Tables were reformatted according to suggestions (to be horizontal)
* Added clarification to the specific points that raised questions for the reviewers
* An introduction/motivation for the paper was added (first paragraph)
* Research contributions were made more explicit (line 52)
* Added limitations and future work paragraph in the conclusion (second to last)
Please let us know if these changes sufficiently address your comments
Best regards,
Thomas Schoegje
Reviewer 3 Report
GENERAL OVERVIEW
In this paper, the authors present how generic task models and domain-specific search engines can be combined by adapting the generic model for the purpose of developing specific applications.
The search tasks of Policy Workers (PWs) were characterized, and a search engine was developed to improve support for these tasks.
The developed search engine better supports simple navigation and resource tasks of PWs, and it was found that complex tasks can be better supported by integrating expert search functionality.
The authors define the two following research questions:
RQ1 - What work and search tasks do PWs at the municipality of Utrecht perform?
RQ2 - How can we design functionality for more effective and efficient completion of the web-based search tasks of PWs?
POINTS TO IMPROVE:
The ideas presented in the paper are really interesting and the paper is well written, however, there are some open issues and typos, which must be modified to improve the quality of the paper:
- Which are the main contributions of this paper? The main scientific contributions of this work should be clearly presented in the Introduction section;
- It is mentioned that additional municipalities can be supported by the search engine by including the authoritative sources relevant to these municipalities. However, it is not clear whether the search engine is limited to Dutch municipalities only. Can the search engine be applied to other countries? This point should be clarified in the paper.
- In the second round of interviews, it is mentioned that five new participants (three female, two male) were approached, working in different fields of expertise. The participants are employees working on municipal policy? Participants include PWs ? Please make this clear in the text.
- For ease of reading, “Table 1. Descriptions of work tasks that involve web search tasks”, “Table 2. High level descriptions of search tasks”, “Table 3. Descriptions of (web) search tasks using a faceted task classification”, and “Table 4. Results from comparing the proposed system to existing search methods. Task completion was defined as when the search ends with a correct, approximately correct or incorrect result. Additionally, tasks could also end when the user decided to ask for advice or reached the time limit. Time spent searching was reported for multiple outcomes, such as task completion and stopping search to ask an expert instead. The number of search engines and search actions were reported per task.” should be positioned horizontally.
- In section “7. Results” the authors should give an introductory text to this section, before subsection “7.1. Effectiveness”.
- In Figure 8, authors should use different line styles and colors to improve readability.
- The authors should propose some ideas for future work in the “Conclusion” section.
Some typos
- Line 19: “Policy workers (PWs)” should be “Policy Workers (PWs)”
- Lines 421-423: “A challenge in this approach is the variety of systems and search methods utilized by PWs, including preference for different search engines (e.g. Google or Bing) and methods of search (e.g. web search or direct navigation to known webpages).” This sentence should be removed because it is a repetition of lines 412-415.
In summary, the paper can be improved if the authors answer the above questions and modify the paper according to the suggestions.
Minor editing of English language required.
Please correct the typos:
- Line 19: “Policy workers (PWs)” should be “Policy Workers (PWs)”
- Lines 421-423: “A challenge in this approach is the variety of systems and search methods utilized by PWs, including preference for different search engines (e.g. Google or Bing) and methods of search (e.g. web search or direct navigation to known webpages).” This sentence should be removed because it is a repetition of lines 412-415.
Author Response
Dear editor(s) and reviewers,
Thank you for your consideration, and for your improvements to the manuscript. All comments were incorporated in this new version, which resulted in the following changes:
* Added small intro in Section 7, before 7.1
* Suggested textual corrections (typo's, a duplicate sentence) were applied
* The Figures were re-added in 300dpi
* Tables were reformatted according to suggestions (to be horizontal)
* Added clarification to the specific points that raised questions for the reviewers
* An introduction/motivation for the paper was added (first paragraph)
* Research contributions were made more explicit (line 52)
* Added limitations and future work paragraph in the conclusion (second to last)
Please let us know if these changes sufficiently address your comments
Best regards,
Thomas Schoegje
