Redesigning Assessments for AI-Enhanced Learning: A Framework for Educators in the Generative AI Era
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThis paper is touching an extremely topical and contemporary topic, therefore its importance and impact is expected to be a great one.
I have read the paper carefully, and in many cases it was not easy to follow, not because of the English but because the paper is not really explaining and citing all information. Let me be more detailed.
To start with the title indicates that this paper will talk about all kinds of assessment in all kind of disciplines, in all universities/ It later ,as one reads the reports it emerges that this study was carried out in Palestine, using Universities from the area. This is not necessarily a problem per se, but I wonder why the author is not making explicit that this is a study that has been carried out is a specific part of te word. As an educator , the author must realise that location of study culture and context do play an important role, especially when you study educational issues.
So I strongly suggest that the title of this article is altered to reflect correctly the content of the paper and the study. If the author believes that some findings could be applied to any educational settings, then this argument must be made explicitly.
Furthermore, I would even consider selecting a specific discipline, (eg Social Science, since it seems to be the largest sample, and use the rest of the data as ‘backup’ to confirm findings from amin group, where applicable, or use the rest of the data for another paper).
Now, the literature review is a paper per paper analysis, which is not even as critical as one expects. It is rather descriptive and there si not a critical analysis nor a critical discussion that demonstrates the issues that will eventually lead to the formation fo the research questions as well as the formation of the interview as well as focus groups questions/directions.
Even so, in the paper per paper approach, which I do not find useful nor fruitful for the objectives of a literature review, still the author fails to explains and present the paper in a rather comprehensive manner and seems to replicate the paper ( quote from lines 214 to 217 of manuscript)
“Swauger (2020) pointed out that the use of AI-supported assessment also raised concerns about restricting certain types of learning. Although one of the promises of this type of assessment is its ability to capture all forms of modern learning, it may, in practice, limit what is assessed by focusing on specific aspects of learning”
(excerpted from lines 214 to 217 of manuscript).
In the quote above, my obvious questions are: what is meant by certain types of learning? Which are these certain types? Later at the end of the quote the term aspects of learning is cited. Which aspects of earning are meant here?
Maybe these questions are answered or are obvious in the paper of Swauger (2020) but not here! This kind of question are repeated all over in the literature review section, but also beyond, and special attention must be paid
Moreover, there is no critical assessment if the author is in agreement with the suggestions of the paper, Does the author agree/disagree/partially agree/disagree etc? Very important point that needs to be addressed in the revision of the paper.
Regarding the design of the study, a few issues are also present. For example, in line 309 to 318 of the manuscript, where the rationale for forming the focus groups:
“Four focus group sessions were conducted, each involving nine participants selected from a pool of 130 individuals, excluding those who participated in the semi-structured interviews. This resulted in a total of 36 participants. Selection prioritized faculty members who demonstrated significant experience with AI-resistant assessment design, as reflected in their modifications to assessments or thoughtful reflections on integrating AI considerations. Additionally, participants were chosen based on their openness to collaboration, ensuring they could contribute meaningfully to interactive discussions by sharing best practices and engaging in collective learning. Focus group sessions provided a platform for collective reflection, enabling participants to share experiences, address challenges, and exchange strategies in a collaborative learning environment (Gundumogula & Gundumogula, 2020)”.
(Excerpted from the submitted paper for review lines 309 to 218))
I am trying to identify the criteria for selecting the members of each focus group, and I see the term that seems to answer this is prioritised with faculty members that demonstrated significant experience with AI resistant assessment design.
Here there are a number of issues for discussion:
Why has the author selected such people, what is the rationale? There is an explanation that the ones selected have shown experience in AI resistant assessment design? However, this is very vague, what does this mean? In terms of educational principles? Learning and teaching objectives? You need to explain.
Secondly regarding the AI resistant assessment, this is s not really a defined concept in your study. This concept is mentioned in the 3.1 section of the paper, but not really explained or elaborated and linked to educational principles and learning/teaching objectives? What is expected gain by adopting this kind of assessment?
Another design point is that the author has not mentioned the objectives of each focus group, since this is very important factor in selecting the members of that group. Depending on what is the focus, the appropriate people ought to be selected.
The lack of a good and rich description of the profiles of the people that participated in the study is also a problem, since it does not allow the reader to independently establish that the participants are able to provide meaning input to the study.
Reading through the paper, I realized that this whole paper is based on the basis of the workshop where the participants were exposed to what is described as theoretical and applicable parts. Again details of this workshop are NOT discussed at all in the paper.
The fact that the paper is based on the workshop and what people designed, means that we do not have result based on factual application of such assessments to students and observe how these performed as opposed to plan.
Given the emphasis and importance on the workshop, the author ought to provide detailed description so at least the reader has the ability to see the design process the steps as well as the logic behind the setup of the assessments proposed.
As such the results are rather shallow and do not provide an rich and deep understanding of the issues faced as well as the solutions proposed, since no such proposal exists.
Fo example, saying the redesigning , promote further the active and authentic aspect of assessment is not really a result of the main focus of the study. It can be argued that it may be a by-product. Moreover, academic integrity, which is a major issue for research as well as practical, there is nothing, really, there!
In short, in my opinion, the author needs to reconsider the paper and really decide which aspects the data can support. There is basis for many papers in this submitted article, and this the author may want to consider a paper on educating the educator to face the GenAI era, where the focus is on the workshop and how it was designed and delivered.
If the author decides to pursue the holistic paper, then the title has to be altered to reflect the geographical location of the HEI, as well as the fact that this was a design aspect of designing GenAI assessments. It is important that the title correctly and accurately reflects the content and essence of the paper. Now, in my opinion, this is not the case.
Author Response
Dear reviewer,
Thank you for providing us with your comments and feedback to improve our article. We addressed all of your comments in the main article with track changes.
This paper is touching an extremely topical and contemporary topic, therefore its importance and impact is expected to be a great one.
I have read the paper carefully, and in many cases it was not easy to follow, not because of the English but because the paper is not really explaining and citing all information. Let me be more detailed.
To start with the title indicates that this paper will talk about all kinds of assessment in all kind of disciplines, in all universities/ It later ,as one reads the reports it emerges that this study was carried out in Palestine, using Universities from the area. This is not necessarily a problem per se, but I wonder why the author is not making explicit that this is a study that has been carried out is a specific part of te word. As an educator , the author must realise that location of study culture and context do play an important role, especially when you study educational issues.
Action
Thank you so much. We have already added the context of the study in the body of the text and stated clearly the location of the study. We did not add it in the title because it will be long.
So I strongly suggest that the title of this article is altered to reflect correctly the content of the paper and the study. If the author believes that some findings could be applied to any educational settings, then this argument must be made explicitly.
Action
We believe that the present title reflects the study's scope and contributions carefully, any concerns about applicability to broader educational contexts are addressed within the manuscript itself, where the framework's adaptability is discussed. therefore, no change to the title is necessary.
Furthermore, I would even consider selecting a specific discipline, (eg Social Science, since it seems to be the largest sample, and use the rest of the data as ‘backup’ to confirm findings from amin group, where applicable, or use the rest of the data for another paper).
Action
Honestly, we requested to extend the deadline of submitting our revision to address this issue and we have granted this extension. As a team we carefully revisit to data and the findings we found it is difficult to separate the data to cover the main points and to build the framework. So we keep all the data in this study as it, but taking into consideration to check the developed the framework in specific context.
Now, the literature review is a paper per paper analysis, which is not even as critical as one expects. It is rather descriptive and there si not a critical analysis nor a critical discussion that demonstrates the issues that will eventually lead to the formation fo the research questions as well as the formation of the interview as well as focus groups questions/directions.
Action
We rephrased the literature review to address your comment. Hope, it meets your satisfaction.
Even so, in the paper per paper approach, which I do not find useful nor fruitful for the objectives of a literature review, still the author fails to explains and present the paper in a rather comprehensive manner and seems to replicate the paper ( quote from lines 214 to 217 of manuscript)
“Swauger (2020) pointed out that the use of AI-supported assessment also raised concerns about restricting certain types of learning. Although one of the promises of this type of assessment is its ability to capture all forms of modern learning, it may, in practice, limit what is assessed by focusing on specific aspects of learning”
(excerpted from lines 214 to 217 of manuscript).
In the quote above, my obvious questions are: what is meant by certain types of learning? Which are these certain types? Later at the end of the quote the term aspects of learning is cited. Which aspects of earning are meant here?
Maybe these questions are answered or are obvious in the paper of Swauger (2020) but not here! This kind of question are repeated all over in the literature review section, but also beyond, and special attention must be paid
Action
We removed this part because we feel, the original did not answer the questions you raised.
Moreover, there is no critical assessment if the author is in agreement with the suggestions of the paper, Does the author agree/disagree/partially agree/disagree etc? Very important point that needs to be addressed in the revision of the paper.
Regarding the design of the study, a few issues are also present. For example, in line 309 to 318 of the manuscript, where the rationale for forming the focus groups:
“Four focus group sessions were conducted, each involving nine participants selected from a pool of 130 individuals, excluding those who participated in the semi-structured interviews. This resulted in a total of 36 participants. Selection prioritized faculty members who demonstrated significant experience with AI-resistant assessment design, as reflected in their modifications to assessments or thoughtful reflections on integrating AI considerations. Additionally, participants were chosen based on their openness to collaboration, ensuring they could contribute meaningfully to interactive discussions by sharing best practices and engaging in collective learning. Focus group sessions provided a platform for collective reflection, enabling participants to share experiences, address challenges, and exchange strategies in a collaborative learning environment (Gundumogula & Gundumogula, 2020)”.
(Excerpted from the submitted paper for review lines 309 to 218))
I am trying to identify the criteria for selecting the members of each focus group, and I see the term that seems to answer this is prioritised with faculty members that demonstrated significant experience with AI resistant assessment design.
Here there are a number of issues for discussion:
Why has the author selected such people, what is the rationale? There is an explanation that the ones selected have shown experience in AI resistant assessment design? However, this is very vague, what does this mean? In terms of educational principles? Learning and teaching objectives? You need to explain.
Action
We rephrased the methodology part
Secondly regarding the AI resistant assessment, this is s not really a defined concept in your study. This concept is mentioned in the 3.1 section of the paper, but not really explained or elaborated and linked to educational principles and learning/teaching objectives? What is expected gain by adopting this kind of assessment?
Action
We added a new section to define AI-resistant assessment and its theoretical foundation which was the social constructivism theory
Another design point is that the author has not mentioned the objectives of each focus group, since this is very important factor in selecting the members of that group. Depending on what is the focus, the appropriate people ought to be selected.
Action
We disagree with the reviewer because we added the rationale of using focus group. Despite that we added new paragraph to clear the rationale of using focus group sessions.
Reading through the paper, I realized that this whole paper is based on the basis of the workshop where the participants were exposed to what is described as theoretical and applicable parts. Again details of this workshop are NOT discussed at all in the paper.
Action
We disagree with the reviewer because there is a section called the context of the study. It described the procedures we did to conduct the training workshops and the expected objectives.
The fact that the paper is based on the workshop and what people designed, means that we do not have result based on factual application of such assessments to students and observe how these performed as opposed to plan.
Given the emphasis and importance on the workshop, the author ought to provide detailed description so at least the reader has the ability to see the design process the steps as well as the logic behind the setup of the assessments proposed.
Action
A detailed description of the workshops provided in the article
As such the results are rather shallow and do not provide an rich and deep understanding of the issues faced as well as the solutions proposed, since no such proposal exists.
Fo example, saying the redesigning , promote further the active and authentic aspect of assessment is not really a result of the main focus of the study. It can be argued that it may be a by-product. Moreover, academic integrity, which is a major issue for research as well as practical, there is nothing, really, there!
In short, in my opinion, the author needs to reconsider the paper and really decide which aspects the data can support. There is basis for many papers in this submitted article, and this the author may want to consider a paper on educating the educator to face the GenAI era, where the focus is on the workshop and how it was designed and delivered.
Action
We hope addressing these issues in the current revision article meet your feedback
If the author decides to pursue the holistic paper, then the title has to be altered to reflect the geographical location of the HEI, as well as the fact that this was a design aspect of designing GenAI assessments. It is important that the title correctly and accurately reflects the content and essence of the paper. Now, in my opinion, this is not the case.
Action
We added our rationale for the current title at the top of our response to the reviewers and editor feedback.
Reviewer 2 Report
Comments and Suggestions for Authors
Reviewer Positionality: I come to this review as a STEM education faculty member who has kept a close eye on developments in AI and their influence on education. Like the authors of this paper, I have run sessions with faculty at my university as well as public school teachers on the topic of generative AI and its implications for teaching and assessment practices. I employ critical perspectives on education technology in my work.
The topic that you address in this study is timely and worthwhile. There are plenty of recommendations and suggestions for how faculty should be adopting various AI technologies or modifying their teaching practices in light of those that exist. Gaining perspectives from faculty members themselves is something that is very much needed. I see potential for this study to be informative. You have collected a robust set of data and the interview/focus group questions you used are sufficient to address your research questions.
While I believe that this manuscript has potential, there are also many areas in need of improvement and further development. In broad terms, the two main areas that need attention:
1. The introductory sections need more clarity and focus: what exactly are you studying here, and why?
2. The methods require further explanation and clarification.
More detailed notes on these areas follow:
1. Terminology: You use “generative AI” throughout the manuscript, even when referring to technologies that are not in fact generative. Many AI-based assessment technologies do not make use of generative AI. Automated scoring, for instance, may utilize machine learning techniques but are not engaged in any sort of generative processes.
2. Focus: Broadly, you are addressing the impacts of AI on assessments in higher education, but there are many ways you could look at this situation. We could look at how instructors use generative AI to create assessments. Or their adoption of other AI assessment systems such as automated scoring or learning analytics. But then there’s the student side of things: how are students using AI during assessment processes? And how might faculty redesign their assessments in light of students’ AI uses? Notice that these are all very different questions! But they are all blended together in the introductory portions of the manuscript. Adding to the confusion, you also discuss limitations in standard assessment practices and contrast those with ones that are more, for instance, authentic and performance-based. That’s an important distinction of course, but it doesn’t necessarily have anything to do with AI.
3. Purpose of the Study: I appreciated your engagement with views on AI and assessment that are critical as well as optimistic. You make clear that within higher education, there are both promises as well as serious concerns over the use of AI in assessment. Now, how are you positioning your study in relation to those conversations and disagreements? Be more clear about how you are speaking to the literature that you survey.
4. Methods: In describing the interview recruitment, you mention eligibility criteria – what were those criteria? How many participants participated in the focus groups? Were these the same people who completed individual interviews (it looks like the answer is no but want to be sure).
5. Analysis: I appreciated the inclusion of coded examples in the appendix. Your analytical process, though, still needs further explanation. Your research questions address three different things: motivations, challenges, and redesign practices. You used thematic analysis to answer these questions (note that your methods citation doesn’t appear in the reference list). How did you separate your thematic codes with respect to those research questions? The coded example you provide does not make that clear. Presenting your code book would be more helpful. Drawing from multiple data sources is a strength of your study, but you don’t explain how you engaged in “triangulation.” Which data did you start with? Did you then look at a different source to confirm your initial codes? Did you compare the coding patterns across the different data sources? Or did you essentially pool them all together for analysis (in which case you aren’t really triangulating)?
6. Results: I found myself questioning how you organized the various subthemes together into your overarching themes. The subthemes did not always seem to fit particularly well. For instance, you placed the concern that students might undermine their learning processes by using AI under “ethical” considerations. This isn’t really a question of ethics, though. I’m also not sure how using AI to promote creativity fits within the “future of work” theme. Throughout your results, you also use words like “most” or “many” participants, but you don’t specify how you quantified your findings in this way. That’s something to address in the methods. The quantifications become confusing once you get to your third research question. You say that the majority of your participants seemed to be Against, but also the majority mentioned Explore? How is that possible?
7. Figure 1: I don’t really understand the vertical axis here. I get that Against and Avoid both are approaches that seek to minimally engage with genAI. But in what sense is Against higher on “academic dishonesty?” A more detailed description of these codes in Table 3 would likely help.
8. Discussion: When interpreting your work, it is important to acknowledge the context in which it occurred. You recruited faculty who participated in your workshops, which means that they are a self-selected group. This isn’t necessarily a problem, but you need to think about the limitations of your sample.
Author Response
Dear reviewer,
Thank, all the changes we did were in track changes. Below our response to your comments one by one:
Reviewer Positionality: I come to this review as a STEM education faculty member who has kept a close eye on developments in AI and their influence on education. Like the authors of this paper, I have run sessions with faculty at my university as well as public school teachers on the topic of generative AI and its implications for teaching and assessment practices. I employ critical perspectives on education technology in my work.
The topic that you address in this study is timely and worthwhile. There are plenty of recommendations and suggestions for how faculty should be adopting various AI technologies or modifying their teaching practices in light of those that exist. Gaining perspectives from faculty members themselves is something that is very much needed. I see potential for this study to be informative. You have collected a robust set of data and the interview/focus group questions you used are sufficient to address your research questions.
While I believe that this manuscript has potential, there are also many areas in need of improvement and further development. In broad terms, the two main areas that need attention:
- The introductory sections need more clarity and focus: what exactly are you studying here, and why?
Action
We rephrased it to address your comments
- The methods require further explanation and clarification.
Action
New explanation was added to the manuscript
More detailed notes on these areas follow:
- Terminology: You use “generative AI” throughout the manuscript, even when referring to technologies that are not in fact generative. Many AI-based assessment technologies do not make use of generative AI. Automated scoring, for instance, may utilize machine learning techniques but are not engaged in any sort of generative processes.
- Focus: Broadly, you are addressing the impacts of AI on assessments in higher education, but there are many ways you could look at this situation. We could look at how instructors use generative AI to create assessments. Or their adoption of other AI assessment systems such as automated scoring or learning analytics. But then there’s the student side of things: how are students using AI during assessment processes? And how might faculty redesign their assessments in light of students’ AI uses? Notice that these are all very different questions! But they are all blended together in the introductory portions of the manuscript. Adding to the confusion, you also discuss limitations in standard assessment practices and contrast those with ones that are more, for instance, authentic and performance-based. That’s an important distinction of course, but it doesn’t necessarily have anything to do with AI.
Action
Thank you for your detailed feedback. Our intention in this manuscript was to address a specific aspect of AI's role in assessment: how faculty can redesign assessments in the Gen AI era to balance the opportunities and challenges posed by this technology. While our work touches on related issues such as how students might use AI in assessments and the limitations of traditional methods, which is out of the scope of our study, but can be addressed in new research and in different context. To improve clarity, we revised the introduction to explicitly outline the scope of our study. Specifically, we emphasize that the primary focus is on faculty-driven assessment redesign in response to Gen AI use. While related topics, such as AI’s role in creating assessments or automated scoring systems, are important, they fall outside the primary focus of this manuscript and will be briefly mentioned as areas for future exploration.
- Purpose of the Study: I appreciated your engagement with views on AI and assessment that are critical as well as optimistic. You make clear that within higher education, there are both promises as well as serious concerns over the use of AI in assessment. Now, how are you positioning your study in relation to those conversations and disagreements? Be more clear about how you are speaking to the literature that you survey.
Thank you
We addressed it
- Methods: In describing the interview recruitment, you mention eligibility criteria – what were those criteria? How many participants participated in the focus groups? Were these the same people who completed individual interviews (it looks like the answer is no but want to be sure).
Action
The participants in the focus group were not participated in the interviews and it was stated clearly in the description of the focus group sessions.
- Analysis: I appreciated the inclusion of coded examples in the appendix. Your analytical process, though, still needs further explanation. Your research questions address three different things: motivations, challenges, and redesign practices. You used thematic analysis to answer these questions (note that your methods citation doesn’t appear in the reference list). How did you separate your thematic codes with respect to those research questions? The coded example you provide does not make that clear. Presenting your code book would be more helpful. Drawing from multiple data sources is a strength of your study, but you don’t explain how you engaged in “triangulation.” Which data did you start with? Did you then look at a different source to confirm your initial codes? Did you compare the coding patterns across the different data sources? Or did you essentially pool them all together for analysis (in which case you aren’t really triangulating)?
Action
We addressed your concern and established a new section for triangulation
- Results: I found myself questioning how you organized the various subthemes together into your overarching themes. The subthemes did not always seem to fit particularly well. For instance, you placed the concern that students might undermine their learning processes by using AI under “ethical” considerations. This isn’t really a question of ethics, though. I’m also not sure how using AI to promote creativity fits within the “future of work” theme. Throughout your results, you also use words like “most” or “many” participants, but you don’t specify how you quantified your findings in this way. That’s something to address in the methods. The quantifications become confusing once you get to your third research question. You say that the majority of your participants seemed to be Against, but also the majority mentioned Explore? How is that possible?
Action
Yes, you are right and agree with you for future work and creativity we mean that students who use AI-resistant assessment could enhance their critical and creativity thinking which is consoder as one of the 21th century skills that are important for future jobs. For the “ethical consideration” it was our mistake and removed it. For. Most, many,… we added a definition in the methodology part
- Figure 1: I don’t really understand the vertical axis here. I get that Against and Avoid both are approaches that seek to minimally engage with GenAI. But in what sense is Against higher on “academic dishonesty?” A more detailed description of these codes in Table 3 would likely help.
Action
We response to you that we have updated table 3 and the response is as follows:
The vertical axis measures the potential for academic dishonesty, which refers to the likelihood of students misusing generative AI tools in assessments. The placement of "Against" higher on the axis compared to "Avoid" highlights the nature of tasks within these categories. For assessments under "Against," such as exams or oral assessments, the strict exclusion of AI tools does not eliminate the possibility of dishonesty entirely (e.g., students using unauthorized external AI assistance). On the other hand, "Avoid" assessments, like performance-based or reflective tasks, are inherently designed to minimize dishonesty due to their reliance on individualized, higher-order thinking that is difficult for AI to replicate without detection.
- Discussion: When interpreting your work, it is important to acknowledge the context in which it occurred. You recruited faculty who participated in your workshops, which means that they are a self-selected group. This isn’t necessarily a problem, but you need to think about the limitations of your sample.
Action
We improved it
Reviewer 3 Report
Comments and Suggestions for AuthorsThe work has a title that is focused on the indicated content in a necessary and urgent area of ​​research. But I am struck by the use of ChatGPT for what the authors indicated in its use to improve the writing, this topic in research should be analyzed because if it is a group of researchers there would be no need for its use.
Regarding the references, it would be appropriate to place the DOI for its search, for example Feuerriegel, S., Hartmann, J., Janiesch, C. et al. Generative AI. Bus Inf Syst Eng 66, 111–126 (2024). they do have: https://doi.org/10.1007/s12599-023-00834-7, that is, the work team must add the DOI to the references for the search of them.
For example, this citation is not found: Noroozi et al., 2024, we can assume that it was an error by the research team but it is likely that it was ChatGPT being strong in the rigorous review accepted. For example, another citation that is not found is (UNESCO, 2023).
On page 3, Bsharat & Khlaif, 2024 is found but it does not exist in the references, it must be misquoted. The same with (Donnell et al., 2024), the reference has been located but it does not exist in Google Scholar, they must add their DOI to quickly locate and be able to read the text.
The work described is good but the theoretical aspect needs to be strengthened, improve the writing and analysis of the introduction and state of the art sections.
For example, in the introduction, improve the use of citations. Also improve 1.1 with an adequate analysis introducing your topic, not the one suggested by the AI.
Likewise in section 2, especially 2.1, 2.2 and 2.3 without the need for AI, since it can be thought that all the content is generative.
On page 8, section 3.4, it is not clear why they started with 58 and then 61, it is better to indicate the final number so that it matches the summary. The same occurs throughout the methodology, in section 3.2, 25 out of 155, but this is not indicated in the summary (different universities but how many each, that is, are they representative samples?). The same occurs in section 3.3, 36 out of 130 participants. It is not clear if they are the same or if it is another group of the faculty. That is, in total of professors, how many are there? And how many participate? Are they all from the same institution?
Figure 1 indicates in the image that it is Adopt Partially, so the framework should be AAAPE instead of AAAE.
Starting on page 11, there are some codes (they must be from the participants) such as F17, F23, F2, FG3 etc. This should be indicated to the reader that it will be located in the appendix, or better not to place it.
Section 4.1.5 has another type of typography, it should be reviewed.
Section 4 should be titled "Results"
Although the work is "qualitative" representative values ​​of the research should be mentioned.
The discussion should be improved and focused on the work carried out, avoid placing subsections 5.1, 5.2 etc. It seems generated by AI.
The conclusions should be expanded, having an extensive work does not contribute with all that was done that is valuable.
Author Response
Dear reviewer
Thank you
We addressed all of your comments in the main article with track changes. Below our response point-by-point
Reviewer #3
The work has a title that is focused on the indicated content in a necessary and urgent area of ​​research. But I am struck by the use of ChatGPT for what the authors indicated in its use to improve the writing, this topic in research should be analyzed because if it is a group of researchers there would be no need for its use..
Action
Yes, I agree with you but we used it because all of us are native in the Arabic language and we have to make sure the language is understandable. We used it just for proofreading
Regarding the references, it would be appropriate to place the DOI for its search, for example Feuerriegel, S., Hartmann, J., Janiesch, C. et al. Generative AI. Bus Inf Syst Eng 66, 111–126 (2024). they do have: https://doi.org/10.1007/s12599-023-00834-7, that is, the work team must add the DOI to the references for the search of them.
Action
We addressed this issue and provided the DoI that is available
For example, this citation is not found: Noroozi et al., 2024, we can assume that it was an error by the research team but it is likely that it was ChatGPT being strong in the rigorous review accepted. For example, another citation that is not found is (UNESCO, 2023).
Action
We added the missing sources
On page 3, Bsharat & Khlaif, 2024 is found but it does not exist in the references, it must be misquoted. The same with (Donnell et al., 2024), the reference has been located but it does not exist in Google Scholar, they must add their DOI to quickly locate and be able to read the text.
Action
We removed it and added a new source
The work described is good but the theoretical aspect needs to be strengthened, improve the writing and analysis of the introduction and state of the art sections.
Action
We rephrased the introduction
For example, in the introduction, improve the use of citations. Also improve 1.1 with an adequate analysis introducing your topic, not the one suggested by the AI.
Likewise in section 2, especially 2.1, 2.2 and 2.3 without the need for AI, since it can be thought that all the content is generative.
Action
We rephrased and add new section in the literature review
On page 8, section 3.4, it is not clear why they started with 58 and then 61, it is better to indicate the final number so that it matches the summary. The same occurs throughout the methodology, in section 3.2, 25 out of 155, but this is not indicated in the summary (different universities but how many each, that is, are they representative samples?). The same occurs in section 3.3, 36 out of 130 participants. It is not clear if they are the same or if it is another group of the faculty. That is, in total of professors, how many are there? And how many participate? Are they all from the same institution?
Action
We fixed this issue. Actually the total number participated in the study was 61
Figure 1 indicates in the image that it is Adopt Partially, so the framework should be AAAPE instead of AAAE.
Action
Integrating Gen AI was full and partially. Therefore we keep the name as it.
Starting on page 11, there are some codes (they must be from the participants) such as F17, F23, F2, FG3 etc. This should be indicated to the reader that it will be located in the appendix, or better not to place it.
Action
Yes, these codes refer to the number of faculty
Section 4.1.5 has another type of typography, it should be reviewed.
Action
We addressed it
Section 4 should be titled "Results"
Action
We changed it
Although the work is "qualitative" representative values ​​of the research should be mentioned.
The discussion should be improved and focused on the work carried out, avoid placing subsections 5.1, 5.2 etc. It seems generated by AI.
We removed
The conclusions should be expanded, having an extensive work does not contribute with all that was done that is valuable.
Action
Expanded
Round 2
Reviewer 2 Report
Comments and Suggestions for AuthorsThank you for the opportunity to review this revised manuscript. The revisions that made to the paper are extensive and definitely improve the manuscript. I have a few smaller suggestions for the paper at this stage.
Introductory Sections: The focus of your study is now much more clear. The revised introductory portions of the manuscript make a more streamlined and focused case for the purpose of your study and the informing literature.
One place I would suggest some further revision is the pair of subsections "Purpose of the Study" and "Contribution of the Study." These two sections were overlapping in a lot of their content, so it might be worth combining them.
There are also some vague statements that make sense only after reading the rest of the paper. For instance, the first mention of workshops is your statement about how you are "drawing on theoretical insights and practical experiences gained from workshops with higher education faculty..." It would really help your reader to provide a more concrete (but still brief) description of the context of your work. It could be as simple as, "In this study, we examine how higher education faculty members are thinking about and responding to GenAI in their assessment practices. These faculty were participants in a series of workshops on GenAI that were conducted by the authors. By examining their views and experiences, we hope to....”
It would also help to revisit the contributions of your study at the end of the literature review section on GenAI in Higher Ed. You summarize a lot of prior work, and it would be helpful to include a brief statement about how your study speaks to the prior research.
Small thing: in section 1.1 you mention that you focus on faculty in the humanities sciences but your participants also come from STEM fields.
Methods: I found the clarifications and added details to be sufficient. I tend to prefer more specific descriptions of qualitative coding processes (yours are generic in that they give an overview of your methods but provide fewer specific details about how those were applied to your specific data), but I think most readers would find the methods description to be adequate.
Small note on Line 460: “Some” is defined twice in this paragraph.
Findings, Conclusions: The organization of the findings made more sense to me in this revised version. I don’t see any further revisions as being necessary.
Author Response
Thank you for the opportunity to review this revised manuscript. The revisions that made to the paper are extensive and definitely improve the manuscript. I have a few smaller suggestions for the paper at this stage.
Introductory Sections: The focus of your study is now much more clear. The revised introductory portions of the manuscript make a more streamlined and focused case for the purpose of your study and the informing literature.
One place I would suggest some further revision is the pair of subsections "Purpose of the Study" and "Contribution of the Study." These two sections were overlapping in a lot of their content, so it might be worth combining them.
Action
We addressed your comments through combining the two sections into one and renamed the title of the section into: Purpose and contribution of the study
This study explores faculty members' motivations for redesigning their course assessments and presents a comprehensive framework for rethinking assessment practices in the era of generative AI (Gen AI). By examining the perspectives and experiences of higher education faculty who participated in a series of workshops on Gen AI conducted by the authors, this research captures the collective wisdom of practitioners regarding the adaptation of assessments to the Gen AI era. These workshops facilitated collaboration and the exchange of ideas, fostering innovative approaches to align course assessments with the evolving educational landscape.
The study contributes to the ongoing discourse on integrating Gen AI into higher education by addressing gaps in understanding AI's transformative role in assessment practices. While existing literature focuses primarily on AI's applications in personalized learning and administrative tasks, this research examines Gen AI’s potential for reshaping assessments at a fundamental level. By engaging early adopters from diverse academic fields, the study highlights the motivations, challenges, and benefits of redesigning assessments to meet the demands of the Gen AI era.
A key outcome of this study is a comprehensive framework, grounded in theoretical insights and practical experiences from faculty training workshops, that provides actionable solutions for designing AI-enhanced assessments. This framework addresses critical issues such as academic integrity, student engagement, alignment with 21st-century skills, and the Sustainable Development Goals (SDGs). Additionally, by contextualizing its findings within the challenges faced by resource-constrained regions, the research promotes equitable and effective AI integration in education, fostering global innovation in teaching, learning, and assessment.
There are also some vague statements that make sense only after reading the rest of the paper. For instance, the first mention of workshops is your statement about how you are "drawing on theoretical insights and practical experiences gained from workshops with higher education faculty..." It would really help your reader to provide a more concrete (but still brief) description of the context of your work. It could be as simple as, "In this study, we examine how higher education faculty members are thinking about and responding to GenAI in their assessment practices. These faculty were participants in a series of workshops on GenAI that were conducted by the authors. By examining their views and experiences, we hope to....”
Action
We rephrased the two sections that include your statement.
It would also help to revisit the contributions of your study at the end of the literature review section on GenAI in Higher Ed. You summarize a lot of prior work, and it would be helpful to include a brief statement about how your study speaks to the prior research.
Action
We added at the end of the literature review the following short statement:
this study builds on prior research by addressing gaps in the integration of Gen AI in assessment practices, focusing on practical solutions grounded in faculty workshops. While existing literature highlights challenges like ethical concerns and plagiarism, our research offers a comprehensive framework promoting academic integrity, student engagement, and alignment with 21st-century skills and SDGs. Additionally, it explores AI-resistant assessments to foster critical thinking and adaptability, contextualizing findings within resource-constrained settings to support equitable AI integration in higher education. This contribution bridges theoretical and practical insights, advancing the discourse on Gen AI's transformative role in education.
Small thing: in section 1.1 you mention that you focus on faculty in the humanities sciences but your participants also come from STEM fields.
Action
We fixed it
It focuses on early adopters in Palestine in humanities science, STEM fields, and Medical sciences
Methods: I found the clarifications and added details to be sufficient. I tend to prefer more specific descriptions of qualitative coding processes (yours are generic in that they give an overview of your methods but provide fewer specific details about how those were applied to your specific data), but I think most readers would find the methods description to be adequate.
Action
We did not add more details about coding because of the length of the manuscript. We consider that readers could be familiar with qualitative data analysis. However, if you still need it we can add it.
Small note on Line 460: “Some” is defined twice in this paragraph.
Action
We fixed it
Findings, Conclusions: The organization of the findings made more sense to me in this revised version. I don’t see any further revisions as being necessary.
Thank you
Author Response File: Author Response.docx