A Study of Student and Teacher Challenges in Smart Synchronous Hybrid Learning Environments
Round 1
Reviewer 1 Report
The topic the article tackles is very relevant for the times we are living, as synchronous hybrid learning environments could offer an excellent solution. The article emphasizes this aspect, but also gives an overview of the resources needed and challenges involved, for institutions, teachers and students, in a very coherent and scientifically supported way. Consequently, from my perspective, the article gives a very useful insight into the know-how with respect to this novel form of education.
My suggestion is that the authors should look into the following minor slips:
L34 - decisions about to adapt - decisions about how to adapt
L41, L42 – that data – those data
L77 - teachers level of experience – teachers’ level of experience
L 95, L 97, L 301 – see punctuation marks
L 127 – teachers actions – teachers’ actions
L 337 - ENAs, a pairwise comparisons - ENAs, pairwise comparisons
My suggestion is that the authors should look into the following minor slips:
L34 - decisions about to adapt - decisions about how to adapt
L41, L42 – that data – those data
L77 - teachers level of experience – teachers’ level of experience
L 95, L 97, L 301 – see punctuation marks
L 127 – teachers actions – teachers’ actions
L 337 - ENAs, a pairwise comparisons - ENAs, pairwise comparisons
Author Response
Dear Reviewer 1,
Based on your suggestion, we have made revisions addressing the issues raised by the Reviewer in their comments, and re-submit this article.
We appreciate the reviewer’s comment raising this issue. We have made all the suggested changes to improve the writing and clarity of the English language.
Thank you,
The authors of the article.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Title: Analysis of orchestration load and teacher agency in smart synchronous hybrid learning environments. The paper is good, and the topic is interest. However, the following are my comments:
1- Abstract needs some improvement.
2- In introduction, authors should specify the research contributions clearly.
3- Authors should provide critical analysis of these related work.
4- The authors should add some figures related to the taxonomy of the analytics techniques done in the literature to be clearer for the readers.
5- In addition, the references used in the literature can be analyzed, for example by year of publication, publisher, …etc.
6- The materials and methods are the core of the paper; therefore, authors should provide a detail of the steps/ approach that follow to be clear for the readers.
7- The aims of using three experiments should be highlighted and explained more clearly.
8- Please highlight what are the novel aspects of this work.
9- In addition, what are the main role that can be provided through this work?
10- It would be much appreciated if authors could include potential future research directions and remaining challenges as a new section, since this would be more meaningful within in the scope of this manuscript.
11- More explanation should be done clearly and in more details. Particularly, how the three experiments exactly can reflect the objectives of this work.
12- Based on the research questions and contributions of this work, please explain what the exact significance of the paper that can be added to the field of study.
Thank you for the work.
Minor editing of English language required.
Author Response
Dear Reviewer 2,
Based on your suggestion, we have made revisions addressing the issues raised by the Reviewer in your comments, and re-submit this article.
What follows is a detailed summary of the changes that have been made in the article. To facilitate the review process, we have included the comments made at the beginning of each section, followed by a description of the changes that we made in response to the issues raised. We have also included the paragraphs where the changes were made, highlighting the changes in bold font.
Thank you,
The authors of the article.
“1- Abstract needs some improvement.”
We appreciate the reviewer’s comment. We have tried to make the contribution of the paper clearer in the abstract. The contribution of the article is to analyse the orchestration load and teachers agency in various SSHLEs.
Abstract
|
The COVID-19 pandemic has led to the growth of hybrid and online learning environments and the trend to introduce more technology into the classroom. One such change would be the use of smart synchronous hybrid learning environments (SSHLE), which are settings with both onsite and online students concurrently, and in which technology plays a key role in sensing, analyzing, and reacting throughout the teaching and learning process. These changing environments and the incorporation of new technologies can place a greater orchestration load on participants and a reduction in teacher agency. In this light, this paper aims to analyse the orchestration load and teacher agency across various SSHLEs. The NASA-TLX model was used to measure the orchestration load in several scenarios. Questionnaires and interviews were used to measure teacher agency. The results obtained indicate that the orchestration load of the teacher tends to be high (between 60 and 70 points out of 100 of the NASA-TLX workload), especially when they lack experience in synchronous hybrid learning environments, and the orchestration load of the students tends to have average values (between 50 and 60) in the SSHLEs analysed. Meanwhile, the teacher agency does not appear to be altered but shows potential for improvement. |
“2- In introduction, authors should specify the research contributions clearly.”
We appreciate the reviewer’s comment raising this issue. We have expanded the introduction. This extension includes a better clarification of the contribution and the novelty of the study.
Section “Introduction”, paragraph 5
|
The main contribution sought in this study is to find the factors that influence the orchestration load of the teacher and students, and teacher agency in the particular context of the implementation of collaborative learning situations in SSHLEs. This contribution is novel because it addresses scenarios not featured in the related literature, and is complemented by different analyses of these scenarios. To this end, two research questions are posed: |
“3- Authors should provide critical analysis of these related work.”
We appreciate the reviewer’s comment. We have examined the literature and, finding a lack of studies in this field, have attempted to analyse some works related to certain aspects of this area. The study by Raes et al., 2020 [3] is the closest to our topic, so we have expanded on its analysis.
Section “Introduction”, paragraph 1
|
... Therefore, the use of SHLEs is not restricted to situations where social distancing restrictions apply but can be beneficial to make access to formal learning more flexible than in traditional educational settings. However, while SHLEs have the potential to support education, they also pose important challenges. For example, the implementation of these environments requires significant investment in technology, infrastructure, and teacher training to ensure an effective learning experience [3]. In the related article by Raes et al. [3], they conduct an in-depth study of various SHLEs, contributing further to a better understanding of their complexities and possibilities. Despite the advantages of SHLEs, more research is needed to better understand their impact on student learning and performance as well as to identify best practices in their implementation [5]. This will enable educational institutions to make more informed and effective decisions about to adapt to the educational challenges posed by the pandemic and to implement more effective long-term learning environments. |
“4- The authors should add some figures related to the taxonomy of the analytics techniques done in the literature to be clearer for the readers.”
We appreciate the reviewer’s comment. Regrettably, conducting an in-depth taxonomy of the analysis proves challenging due to the dearth of literature directly related to the subject matter under investigation. Despite diligent exploration, only a limited number of studies pertinent to this topic are, to our current understanding, published and available for review.
“5- In addition, the references used in the literature can be analyzed, for example by year of publication, publisher, …etc.”
We appreciate the reviewer’s comments raising this issue. Unfortunately, it is not possible to conduct a very profound analysis on the existing literature given the scarcity of literature directly related to our topic and few studies published, to the best of our knowledge. Nevertheless, the references used are recent (Raes in 2020, Tabuenca in 2021, Bulow in 2022).
“6- The materials and methods are the core of the paper; therefore, authors should provide a detail of the steps/ approach that follow to be clear for the readers.”
We appreciate the reviewer’s comment. To enhance the clarity of the materials and methods section, a new paragraph has been added specifying the content and purpose of this section.
Section “Materials and Methods”, paragraph 1
|
This section lays out the foundation of the research process. It provides an explanation of the methodological approaches taken in this study, thus providing the necessary context for understanding the findings and subsequent conclusions. |
“7- The aims of using three experiments should be highlighted and explained more clearly.”
We appreciate the reviewer’s comment raising this issue. As the objectives of the scenarios were not clearly defined, they have now been specified in various parts of the Materials and Methods section, under the Experiments subsection.
Section “Materials and Methods”, subsection “Experiments”, paragraph 1
|
The three experiments conducted are summarised in Table 2. The first experiment was conducted at the Catholic University of Louvain (KU Leuven, Belgium) due to their experience in SSHLEs and the available classrooms with the appropriate technology for these environments [3]. The aim of this experiment was to study a SSHLE, in this case a classroom with greater incorporation of specific technology to cover hybrid learning, and where the teacher and students had more experience in this type of environment. The experiment was carried out in a session of the university course where a collaborative learning situation was to be implemented... |
Section “Materials and Methods”, subsection “Experiments”, paragraph 2
|
The second experiment was conducted at Universidad Carlos III de Madrid (UC3M, Spain) and involved participants from Universidad de Valladolid (UVa, Spain), Universitat Pompeu Fabra (UPF, Spain) and UC3M. The aim of this experiment was to study a classroom with technologies usually available (blackboard, projector, speakers, and computers) transformed into a SSHLE with the addition of software to support it. To do this, a one-hour workshop was carried out with online (6 students) and onsite (11 students) participants in this classroom… |
Section “Materials and Methods”, subsection “Experiments”, paragraph 3
|
The last experiment was conducted at UC3M and involved participants from UVa and UC3M. The aim of this experiment was to study and repeat the approach of the second experiment, a simple classroom transformed into an SSHLE with the addition of software to support it, but with participants and a teacher more familiar with the SSHLEs. This experiment was intended to collect information from participants with greater experience in these environments for a more effective comparison. A one-hour workshop was planned with 3 online students and 9 onsite students… |
“8- Please highlight what are the novel aspects of this work.”
We appreciate the reviewer’s comment. We modified the introduction and conclusions to clarify the novel aspects of this work. These changes highlight the characteristics that make this study novel.
Section “Introduction”, paragraph 5
|
The main contribution sought in this study is to find the factors that influence the orchestration load of the teacher and students, and teacher agency in the particular context of the implementation of collaborative learning situations in SSHLEs. This contribution is novel because it unfolds in scenarios not featured in the related literature, and is complemented by different analyses of these scenarios. To this end, two research questions are posed: |
Section “Conclusions”, paragraph 1
|
... In general, the main innovation in this study refers to the introduction and analysis of scenarios that are not addressed in the related literature. |
“9- In addition, what are the main role that can be provided through this work?”
We appreciate the reviewer’s comment raising this issue. We modified the introduction and conclusion sections to clarify the contributions of this paper. These modifications serve to emphasize the unique aspects that distinguish this study. Consequently, we hope to have detailed the contribution sufficiently.
Section “Introduction”, paragraph 5
|
The main contribution sought in this study is to find the factors that influence the orchestration load of the teacher and students, and teacher agency in the particular context of the implementation of collaborative learning situations in SSHLEs. This contribution is novel because it unfolds in scenarios not featured in the related literature, and is complemented by different analyses of these scenarios. To this end, two research questions are posed: |
Section “Conclusions”, paragraph 1
|
... In general, the main innovation in this study refers to the introduction and analysis of scenarios that are not addressed in the related literature. |
“10- It would be much appreciated if authors could include potential future research directions and remaining challenges as a new section, since this would be more meaningful within in the scope of this manuscript.”
We appreciate the reviewer’s comment raising this issue. As recommended, a new subsection was created to present the future work. Additionally, this has been leveraged to provide more clarity to the text and better specify the next steps to be taken.
Section “Conclusions”, subsection “Future work”, paragraph 1
|
The following steps include conducting further experiments in other SSHLEs with a different distribution of the hybrid learning environment, for example, in telepresence classrooms. These classrooms consist of two rooms, one with the teacher and a group of students, and the other with the remainder of the students. On one of the walls of each room, there is a projection of the other room, simulating a connection between the two. Another area for future work is the incorporation of the features recommended by teachers in the SSHLEs and evaluating their impact on improvement. The final proposed future work is to replicate these experiments in more realistic scenarios, with participants from university courses as in Experiment 1 |
“11- More explanation should be done clearly and in more details. Particularly, how the three experiments exactly can reflect the objectives of this work.”
We appreciate the reviewer’s comment. As the objectives of the scenarios were not clearly defined, they have now been specified in various parts of the Materials and Methods section, in the Experiments subsection. This is expected to make it clearer how it influences the final aim of the study.
Section “Materials and Methods”, subsection “Experiments”, paragraph 1
|
The three experiments conducted are summarised in Table 2. The first experiment was conducted at the Catholic University of Louvain (KU Leuven, Belgium) due to their experience in SSHLEs and the available classrooms with the appropriate technology for these environments [3]. The aim of this experiment was to study a SSHLE, in this case a classroom with greater incorporation of specific technology to cover hybrid learning, and where the teacher and students had more experience in this type of environment. The experiment was carried out in a session of the university course where a collaborative learning situation was to be implemented... |
Section “Materials and Methods”, subsection “Experiments”, paragraph 2
|
The second experiment was conducted at Universidad Carlos III de Madrid (UC3M, Spain) and involved participants from Universidad de Valladolid (UVa, Spain), Universitat Pompeu Fabra (UPF, Spain) and UC3M. The aim of this experiment was to study a classroom with technologies usually available (blackboard, projector, speakers, and computers) transformed into a SSHLE with the addition of software to support it. To do this, a one-hour workshop was carried out with online (6 students) and onsite (11 students) participants in this classroom… |
Section “Materials and Methods”, subsection “Experiments”, paragraph 3
|
The last experiment was conducted at UC3M and involved participants from UVa and UC3M. The aim of this experiment was to study and repeat the approach of the second experiment, a simple classroom transformed into an SSHLE with the addition of software to support it, but with participants and a teacher more familiar with the SSHLEs. This experiment was intended to collect information from participants with greater experience in these environments for a more effective comparison. A one-hour workshop was planned with 3 online students and 9 onsite students… |
12- Based on the research questions and contributions of this work, please explain what the exact significance of the paper that can be added to the field of study.
We appreciate the reviewer’s comment raising this issue. As already indicated in the response to comment 9, we modified the introduction and conclusion sections to clarify the contributions of this paper. These modifications serve to highlight the unique aspects that distinguish this study. As a result, we hope that we have sufficiently detailed the contribution.
Section “Introduction”, paragraph 5
|
The main contribution sought in this study is to find the factors that influence the orchestration load of the teacher and students, and teacher agency in the particular context of the implementation of collaborative learning situations in SSHLEs. This contribution is novel because it unfolds in scenarios not featured in the related literature, and is complemented by different analyses of these scenarios. To this end, two research questions are posed: |
Section “Conclusions”, paragraph 1
|
... In general, the main innovation in this study refers to the introduction and analysis of scenarios that are not addressed in the related literature. |
Author Response File: Author Response.pdf
Reviewer 3 Report
Dear Authors,
Thanks for giving me a chance to read this manuscript, “Analysis of orchestration load and teacher agency in smart synchronous hybrid learning environments”. The current paper analyses the orchestration load and teacher agency in different SSHLEs and reports the orchestration load of the teacher tends to be high (between 60 and 70 10 points out of 100 of the NASA-TLX workload), especially when they lack experience in synchronous hybrid learning environments, and the orchestration load of the students tends to have average values (between 50 and 60) in the SSHLEs analysed.
This is an interesting and significant topic in the field of smart education. There are minor issues in the current manuscript that should be carefully addressed to be further considered.
1. Literature
· Authors are advised to explicitly describe the contribution of the current study. For example, introduction does not explicitly emphasize the research gap and related contribution. Please further discuss it with previous research.
· Synchronous hybrid learning environments are applications in smart education. Authors are also advised to compare this form with robot-assisted environment (Song et al., 2023), which is also an emerging area for smart education
Ref:
Song, Y., Tao, D., & Luximon, Y. (2023). In robot we trust? The effect of emotional expressions and contextual cues on anthropomorphic trustworthiness. Applied ergonomics, 109, 103967.
2. Method
· How was the sampling conducted? The authors just mentioned recruiting students on campus.
· Is it a convenience sampling? How would you ensure the reliability and validity of your sampling techniques.
To sum up, I personally like this paper. However, the problems should be addressed in order to be further considered. Hope these suggestions help.
sufficient for publication
Author Response
Dear Reviewer 3,
Based on your suggestion, we have made revisions addressing the issues raised by the Reviewer in their comments, and re-submit this article.
What follows is a detailed summary of the changes that have been made in the article. To facilitate the review process, we have included the comments made at the beginning of each section, followed by a description of the changes that we made in response to the issues raised. We have also included the paragraphs where the changes were made, highlighting the changes in bold font.
Thank you,
The authors of the article.
“Authors are advised to explicitly describe the contribution of the current study. For example, introduction does not explicitly emphasize the research gap and related contribution. Please further discuss it with previous research.”
We appreciate the reviewer’s comment raising this issue. We modified the introduction and conclusion sections to clarify the contributions of this paper. These modifications serve to emphasize the unique aspects that distinguish this study. Consequently, we hope to have detailed the contribution sufficiently.
Section “Introduction”, paragraph 5
|
The main contribution sought in this study is to find the factors that influence the orchestration load of the teacher and students, and teacher agency in the particular context of the implementation of collaborative learning situations in SSHLEs. This contribution is novel because it addresses scenarios not featured in the related literature, and is complemented by different analyses of these scenarios. To this end, two research questions are posed: |
Section “Conclusions”, paragraph 1
|
... In general, the main innovation in this study refers to the introduction and analysis of scenarios that are not addressed in the related literature. |
“How was the sampling conducted? The authors just mentioned recruiting students on campus.”
We appreciate the reviewer’s comment. It is true that it was unclear how participants were selected or if there were any restrictions. To clarify this, a brief explanation has been included in the description of each of the three experiments.
Section “Materials and Methods”, subsection “Experiments”, paragraph 1
|
… The experiment was carried out in a session of the university course where a collaborative learning situation was to be implemented. There were no restrictions for participating in this session; so all the students registered in this course were able to participate. This session lasted 2 hours and was attended by 22 online students and 24 onsite students… |
Section “Materials and Methods”, subsection “Experiments”, paragraph 2
|
… To do this, a one-hour workshop was carried out with online (6 students) and onsite (11 students) participants in this classroom. An open call was launched, seeking individuals from collaborating universities to participate in this workshop, with no restrictions placed on participation. The software Engageli was used to support this SSHLE… |
Section “Materials and Methods”, subsection “Experiments”, paragraph 3
|
… This experiment was intended to collect information from participants with greater experience in these environments for a more effective comparison. A one-hour workshop was planned with 3 online students and 9 onsite students. An open call was launched, seeking individuals from collaborating universities to participate in this workshop, with no restrictions placed on participation. The software Engageli was also used to support this SSHLE, and in addition, the teacher had gained more experience as it was the same as in Experiment 2… |
“Is it a convenience sampling? How would you ensure the reliability and validity of your sampling techniques.”
We appreciate the reviewer’s comment raising this issue. The challenge of conducting real scenarios limits the selection of participants. In the paper, we specify that in the first scenario all students from the course could participate, and for the second and third scenarios an open call was made in the collaborating universities. Moreover, due to this limitation, we have added the execution of additional experiments in real situations as future work.
Section “Conclusions”, subsection “Future work”, paragraph 1
|
The following steps include conducting further experiments in other SSHLEs with a different distribution of the hybrid learning environment, for example, in telepresence classrooms. These classrooms consist of two rooms, one with the teacher and a group of students, and the other with the remainder of the students. On one of the walls of each room, there is a projection of the other room, simulating a connection between the two. Another area for future work is the incorporation of the features recommended by teachers in the SSHLEs and evaluating their impact on improvement. The final proposed future work is to replicate these experiments in more realistic scenarios, with participants from university courses as in Experiment 1. |
Author Response File: Author Response.pdf
