Developing a Performance Evaluation Framework Structural Model for Educational Metaverse^†

Round 1

Reviewer 1 Report

The manuscript introduces a novel and timely framework for evaluating performance within the educational Metaverse, which is both innovative and relevant to current technological trends. The authors' integration of the Theory of Planned Behavior, the Unified Theory of Acceptance and Use of Technology, and the Community of Inquiry Framework into a comprehensive model is commendable. This approach provides a solid theoretical foundation for their proposed Key Performance Indicators, offering a nuanced understanding of avatar-mediated educational interactions in the Metaverse.

However, a few areas may require further attention before publication:

1. Empirical Validation: The manuscript outlines a strong theoretical framework but seems to lack empirical validation of the proposed model. Future research directions should emphasize the need for empirical studies to test the framework's applicability in real-world educational settings within the Metaverse.
2. Technological Advances: Given the rapid pace of technological advancement in virtual environments, it would be beneficial if the authors could discuss how their model adapts to or incorporates emerging technologies beyond the scope of their current review.
3. Diversity and Inclusion: The manuscript could further address how the model accounts for diversity and inclusion within the Metaverse, considering the global and culturally diverse nature of virtual learning environments.
4. Practical Implementation: Additional insights into the practical implementation of the model, including challenges and opportunities for educators and institutions, would enhance the manuscript's applicability and relevance to practitioners.
   
   The authors should consider broadening their research to include references from related fields such as game-based learning and serious games, for instance, CMX MMORPG. It would enrich the study to explore how these domains have applied design and evaluation frameworks, as well as how learning analytics have been utilized within these contexts. This addition would not only deepen the theoretical foundation of the manuscript but also provide a more comprehensive view of the application of similar frameworks in various educational technologies, enhancing the relevance and applicability of the proposed model.
   
   

The manuscript is well-written and demonstrates a good command of English language, facilitating clear communication of complex ideas and theoretical concepts. However, to further enhance the readability and academic rigor of the text, I recommend a careful review for minor grammatical and syntactical improvements.

Author Response

We addressed Reviewer 1's feedback comprehensively, detailing our approach to empirical validation with Structural Equation Modeling (SEM) in a Metaverse-based educational setting at Neapolis University Pafos. Our manuscript updates showcase the model's adaptability to emerging technologies, integrating theoretical frameworks like TPB, UTAUT, and CoI. We revised the discussion on diversity and inclusion, emphasizing adaptable models for diverse educational contexts and the importance of creating inclusive content. Our practical implementation is highlighted through a case study, providing clear adoption guidelines. Additionally, our literature review was expanded to include game-based learning and serious games. Finally, we meticulously reviewed and corrected grammatical errors to enhance the manuscript's clarity and precision.

Author Response File: Author Response.pdf

Reviewer 2 Report

The main objective of this study is to propose a performance evaluation framework for an educational meta-universe environment. It is a very important research topic in the educational field. In the literature review section, the theoretical foundations and orientations are clearly presented. The literature review clearly discusses the relationship between variables in different theories.

The article mentions that structural equation modeling will be used to build the whole evaluation framework, but this is weakly related to this study. Structural equation modeling is a statistical approach to building and validating models, but in this study, no related statistical data were analyzed and presented.

Author Response

We deeply appreciate your insightful feedback and constructive criticism on our manuscript. Your comments have been instrumental in refining our research and presentation. Here's a brief overview of the actions we've taken in response to your review:

• Empirical Validation and SEM Application: We've enriched the manuscript with a clearer articulation of our Structural Equation Modeling (SEM) approach and detailed our plan for empirical validation. This initiative is part of a broader PhD project, with empirical validation actively underway. We aim to provide substantial evidence of the SEM's effectiveness within a Metaverse-based educational setting.
• Revisions Highlighted: The revisions and enhancements made in response to your feedback are meticulously marked in the resubmitted manuscript. We believe these changes significantly improve the clarity, depth, and rigor of our study.
• Commitment to Clarity and Rigor: Although the quality of English was not within your evaluation scope, we undertook a thorough review to ensure the manuscript's language is academically sound and accessible.

Author Response File: Author Response.pdf

Reviewer 3 Report

The article entitled "Developing a Performance Evaluation Framework Structural Model for Educational Metaverse" is relevant to the mission of the journal, as it contributes to increasing the field of knowledge in relation to the management of virtual environments, combining traditional pedagogical principles with contemporary evaluation methodologies.

The abstract of the article is adequate and briefly explains the content of the study conducted. In the abstract, the semicolon should be removed.

The key words are also adequate.

The structure of the article is also in line with the structure of a scientific paper.

The title of the article corresponds to the content of the document and its main objective is none other than to offer a causal model of the relationships between the students' self-perception of their technical and cognitive skills and learning path variables such as the semester and, from this, the development of student typologies.

The document is well structured, facilitating the understanding of the study carried out. The theoretical foundation is based on the study of the metaverse as a mixture of 3D virtual spaces shared in the development of tools for the evaluation of educational experiences. The evaluator also highlights table 1, which presents the key performance indicators proposed by the authors. Table 2 shows the latent variables.

Method: the method is a descriptive study with the aim of proposing a structural model to evaluate the performance of the academic avatar within the educational metaverse.

The research phases are presented in a clear and structured way. 

The figures and tables used contribute to facilitate the reader's understanding.

This evaluator considers that the discussion of the model presented is adequate. Likewise, I consider that the authors should make explicit possible lines of research, as well as an exclusive section called "contributions to educational practice". 

In short, I consider that this is a work that will undoubtedly contribute to performance management for one in virtual platforms for a better assessment of skills and workforce.

Author Response

We have meticulously revised our manuscript to clarify the abstract and further illuminate our research's implications for educational practice. We are committed to engaging in further discussions and are ready to provide additional information or clarification as needed. Thank you for guiding us towards enhancing the impact and clarity of our work.

Author Response File: Author Response.pdf

Reviewer 4 Report

According to the authors, this study introduces a novel Performance Management framework for virtual learning environments in the Metaverse era, utilizing Structural Equation Modeling (SEM). The model integrates the Theory of Planned Behavior, Unified Theory of Acceptance and Use of Technology, and Community of Inquiry Framework, synthesizing five Key Performance Indicators to assess academic avatar performance. Focused on Content Delivery, Student Engagement, Metaverse Tool Utilization, Student Performance, and Adaptability, the model advances understanding and enhances avatar efficacy in virtual educational settings. This research contributes to Performance Management discourse in digital education, laying a foundation for future empirical studies and informing educational assessment practices and HRM strategies in the evolving metaverse.

While there appears to be a gap in the Performance Management framework for virtual learning environments, especially in the Metaverse era, this article introduces a novel framework. However, a systematic literature review to identify principal factors in educational metaverse performance is merely the beginning of research activity. The authors' proposed structural model introduces a novel approach to evaluating academic avatar performance in the educational Metaverse through the integration of theories such as the Theory of Planned Behavior (TPB), Unified Theory of Acceptance and Use of Technology (UTAUT), and Community of Inquiry Framework etc. This insight is commendable but insufficient. Longitudinal studies are needed to examine the adaptability and scalability of the proposed structural model over time. At least an initial empirical validation is required to validate the proposed model's effectiveness. This article provides valuable initial insight but lacks empirical evidence to support the proposed model.

Minor issues.

Author Response

Thank you for your thorough and insightful feedback. Your comments have been instrumental in refining our manuscript, particularly in expanding our literature review and improving our methodology description. We've taken to heart your suggestions on empirical validation and have detailed our approach to address this critical aspect. Additionally, we've corrected the minor grammatical issues you highlighted, aiming to enhance the clarity and readability of our work. 

Author Response File: Author Response.pdf

Round 2

Reviewer 4 Report

While several paragraphs have been added to the manuscript, specifically lines 539-575, there is not a single-subject experiment (e.g., a case study). Consequently, NO DATA WERE COLLECTED, and consequently, there is no data analysis and no minimum empirical validation. The authors did not address the following major recommendation.

"At least an initial empirical validation is required to validate the proposed model's effectiveness. This article provides valuable initial insight but lacks empirical evidence to support the proposed model."

Author Response

None