Exploring User Experience and Usability in a Metaverse Learning Environment for Students: A Usability Study of the Artificial Intelligence, Innovation, and Society (AIIS)

Abstract

The metaverse, a rapidly evolving research area, is a virtual, interconnected universe that seamlessly integrates digital and physical realms, facilitating immersive experiences, social interaction, and economic activities across various fields, including computer science, business, and education. Despite its potential to transform current pedagogies and learning experiences through engaging, interactive, and fascinating educational techniques, research is limited in exploring its application, usability, and user experience, particularly for Thai undergraduate students in learning computer engineering courses. In response, we designed and developed an innovative metaverse-based learning system called the AIIS collaborative learning interface, tailored to European undergraduate students. While the original intent of AIIS was to cater to medical students, in this study, our objective was to assess its usability and applicability for computer science and engineering students, specifically focusing on the Asian demographic. After an initial pilot with European students in their local context, the research was extended to Thailand, where 21 Thai undergraduate students evaluated the platform’s usability. The observations from this study indicate that AIIS provides a dynamic, user-friendly learning environment. Nevertheless, the evaluation process unveiled some usability flaws that offer informative directions for future enhancements. Drawing from our observations, we formulated usability guidelines to guide designers and developers toward crafting more efficient metaverse systems, particularly those intended for educational use. Our results also underline the considerable promise that metaverse technologies hold for advancing higher education globally.

1. Introduction

In recent decades, experts in research, development, and industry, particularly those concentrating on human–computer interaction, computer engineering, and information systems, have effectively utilized the potential of cutting-edge technologies to enhance educational experiences for both students and educators [1]. For instance, researchers have employed digital games to enrich students’ learning experiences (e.g., motivation and engagement) and boost their academic performance [2]. Researchers have also looked into gamification, which is the use of game design concepts in contexts other than games [3], as an alternative approach to motivate and engage students in the learning processes. As highlighted by its applications in medicine [4], Virtual Reality (VR)—a simulated digital environment with which users can interact—has piqued the interest of researchers, educators, and students in the educational field, significantly enriching their instructional and learning experiences. Similarly, additional immersive technologies have captivated researchers across education and other disciplines. These include Mixed Reality (MR), a novel combination of VR and AR that enables user interaction with both physical entities and virtual elements, and Augmented Reality (AR), a cutting-edge technology that superimposes digital data onto our material world. Extended Reality (XR) is an umbrella term for the full spectrum of modern and cutting-edge technology, signifying a fusion of the real and the virtual. This technology has the ability to redefine how we teach, learn, and explore in a variety of contexts in addition to offering an enhanced interactive environment.

Researchers and educators have recently incorporated VR technologies into existing teaching methodologies and pedagogies, recognizing their potential to create dynamic, entertaining, and engaging learning environments for students that foster a deeper understanding of various subjects (e.g., vocational training) [5]. For instance, vocational schools in Finland have recently started using and integrating VR-based learning for students [6]. Furthermore, in the business landscape, corporations have leveraged VR applications for commercial use in education. For example, to promote the interactive learning experience, ADE has innovatively engineered a VR system for training in firefighting at construction sites. This state-of-the-art system integrates a fire extinguisher with VR glasses, elevating the training experience to new heights. In ADE’s case study [7], the fire extinguisher, outfitted with multiple sensors, allows for real-time monitoring of key actions like pin removal and foam distribution. This innovatively designed extinguisher can be utilized throughout various firefighting drills, ensuring comprehensive and effective training [7].

Similarly, augmented reality (AR) technology has significantly transformed medical education, allowing healthcare practitioners, educators, and students to interact with virtual components within actual environments [8]. As technology continues to evolve, AR is set to play an increasingly important role in streamlining and developing medical learning processes. However, it is worth noting that the market still lacks robust, intuitive AR devices specifically tailored to the unique requirements of medical education and associated fields. This presents a substantial opportunity for technological advancements and market development. Considerations might include creating devices with heightened precision for surgical simulations, increased comfort for prolonged usage, or superior integration capabilities with existing medical systems and software for a seamless user experience. Leveraging these ideas could help solidify the role of immersive technologies in the future of medical education [8].

Recently, the metaverse—a virtual realm with three-dimensional dimensions where real individuals manifest as avatars within [9]—has captivated researchers and developers across diverse fields and disciplines, fueling interest in its potential applications in various domains such as education, entertainment, and enterprise. The Metaverse, a virtual shared digital space where real individuals appear as avatars in a three-dimensional realm, is garnering interest across diverse fields for its potential applications in education, entertainment, and enterprise, offering an interconnected virtual universe for socializing, gaming, and commerce [10]. Blending physical and virtual realities creates an immersive digital universe for users facilitated by VR and AR [11]. This interconnected network of dynamic virtual spaces enables real-time user interaction with digital content and virtual objects. Initially comprising avatar-based virtual worlds, the metaverse has evolved to encompass social VR platforms for large-scale multiplayer games and cooperative virtual and AR environments [11]. As per [12], the metaverse concept can be viewed as expanding the Virtual Continuum (VC) idea. According to [12,13], the authors highlight the importance of creating an optimal virtual space that incorporates crucial aspects of reality to facilitate communication within a cohesive visual display environment.

The study by Said [14] explores the metaverse—a sophisticated integration of information technology with human–computer interaction at its core—and its vast potential to reshape the educational landscape. Investigating the opportunities and challenges of applying metaverse technology in education, Said gathered insights from 19 experts and identified five challenges: immersive design, privacy and security, universal access, health concerns, and governance. The study also underscores the potential for hands-on training, game-based learning, and collaborative knowledge creation within the metaverse. Its findings aim to enhance our comprehension of online learning in the metaverse, thus encouraging further exploration of its viability as a virtual communication and learning platform. A subsequent study [15] focuses on the application and advantages of the metaverse within the educational sector. Given the fact that the metaverse encompasses all potential digital aspects of our future, significant restructuring within fields like education is essential to ensure lasting accessibility and viability. Although digital resources offer benefits such as interactivity and portability, they are not without challenges, including technical, infrastructural, and managerial hurdles. As such, this study dives into several facets of integrating the metaverse into education. Lastly, study [16] explores the escalating interest in and daily usage of the metaverse. Examining its historical development, architecture, and applications in education, this research underscores the benefits and challenges of the metaverse. With the metaverse still being a new concept and its resources currently being scarce, the study suggests that it holds the potential to revolutionize educational technologies. However, as the technology underpinning the metaverse is still in development, creating strategic plans for its integration into education and analyzing its broader implications as it matures is critical.

Based on the existing literature, there is a growing trend in using metaverse technology in various fields, including retail, education, entertainment, and manufacturing. For instance, integrating the metaverse in retail can transform consumer experiences by offering personalized, engaging, and interactive shopping environments, as evidenced by a study examining the impact of virtual reality on consumer behavior and decision making [17]. According to Nokia, one of the pioneers in using metaverse technology in business and industry, the metaverse is a transformative digital space that combines cutting-edge technologies, driven by human augmentation and the fusion of digital and physical, with multiple metaverses catering to different sectors, relying on robust networks and connectivity, and culminating in a generational leap with the arrival of 6G, emphasizing collaboration, openness, safety, security, sustainability, and inclusivity for its full realization [18]. In [19], the usability of virtual-reality-based systems is compared through comparing Spatial.io and Gather.Town with Zoom. The findings show the potential of such platforms for student engagement, enjoyment, and sociability in online learning experiences. Also, in [20], researchers investigate user participation and behavior patterns during VR live events, aiming to identify factors that enhance user experience and promote better VR event design in the future. Study [21] discusses the ways in which luxury brands could embrace emerging digital technologies such as metaverse, digital fashion, and non-fungible tokens (NFT) to create new opportunities and enhance customer or consumer experiences.

Metaverse technology has the potential to revolutionize sectors including gaming, entertainment, education, and the workplace by introducing novel interaction and engagement avenues. According to [22], the metaverse offers features that facilitate immersive encounters, digital twin integration, and social interactions. In 2021, the “metaverse” concept became notably prevalent. Microsoft, for example, envisions a corporate metaverse integrating digital twins, IoT, and mixed reality as a foundational layer [23]. PwC’s study emphasizes the potential of VR-driven learning to usher in a transformative phase in corporate training, offering an engaging, affordable, and effective method to impart soft skills to employees [24]. Highlighting its commitment to the metaverse, Facebook rebranded to Meta [25], anticipating a 10–15-year timeline for the full fruition of metaverse offerings [26]. Nokia segments the metaverse market into consumer, enterprise, and industrial categories [27,28]. Major tech firms, as indicated by [29], have launched bold endeavors to advance the metaverse, notably in healthcare [29,30,31]. Further, [32] suggests that enhancing life quality entails creating offerings that ease the blending of the virtual and tangible worlds in both daily activities and relaxation. Innovations like the metaverse are instrumental in enhancing experiences in this hybrid environment, paving new paths for professional and recreational pursuits.

This study primarily focuses on employing metaverse technology in the educational sector, a burgeoning yet highly prospective field of research. The expanding body of literature reinforces this perspective. It underscores the need to probe into novel methodologies, cross-disciplinary strategies, and the influence of breakthrough technologies in nurturing an advanced educational milieu. Furthermore, there has been a recent surge in the transformation of traditional learning environments by researchers, developers, educators, and instructional designers. This transformation has been made possible by leveraging emerging technologies (e.g., metaverse), thus leading to more immersive, engaging, and inspirational educational settings for students. According to [5], implementing the metaverse in education can revolutionize conventional learning experiences by providing immersive, interactive, and personalized educational context, as demonstrated by a study on virtual reality in educational settings. In study [33], authors explore the application of the metaverse in education—a 3D virtual learning platform fostering interaction among students, educators, and digital objects. This platform’s potential and AI’s role in augmenting educational experiences are emphasized. Critical issues such as user experience, privacy, and content creation warrant further research to fully harness the metaverse’s potential in education.

Addressing numerous challenges and ethical considerations is crucial as the metaverse develops to ensure its safe and responsible integration into our daily lives, especially in education. Although the potential of using the metaverse for educational purposes is promising, research on its usability and application in learning is still in its infancy. Moreover, existing studies have yet to fully explore the interactions between users, specifically teachers and students, within this innovative virtual environment regarding education. It is also uncertain how existing user interface guidelines can be applied to the design of metaverse-based learning systems. To address these research gaps, this study presents the design and development of a metaverse-based system called Artificial Intelligence, Innovation, and Society (AIIS) Collaborative Learning Interface—further referred to as the AIIS interface or AIIS environment in this article. This immersive platform caters to graduate and undergraduate students, particularly in medicine and nursing. However, half the learning content in AIIS is related to computer science (e.g., introduction to artificial intelligence). Whether AIIS is suitable for computer engineering undergraduate students, particularly in Asia, has not yet been studied. Hence, to bridge these research gaps, we outlined the following research objectives in this study. The primary objectives of this research are: (1) to assess the usability of the AIIS interface for undergraduate students in Thailand, (2) to delve into their user experiences while using AIIS, (3) to understand whether the AIIS interface is suitable for computer engineering students, and (4) to offer design recommendations for creating metaverse-based learning systems. By exploring these aspects, the study aims to provide valuable insights for researchers and designers in developing user-friendly metaverse systems that cater to students in an educational context. Additionally, it opens possibilities for further investigation into user-centered design, usability, user experience, accessibility, and inclusivity within the metaverse, ensuring a more comprehensive understanding of ways in which this technology can benefit education and learning in the future.

2. Artificial Intelligence, Innovation, and Society (AIIS)

2.1. Background of AIIS

The main goal of the AIIS project is to offer a comprehensive medical and health technology program for engineering students, encompassing AI, innovation, and soft skills. This aims to foster its inclusion in European university curricula, possibly expanding to Asian universities in the future (e.g., Thailand and Singapore). The project seeks to utilize metaverse technology to facilitate collaboration between students from medical and engineering disciplines and simulate real-life problem-solving scenarios. This approach offers participants unique opportunities to enhance their professional understanding and multidisciplinary knowledge. The AIIS learning program consists of theoretical and practical components—all students in the AIIS learning program must complete both components. The practical (applied) part involves a machine learning task with accurate medical data, conducted outside the learning interface with the guidance of mentors, and accounts for one third of the workload. The other two thirds of the workload consist of the theoretical portion, which students independently explore through the AIIS learning interface within the metaverse. The theory material is accessible to students via desktop mode or a VR headset, presented in short (maximum 15 min) videos linked to interactive tasks for deeper learning of the topics. The AIIS program pedagogy follows a problem-centered approach focusing on the learner by considering their interests, study habits, and prior knowledge. The learning program pivots on a tri-axial teaching approach incorporating collaborative, reflective, and inquiry-based pedagogy. Backed by Erasmus+, the AIIS initiative introduced a cooperative learning platform on TUAS’s VR Social Platform. This platform educates medical students about artificial intelligence significance and soft skills in their forthcoming healthcare or engineering careers. A primary goal of this project is to investigate the feasibility of teaching theoretical content in a shared metaverse setting, ensuring a seamless integration of theory with practice. The AIIS curriculum, recognized for 3 ECTS, operates within a 3D space. Here, students can navigate and approach various stations presenting theoretical assignments on specific AI and soft skills topics (refer to Figure 1). The course’s teaching approach prioritizes the use of TUAS’s VR Social Platform’s interactive capabilities. In the fall of 2022, a comprehensive trial was undertaken with 115 medical and 10 engineering students from five European academic institutions.

2.2. AIIS Learning Contents and Pedagogical Design

The AI and soft skill modules contain six topics, each containing three to five instructional videos, followed by tasks for students to complete. The curriculum includes 59 theory tasks, 29 focusing on AI and 30 on soft skills. The AI module covers subjects such as AI introduction, expert systems in healthcare, machine learning, its application in healthcare, machine vision basics, and image recognition within the healthcare industry. The module aims to provide students with an extensive understanding of AI in healthcare. It empowers them to assess AI solutions critically, engage in informed discussions, adapt their practices to incorporate AI, and propose innovative applications for existing AI techniques. Although the AI module does not directly teach AI technique development, it imparts essential knowledge on the underlying architecture, applied mathematics, and programming algorithms, laying the foundation for a comprehensive understanding of AI techniques used in healthcare. The soft skills module addresses self-awareness, adaptability, communication, teamwork, work organization, and ethics, helping students cultivate critical interpersonal and professional skills for career advancement, stress management, and conflict resolution. Both modules are enhanced with real-world examples and case studies to deepen student understanding of concept relevance and applicability. Figure 2 displays a screenshot of AIIS’s AI training.

The soft skills module emphasizes the vital role of soft skills in professional development of students (see Figure 3). By engaging with the soft skills module, students can develop and practice highly sought-after skills within their respective fields (e.g., medicine, computer engineering). This module enables students to understand their thought processes, cultivate self-regulation techniques, and promote personal well-being. Additionally, students improve their communication skills, forge connections through various dialogues, work in groups, and engage with larger audiences. They also acquire strategies to improve focus, time management, and efficient planning. A key component of the module is understanding engineering ethics, especially concerning applied artificial intelligence. To fulfill the AIIS certificate requirements, students must collaboratively complete at least ten of the 59 theory tasks. Beyond the theory topics, students must select one specific challenge from various options, each tied to a subject area. These challenges involve applying foundational AI knowledge from the AIIS theory component to undertake a fundamental machine learning issue using real-world datasets. Ideally, groups of ten students work under a mentor’s guidance to tackle the chosen challenge, fostering the practical application of soft skills acquired through the AIIS learning interface’s theory component.

2.3. AIIS User Interfaces

Upon enrolling in the AIIS course, students receive a welcome email with a link to download the local client and a student authentication token. This token grants access to the learning interface, modules, and challenge selection area, while other tokens, like teachers and visitors, have distinct access rights. After installing the client and registering with their token, students enter the collaborative virtual learning environment (CVLE), represented by an avatar. They can choose between a first- or third-person view and communicate with others using push-to-talk when nearby. Initially, students arrive in a tutorial room to practice accessing and completing a topic task. After completing or skipping the tutorial, they can access the AIIS virtual environment on the main floor. Figure 4 shows the first- and third-person views of the AIIS interface on the ground floor. The three-story virtual environment houses AI learning materials on the ground floor, soft skills materials on the next level, and a rooftop challenge selection area. However, the challenge selection rooftop remains restricted during the first three weeks, allowing students the time to acquire foundational AI knowledge before choosing a challenge.

To acquire AI and soft skills knowledge, students must complete tasks which they can access via tablets dispensed from machines. Each AI and soft skills topic has its tablet dispenser throughout its respective module floors. Specifically, the ground floor houses six dispensers dedicated to AI, while the floor above accommodates an equal number of dispensers focusing on soft skills. Students can complete any task from any dispenser at any time, meaning the virtual learning environment pedagogy does not follow a linear learning structure. Each topic task comes with a theory content video lecture and a corresponding accompanying activity. The only compulsory mechanic for students is that every dispenser has an unlock task they must complete before the rest of the functions for that specific dispenser become available. However, students do not necessarily have to complete the unlock task successfully; they only need to submit an effort. Figure 5 illustrates the tablet dispenser for the Intro to AI topic, with three tasks that students must complete. Students can approach the dispenser and choose the task they want to do. The dispenser then releases a tablet that students can move around with freely. The interactive tablet includes a theory video for students to watch and an activity related to the video content that they need to complete. The AIIS learning content currently features 11 activity types, including crossword puzzles, diagram labeling, matching columns, concept mapping, synonym bingo, and more.

Each theory task in the AIIS environment is assigned a certain number of micro-credentials based on the estimated completion time, with one micro-credential equating to roughly 30 min of work. To complete the AIIS theory material (excluding the practical challenge), students must accumulate 105 micro-credentials from the 59 available tasks. They can repeat tasks until they submit a successful entry and earn the designated micro-credentials. Students can track their micro-credential progress using a dashboard accessible throughout their time in the AIIS environment. The dashboard features a general view (see Figure 6) displaying an overview of module-level learning progress and detailed views for the AI (see Figure 7) and soft skill modules, illustrating task-by-task progress. The dashboard also includes explanatory videos to help students navigate the AIIS environment mechanics and introductory videos for each module.

In addition to acquiring 105 micro-credentials, students are expected to accumulate at least ten collaborator points. These points are granted when students actively participate in tasks initiated by their peers. When a task commences, an indicator pole in the AIIS environment displays the available seats for collaboration. Each task allows for two open seats, and interested students can join the task area and select the collaboration option. Collaborators communicate with the task initiator through push-to-talk, with conversations limited to those within the immediate circle. Collaborator points are awarded to active collaborators in collaboration mode only when the initiating student submits a correct solution. Leaving the collaboration before submission results in no points being granted. The initiating student only earns micro-credentials for the task—no collaborator points are awarded to the initiator, regardless of whether others collaborated on the task. The AIIS environment also includes reflection zones adjacent to each topic’s task dispenser. These zones feature informative posters showcasing additional details, recent advancements, or news related to the completed topic. While participation in these reflection zones is optional, they serve as points of interest for students to read and contemplate.

Furthermore, the environment incorporates an interactive feature accessible through a control panel, enabling users to explore various physiological layers of a human head. This interactive feature is purely ornamental and does not affect the completion of assigned tasks. Its primary purpose is to observe how students engage with such an element within the environment (see Figure 8).

2.4. Learning Content and Pedagogy Summary

The AIIS program component completed within the AIIS digital learning environment consists of two modules (AI and Soft Skills), with six topics each. Each topic is broken down into tasks that students must complete. Each task has an accompanying theory video to help students complete the respective task. Students can jump between topics and tasks in any sequence (i.e., the AIIS program has non-linear learning content). In conducting a task, students earn micro-credentials for that specific task. Students must earn all 105 micro-credentials to complete the theory component of the AIIS program. Each topic and its associated tasks strive toward meeting the overall outcomes of the AI and soft skills modules.

Table 1. Topic Summary for the AIIS Program Theory Content.

Module	Topic	Number of Topic Tasks	Number of Topic Micro-Credentials
AI	Introduction to AI	3	4
	Expert systems and their role in the health care sector	5	50
	Introduction to machine learning	5	9
	Machine learning in the health care sector	5	9
	Introduction to machine vision	5	9
	Image recognition in the health care sector	6	10
Soft Skills	Self-knowledge and initiative	5	9
	Capacity to adapt to different situations	5	9
	Communication	5	9
	Teamwork	5	9
	Work organization	5	9
	Work ethic	5	9

presents the modules, topics, number of functions, and total micro-credentials for each topic.

The AI module intends to offer participants a high-level understanding of the prevalent AI in the healthcare sector. Upon completing this module, participants will be able to

• critically assess the contribution various AI solutions make to their work environments,
• have meaningful deliberation on AI propositions for the healthcare sector,
• adapt their working practices to facilitate the integration of AI into their workplace, and
• propose new use cases within the healthcare sector for existing AI techniques.

Although the AI module does not teach the skill for developing and applying relevant AI techniques, some essential knowledge of underlying architecture, applied mathematics, and programming algorithmics is introduced. This offera learners the foundational building blocks for a well-rounded understanding of the AI techniques utilized within the healthcare sector. The objective of the soft skills module is to make medical students aware of the importance of soft skills in their profession. Completing these topics allows them the development and practice of the most valued skills in the sector through different tasks. After completing this module, students

• become aware of their thought patterns and learn some internal self-regulation mechanisms that allow them greater personal well-being,
• learn techniques to better connect with people verbally and in writing in two-way interactions, small work groups, and large audiences,
• communicate with patients, family members, professionals, and the media,
• recognize and apply some strategies to improve concentration at work and their use of time and organizational planning, and
• become aware and reflect on medical ethics, especially in artificial intelligence.

2.5. AIIS First Iteration

The AIIS collaborative virtual learning environment was piloted with 115 medical students from four partner universities and ten health technology engineering students from Turku University of Applied Sciences. The engineering students were evenly distributed among different challenges, ensuring at least one individual with technical expertise in each group for smooth setup and use of the necessary architecture for machine learning challenge tasks. These engineering students also completed the AIIS theory material. Of the 125 students participating in the pilot, 101 earned certificates for completing the theory component, resulting in an approximate completion rate of 81%. A usability and user experience survey among participants indicated a consensus that the AIIS collaborative virtual learning environment offered a positive overall experience. A comprehensive analysis of the pilot and user experience survey will be made available upon thoroughly examining the respective data sets.

3. Method

The primary objective of this study was to evaluate the usability of the AIIS system in Thailand and Myanmar to gain insights into student experiences while interacting with the system for learning purposes. A mixed methods approach was adopted, incorporating quantitative (e.g., questionnaire) and qualitative (e.g., interview) research methods. Due to its capacity to offer general and in-depth insights into the use of the metaverse-based learning system, the mixed methods technique was chosen for this study. While qualitative interviews enabled an examination of the experiences and feelings of the students, quantitative data from questionnaires provided objective usability evaluations. Together, they made it easier to evaluate the system thoroughly, and using three different sources of information increased the study’s validity. This balanced strategy worked effectively for the sophisticated evaluation of the learning system.

The study sample consisted of 21 undergraduate students from Thailand and Myanmar, aged between 18 and 25 years, currently enrolled in computer engineering-related programs and courses. Some participants are now working as IT professionals in Thailand and Myanmar. In this research, we selected participants from Myanmar and Thailand, even though the AIIS system was initially designed for European students. Our rationale behind this choice was to assess the system’s universality and relevance for a diverse global student population. Additionally, given the scarcity of research on metaverse-based learning in Southeast Asia, our study gains particular importance as it can have a substantial impact. Consequently, our findings bridge this knowledge gap and enhance the existing literature on metaverse-based learning and students’ learning experiences worldwide.

Participants were recruited through social media posts and groups (e.g., international student groups). Usability testing sessions were conducted in both online and onsite settings. The test design and procedure encompassed the following sequential steps. First, participants were provided with an introduction to the AIIS system and informed about the objectives of the test. This segment lasted approximately 10 min. Second, each participant obtained informed consent before proceeding with the study. Third, a tutorial session was conducted between 5 and 10 min to familiarize the participants with the AIIS system. Fourth, pre-study questions were administered to collect demographic data and information on participants’ prior experiences with VR, AR, and metaverse systems. Fifth, participants were instructed to perform a series of tasks within the AIIS system, including logging in, exploring the metaverse world, engaging with “Introduction to AI” at Level 1, followed by a quiz, and advancing to Level 2 to learn “Social Skills” and completing the corresponding quiz. The task execution phase took approximately 30 min. Simultaneously, a researcher observed and documented the participants’ user experiences, interaction patterns, and facial, emotional, and verbal expressions during the test. Upon task completion, participants were asked to complete a series of post-test questionnaires to assess their experiences with the AIIS system. These included the Game Experience Questionnaire (GEQ), in-game and post-game questionnaires, and Nielson’s 10 Heuristics questionnaire. All questionnaires employed a 5-point Likert scale, and this segment took approximately 15 min. Finally, a short debrief and semi-structured interview session lasted approximately 5 to 10 min to gather participants’ opinions on using the system and explore potential improvement recommendations. The entire usability testing session lasted approximately one hour per participant. Data collection methods included observation and note-taking techniques. The study adhered to the ethical guidelines established by the affiliated university.

Table 2. Usability Testing Design and Procedures.

Phase	Tasks	Duration
Pre-Test	Introduction to the study	15 min
	Obtention of participant informed consent
	Pre-Test Questions
In-Test	Tutorial	30 min
	User Tasks (Log-in, Navigation, Learning Video and Quiz for AI, Learning Video and Quiz for Social Skills, Exploration, and Exit)
Post-Test	Heuristics Questionnaire	20 min
	In-game and Post-game Questionnaire
	Post-Test interview
Debrief	Debrief session	10 min

shows the usability test design and procedures.

In this study, we adopted specific instruments to evaluate the usability and user experiences of AIIS. The In-game and Post-game Game Experience Questionnaires (GEQ), developed by the researchers at Eindhoven University of Technology [34,35], were utilized for this purpose. These questionnaires offer comprehensive assessment methods suitable for the gaming industry and academic research. The In-game GEQ focuses on capturing immediate experiences and evaluates seven components of the game experience, encompassing immersion, competence, flow, tension, challenge, negative affect, and positive affect [35]. By utilizing these questionnaires, developers and researchers comprehensively understood players’ in-game experiences, enabling them to enhance different aspects of gameplay. The In-game and Post-game Game Experience Questionnaires (GEQ) are suitable for evaluating AIIS, the metaverse-based learning system, because they provide comprehensive insights into student experiences during and after their interactions with the system, allowing a holistic assessment of its effectiveness and impact on learning.

In the context of the In-game GEQ, the term “immersion” pertains to the degree of engagement and absorption players experience in the game world. It encompasses the sense of presence, emotional connection, and overall attachment to the game’s environment and characters. A high level of immersion contributes to a more captivating and memorable gaming experience. The component of “competence” evaluates players’ perceived skills and abilities in performing in-game tasks. The feeling of competence boosts players’ motivation and satisfaction with the game, making them feel accomplished. ‘Flow, ‘another component of the In-game GEQ, represents a psychological state in which players become fully engrossed in their gameplay, losing track of time and external distractions. Achieving a flow state requires a harmonious balance between challenge and skill and a seamless integration of action and awareness. This state is essential for maintaining player engagement and maximizing enjoyment. Lastly, “tension” captures the emotional stress, frustration, or anxiety experienced by players during gameplay. While a certain level of tension can enhance the overall experience, excessive tension can lead to negative emotions and reduced enjoyment.

Concerning “challenge”, this component measures the perceived difficulty of the game and the extent to which players feel tested by in-game tasks. A well-balanced challenge can promote engagement and the game experience, whereas an imbalance can lead to frustration or boredom. The “negative affect” component gauges the ways in which players experience negative emotions during gameplay, such as anger, sadness, or disappointment. Highly negative effects can harm the overall game experience and lead to disengagement. Conversely, the “positive affect” component evaluates the ways in which players experience positive emotions, such as happiness, excitement, or satisfaction, while immersed in the gameplay. This component is crucial for maintaining player motivation and overall game enjoyment. In summary, the In-game GEQ is a valuable instrument for evaluating and optimizing various aspects of the game experience. By considering the seven components, developers and researchers can better understand the gaming experience and strive to create more engaging, satisfying, and immersive digital games.

The Post-game GEQ assesses players’ overall evaluation after playing, focusing on competence, positive and negative affect, and tiredness [34,35]. The Post-game Game Experience Questionnaire (GEQ) is a crucial instrument designed to measure players’ overall evaluation of their gaming experiences after gameplay. This research aims to explore the effectiveness of the Post-game GEQ in assessing four key aspects of post-game experiences: returning to reality, positive and negative affect, and tiredness. By analyzing these components, we hope to better understand the factors that contribute to a satisfying and engaging gaming experience and the potential impact of gameplay on player well-being. Regarding the “returning to reality” aspect, transitioning from the virtual game world back to everyday life is crucial for the post-game experience. This study investigates how easily players can disengage from the game and resume their daily activities while examining the factors that may facilitate or impede this process. For the “positive affect” component, as part of the research, we analyze the extent to which players experience positive emotions, such as joy, excitement, and satisfaction, following gameplay. Understanding the elements that contribute to positive affect can help developers design games that leave players with a lasting sense of enjoyment and accomplishment. Likewise, we examine the “negative affect” component to note negative emotions in players’ post-game evaluations, such as frustration, disappointment, or sadness. Identifying the aspects of gameplay that contribute to negative effects can guide developers in adjusting to minimize negative experiences and enhance overall game satisfaction. Finally, for the “tiredness” component, this study explores the relationship between gameplay and players’ perceived tiredness. We examine whether extended gameplay sessions, intense challenges, or other gameplay factors contribute to fatigue and the potential implications of tiredness on players’ well-being and future game engagement. Through this research, we aim to contribute to the growing body of knowledge on post-game experiences and further validate the Post-game GEQ as a reliable tool for assessing players’ overall evaluations after gameplay. By understanding the factors influencing returning to reality, positive and negative affect, and tiredness, game developers and researchers can better design games that offer enjoyable and fascinating experiences while mitigating potential adverse effects on player well-being. Validated and reliable across various contexts, In-game and Post-game GEQ are widely used in game user research to understand player experiences and preferences, aiding in the design and improvement of digital games [34].

In this research, we incorporate Nielsen’s 10 Heuristics as a tool for participants to assess the AIIS system, a framework introduced by usability specialist Jakob Nielsen in 1994. As pointed out in [36], Nielsen’s 10 heuristics can be proficiently employed by non-designers. These rules offer a comprehensive guide for user interface design, formulated in a way that is both understandable and relevant for non-experts conducting a usability evaluation of a product. Nielsen’s 10 heuristics are suitable for the above study because they provide a well-established framework to identify and address usability issues within the AIIS metaverse-based learning system’s design and interface.

The principles are a collection of usability guidelines meant to assess user interface designs. They stress the importance of system status visibility, real-world terminology, standard practices, user empowerment, uniformity, error avoidance, prioritizing recognition over memory, adaptability, visual appeal, informative error notifications, and easily accessible guides. Recognized as foundational criteria for usability assessments, these heuristics aid in pinpointing and rectifying potential problems, enhancing the user’s overall experience [37,38]. Thus, given these factors, these tools are dependable and apt for appraising a metaverse system. The questionnaires utilize a 5-point Likert scale, with scores ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). Follow-up interviews are also organized to capture users’ comprehensive feedback on their AIIS interactions. These interviews aim to elicit valuable suggestions and recommendations for potential improvements and enhancements to the system. While assessing AIIS’s usability, we actively engage participants by soliciting their input through follow-up interview questions. These questions encompass a variety of topics, such as “Q1. What are your positive experiences using this system? Q2. What are your negative experiences using this system? Q3. What are your overall learning experiences using this system? Q4. What are your suggestions/recommendations for improving this system?” We employ Google Forms to administer both the survey and interview questions in our assessment process. For instance, after completing the AIIS exam, participants were instructed to sign into Google Forms and answer the questions. The researcher was on hand to help participants if they encountered any problems or uncertainties, such as not understanding a particular question or how to respond.

4. Results

4.1. Pre-Study Findings

The pre-study findings of the usability evaluation for the AIIS system yielded additional noteworthy insights. The study involved a diverse group of 21 participants, encompassing 20 undergraduate students and one graduate student. The average age of the participants, which was found to be 22 years old, suggests that the study predominantly included young individuals in the higher education system. This age range indicates that the participants are likely technologically adept and comfortable with digital platforms, including smartphones and digital games. Furthermore, 20 participants are currently studying in a computer engineering program in Thailand or Myanmar, while one recently graduated from a similar program.

Regarding the gender distribution, the study revealed that nine participants identified as male, accounting for 42.9% of the total sample, while 12 participants identified as female, constituting 57.1% of the participants. This balanced gender representation within the study is crucial for obtaining comprehensive insights into the usability of the AIIS learning system across different genders, ensuring that the evaluation captures a diverse range of perspectives and experiences. Furthermore, the participants’ educational background, comprising primarily undergraduate students, suggests that they are familiar with the traditional learning methods in academic settings. This information is essential for assessing the effectiveness of AIIS as a metaverse-based learning platform. These preliminary conclusions provide valuable contextual information about the participants, including their age, gender distribution, and educational background. These insights lay the foundation for a comprehensive usability evaluation of the AIIS learning system, enabling a deeper understanding of its potential strengths and areas for improvement within the target user population.

When participants were asked about their VR/AR/XR usage frequency, the responses revealed the following distribution: 42.9% disclosed never having used these technologies. Similarly, another 42.9% stated that they used them rarely, while a small percentage, 9.5% of the participants, indicated that they sometimes used these technologies. Merely 4.8% reported using VR/AR/XR often, and none of the participants reported using them very often. These results demonstrate that most participants either have no experience with these technologies or engage with them infrequently, with only a tiny fraction of users utilizing them more regularly. Regarding the participant’s usage of the metaverse, the results indicated that 66.7% reported never using it, while 33.3% mentioned rare usage. None of the participants reported using the metaverse sometimes, often, or very often. These results suggest that most participants have not yet been exposed to the metaverse, and even among those who have, its usage remains infrequent. Our study reveals that the participants’ adoption and use of VR/AR/XR technologies and the metaverse remain limited. These findings shed light on potential areas for further investigation, such as understanding the barriers to adoption and identifying strategies to encourage more regular engagement with these innovative technologies. There is ample room for growth and exploration in terms of increasing the utilization and integration of these technologies into everyday life. By addressing the factors that hinder their adoption, we can pave the way for a more immersive future and unlock the full potential of VR, AR, XR, and the metaverse, particularly in educational settings.

4.2. Heuristics Evaluation

The AIIS heuristic evaluation, a metaverse with a learning mechanism, offers significant insights into a number of system usability factors. These characteristics include user freedom and control, consistency, and standards, error prevention, recognition rather than recall, adaptability and efficiency of use, minimalist aesthetics, visibility of system status, alignment with the real world, user control and freedom, flexibility and efficiency of use, error recovery, and accessibility to help and documentation. Beginning with system visibility (M = 3.7; SD = 1), the data indicate a need for enhancement. During initial interactions, some participants overlooked essential aspects of the system, suggesting that key system elements should be more conspicuous. To achieve this, we could improve the intuitiveness of user interfaces, offer clearer instructions, and potentially introduce visual or auditory cues to guide user interaction.

The system’s alignment with the real world (M = 3.3; SD = 0.9) could be improved for a more engaging user experience. The metaverse should ideally mimic real-world environments, like classrooms or breakout rooms, to enhance the learning experience and encourage user engagement. Regarding user control and freedom (M = 3.7; SD = 0.9), the system performs reasonably well, but there is room for advancement. Enhancing the intuitiveness of interactions and facilitating smooth transitions back to previous states can further improve the user experience. The system’s consistency and standards (M = 3.7; SD = 0.9) can also be improved. Introducing discipline-specific designs and colors for educational tools, such as differentiating between AI and social skills tools, could improve navigability and facilitate more effective learning.

For recognition over recall (M = 3.4; SD = 0.7), enhancements can be made to reduce the cognitive load on users. An intuitive and flowing interaction experience, which reduces the need for users to remember specific instructions or processes, would make subsequent usage much easier. Concerning flexibility and efficiency of use (M = 3.4; SD = 0.9), further optimization of user settings and controls can foster more significant user interaction, thereby improving the system’s overall flexibility and efficiency. Error prevention (M = 2.8; SD = 1.2) requires notable improvement. Proactive measures that predict and prevent user errors could significantly enhance the system’s usability. Even though aesthetics and minimalist design (M = 3.9; SD = 1.1) scored relatively high, there is room for further refinement. Balancing visual appeal and simplicity can improve navigation and enhance user engagement.

Lastly, help and documentation (M = 3.3; SD = 1.1) can be more comprehensive and user friendly. This would shorten the learning curve and improve system accessibility. In summary, while AIIS excels in certain aspects, system visibility, real-world resemblance, user control, and error prevention require significant improvements. By enhancing aesthetics, simplicity, flexibility, and user-friendliness and iterating based on user feedback, the AIIS system can evolve into a more engaging and effective learning platform. The thorough analysis and results from this usability evaluation study of the AIIS metaverse-based learning system reveal a solid basis for a compelling, user-friendly platform. The AIIS learning environment can mature into a more sophisticated, engaging metaverse experience by addressing the identified areas for improvement. This can effectively support the platform’s educational objectives and elevate the overall user experience, ensuring the AIIS metaverse-based learning system remains a pioneer in innovative education.

Table 3. Heuristics Evaluation of AIIS.

Heuristics	Mean	Standard Deviation
The system provides the visibility of the status.	3.7	1.0
The system does match the real world.	3.3	0.9
The system provides user control and freedom.	3.7	0.9
The system provides consistency and standards.	3.7	0.9
The system provides error prevention.	2.8	1.2
The system provides recognition rather than recall.	3.4	0.7
The system provides flexibility and efficiency of use.	3.4	0.9
The system provides aesthetics and minimalist design (the design is simple and pleasant).	3.9	1.1
The system helps users recognize, diagnose, and recover from errors.	3.4	0.9
The system provides help and documentation.	3.3	1.1

presents the heuristics evaluation of AIIS.

4.3. Users’ In-Game Experiences

One primary objective of the current study was to evaluate users’ in-game experiences while interacting with the AIIS system. The analysis focused on seven key dimensions: competence, sensory and imaginative immersion, flow, tension, challenge, negative affect, and positive affect.

As shown in

Table 4. Users’ In-game Experiences in AIIS.

Category	Items	M	SD
Competence	I felt successful when using AIIS.	3.14	0.9
	When I was using the AIIS system, I felt skillful.	3.4	0.8
Sensory and Imaginative Immersion	I was interested in the AIIS system’s story.	3.5	1.1
	I found it impressive to use the AIIS system.	3.9	1.0
Flow	When I was using the AIIS system, I forgot everything around me.	2.6	1.0
	When I was using the AIIS system, I felt completely absorbed.	3.2	0.9
Tension	When using the AIIS system, I felt frustrated (felt bad).	2.2	1.0
	When I was using the AIIS system, I felt irritable (annoyed).	1.9	0.9
Challenge	When I was using the AIIS system, I felt challenged.	3.9	0.6
	When using the AIIS system, I had to put a lot of effort into it.	2.9	1.0
Negative affect	I felt bored when using AIIS.	2.0	1.0
	I found it tiresome when using the AIIS system (I felt tired of using it).	2.2	1.0
Positive affect	When I was using the AIIS system, I felt content (satisfied).	3.5	0.8
	When I was using the AIIS system, I felt good.	3.7	0.7

, participants’ self-assessed competence in the system suggested that they perceived themselves as moderately successful (M = 3.14, SD = 0.9) and skillful (M = 3.4, SD = 0.8) while engaging with the AIIS system, as shown in Table 3. The findings imply that AIIS effectively imparted a sense of competence among participants, although it did not exceed the system’s expected skill enhancement targets. This suggests room for improvement to make the system more user-friendly or intuitive.

In terms of “sensory and imaginative immersion”, participants demonstrated substantial interest in the AIIS system’s narrative (M = 3.5, SD = 1.1) and found the overall experience impressive (M = 3.9, SD = 1). These outcomes suggest that the AIIS system effectively fosters an immersive environment, capturing and maintaining players’ attention.

As for “flow”, participants reported only moderate absorption (M = 3.2, SD = 0.9) and did not experience complete detachment from their surroundings (M = 2.6, SD = 1). This indicates that while the AIIS system engages players to a certain degree, it is not yet successful in inducing a comprehensive flow state. This could be due to factors like interface complexity or environmental disruptions, warranting further investigation for improvements in the system’s immersive design.

The study uncovered low levels of “tension” among participants, with frustration (M = 2.2, SD = 1) and irritability (M = 1.9, SD = 0.9) scores registering below the midpoint. This indicates that the AIIS system is generally free of frustrating or irritating elements. However, a degree of tension can benefit learning, as it can stimulate problem solving and deepen understanding. Therefore, it might be worth investigating ways to incorporate an optimal level of “healthy tension” into the system.

The AIIS system strikes a commendable balance for the “challenge” component. Participants felt a sense of challenge (M = 3.9, SD = 0.6) but did not perceive the tasks as overly demanding (M = 2.9, SD = 1.0). This balance is crucial for promoting flow and engagement, which should be emphasized as a significant strength of the AIIS system.

Regarding “negative affect”, participants reported minimal boredom (M = 2.0, SD = 1.0) and tiredness (M = 2.2, SD = 1.0) during their interactions with the AIIS system. Although this might suggest that the AIIS experience is neither monotonous nor fatiguing, future iterations could introduce a wider variety of tasks or content to decrease these scores further.

As for “positive affect”, the system elicited favorable emotions among users. Participants reported feeling content (M = 3.5, SD = 0.8) and optimistic (M = 3.7, SD = 0.7) throughout their engagement. This highlights the system’s potential to facilitate an enjoyable experience, which can boost motivation and, potentially, learning outcomes.

In summary, while the AIIS metaverse-based learning system shows promising performance in areas like competence, immersion, challenge, and positive affect, the study’s findings suggest the potential for further enhancements. These might include refining the system’s design to boost flow and engagement, introducing an optimal level of challenge, and diversifying the content to minimize negative affect. Future research should also delve deeper into the implications of the positive emotions evoked by the system for the learning experience, such as their impact on information retention or user motivation.

4.4. Users’ Post-Game Experiences

Examining the AIIS system’s usability and user experience with user experiences broken down into four categories—positive experience, unpleasant experience, fatigue, and resuming reality—was the study’s secondary goal. For each group, descriptive statistics such as the mean and standard deviation were computed.

Participants generally reported having positive experiences with the AIIS system, as seen in

Table 5. Users’ Post-game Experiences in AIIS.

Category	Items	M	SD
Positive Experience	After using the AIIS system, I felt revived.	3.4	0.9
	After using the AIIS system, I felt like a victory.	2.6	0.9
	After using the AIIS system, I felt energized.	2.9	0.8
	After using the AIIS system, I felt satisfied.	3.5	0.9
	After using the AIIS system, I felt powerful.	2.6	1.2
	After using the AIIS system, I felt proud.	3.1	1.1
Negative Experience	After using the AIIS system, I felt bad.	1.8	1.1
	After using the AIIS system, I felt guilty.	1.2	0.4
	After using the AIIS system, I found it a waste of time.	1.6	0.9
	After using the AIIS system, I felt I could have done more useful things.	1.5	0.8
	After using the AIIS system, I felt regret.	1.5	1.0
	After using the AIIS system, I felt ashamed.	1.3	0.6
Tiredness	After using the AIIS system, I felt exhausted.	2.0	1.0
	After using the AIIS system, I felt weary.	2.3	1.3
Returning to Reality	After using the AIIS system, I found it hard to get back to reality.	2.0	0.8
	After using the AIIS system, I felt disoriented.	2.0	0.8
	After using the AIIS system, I felt I had returned from a journey.	3.2	1.2

under the category “positive experiences”. According to Table 4, they felt satisfied (M = 3.5, SD = 0.9) and revitalized (M = 3.4, SD = 0.9). Empowerment (M = 2.6, SD = 1.2) and triumph (M = 2.6, SD = 0.9) were less noticeable than anticipated. This can indicate that the system does not provide users with the triumphal or overwhelming sensation of achievement they expect, which could be rectified. As for “negative experiences”, these were notably less common than positive experiences. Users did not report significant negative feelings such as feeling bad (M = 1.8, SD = 1.1), guilty (M = 1.2, SD = 0.4), or regretful (M = 1.5, SD = 1). This result suggests that the system generally avoids inducing negative emotions, a positive aspect of user experience. However, the study could further investigate why users did not feel regretful or guilty—were they fully engaged in the learning experience, or were they not emotionally invested?

In terms of “tiredness”, participants reported feeling moderately exhausted (M = 2, SD = 1.0) and weary (M = 2.3, SD = 1.3) after using the AIIS system. This fatigue could be attributed to the intense concentration or effort required to use the system. Future design improvements could incorporate regular breaks or relaxing activities to mitigate this issue.

Regarding the category of “returning to reality”, user experiences varied. While some found it challenging to return to reality (M = 2.0, SD = 0.8) and felt disoriented (M = 2.0, SD = 0.8), others experienced a sense of having returned from a journey (M = 3.2, SD = 1.2). This diversity indicates that the transition from the AIIS system back to the real world can impact users differently. The system’s design might consider strategies to facilitate this transition more smoothly, possibly by incorporating a “cool-down” phase or clear transition cues.

In conclusion, while users generally had positive experiences with the AIIS metaverse-based learning system and negative experiences were less common, there are areas for potential improvement. Users reported moderate tiredness, and their experiences upon returning to reality were mixed. The findings suggest that future system refinements should enhance the sense of accomplishment, reduce user fatigue, and facilitate a smoother transition back to reality. These enhancements could improve overall usability and user satisfaction. Additional research should further investigate these issues and their impact on learning outcomes.

The qualitative results from the brief interview segment offer insightful information about how participants used the AIIS, a learning system built in the metaverse. Participants reported having an immersive learning experience while learning about topics like social skills and machine learning in a virtual setting. One participant (P1) said: “It’s a fascinating blend of learning and interaction that keeps you engaged”. They valued the interaction. The qualitative results from the brief interview segment offer insightful information about how participants used the AIIS, a learning system built in the metaverse. Participants reported having an immersive learning experience while learning about topics like social skills and machine learning in a virtual setting. One participant (P1) said: “It’s a fascinating blend of learning and interaction that keeps you engaged”. They valued the interaction. Participants noted areas for improvement, notably in terms of usability and user experience, despite the generally good response. They pointed out that the interaction and navigational directions could be improved for a smoother, more logical journey. According to Participant 3, “Sometimes, I was not quite sure where to go next or how to interact with some objects in the world”. Overall, the learning process was enthusiastically embraced by the participants. Their curiosity about the possibilities of the platform is shown by their eagerness to continue investigating the AIIS metaverse. Participant 4 emphasises the point, by stating that while there are undoubtedly areas for improvement, the students are enthusiastic to learn what will be next for the AIIS interface. We anticipate additional meetings. As a result, in addition to appreciating the novelty and immersive qualities of learning in the AIIS metaverse, users also provided insightful input for further system improvement.

5. Discussion

Our study’s usability evaluation of the AIIS metaverse-based learning system revealed crucial insights into user experiences with immersive technologies, such as virtual reality (VR), and the metaverse in an educational context. These insights highlight potential strengths and areas that could be further improved, thus directing the path of subsequent research within the realm of educational technologies.

The demographic of the study participants primarily comprised young undergraduate students with a balanced gender distribution. This diversity of perspectives facilitated a comprehensive evaluation of the AIIS system. Yet, even amidst a cohort with robust technological proficiency, the limited exposure to VR/AR/XR technologies and the metaverse exposed potential hurdles for broader adoption, such as accessibility issues, cost constraints, lack of awareness, and specialized equipment requirements. As a result, subsequent research could focus on developing strategies to overcome these barriers, such as rendering hardware more affordable, enhancing public awareness, and refining interfaces to make them more user friendly for educational systems. Also, our research illuminated the opportunities and challenges of AIIS metaverse-based learning for digitally adept young individuals, emphasizing the imperative to tackle accessibility, affordability, and interface concerns to bolster its integration into educational settings. Educators recognize AR’s potential in classrooms, but often prioritize content over its learning benefits, indicating a need for professional development to maximize AR’s educational impact [39]. This highlights the importance of not only improving technology but also equipping educators with the necessary skills and knowledge to integrate immersive technologies effectively into their teaching practices. As we move forward, addressing this aspect is crucial for the successful adoption of AR and other immersive technologies in educational settings. Based on our AIIS study findings, integrating simulation technologies and the metaverse into existing educational structures is a promising direction for future inquiry. This combination could improve both traditional classroom teaching and digital platforms like Zoom. Crucially, our results emphasize that as we move forward, researchers and educators must engage in thoughtful discussions about this integration’s ethical and societal ramifications, especially concerns related to privacy, data protection, and addressing the digital divide.

Nielsen’s ten heuristic principles formed the bedrock of our design strategy for the AIIS metaverse-based learning system. The visibility of system status, user control, freedom, consistency, standards, and minimalist aesthetic design embedded within this system substantially enhance usability, user satisfaction, and learning efficacy. However, further opportunities remain to fortify the AIIS system by applying Nielsen’s principles more extensively, such as aligning the system more closely with real-world conditions, privileging recognition over recall, and bolstering flexibility, efficiency, help, and documentation features while minimizing error occurrences. Therefore, it becomes evident that each of Nielsen’s heuristics has a significant role in creating an engaging, user-friendly, and intuitive metaverse learning system. Our study, therefore, underscored the crucial role of design heuristics in shaping metaverse systems and validated the continued relevance of established heuristics in developing efficacious metaverse systems [37]. With its distinctive digital environment, the metaverse presents difficulties and chances for user involvement. Despite being developed for early web design, Nielsen’s heuristic concepts are still helpful in creating this new experience. However, given the unique characteristics of the metaverse, designers might need to modify these guidelines. Emulation of the natural world must be balanced with the distinctive aspects of the metaverse. Nielsen’s principles must be applied more precisely as metaverse systems develop. The metaverse can turn mistakes into interesting learning opportunities instead of just minimizing them. In essence, while Nielsen’s recommendations offer a strong foundation, the nature of the metaverse forces designers to consider user experience in novel ways.

The primary objective of our study was to evaluate users’ in-game experiences with the AIIS system, focusing on seven key dimensions. The results paint a picture of potential in several areas, although some aspects could be further refined to optimize user experience. Concerning “competence”, participants reported feeling moderately successful and skilled, pointing to the system’s effectiveness in fostering a sense of achievement. However, further enhancements could be made in this area, possibly by offering more detailed feedback or adapting difficulty levels to match individual skill levels better. The AIIS system also succeeded in capturing user attention and created a stimulating environment for sensory and imaginative immersion, primarily due to the engaging narrative and the immersive experience it offers. To build on this success, developers could consider integrating additional sensory elements, such as sound and haptic feedback, to enrich the immersive experience further.

Additionally, the emphasis on “competence” underlined the importance of fostering user agency and empowerment in game-based learning environments. User motivation to engage and continue within the platform increased when they felt adept and competent. The fact that the AIIS system garnered attention points toward its inherent capability to resonate with its user base. To leverage the rapid evolution (such as spatial audio and nuanced haptic feedback) of sensory technologies can amplify immersion and bridge the cognitive connection between users and the learning material, cementing the knowledge transfer process. According to [40], metaverse services have the potential to revolutionize the way healthcare is delivered, enabling new and innovative forms of care that are more accessible, efficient, and effective.

With regard to “flow”, the results present a mixed picture. While participants reported feeling moderately absorbed, they did not experience a complete disconnect from their surroundings, suggesting an area for system improvement. Implementing more captivating gameplay elements or refining the user interface to minimize distractions could facilitate users to maintain a deeper focus, thereby enhancing the “flow”. Despite some areas for improvement, the study revealed low tension among participants, indicating that the AIIS system largely avoids frustrating or irritating elements. To sustain this balance, a vital aspect is striking the right balance between “challenge” and “frustration” to ensure that tasks are stimulating but not overly stressful. The “flow” observations underscore the need for a seamless user experience in the AIIS system, where enhanced engagement mechanisms can help users transition from passive observers to deeply immersed participants without undue stress. In the context of “challenge”, the AIIS system appears to find an optimal balance, presenting engaging tasks without being excessively demanding. One way to maintain this balance could be implementing adaptive difficulty levels that modify according to users’ skill levels, ensuring that tasks remain suitably stimulating for everyone. The AIIS system managed to keep “negative affect”, such as boredom and tiredness, minimal, which suggests that the system is not perceived as monotonous or fatiguing. Developers could incorporate a wider range of content and activities to maintain this positive outcome to sustain user engagement and motivation.

Drawing from IJsselsteijn et al.’s [34] research on the GEQ, which delves deep into player experiences and engagement in video games, we can infer its potential applicability to the AIIS metaverse-based learning system. The existing studies underscore the relevance of the GEQ framework for evaluating user experiences in educational metaverse systems. The AIIS study, like GEQ, gauges users’ in-metaverse experiences, mainly focusing on dimensions like competence, immersion, and challenge [34,35]. The AIIS metaverse-based learning system demonstrates promising performance in several key dimensions, including competence, immersion, challenge, and positive effects, while maintaining low tension and negative effect levels. Specifically, regarding “positive affect”, the AIIS system elicited positive emotions among users, suggesting its potential to deliver an enjoyable learning experience. Encouraging such “positive affect” could involve integrating more rewarding elements or positive reinforcement mechanisms, such as achievements or personalized feedback. The AIIS system adeptly blends challenge and engagement in its learning environment, evoking positive emotions and minimal negative effects. However, to optimize the user experience, refinements to the system’s design should be explored, particularly in enhancing the flow state and player engagement. Refining its design to amplify flow and user engagement remains crucial for an optimized experience. Using such tools is pivotal in refining the design and usability of digital platforms, ensuring heightened user satisfaction regardless of their primary purpose—gaming or education.

Our findings, centered on undergraduate students’ experiences with the AIIS metaverse-based learning system, resonate with previous research on teachers’ perspectives of AR tools like Metaverse [41]. That study revealed that while teachers across experience levels were optimistic about AR tools, only the experienced ones had tangible ideas for classroom integration. Similarly, our results indicate that while undergraduates are technologically proficient and enthusiastic about immersive technologies, actual implementation and adoption have hurdles. The consistent technological confidence across age groups, as highlighted in the previous research [41], further underscores that experience is pivotal for maximizing the potential of new educational tools beyond mere proficiency. This suggests that future research should blend technological advancements with real-world application insights. A thorough usability study was carried out as part of the AIIS research to shed light on the possibilities and difficulties of integrating the metaverse into a learning environment. The findings emphasized the areas needing additional refining and the anticipated benefits of immersive technologies. This is consistent with earlier research highlighting the metaverse’s emerging significance in changing e-learning systems [42]. Notably, the literature describes ELEM as a ground-breaking tool created expressly to simplify e-learning applications within the metaverse, combining various e-learning and virtual learning settings while utilizing the special technology connected to the metaverse. The studies highlight the crucial part played by the metaverse in contemporary pedagogical applications, pointing to the necessity for further research and development. The UAE study, which emphasizes user satisfaction and incorporates constructs like the Technology Acceptance Model and the AIIS research’s comprehensive evaluation of metaverse usability in education are complementary, demonstrating the concerted focus of both investigations on understanding user experiences and the crucial dynamics of metaverse adoption in educational contexts [43].

In conclusion, our research indicates that the AIIS metaverse-based learning system delivers compelling usability with positive user interactions within in- and post-game scenarios. It offers impactful, relevant, and forward-looking educational experiences specially tailored for undergraduate computer engineering students in Thailand. Nonetheless, there is room for enhancement, particularly in accessibility, deepening engagement, and user fatigue reduction. A primary direction for further advancements is bridging the gap between the metaverse and real-world experiences, ensuring a fluid transition. The emphasis should be on amplifying user engagement and maintaining an uninterrupted flow state. Addressing user fatigue and finding the right equilibrium between challenges and user frustration is essential. Our ultimate goal, guided by these insights, is to craft a metaverse environment that excels in usability, user engagement, and educational results, thereby creating a comprehensive educational journey. As we move forward, we are excited about potential collaborations with SurgeryVision, a frontrunner in VR software for surgical planning, to venture into groundbreaking solutions in surgical procedures and enhance patient outcomes. We additionally recommend the continued evaluation of the applicability of usability assessment instruments, such as the System Usability Scale (SUS), for assessing metaverse systems. Given its reputation as a reliable, industry-recognized approach for assessing various interface attributes, the SUS should be employed to scrutinize further the AIIS system’s usability aspects [44]. The SUS’s demonstrated efficiency and responsiveness to alterations make it essential for gaining insights into user experiences and enhancing the system’s design.

6. Design Recommendations

Moreover, considering the outcomes of our observational study conducted during the usability test, we put forward the subsequent design guidelines and suggestions to enhance the AIIS metaverse-based learning system and other comparable metaverse or mixed-reality systems in the future as follows.

• Enhancing Usability and User Interaction

For metaverse-based learning and education systems, it is essential to focus on improving user interaction and usability. This can involve refining interfaces, offering alternative interaction modalities, employing clear terminology, providing concise instructions, addressing interaction issues, and enhancing help functionality to contribute to a seamless and engaging user experience. Learners can traverse the metaverse-based platform more efficiently and gain instructional value from its content.

• Incorporating Adaptive Difficulty Settings

Integrating adaptive difficulty settings for tasks and activities ensures a customized learning experience tailored to individual user needs and skill levels, catering to diverse learners. By adjusting to each learner’s pace and ability, such settings can enhance engagement, reduce frustration, and optimize knowledge retention.

• Facilitating Acclimatization for First-Time Users

During our investigation, the AIIS interface that was developed did feature an instructional scenario. However, this guide was not immediately made available when the users first interacted with the system. As such, it is evident that beginners in our study could have significantly benefited from a step-by-step, initial walk-through tutorial. This would introduce them to the platform and ensure they understood its various functionalities and could navigate it confidently. A readily available introductory tutorial can drastically reduce the learning curve, ensuring users feel supported and equipped from the outset, boosting their confidence and engagement with the platform.

• Improving User Comprehension and Navigation

Increasing the visibility of essential controls, utilizing explicit terms in prompts, and adopting intuitive color-coded systems for buttons and menus can contribute to a more user-friendly experience and promote user comprehension and navigation within the system. Such design enhancements can reduce users’ cognitive load, allowing them greater focus on the content and activities rather than struggling with the system’s mechanics.

• Promoting Social Interaction

Although an element of collaboration is required to complete the AIIS learning program, it is somewhat understated. Further encouragement of social interaction (perhaps through creative group activities) within metaverse-based learning systems can foster community and facilitate peer-to-peer learning. Emphasizing collaborative activities can deepen the learning experience and simulate real-world teamwork scenarios, preparing students for collaborative endeavors in their future professional lives.

• Enhancing Accessibility and Customization

To enhance accessibility and customization, it is recommended to incorporate a customizable interface that enables users to personalize control settings, key mapping, and visual elements. Additionally, integrating voice commands and speech recognition technology can enhance accessibility and expand the system’s user base.

• Incorporating a Comprehensive Feedback System

Implementing a robust immediate feedback system during task completion interactions and including diverse learning materials can elevate the educational experience offered by the platform, ensuring that the learning environment remains responsive to the evolving needs of users and continuously improves based on their feedback.

• Improving Object Affordances

Streamlining user experience and facilitating more intuitive navigation in metaverse-based learning systems can be achieved by enhancing object affordances through visual and auditory cues. This enables users to differentiate between interactive and non-interactive objects within the virtual environment readily. Implementing clear visual and auditory cues simplifies user interaction and reduces cognitive load, allowing users greater focus on the learning content rather than navigating the platform.

We examine ways in which design recommendations can be applied to improve educational technology, particularly in VR/AR/metaverse learning environments. Incorporating design strategies focused on boosting usability and user interaction, refining educational interfaces, and implementing adaptive difficulty settings enables the creation of tailored learning experiences that cater to unique needs and abilities of individual students. In addition, integrating guided tutorials aids new users, while intuitive design features enhance understanding and navigation in educational technology. Furthermore, fostering social interaction promotes community development and collaborative learning while incorporating accessibility and personalization options to accommodate diverse learners. Lastly, implementing a robust feedback system supports ongoing improvement, and optimizing object affordances ensures a more seamless user experience in the educational environment. Implementation of these design concepts can make VR/AR/metaverse learning environments more engaging, user-centric, and efficient, offering dynamic educational experiences. It is also in line with [41] in which DigiVision aims to create a new digital service, titled the continuous and flexible learning tray, which seamlessly combines the constant learning opportunities provided by Finnish higher education institutions into one accessible platform, allowing learners easy exploration and comparison of non-degree education options, search of programs that suit their individual needs, and choice of a trusted and high-quality partner institution for their continuous learning journey while enabling institutions to expand their learner base, improve operational efficiency through data utilization, and develop a versatile offering tailored to the diverse motivations and interests of continuous learners. Addressing these aspects and integrating improvements will lead to more accessible and immersive learning environments in the educative metaverse and similar platforms. Ongoing research and user feedback are crucial for fine-tuning these systems, ensuring they cater to the varied needs of learners, and ultimately benefiting both individual users and society at large. In our upcoming work, we will use the suggestions offered to improve our metaverse-based learning systems. These improvements will improve user interaction and usability, incorporate adaptable difficulty settings, and make it easier for new users to start by providing thorough tutorials. By making controls more visible and encouraging social engagement, we hope to increase user comprehension. We also aim to improve system accessibility through customizable choices and cutting-edge technology. In our ongoing projects, we also prioritize adding a thorough feedback system and enhancing object affordances for more straightforward navigation.

7. Conclusions

This study offered an in-depth analysis of the usability and user experiences associated with the AIIS learning program, an avant-garde metaverse-based learning platform tailored for undergraduate students in computer engineering, medical, and nursing fields, predominantly in Asia. Our driving motivation was to appraise the immersive quality of the AIIS interface, with a particular emphasis on learners from Thailand and Myanmar. To assess the system’s usability and user experience, we engaged 21 participants from distinguished universities in Thailand and Myanmar. Our methodological approach embraced a multifaceted set of evaluation techniques, including In-game and Post-game GEQ questionnaires, heuristic evaluation, and semi-structured interviews. This approach endowed us with a rich tapestry of feedback, painting a detailed picture of the system’s performance metrics and users’ subjective perceptions. The empirical results from our study advocate that AIIS is not merely user friendly but excels in its capacity to captivate learners and cultivate an electrifying learning atmosphere. This was further substantiated by soaring levels of learner satisfaction and overwhelmingly affirmative experiences. In parallel, the qualitative leg of our study shed light on potential areas of improvement, offering a blueprint of specific design recommendations to fine-tune metaverse systems, particularly within academia. Incorporating the feedback, our short-term strategic objectives encompassed fortifying the system’s interface usability, introducing innovative learning modules to infuse diversity in content, and broadening the participant spectrum to capture more holistic feedback. We also recognize the value of incorporating adaptive learning algorithms to personalize the experience based on individual learning patterns and integrating gamification elements to heighten engagement levels. Yet, it is imperative to approach these findings with a measure of circumspection. Limitations such as a constricted sample size, the concise duration of testing, demographic specificity, and the non-usage of immersive head-mounted displays during trials do cast certain shadows. These caveats accentuate the necessity for continued exploration and provide a roadmap for future investigations. Future studies should redress these shortcomings, venturing into innovative design paradigms and leveraging cutting-edge technologies to augment the learning matrix. We are particularly enthused about embracing augmented reality (AR) elements and integrating real-time feedback mechanisms. Additionally, the prospect of exploring a more expansive participant demographic and amalgamating state-of-the-art immersive tools promises to elevate the robustness of our conclusions, heralding a transformative phase for metaverse-driven pedagogy.