Model for Designing Gamified Experiences Mediated by a Virtual Teaching and Learning Environment

Abstract

Higher Education Institutions (HEIs) face new challenges in regard to technological development in light of necessary pedagogical and didactic innovations in educational action. This article proposes a Technological–Pedagogical Gamification Model (MGTP) that guides the design of gamified educational practices in Virtual Teaching and Learning Environments (EVEAs). The MGTP proposal is based on theoretical cores of Pedagogy and Computer Science theories, as well as works related to gamified experiences in EVEA where the social, cognitive, and teaching presences were analyzed. This work also presents an initial validation of the MGTP through expert judgment, and its results are analyzed from both a qualitative (content analysis and comments) and quantitative (using the Content Validity Coefficient method) perspective. These results reveal a high level of acceptance of the model by experts that is corroborated by reliability tests (Cronbach’s alpha and split-half reliability test). The results facilitated the development of a final version of the model for its subsequent application and evaluation in university practice.

1. Introduction

The processes of historical innovation that the educational system undergo as a result of social needs lead to changes and reformulation of established systems [1]. Thus, the daily education provided in Higher Education Institutions (HEIs) has been assumed by many students to be monotonous and sometimes ineffective [2,3]; this has generated questions in regard to emerging methodologies that promote significant improvements in learning and overcome the limitations of traditional learning while being linked to memorization and other aspects related to the acquisition of contents of different disciplines. Considering that university teaching has been the object of severe criticism that questions the role that teachers must assume in the face of these challenges [4], it is necessary to appropriate strategies that contribute to these new needs and improve teaching and learning processes in an efficient and quality manner [1,5,6].

Virtual Teaching and Learning Environments (EVEAs) have become ubiquitous in HEIs. They are mainly used by educational institutions to complement face-to-face classes with a blended learning approach or to enable distance learning modalities [7,8]. This implies the creation of innovative EVEAs that contribute to the improvement of student experiences and outcomes [9] through learning strategies that promote participation and enjoyment, such as gamification [10].

Gamification is defined as the use of game design elements in non-game environments [11]. It seeks to promote student participation and involvement [12] with the aim of influencing people’s behavior through actions on their motivation [13]. The publications of the scientific community refer to several approaches to the game elements applied in gamified experiences [14,15]. In this paper, we adopt the following categorizations, which have been presented in Teixes [13], for the elements that are of importance when creating a game: mechanics and components, dynamics and aesthetics. The mechanics and components refer to the elements that integrate the game and make it possible for progress in the gamified activity to be visible [16]. Meanwhile, the dynamics allow the mechanics and components of gamification to become meaningful to the players, and this helps in avoiding monotonous tasks [13]. Aesthetics mobilize emotions and are linked to what the gamified system generates in the learner through interactions with the mechanics and dynamics [17].

The Community of Inquiry (CoI) model of Garrison et al. [18], which is of current relevance, assumes that learning occurs in a community in which teachers (facilitators) and students interact. It indicates that this community is nurtured by the following three types of presence: social (SP), cognitive presence (CP), and teaching presence (TP). A teacher present in the virtual environment must have a mastery of the contents, technological skills, and didactic knowledge to plan activities, guide strategies, and achieve the expected results [19]. This CoI model has made it possible to analyze and design educational practices mediated by digital technologies, particularly in EVEAs [20,21,22], focusing on the development of critical thinking, interaction, and dialogue among participants and the guidance, structure, and proposals made by the teacher as a facilitator.

While there is some work linking the CoI model to gamification experiences in EVEAs with presence analysis [20,21,22,23,24,25], only the work of Utomo et al. [26] is oriented to present a model that integrates gamification strategies with presences. The model presented in [26] considers the following four components: the user (facilitators and students), the learning process (including triggers, discussions, challenges and tests with gamification elements to analyze understanding of the topics, feedback from facilitators through gamification elements, and problem solving with application of what is understood in the process), the objective (goals to be achieved), and the environment (the EVEA; in this case, the authors use Moodle, and it is linked to the materials and resources to generate discussions and interactions). After presenting the model, the authors describe a gamification experience that draws on the model and obtains results. Thus, the model of Utomo et al. [26] has not been designed as a guide for teachers/institutions that want to design gamified proposals where CoI presences are considered but rather as a theoretical model of the components to be considered. The proposal of Utomo et al. [26] is a background of interest for this work. However, a more comprehensive model (MGTP) is proposed here to guide teachers/researchers and institutions in the design of gamified experiences in EVEA and to consider the evaluation through the presences presented by Garrison et al. [18,27]. The latter can contribute to analyzing how the design and implementation of the gamified experience (designed from the MGTP) impacts on the SP (in the link with teachers and peers) and the student’s CP (with higher order cognitive activities), as well as to analyzing how the TP is perceived and intervenes as a fundamental link to drive and accompany the process. Consequently, this MGTP aims to guide the design, development, and evaluation of gamified experiences in EVEAs while considering the presences of the CoI model for the promotion of motivation, involvement, and commitment of students towards the construction of higher order cognitive levels.

In this way, this paper is organized as follows: Section 2 focuses on presenting some key concepts and background information found in the literature that contribute significantly to this study; Section 3 describes the methodology used for the development of this work; Section 4 presents the MGTP model and its rationale; Section 5 details the validation process carried out and the results of the MGTP evaluation by means of expert judgment and discussion. Finally, Section 6 presents the conclusions and lines of future work.

2. Background and Related Work in EVEA

The research reviewed provides a variety of approaches and methodologies that highlight how gamification can increase student interaction and engagement. However, seven studies that are considered relevant to this work are analyzed throughout this section, providing knowledge regarding the presence of the CoI model as part of a gamification proposal in EVEA. First, Utomo et al. [26] highlights a learning model that combines gamification strategies to integrate the three presences of the CoI model. The authors of [26] present an improvement in collaboration and communication between students and teachers as a result. This approach not only boosts student engagement but also motivates them to be more active and share ideas.

Similarly, the reviewed studies use a diverse range of EVEAs to mediate their courses, from Moodle [20,21,22] and Open edX [23] to customized environments developed specifically for gamification projects. Petroulis et al. [24] implement a reward system in blended courses delivered through Moodle, while, Antonaci et al. [23] study gamification in massive open online courses (MOOCs) where their results reveal an increase in social presence and sense of community.

However, learning in gamified EVEAs is directed by means of strategies that integrate the three presences of the CoI model [21]. The experiences analyzed show that a greater social presence is achieved through collaborative and competitive activities. CP is strengthened through challenges and problems that require the use of critical thinking skills, while PD is manifested in the guidance and continuous feedback provided by the facilitators. Hence, Utomo et al. [25] found that these gamified approaches increase participation in the system and improve learning.

Also, the most common strategies include the use of point systems, levels, rewards and challenges, as in the study by Tzelepi et al. [28], who suggest the following two types of gamification elements: individual badges and community progress, making it known that individual elements are more effective for critical thinking. Utomo et al. [26] implemented assignments and discussions in Moodle that were enriched with gamification, such as badges and scores, with the purpose of encouraging students’ participation and critical thinking.

For their part, Mese and Dursun [20] found that blended learning environments enriched with gamification have a significant impact on these presences, which improves student engagement and motivation. Something similar occurs in the studies of Papanikolaou et al. [21], who integrate gamification with social network metrics to increase the effectiveness of online learning communities.

Consequently, the reviewed studies show a significant positive impact by promoting social, cognitive, and teaching presence. Additionally, Utomo et al. [26] provided evidence that the integration of gamification increases student participation and engagement. Tzelepi et al. [22], concluded that individualized gamification elements are more effective for deep learning, while Antonaci et al. [23] found that gamification in MOOCs enhances social presence and the sense of community.

3. Methodology

This work addressed a Systematic Literature Review (SLR), following the protocol presented by Kitchenham [29], which made it possible to investigate studies that combine the main study variables, EVEA, gamification, and CoI, as central axes of the proposal under investigation. The results of the SLR were published in [27]. Based on these results, a model proposal is created that sifts the pedagogical and technological dimensions, where the presence of the CoI model in the evaluation phase is considered. Thus, the MGTP model proposal emerges.

The construction of a model that integrates pedagogical and technological dimensions, and that considers the presence of the CoI model in the evaluation phase, requires a rigorous and well-founded approach. The MGTP was created based on the background found and from the study of the theoretical cores of Pedagogy and Computer Science. Pedagogy, as a science that studies the education of people [29], is integrated with Computer Science, which focuses on computational processes oriented to the creation of software to solve specific problems in the environment [30]. This interdisciplinary approach allows for the development of a model that seeks to meet the growing needs of higher education [31].

In order to carry out the creation of the model, in addition to considering the studied antecedents with their potentialities and gaps, an iterative process was carried out among the authors. To evaluate the model, the first step was to define the object and objective of the evaluation. It was decided that an expert judgment needed to be carried out. An expert judgment constitutes a valid and reliable methodology that allows for an obtained qualified opinion on an object to be evaluated through a series of procedures to be used to collect information from the experts [32,33]. Expert selection was approached using biograms and a coefficient of knowledge, as well as the development and application of the assessment instrument. Additionally, instrument validity and reliability were calculated.

4. MGTP Model

The Technological–Pedagogical Gamification Model (MGTP) aims to integrate gamification strategies and presence indicators to enable gamified educational practices in EVEAs within HEIs, and it is aimed at impacting the cognitive, social, and teaching presence in these environments. It targets those interested in conducting gamified experiences in EVEAs. Therefore, HEIs require specific actors who can approach the use of the model as a guide for designing, developing, and evaluating gamified experiences in EVEA, taking into account the CoI model’s presence.

The main actors of the MGTP include researchers interested in studying this topic, coordinators with expertise in gamification and Pedagogy who will institutionalize the model, teachers who will participate as actors throughout the methodological process of the model, and students involved in developing the gamified experience in EVEAs. HEI teachers can utilize a MGTP to design and implement gamified pedagogical activities, while researchers can employ it to analyze and enhance educational practices. Students will actively participate in the implementation and evaluation phases to construct meanings, knowledge, and skills, as well as to express their opinions about gamified experiences based on the analysis of presence indicators. This approach allows HEIs to adopt the model to establish educational practices that consider necessary dimensions to enhance quality.

The MGTP is structured around subsystems, their components, and their relationships. The relationships represent the existing links between the subsystems and finally, the novel essence, which arises from the interactions of components–structure–relationships, the result of which is superior to the isolated elements that form it [34]. In an integral look at the model, Figure 1 presents its structure in subsystems, which include components that are related, from the logic of the pedagogical organization of a gamified EVEA, to direct mediation of digital technologies.

Subsystems comprise interrelated components that form a cohesive system as referred to by Bertalanffy [34]. The organization of the MGTP model into subsystems comprises the following:

Subsystem I: Determination of gamification needs. This subsystem covers the analysis of the gamification content requirement, the technological and professional diagnosis of the teaching and learning process, and the decision making required to gamify the content. It includes an analysis of the justification of the needs to implement gamification in EVEAs. It may involve the institution and/or researchers who want to address gamified educational practices and teachers. It is based on Maslow’s “pyramid of human needs [35].” and Leontiev’s activity theory [36], which affirm that, by satisfying basic needs, higher desires are demanded and developed that can make self-actualization possible. In this subsystem, the MGTP proposes interviews and surveys which evaluate the motivation and digital competences of teachers before starting the gamification process. These instruments are found in the Supplementary Materials at the end of the document and are identified as follows: Instrument S1: semi-structured interview: motivation to gamify learning contents; Instrument S2: Survey of Digital Competence of Teachers.

From this subsystem, there will be a diagnosis of gamification needs and training needs for teachers in charge of planning and carrying out the gamified experience.

Subsystem II: Technological–pedagogical gamification projection. This subsystem focuses on the planning of the gamified teaching and learning process, where clear objectives are established that determine resources and specific activities, gamification rules and guidelines are offered, and the gamified experience is designed. Planning includes the selection of appropriate tools and applications for gamification, where it ensures that the activities and resources are effective and motivating for students. This planning, according to Álvarez [29], is directly linked to the methodically organized procedure of the educational process to obtain a given objective. Foncubierta y Chema [37] stresses the importance of an adequate design to avoid boredom and ensure the success of the gamified activity. Strongly involved in this subsystem are the teachers who will depend on the resources and policies of the institution to make decisions. Researchers can guide the use of the model at this stage by analyzing the teachers’ decision making. The model provides teachers with a compilation of good gamification practices so that the teacher has examples. In addition, a document is provided with a list of possible mechanics to consider for the design of the gamified experience. As an instrument, the teacher generates a logbook that allows him/her to record this process (Instrument S3: logbooks of the teacher’s work).

From this subsystem, we have all the planning of the experience, including its mechanics, dynamics, aesthetics, and other components.

Subsystem III: Gamified development of the EVEA experience. In this subsystem, the implementation and evaluation of the gamified experience is carried out. The following four phases are considered for the development of the experience, following aspects of the CoI model: triggering event, exploration and integration, resolution and feedback. The initial phase is oriented to capture students’ interest through problems or dilemmas that require resolution, which encourages critical thinking. During exploration and integration, students actively participate in activities, exchange ideas, and develop solutions. The resolution phase involves the practical application of what has been learned, while feedback focuses on constructive evaluation and continuous improvement of the educational process [31].

The evaluation phase is elementary and stands out in the proposed model due to its significant contribution. The evaluation is performed through the consideration of the CoI model, which analyzes the social, cognitive, and teaching presence. The MGTP suggests using a validated instrument in the evaluation process of the gamified experience. This is the CoI survey designed by Richardson et al. [38]. This consists of 34 items organized by dimensions and evaluative categories. In this case, the instrument will allow for analyzing the interactions and perceptions of the students regarding the gamified experience in the EVEA according to the three presences of the CoI model. In this sense, the cognitive presence dimension focuses on analyzing the students’ understanding and critical reflection on the curricular contents; the gamified activities promote inquiry and knowledge integration. Social presence, on the other hand, evaluates group cohesion, open communication, and affective expressions; it will be considered whether gamified strategies promote interaction and collaboration among students. The teaching presence dimension analyzes the role of the teacher in facilitating discourse, content organization, and feedback. Here the teacher is responsible for guiding and motivating students through proactive and constructive interventions. This dimension of the evaluation will make it possible to know whether the students consider the proposals and interventions made by the teachers in the gamified proposal to be effective. This instrument is available in the Supplementary Materials at the end of the document and are identified as follows: Instrument S4: Inquiry Community Survey. In addition, as part of the evaluation process, the MGTP considers using interviews with teachers to know their opinion as well. This instrument is available in the Supplementary Materials at the end of the document and are identified as follows: Instrument S5: Assessment interview on the impact of the gamified experience. According to Bertalanffy [34], a feedback mechanism can “reactively” reach a higher state of organization (p. 156).

From this subsystem, the development and evaluation of the experience will be achieved for its subsequent analysis and decision making for improvement Thus, the MGTP system has a bidirectional relationship (arrows) between the components of the model’s subsystems, suggesting a relationship of interdependence and/or mutual feedback between them. Thus, activities performed in one component may have an impact on the next component and vice versa, ensuring the development of continuous adjustments and improvements, based on interaction, which ensures a dynamic and effective integrated approach. Thus, the circular date located between the subsystems represents the cyclical, continuous and adaptive nature of the MGTP through action (active participation of the actors), reflection (evaluation and analysis of the processes), and adaptation (adjustments and improvements based on feedback), which ensures that the gamified experience in the EVEA, developed with the guidance of the proposed model, is flexible and adaptable to various needs and contexts where it is applied to maintain the interest and motivation of students [39].

It is important to highlight that, from the relationships between the components of each subsystem, a new quality emerges as a fundamental contribution to science. Consequently, from these relationships a general quality is revealed that is linked to the processual nature of gamification in an EVEA in regard to the determination of needs, action planning and pedagogical and technological interaction with the use of gamification strategies (see Figure 2).

5. Expert Judgment as a First Validation of the Model

In the following, the application of the expert judgment is described first, followed by the presentation of the results and their analysis and discussion.

5.1. Application of Expert Judgment

In this work, expert judgment is used and the content validity method identified as Content Validity Coefficient (CVC) has been chosen. This method determines the content validity for each item and in general, based on the level of agreement between experts. Expert judgment and the CVC validity method are widely used in research and have many advantages over other validation procedures [40].

The methodology followed for this study is shown in

Table 1. Description of methodological steps applied in the expert judgment.

Steps	Description
Define the object to be evaluated.	Technological–Pedagogical Gamification Model for a Virtual Teaching and Learning Environment (MGTP).
2. Define the objective of the evaluation by expert judgment.	Validate the content and reliability of the MGTP through the measurement instrument.
3. Define the method to be used in the expert judgment.	Content Validity Coefficient (CVC) by Hernandez-Nieto [40]
4. Select the experts involved in the process (Steps 4 and 5 can be performed in the reverse order, if applicable).	Selection of experts in EVEA Pedagogy and Gamification through the biogram and Knowledge Quotient.
5. Elaborate the evaluation instrument.	The instrument consisted of four sections: Section 1. Analysis of Subsystem I Section 2. Analysis of Subsystem II Section 3. Analysis of Subsystem III Section 4. Opinion on the Adequacy of the MGTP
6. Communicate to the experts the evaluation methodology to be followed.	Communication process and methodological follow-up of the expert judgment developed.
7. Apply the instrument.	E-mail sent to the experts for the development of the validation process by expert judgment.
8. To carry out the general follow-up of the process.	Communication process and methodological follow-up of the expert judgment developed.
9. Calculate content validity and interpretation of results.	Procedure to calculate the CVC for each of the items and the general instrument and interpretation of the CVC, based on the scale of values established by Hernandez-Nieto [40].
10. Conduct qualitative review of the items.	Verification of the concordance of the items for elimination, modification or approval. Review of considerations on comments or improvements suggested by experts.
11. Calculate the reliability of the instrument.	Procedure to calculate the reliability of the results obtained from the pilot test by means of a Cronbach’s Alpha Coefficient and a Test of Two Halves.
12. Communicate the results of the evaluation to the participating experts.	Once the application and analysis of the results obtained have been completed, the experts are informed of the entire process.

Adapted from [41,42,43,44].

.

The process began with defining the object of evaluation, in this case the MGTP, and the evaluation objective, which was to validate the content and reliability of the model using a measurement instrument. Expert judges in EVEA Pedagogy and Gamification were selected based on criteria established in the biogram and the Knowledge coefficient (Kc) to conduct the assessment.

An evaluation instrument divided into four sections was developed covering the analysis of the three subsystems of the model and opinions on the MGTP in general. This instrument was designed according to the proposal of Hernández-Nieto [40]. The assessment methodology was communicated to the experts along with detailed instructions on the assessment process, which included as its decision criteria the following: CVC < 0.600 = Unacceptable; CVC ≥ 0.600 and ≤0.700 = Poor; CVC > 0.71 and ≤0.800 = Acceptable; CVC > 0.800 and ≤0.900 = Good; and CVC > 0.900 = Excellent. The evaluation methodology was explained to the experts through e-mails, which included the validation format and the link to the instrument in digital format.

Subsequently, the instrument was applied using a Google form, which allowed the experts to evaluate the items according to criteria such as relevance, wording, clarity, coherence, and representativeness with a Likert-type scale of 1 to 5. The instruments are available in the Supplementary Materials at the end of the document and are identified as follows: Instrument S6: MGTP validation format, Input S1: Cards of the validation of the scores obtained by the experts. The process was continuously monitored to ensure the active participation of the experts and to resolve any doubts they might have. Once the evaluation was completed, the CVC was calculated for each item, and for the instrument in general, as well as for the reliability of the instrument, by means of Cronbach’s Alpha Coefficient. Finally, the results obtained were communicated to the experts, who were informed of the entire process and thanked for their participation.

5.2. Results and Discussion

The results of applying the evaluation methods discussed in the previous section to the model are presented below. The selected evaluation methods (expert judgment, Content Validity Coefficient (CVC), Cronbach’s Alpha Coefficient and the test of two halves) are justified and the methodological process that was followed is described. The expert judgment resulted in an overall CVC of 0.902, indicating high validity and agreement among the experts based on the Hernández-Nieto [40] scale of values. All items assessed obtained CVC values above 0.8, indicating that the instrument has an adequate relationship between the items, components, and subsystems of the MGTP. In addition, an overall CVC of 0.949 was obtained, indicating excellent validity and agreement of the instrument. Therefore, the results of the validation process for each of the components and subsystem relationships of the MGTP yielded significant values of relevance, wording, clarity, coherence, and representativeness for the proposed items.

Regarding qualitative assessment, the experts contributed the following improvements: The learning objectives should be clearly defined and aligned with the social task and in relation to the contents of the subject, which is related to what Vygotsky [45] mentions, who emphasizes that learning occurs through social interaction in a given environment and with the contents of the culture offered in the curriculum.

In addition, the experts proposed that the learning process should begin with a triggering event, such as a problem or dilemma, that captures the interest of the students and encourages critical thinking through communication between the actors in the educational process, aspects that, from the theory of presences of Garrison et al. [32], also indicate that the student is in interaction with others, within the learning communities, to build higher-order knowledge [33,46]. The rationale for the quality among the components of subsystem III was also modified to include the student’s reality; and an interview was added to assess the impact of the gamified experience from the teacher’s point of view. It should be clarified that the model presented in Section 3 already includes the improvements indicated by the experts.

Although the experts did not consider it necessary to include additional subsystems or components for the MGTP, they provided valuable qualitative input that enriched its content and led to the design of an additional instrument to assess the impact of the gamified experience from the teacher’s perspective.

Cronbach’s Alpha Coefficient (0.983) and the test of two halves (Cronbach’s Alpha above 0.9 for each half) confirmed the reliability of the instrument, reflecting a very satisfactory internal consistency. This indicates that the instrument was well understood by the experts and is considered for application in HEIs.

In summary, the MGTP was favorably evaluated by the experts and opportunities for improvement were also found. After the analysis of the experts’ opinions, the following modifications were addressed in terms of wording (components and relationships), and regarding the implementation of qualitative characteristics (assessment interview on the impact of the gamified experience), but not in its structural design (number of components and subsystems). Below, for more detail, Figure 3 shows the previous aspects of the MGTP, presented in white color, and the modifications subsequent to the expert judgment, distinguished by their gray color.

The MGTP is presented as a promising tool for the integration of gamification in EVEA. Its validation through expert judgment and reliability testing supports its robustness and applicability. The incorporation of pedagogical and technological elements, together with assessment based on the CoI model, ensures a comprehensive approach to gamification in higher education. The results of this study open new possibilities for research and teaching practice and lay the groundwork for future research exploring the impact of MGTP in different educational contexts and disciplines. It is important to note that the MGTP has been used in two case studies, the results of which are being analyzed and will be the subject of a forthcoming publication.

6. Conclusions

The MGTP is presented as an innovative and promising tool to enrich the educational experience in virtual environments. Its design, validated by experts and supported by pedagogical and technological theories, offers a comprehensive guide for teachers and institutions seeking to implement gamification effectively [47]. The MGTP not only ensures a structured process but also promotes continuous improvement through constant feedback and analysis of social, cognitive, and teaching presences.

The model proposal is supported by the results obtained from expert judgment and reliability tests, demonstrating its validity and applicability in real contexts. The integration of technological tools and resources, coupled with evaluation based on the CoI model, ensures an experience that considers both technological and pedagogical aspects, incorporating the best practices identified in the literature. The MGTP not only contributes to educational innovation but also lays the groundwork for future research and developments in the field of gamification in higher education.

During the research development, a methodological gap was identified regarding the lack of studies utilizing models that integrate quantitative and qualitative approaches to comprehensively understand the effects of gamification on EVEA teaching, learning processes, and motivation. The next steps of this research will focus on evaluating the implementation results of the MGTP across diverse educational contexts, encompassing various academic levels, disciplines, and teaching modalities. The study will assess its impact from both student and teacher perspectives.