Using Advanced Learning Technologies with University Students: An Analysis with Machine Learning Techniques

Abstract

The use of advanced learning technologies (ALT) techniques in learning management systems (LMS) allows teachers to enhance self-regulated learning and to carry out the personalized monitoring of their students throughout the teaching–learning process. However, the application of educational data mining (EDM) techniques, such as supervised and unsupervised machine learning, is required to interpret the results of the tracking logs in LMS. The objectives of this work were (1) to determine which of the ALT resources would be the best predictor and the best classifier of learning outcomes, behaviours in LMS, and student satisfaction with teaching; (2) to determine whether the groupings found in the clusters coincide with the students’ group of origin. We worked with a sample of third-year students completing Health Sciences degrees. The results indicate that the combination of ALT resources used predict 31% of learning outcomes, behaviours in the LMS, and student satisfaction. In addition, student access to automatic feedback was the best classifier. Finally, the degree of relationship between the source group and the found cluster was medium (C = 0.61). It is necessary to include ALT resources and the greater automation of EDM techniques in the LMS to facilitate their use by teachers.

1. Introduction

Today’s society is constantly evolving. Technological advances, together with the current COVID-19 crisis, are underlining the need for a change in the teaching–learning process in education. This change is particularly significant in university education. It is centred on the development of instructional modalities, such as e-Learning and Blended Learning (b-Learning). Both implement learning management systems (LMS). These systems can integrate a wide variety of educational tools, such as virtual reality, augmented reality, robotics, artificial intelligence, holograms, virtual laboratories, etc. [1,2,3]. However, in order to facilitate effective learning in students, these technological tools must be implemented with appropriate pedagogical designs [4,5]. Consequently, university education today faces many challenges, one of which is the integration of technological resources in LMS to improve teaching [6]. Among these methods is the facilitation of self-regulated learning (SRL) through the use of advanced learning technologies (ALTs) in LMS. This will help in providing students with personalised help [7,8], which increases their motivation [9]. In particular, teachers, through the use of different didactic resources (e.g., virtual labs, infographics, flipped learning chatbot experiences, serious games, multimedia resources [10], online project-based learning (OPBL) methodology) [11,12] and intelligent tutoring systems (ITS) [13], can promote individual and group work [14]. In addition, the joint use of ALT and LMS allows the recording of interactions during the teaching–learning process. These records can be analysed with statistical and educational data mining (EDM) techniques [15]. All of this will allow the teacher to follow the learning trajectory of each of their students [16]. Another advantage is that the use of ALT facilitates SRL in the student in real time [17]. Similarly, the use of ALT in LMS will facilitate collaborative work [18,19]. However, the implementation of these technological resources currently requires constant human supervision. To solve this problem, the implementation of intelligent multi-agents based on natural language is being initiated [20]. The results of their use are promising with respect to improving academic performance and perceived usability in students [21]. Specifically, all ITS have one feature in common: they all provide real-time feedback to the learner, which facilitates the personalisation of learning [17,20]. The use of these resources has shown promise [22], because they enhance dynamic scaffolding, which can help students learn more effectively. Such systems are highly interactive and employ artificial intelligence, and some of them can be easily integrated into LMS. However, they require the teacher to have digital skills and the ability to use EDM techniques (supervised and unsupervised learning). This aspect could slow down their implementation in educational contexts [23]. However, the use of these systems has more advantages than disadvantages; for example, the use of Intelligent Personal Assistants (IPAs) seems to improve students’ listening and speaking skills without the presence of a physical teacher [24]. In addition, the use of other ITS resources seems to facilitate students’ access to LMS and their motivation towards learning [8]. This promotes self-reflection on their own practice, which enhances the acquisition of new concepts and problem solving [25]. However, as discussed above, digital competencies are required for successful use [26]. This poses a challenge for academic leaders in universities when trying to promote the training of teaching staff in this area [27].

In particular, this training plan should focus on the use of IPAs, as their future in the educational field is promising [28]. The advantages of IPAs compared to other ITS are that they establish conversational empathy with the user [29] and are highly useful with students with special educational needs (e.g., people with visual impairments or attention deficits) [30]. Additionally, IPAs provide learners with web navigation aids, which enhance the personalisation of learning [31,32]. However, IPAs should be adapted to the requirements of each task, and not have a generalised structure [33]. Researchers also stress the need for more research in this area [34]. Another aspect that should be included in university teacher training is the use and interpretation of EDM techniques [35], specifically those related to supervised predictive and unsupervised clustering learning techniques [36]. Through these techniques, it is possible to determine the learning patterns of students and detect students at academic risk [37]. These data will provide the teacher with information helping them to give personalised education to a student or group of students with similar characteristics.

In conclusion, many educators do not feel qualified to use these technologies independently, due to the difficulties of applying ALT in LMS and the absence of machine learning techniques useful for the interpretation of the findings. Figure 1 shows a summary of the preceding, and

Table 1. Summary of the benefits and drawbacks when using digital resources in the teaching–learning process.

Resources	Advantages		Challenges
	Students	Teachers	Students	Teachers
ALT	Personalized help Easy to get personalized feedback in real time	Make it simpler to figure out how a learner learns Aid students based on their requirements Personalise teaching Simplify the creation of automated personalised teaching Feedback processes	They require certain knowledge of, as well as the need for, skills on the application of LMS	Training in digital skills and EDM is required
LMS	Personalise learning Increase motivation Individual work Group work Use different didactic resources	Personalise teaching Enhance individual and group work Include didactic resources
ATL and LMS		Enables the recording of interactions (logs)
EDM		Know students’ learning patterns Easy to find out which of the resources implemented is more effective in increasing the learning results in the students Simplifies groupings of students to offer educational answers adjusted to each of them
ITS	Personalised learning Increased motivation	Make it easy to design personalised automatic feedback processes
IPA and chatbot	Facilitate attention to diversity and help students with special educational needs Enhance personalised learning Facilitate an increase in motivation	Facilitate the educational response in students with special educational needs
All of the above	Promote the use of metacognitive self-assessment strategies in students Facilitate effective learning	Increase constructive and meaningful learning in students Increase learning outcomes	They require certain knowledge of, well as the need for, skills related to multiple digital resources	They involve teaching with active digital methodologies

presents the benefits and challenges of digitalization in the teaching–learning process.

The world is becoming increasingly digital, indicating the need for a number of changes in teaching design and the usage of educational resources in university teaching. Experts recommend establishing a Smart University with technology-based pedagogy [38]. The research should be focused on evaluating the efficacy of these technological resources at various levels and with different types of students. In this regard, the European Agenda 2030 [39] includes objectives to achieve quality education, such as increasing digital competences and promoting equal opportunities via the development of sustainable and inclusive quality education [40].

As the use of these resources in education is still an emerging field, with few empirical studies rigorously analysing their impact on student learning outcomes, more research is needed [20].

Based on the above state of the art, the research objectives of this study were:

RO1.

To find out whether the combination of ALT resources predicts students’ satisfaction and behaviours in the LMS;

RO2.

To find out which is the best cluster with respect to student satisfaction and behaviour in the LMS;

RO3.

To find out whether different clusters are related to the ALT resources used in the different intervention groups.

2. Materials and Methods

2.1. Participants

Over the course of two academic years, we worked with 225 third-year Health Sciences students in two subjects. Both tests were conducted during the COVID-19 pandemic (academic years 2019–2020 and 2020–2021), with 98 students from the Occupational Therapy program and 127 students from the Nursing program (see

Table 2. Descriptive statistics for the sample of participants.

Group	Subject	Health Sciences Students (N = 225)							Rate of Return Degree	Subject Performance Rate
		N	n	Men		n	Women
				Mage	SDage		Mage	SDage
Group 1	Subject 1	46	8	21.63	1.77	38	22.42	2.25	90.24%	100%
Group 2	Subject 2	61	5	21.40	0.89	56	23.54	6.30	95.83%	100%
Group 3	Subject 1	52	7	21.57	0.79	45	22.64	4.72	90.14%	100%
Group 4	Subject 2	66	7	25.71	7.39	59	23.44	5.51	96.83%	100%

Note. Group 1 = students from the Occupational Therapy course, academic year 2019–2020; Group 2 = students from the Nursing course, academic year 2019–2020; Group 3 = students from the Occupational Therapy course, academic year 2020–2021; Group 4 = students from the Nursing course, academic year 2020–2021. Performance rate: percentage ratio between the number of credits passed by the total number of students enrolled in a given academic year with respect to the number of credits enrolled by these students in the same year. Source: University of Burgos Information System.

). The sample was chosen via convenience sampling.

2.2. Instruments

• Learning Management System-LMS-. We use a LMS based on Moodle v.3.9: Virtual Learning Platform from the University of Burgos, UBUVirtual.
• Online Project-Based Learning (OPBL). All student groups worked with the OPBL methodology in small groups (3 to 5 members).
• Virtual laboratories. Ad-hoc-oriented and open access in the Repository of the University of Burgos. SRL methodology was used for all of them [41,42].
• Quiz-type questionnaires with process-oriented feedback. The learning check questionnaires included multiple-choice (four options) questions with one correct option. All of them included automatic process-oriented feedback, in which the student was given information about the correct answer, the reason for it, and where they could find the theoretical justification within the material given.
• Intelligent Personal Assistants-IPA- Students could access the main dates of the course (delivery of practices, completion of questionnaires, project delivery, etc.) using the Amazon Alexa application (mobile, tablet, or computer). Amazon Alexa skills were developed and deployed via the Amazon Web Service (AWS). Students first had to prove their identity to be able to use this application. This process was initiated via UBUVirtual, the learning platform (LMS) of the University of Burgos. To access the Alexa resource, students had to provide valid credentials of their identity within the LMS. Students could then utilize the skill without needing to access the LMS again after the first successful validation. As a result, the connection was secure, and personal data were secured [43,44]. During the academic year 2019–2020, the device was used with Group 1 of students from the Occupational Therapy degree.
• Flipped classroom experiences. Flipped classroom classes were carried out in the four intervention groups. These included the creation of videos based on the topics’ thematic units, with three lessons applied in each of the four intervention groups. After each lesson, students could take a quiz-type questionnaire with process-oriented feedback (see point 4).
• Gamification with feedback on the results. H5P, featuring in the most recent versions of Moodle, was used to create the gamification activities. The following games were used: crossword, find the words, memory game, speak the words set, and true/false questions. All these activities included process-oriented feedback and a progress bar. They were also divided into three levels of difficulty (beginner, intermediate, and advanced). The gamification activities in both degrees (Occupational Therapy and Nursing) were performed during the final four weeks of the academic year 2020–2021.
• Laboratory simulation. The simulation practices were designed at the simulation laboratory of the Faculty of Health Sciences from the University of Burgos. This facility has rooms with a Gesell Chamber-type one-way mirror, wherein students can perform simulations with dummies in clinical practice environments. An example of the procedure is available at https://youtu.be/C8XGemeBuOM (access on 26 October 2021) which was provided to students from the Occupational Therapy and Nursing degrees during the 2020–2021 academic year.
• Survey of general satisfaction with the training activity [45]. An ad hoc survey with 19 closed-ended questions assessed on a Likert-type scale of 1 to 5 (1, do not agree at all; 5, strongly agree) and three open-ended questions related to strengths, weaknesses, and proposals for improvement. In this study, the survey had a Cronbach’s reliability coefficient of α = 0.93 (see Appendix A 

  Table A1. Coefficients in the prediction of type of group (combining ALT resources) with LO, feedback, LMS access, and MS.

  	Unstandardized Coefficients		Standardized Coefficients	t	p	95.0% Confidence Interval for B		Correlations			Collinearity Statistics
  	B	SE	Beta			Lower Bound	Upper Bound	Zero-Order	Partial	Part	Tolerance	VIF
  (Constant)	2.49	1.22		2.05	0.04 *	0.09	4.89
  LO	0.38	0.08	0.26	4.49	0.00 *	0.21	0.55	0.35	0.29	0.25	0.94	1.07
  Feedback	0.00	0.00	0.28	4.70	0.00 *	0.00	0.01	0.41	0.30	0.26	0.89	1.12
  LMSA	0.00	0.00	0.23	4.06	0.00 *	0.00	0.00	0.31	0.26	0.23	0.96	1.05
  MS	−0.53	0.21	−0.14	−2.46	0.01 *	−0.95	−0.10	−0.21	−0.16	−0.14	0.97	1.03

  Note. LO = learning outcomes; LMS access = learning management system access; MS = mean satisfaction teaching; SE = standard error; VIF = variance inflation value. * p < 0.05.

  ).
• Learning outcomes, measured on a scale of 0 to 10. This measurement considered the work done by students using a project-based learning approach. In the final grade, the elaboration of the project was allocated a weight of 25%, and the defence of the project a weight of 20%. A test with multiple choice questions and a single correct answer was also employed, accounting for 30% of the overall weight. Finally, participation in co-evaluation activities was given 15% of the weight (comprising responses to questionnaires on the evolution of the teaching–learning process). In each group, the same percentages were used.
• Plugin “eOrientation” [46,47]. This Moodle plugin was developed as part of a research project funded by the Junta de Castilla y León (Spain). This plugin can be used to collect data related to personalised access to the pedagogical resources utilized by students in each topic over the course of an academic year. It also allows the teacher to communicate with each student individually via email, so as to provide feedback on the results of the learning process. Furthermore, the logs can be exported in multiple file formats (.csv, .xlsx, HTML table, .json, .ods, and .pdf).

2.3. Procedure

Prior to the beginning of this study, a positive report was obtained from the Bioethics Committee of the University of Burgos (No. IR 30/2019), as was the written informed commitment of all research participants. During the second semesters of both academic years, we worked with four groups of third-year students undertaking Health Sciences degrees (Degree in Occupational Therapy and Degree in Nursing). The first academic year (2019–2020) coincided with the COVID-19’s confinement period, hence the teaching had to be virtual, and was conducted via e-Learning. The second course (2020–2021) used a blended learning approach, with in-person course activities and virtual theoretical classes. All of the groups used the same set of methodological resources: OPBL, quiz-type surveys with process-oriented feedback (in some cases for the teacher’s evaluation of knowledge and in others for the student’s self-evaluation of knowledge), and flipped classroom experiences. In addition, depending on the group, various teaching resources were used (virtual laboratories, IPA, gamification with feedback on the results, and laboratory simulation [48]). A convenience sampling method was applied for the distribution of the groups. The “eOrientation” tool [47] was also used to keep track of the students’ learning progress in the LMS.

Table 3. Teaching methodology applied in the intervention groups.

Teaching Methodology with ALT	Group 1	Group 2	Group 3	Group 4
e-Learning	Yes	Yes	No	No
b-Learning	No	No	Yes	Yes
OPBL	Yes	Yes	Yes	Yes
Virtual laboratories	Yes	No	Yes	No
Quiz-like questionnaires with feedback oriented to the evaluation processes	Yes	No	Yes	No
IPA	Yes	No	No	No
Flipped classroom experiences	Yes	Yes	Yes	Yes
Quiz-type questionnaires with feedback oriented to the self-evaluation processes	No	Yes	No	Yes
Gamification with feedback on results	No	No	Yes	Yes
Laboratory simulation	No	No	Yes	Yes

shows a summary of the resources applied in each category.

2.4. Research Design

A descriptive-correlational design was used [49] and the factors applied were: teaching methodology in LMS (e-Learning vs. b-Learning); use of IPA vs. non-use; use of gamification vs. non-use; use of laboratory simulation vs. non-use. The analyses were performed with the statistical package SPSS v.24 [50].

2.5. Data Analysis

To test RO1, the supervised learning techniques of regression (multiple linear regression) were applied. To test RO2, the supervised learning techniques of classification (CHAID decision tree and k-nearest neighbour, or k-nn) were applied. To check RO3, unsupervised learning techniques (k-means) were applied.

3. Results

3.1. Contrasting RO1

To test RO1, supervised learning regression techniques were used to study the degree of prediction of learning outcomes and student satisfaction with the ALT resource combinations used. An R² = 0.31 was found, indicating that student group type explains 31% of student learning outcomes. Specifically, group type predicts learning outcomes at 26.1%, access to automatic feedback from different resources at 28%, access to the LMS at 23%, and student satisfaction at 14%, all parameters being significant at 95%. The tolerance indicators were not close to 0, so the independent variables were not considered redundant, and none had to be eliminated with respect to the dependent variable (type of group). Likewise, the VIF values were no greater than 10, meaning they were within the fit values (1–10) (see Table A1). In the collinearity analysis, dimension 2 obtained a variance proportion of 0.92 with respect to accesses to automated feedback, while dimension 3 obtained a variance proportion of 0.91 with respect to the accesses to the LMS (See

Table A2. Teaching methodology applied in the intervention groups.

Condition				Variance Proportions
Dimensions	Eigenvalue	Index	(Constant)	LO	Feedback	LMSA	MS
1	4.72	1.00	0.00	0.00	0.01	0.01	0.00
2	0.15	5.61	0.00	0.00	0.92	0.05	0.00
3	0.12	6.34	0.00	0.01	0.00	0.91	0.01
4	0.01	22.34	0.00	0.66	0.07	0.00	0.34
5	0.003	40.00	0.99	0.33	0.00	0.03	0.65

Note. LO = learning outcomes; LMSA = learning management system access; MS = mean satisfaction teaching.

).

3.2. Contrasting RO2

For RO2, supervised classification learning was applied—specifically, the CHAID decision tree algorithm. The dependent variable was the type of group and the independent variables were learning outcomes, access to automatic feedback from different resources, access to the LMS, and student satisfaction with teaching. Cross-validation was applied. (This makes it possible to evaluate the robustness of the tree structure. Cross-validation divides the sample into a subsample number, followed by the creation of tree models that do not include the data for each subsample. In SPSS, the first tree is based on all cases except those corresponding to the first fold of the sample; the second tree is based on all cases except those of the second fold of the sample, and so on. For each tree, the risk of misclassification is calculated by applying the tree to the subsample that was excluded when it was first created. A maximum of 25 sample folds can be specified. The higher the value, the lower the number of cases excluded from each tree model. Cross-validation generates a single, final tree model. Cross-validation risk for the final tree is calculated as an average of the risks of all trees. Specifically, in this study, the fold cross-validation used was 10). The ranking variable was access to automatic feedback from different resources, isolating three nodes (see Figure 2). In the lowest node (values below 127.0), 67.2% are members of Group 1. In the intermediate node (values between 127.0 and 202.0), 63.2% are members of Group 4, and in node 3 (values higher than 202.0), 61.1% are members of Group 2. Therefore, it can be inferred that access to automatic feedback was the independent variable with the greatest effect on the differences between the types of group, and the group in which the best results were obtained in this variable was Group 2 (in which the following ALT resources were applied: e-Learning, OPBL, quiz-type questionnaires with process-oriented feedback, and flipped classroom experiences). The group that obtained intermediate results was Group 4 (in which the following ALT resources were applied: b-Learning, OPBL, virtual laboratories, quizzes with feedback oriented towards self-assessment processes, gamification with feedback on the results, and laboratory simulation). Finally, the group that registered a higher percentage in the lower node was Group 1 (in which the following resources were applied: ALT e-Learning, OPBL, virtual laboratories, quizzes with process-oriented feedback, and IPA) (see Figure 2).

Combination of ALT resources used: Group 1—e-Learning, OPBL, virtual laboratories, quiz-type questionnaires with process-oriented feedback, IPA, flipped classroom experiences; Group 2—e-Learning, OPBL, quiz-type questionnaires with process-oriented feedback on self-assessment, flipped classroom experiences; Group 3—b-Learning, OPBL, virtual laboratories, quiz-like questionnaires with process-oriented feedback, flipped classroom experiences, gamification with feedback on results, laboratory simulation; Group 4—b-Learning, OPBL, quiz-like questionnaires with process-oriented feedback on self-assessment, flipped classroom experiences, gamification with feedback on results, laboratory simulation.

The k-nn algorithm was also applied, where the focal case identifier used was the type of group. The features were learning outcomes, access to automatic feedback from different resources, access to the LMS, and student satisfaction with teaching. Four predictors were applied, of which three were isolated: access to automatic feedback from different resources, learning outcomes, and access to the LMS (see Figure 3). Regarding the focal analysis of the features, a greater dispersion was found in the learning outcomes and in the students’ satisfaction with teaching (see Figure 3).

The focal analysis highlights cases of particular interest by displaying the k closest neighbours in a graph of space of attributes, homologues, and quadrants, as well as the distances between them. Regarding the features, a greater dispersion was found in the learning outcomes and in the students’ satisfaction with the teaching (see Figure 4).

3.3. Contrasting RO3

Finally, to contrast the RO3, unsupervised learning clustering techniques were applied, and specifically the k-means algorithm was used. Four clusters were found. Cluster 2 included the students with the best learning results; with greater access to automatic feedback and with higher averages of satisfaction with the teaching process, it was considered an Excellent cluster. Cluster 4 included students with greater access to the LMS, and showed greater access to automatic feedback and the second-best learning outcomes after the students in cluster 2, and was therefore considered a Very Good cluster. Cluster 3 included students with high values for access to the LMS, but lower values of LO, feedback and MS, so it was considered a Good cluster, and cluster 1 was the cluster with the lowest values in all variables, so it was considered a Low cluster (see

Table 4. Centres of final clusters for the variables LO, feedback, LMS access, and MS.

	Cluster 1 (Low) n = 51	Cluster 2 (Excellent) n = 79	Cluster 3 (Good) n = 77	Cluster 4 (Very Good) n = 18
LO	8.36	10.00	8.67	9.21
Feedback	63	333	171	255
LMS access	8	668	1338	2045
MS	3.0	4.2	4.2	3.7

Note. LO = learning outcomes; feedback = accesses to automatic feedback resources; MS = mean student satisfaction with teaching.

). Next, a cross table was constructed between participants’ assignments to the clusters and the groups to which they belonged. A contingency coefficient C = 0.61, significant at 95% p = 0.000, was found, indicating a medium degree of coincidence in the relationships (see

Table 5. Percentage of membership for each intervention group with respect to each cluster.

	Cluster Under n = 51	%	Cluster Excellent n = 79	%	Cluster Well n = 77	%	Cluster Very Good n = 18	%
Group 1 (n = 46)	33	71.74	5	10.87	8	17.39	0	0
Group 2 (n = 61)	5	8.20	43	70.49	11	18.03	2	3.28
Group 3 (n = 52)	6	11.54	15	28.85	19	23.08	12	23.08
Group 4 (n = 66)	7	10.61	16	24.24	39	59.10	4	6.06

Note. Combinations of ALT resources used—Group 1: e-Learning, OPBL, virtual laboratories, quiz-type questionnaires with process-oriented feedback, IPA, flipped classroom experiences; Group 2—e-Learning, OPBL, quiz-type questionnaires with process-oriented feedback on self-assessment, flipped classroom experiences; Group 3—b-Learning, OPBL, virtual laboratories, quiz-like questionnaires with process-oriented feedback, flipped classroom experiences, gamification with feedback on results, laboratory simulation; Group 4—b-Learning, OPBL, quiz-like questionnaires with process-oriented feedback on self-assessment, flipped classroom experiences, gamification with feedback on results, laboratory simulation.

).

It was found that 64.71% of the members of the low cluster belonged to Group 1 (in which the following resources were applied: e-Learning, OPBL, virtual laboratories, quiz-type questionnaires with process-oriented feedback, and IPA), representing 71.74% of the total group. Likewise, 50.65% of the members of the good cluster (which applied b-Learning, OPBL, virtual laboratories, quiz-type questionnaires with process-oriented feedback, gamification with feedback on the results, and laboratory simulation) belonged to Group 4, which represented 59.10% of the total group. Regarding the very good cluster, 66.66% of the members belonged to Group 3 (b-Learning, OPBL, virtual laboratories, quizzes with process-oriented feedback, flipped classroom experiences, and gamification with feedback on results), and this in turn accounted for 23.08% of the total group. Finally, 50.43% of the members of the excellent cluster were members of Group 2 (e-Learning, OPBL, quiz-type questionnaires with feedback oriented to self-assessment processes, and flipped classroom experiences), which represented 70.49% of the total group. In addition, significant differences between the clusters with respect to the variables studied were found (see

Table A3. ANOVA between clusters with respect to the variables LO, feedback, LMS access, and MS.

ANOVA
	Cluster		Error		F	p
	Mean Square	df	Mean Square	df
LO	3.20	3	0.95	221	3.38	0.02 *
Feedback	159,903.40	3	5924.35	221	26.99	0.000 *
LMS access	6,809,384.83	3	13,042.17	221	522.11	0.000 *
MS	0.3	3	0.14	221	2.71	0.05 *

Note. LO = learning outcomes; LMS access = learning management system access; MS = mean satisfaction teaching; df = degrees of freedom. * p < 0.05.

).

4. Discussion

The type of ALT used in the four groups predicted learning outcomes, access to automatic feedback, access to the LMS, and student satisfaction at 31%. The highest partial prediction was detected in the variable accesses to automatic feedback (28%), followed by the variable learning outcomes (26.1%) and accesses to the LMS (23%), and to a lesser extent in student satisfaction with teaching (14%). The access to automatic feedback was the methodological resource based on ALT that had the highest classification weight in the decision tree algorithm, and the highest percentage of students in this node corresponded to Group 2, in which the following ALT resources were applied: e-Learning, OPBL, quiz-type questionnaires with feedback oriented to self-assessment processes, and flipped classroom experiences. Regarding the application of the k-nn algorithm, the variables with the most weight for the classification were access to automatic feedback, learning outcomes, and access to LMS, with a greater dispersion of participants in terms of learning outcomes and satisfaction with teaching. These results seem to indicate that there is a difference in the responses to and effectiveness of the ALT resources applied, depending on whether the teaching is carried out in an e-Learning or a b-Learning modality. This is an important fact for future research, as analysing which ALT resources are the most effective in each teaching modality will guide teachers in their future use under each of these modalities. In addition, these results help with research regarding latent variables that occur, especially in the fields of b-Learning teaching, because we also assessed face-to-face teaching, wherein there may be latent variables that influence learning behaviours, learning outcomes, and student satisfaction. For such reasons, these aspects will be analysed further in subsequent studies.

We did not find a cluster that contained all the highest values in all the dependent variables, although the cluster that came closest, the Excellent one, gathered the highest values in the variables of learning outcomes, access to automatic feedback, and average satisfaction with teaching, in which 35.11% of all participants were located. In this cluster, 50.43% were members of Group 2, in which e-Learning teaching was applied with the implementation of the following ALT resources: e-Learning, OPBL, quiz-type questionnaires with feedback oriented to self-assessment processes, and flipped classroom experiences. The percentage of the total members of Cluster 2 was 70.49%. Likewise, in the next cluster, the Very Good cluster, the highest values were detected in access to the LMS, but a lower degree of student satisfaction with teaching was found (3.7 out of 5). In this cluster, the highest percentage of students (66.6%) belonged to Group 3, which used the following ALT resources: b-Learning, OPBL, virtual laboratories, quizzes with process-oriented feedback, flipped classroom experiences, and gamification with feedback on results. However, this only corresponded to 23.08% of the total members of Group 3, which indicates that there was greater homogeneity in the Excellent cluster. In summary, it seems that the most important aspects regarding the achievement of better academic results are related to the use of ALT resources that offer automatic feedback on the processes regarding the students’ learning responses, which in turn is related to a higher satisfaction of the students with the teaching–learning process. In contrast, greater access to the LMS is not related to better learning outcomes or greater student satisfaction. These differences can be explained by the type of teaching modality; in the first case, an e-Learning modality was used, and in the second a b-Learning modality. In addition, it is necessary to consider that the time of teaching in the first case coincided with the most restrictive period of the pandemic, which necessarily implied strict confinement, and in the second case the teaching was not subjected to strict confinement except in cases of SARS-CoV-2 positivity. This fact may be related to student perceptions of distress or anxiety about the health situation [43]. In the first case, there was no choice of teaching modality, so offering students the option to continue teaching through ALT media allowed for continuing their learning in a context as close to “normality” as possible. This fact could have produced in them a more positive attitude towards the teaching modality. Thus, this circumstance did not occur in Group 1 students who experienced the same situation. Consequently, other variables such as the type of degree may be influential, since Group 1 was formed of Occupational Therapy students and Group 2 of students from Nursing. Furthermore, these results were not repeated during the 2020–2021 academic year in Group 3 (students of Occupational Therapy) or Group 4 (students of Nursing), in which other ALT resources, such as gamification and simulation in the laboratory, had also been implemented.

Finally, it should be noted that although the results are not homogeneous in all groups, the use of ALT resources is effective on the academic performance of students with respect to the combinations of these resources applied, since the performance rate in all groups was 100% compared to the general rates of these groups in the degree (Table 1). The difference in percentage points was 9.76 for Group 1, 4.17 for Group 2, 3.17 for Group 3, and 8.86 for Group 4, which are important values to consider, as they concern the same students in different subjects—those who experienced a pedagogical design based on ALT vs. those who did not.

5. Conclusions

The variability in the intrinsic and extrinsic characteristics of students, which can influence their learning process, makes it difficult to generalize conclusions. In addition, the comparison between the e-Learning and b-Learning teaching modalities is also an important variable. It is therefore difficult to conclude which are the best ALT resources, or combinations of them, to apply in the context of teaching–learning in virtual environments. This variability is related to the motivation of the students with respect to the learning resource, to the moment in which the teaching takes place, and to the teaching modality. In this study, we worked in a pandemic context, which may have caused anxiety in students—not towards the subject or the ALT resources, but towards the uncertainty about their life and future [43,48]. However, the presence of a large number of resources that apply ALT does not ensure better academic results or greater satisfaction. The resources that have been shown to be most effective are those that contain automatic process-oriented feedback [11,27]. Their use seems to be directly related to better learning outcomes and higher student satisfaction [11,27]. As a result, researching the development of automatic feedback using ALT resources is both a challenge and an area requiring continuous focus in this field. Moreover, the use of IPA has not been shown to be highly effective, a possible reason being that it only guides students in very basic issues related to their calendar of events. The use of an IPA in which automatic feedback actions can be applied on the learning processes would probably improve these results. However, since it involves advanced artificial intelligence technology, implementing this capability in LMS is difficult and still in its early stages of development [20,21,28,31]. It should also be emphasized that using ALT materials improves students’ performance rates. Although we are still at the beginning of our journey toward teaching with digital resources, and it is difficult to say which of these resources is the best (as this depends on a variety of factors, such as the context and the characteristics of the students), it appears that their use actually enables better performance.

Because we worked with a convenience sample of students within a certain field of knowledge, Health Sciences, the outcomes of this study should be interpreted with caution. Students in this field are also more likely to have a higher level of vocational motivation for their future career. Future research will look into the impacts of multiple types of ALT on learners at different levels and within different disciplines.

In short, in the digital society of the 21st century, the use of ALT resources for teaching still has a long way to go, and requires substantial research, in terms of both promoting the technology and analysing how its application can improve student learning outcomes. Using resources alone does not guarantee better learning outcomes or motivation. Academic managers confront numerous obstacles when addressing this challenge [27], the first of which is the development of ALT resources within the LMS that are simple for teachers to use. Currently, using these tools requires medium–high-level digital abilities, which most teachers lack. In addition, the interpretation of the results on effectiveness requires the use of EDM techniques that are often not included in the LMS. For this reason, if educators want to learn about the behavioural patterns of the students regarding the use of ALT resources [8,37], they may need to be well-versed in the use of EDM techniques [26]. This is a new obstacle to the effective use of technology aimed at personalized learning [11].

To this end, there is a pressing need to address all these challenges in an increasingly digital teaching–learning context, as well to achieve the goals of the 2030 Agenda [39].

6. Patents

Ochoa-Orihuel, J., Marticorena-Sánchez, R., Sáiz-Manzanares, M.C. UBU Voice Assistant Computer application Nº 00/2020/2502; General Registry of Intellectual Property: Madrid, Spain, 29 July 2020 [44].

Sáiz-Manzanares, M.C., Marticorena-Sánchez, R., et al. eOrientation Computer Software for Moodle. Detection of the student at academic risk at University No. 00/2020/588; General Registry of Intellectual Property: Madrid, Spain, 16 January 2020 [47].