AI-Enhanced Decision-Making for Course Modality Preferences in Higher Engineering Education during the Post-COVID-19 Era

Abstract

The onset of the COVID-19 pandemic has compelled a swift transformation in higher-education methodologies, particularly in the domain of course modality. This study highlights the potential for artificial intelligence and machine learning to improve decision-making in advanced engineering education. We focus on the potential for large existing datasets to align institutional decisions with student and faculty preferences in the face of rapid changes in instructional approaches prompted by the COVID-19 pandemic. To ascertain the preferences of students and instructors regarding class modalities across various courses, we utilized the Cognitive Process-Embedded Systems and e-learning conceptual framework. This framework effectively delineates the task execution process within the scope of technology-enhanced learning environments for both students and instructors. This study was conducted in seven Iranian universities and their STEM departments, examining their preferences for different learning styles. After analyzing the variables by different feature selection methods, we used three ML methods—decision trees, support vector machines, and random forest—for comparative analysis. The results demonstrated the high performance of the RF model in predicting curriculum style preferences, making it a powerful decision-making tool in the evolving post-COVID-19 educational landscape. This study not only demonstrates the effectiveness of ML in predicting educational preferences but also contributes to understanding the role of self-regulated learning in educational policy and decision-making in higher education.

1. Introduction

The COVID-19 outbreak has revealed the complex decision-making processes universities must undertake when prompt and judicious actions are required, and decisions must be made under time pressure [1,2]. At the same time, the transition to remote instruction during the pandemic has revealed opportunities to expand the use of online learning, while also highlighting the drawbacks when the course modality does not align with instructor and student preferences. This decision-making process involves trade-offs between safeguarding students and enhancing access to education while maintaining high-quality, transformative learning experiences [3,4]. The swift transition from in-person to online classes during the COVID-19 pandemic has highlighted the importance of decision-making that considers the preferences of both instructors and students, given that the ability to incorporate students into the decision-making enhances the likelihood of successful school reform and student learning outcomes [5].

One benefit of the increase in online learning is the opportunity to leverage artificial intelligence (AI)’s ability to process large amounts of data to make timely decisions without the need to delay decision-making for human-controlled data analysis [6]. For instance, the decision to move instruction online or to being in person must be made quickly and should consider instructors’ and students’ preferences [7]; however, the time needed to survey both instructors and students may delay decision-making by several weeks. The affordances provided by AI can potentially enable administrators to make well-informed decisions that align with the needs of instructors and students while also optimizing the learning experience [8]. Through the utilization of AI, academic institutions can tailor course design and instructional modalities by using students’ and faculty preferences in real-time, significantly enhancing educational equity within the university ecosystem [9,10].

Incorporating technological advancements within the educational milieu necessitates the establishment of robust and adaptive quality assurance protocols that are tailored to the needs of students and instructors within dynamically changing contexts [11,12,13]. Achieving equilibrium between individual preferences and optimal pedagogical outcomes demands the conceptualization and implementation of a theory-driven framework for decision-making and fortifying the quality and efficacy of technological integrations into the course fabric [11,14]. At the same time, the self-regulated learning (SRL) process, where students manage their learning through cognitive and motivational strategies, plays a key role in shaping course modality factors, further informing AI-driven decision-making models [15].

This paper aims to emphasize the potential of an AI tool that utilizes machine learning (ML) techniques to address challenges associated with decision-making involving large datasets. We ground this discussion by describing a study that developed an AI tool that predicts preferences for course modality (e.g., online or in-person) within seven Iranian Universities; however, this study aims to make a larger contribution to the potential for AI and ML in data-driven decision-making in higher education. The findings of this study can inform decision-making processes in higher-education institutions and contribute to understanding the factors influencing students’ preferences for online learning. By identifying instructor and student preferences from existing large datasets available to institutions, decision-makers can maximize the benefits of technology-enhanced learning by making decisions about instructor and student preferences using a theory-driven data collection approach. This paper describes such an ML approach that addresses the following research questions:

RQ1: To what extent can machine learning (ML) predict student and instructor preferences for course modality in a post-Corona learning context?

RQ2: To what extent can psychological constructs associated with self-regulated learning (SRL) predict student and instructors’ preferences for course modality?

2. Literature Review

2.1. Machine Learning in Education

Within education, ML methodologies play a pivotal role in the comprehensive examination of student performance [16], facilitation of learning processes, provision of nuanced feedback, and personalized recommendations [17,18], as well as educational administration and decision-making [19]. A wide variety of approaches have been proposed for how to employ ML in educational settings.

2.1.1. Machine Learning Methods

Within educational studies, most ML techniques applied for clustering and prediction purposes contain Artificial Neural Networks (ANN), decision trees (DTs), random forests (RFs), and support vector machines (SVMs) [20]. Most articles compare the accuracy of some of these methods to control the accuracy and interpretability of the model [18,21,22,23]. The appropriate model for improving prediction accuracy with categorical items depends on factors such as sample size, the number of variables, and theoretical context in handling the data regarding the variables [24,25,26,27]. Studies have shown that SVM and RFs are accurate for clustering purposes with multidimensional and large data, such as predicting performance through activities and quarterly tests [18,28]. Conversely, methods such as regression tree, RF, and C4.5 (i.e., an extension of the Iterative Dichotomiser 3 algorithm) are highly effective for clustering datasets containing errors or random variations that obscure underlying patterns. These methods are particularly advantageous for datasets that are limited in size but require detailed, context-specific analysis [29]. For instance, ref. [30] used DTs to predict student performance to help teachers provide adapted instructional approaches and supplementary practice for students likely to have difficulty understanding the content.

2.1.2. Variable Selection in Machine Learning

In machine learning (ML), the learning model requires the inclusion of relevant variables and excluding the unhelpful variables to identify the most relevant variables that contribute to accurate predictions [31,32]. In this context, wrapper methods, such as Recursive Feature Elimination (RFE) and Boruta, have proven to be effective by considering the intricate interactions between variables and predictive models [33], while filter methods, such as chi-square and mutual information, focus on evaluating the individual relevance of predictors, shedding light on their independent contributions, as pointed out by [34]. This capability makes these methods ideal for capturing and simplifying complex relationships and interactions [35]. Employing a combination of filter and wrapper methods in variable selection is essential for leveraging diverse ML approaches [36], often using a strategic approach like majority voting identified by multiple selection methods [37].

2.2. Related Works

Faculty readiness for online teaching, particularly the challenges faced by those unfamiliar with the format, was extensively investigated by Singh et al. [12]. They notably focused on integrating laboratory components, distinct from other types of classes, within e-learning systems frameworks during the COVID-19 pandemic. In subsequent work, Singh et al. [38] investigated the impact of various course modalities on different aspects of learning, highlighting the importance of educational environment decisions. Carmona et al. [39] applied K-dependence Bayesian classifiers in machine learning to facilitate resource selection tailored to an individual student’s prior knowledge, showcasing the integration of data-driven decision-making into teaching approaches and teacher preferences. Conversely, Kotsiantis et al. [40] pursued a methodological approach involving decision trees and neural networks. Their primary goal was to predict students’ educational material preferences through designed surveys by considering student preferences. Hew et al. [41] used gradient-boosting tree methods with great precision to bridge the gap between expected and actual data about predicting students’ success in Massive Open Online Courses (MOOCs), aiming for a cohesive approach to improve the reliability and accuracy of their research findings. Remarkably, to our knowledge, no existing literature has yet harnessed the potential of ML techniques to predict student and instructor preferences within the context of course modality using theory-driven approaches. Hebbecker et al. [42] delved into the theoretical foundations of classroom-level data-based decision-making while also examining the impact of teacher support on this process. Through latent mediation analyses on longitudinal data from teachers and students, the study explores the connections between instructional decision-making based on pedagogical approaches and students’ reading progress.

2.3. Theory-Driven ML in Educational Research

Two main threats that reduce the repeatability and reliability of ML are the overfitting and underfitting of models due to limited sample sizes for training and largely defined variables [43]. Theory-driven ML approaches have been proposed to overcome the problem of inappropriate model fitting [9]. Theory-driven ML leverages existing theoretical frameworks (e.g., self-regulated learning) to scaffold models by identifying relevant variables and collecting relevant data [9,44]. Theory-driven ML models use theories of learning to define variables (or features) to include in predictive models. Variable selection methods assist theory-driven ML in determining whether to reduce the problem’s variables based on their importance for model predictability and their interdependence with other variables [35,44,45].

3. Theoretical Framework

Within a course, the conditions related to the student are overshadowed by the course-related conditions set by the instructor. E-learning systems have enhanced educational accessibility and introduced diverse avenues through which learners can interact with educational materials, thereby contributing substantively to the advancement of educational equity among a broad spectrum of students [46,47]. So, it is necessary to increase the scope of investigation in Technology-Enhanced Learning (TEL) as a requirement. TEL provides a framework for how an e-learning system meets the requirements of educational stakeholders [48]. People engage with e-learning systems, while e-learning technologies facilitate both direct and indirect interactions among various user groups that affect learning. E-learning systems encompass all activities aligned with pedagogical models and instructional strategies. Course conditions can be seen as part of the e-learning services component, which encompasses all activities aligned with pedagogical models and instructional strategies and controls the task and cognitive condition [46,49]. The external evaluation part of Cognitive Process-Embedded Systems (COPES) can be seen as part of the people component, as it involves learners engaging with the e-learning system, and the quality of the e-learning system affects their evaluation. By adding the conditions of the e-learning systems framework like pedagogy, instructional strategies, and the quality of Information Communication Technology (ICT), we can investigate most factors related to the course modality [46].

3.1. Self-Regulated Learning (SRL)

The concept of self-regulated learning involves perceiving learning as a dynamic process that entails the adaptive adjustment of cognition, behavior, and motivation to the content and the educational environment [15]. Refs. [50,51] proposed an SRL theory as a guiding framework in different learning environments for preferences of course modality, considering different conditions like students and instructors. One widely used theoretical model for investigating learning within self-regulated contexts is the Cognitive Process-Embedded Systems (COPES) model [46]. The COPES model investigates the process of learning during a task, considering the conditions that shape the satisfaction of tasks according to the learning goal and standards [46,52]. Conditions encompass both internal factors, such as the learner’s characteristics and knowledge about the topic, and external factors, including environmental variables that are believed to impact the task-related internal conditions [46]. Conditions like the skill and knowledge of students, motivation models, and task content are inputs in information processing and decision-making, and the outcomes will be used to evaluate task success based on predefined standards and individual goals [15,46]. Given the success of the COPES model in explaining student behavior in self-regulated learning contexts, the conditions from the COPES model serve as useful variables to aid prediction in an ML model [15,46].

3.2. Course Modality as a Part of Technology-Enhanced Learning

Technology-enhanced learning (TEL) is the incorporation of technology in learning environments to promote the process of teaching and learning [53]. Any tool that aids in improving decision-making and learning experiences or adds value to educational environments by aligning the environment, tools, and content together can be classified as TEL [54]. Therefore, the preference for different course modalities is related to individual beliefs and experience with, and preferences for, the integration of technology within the classroom [54,55]. As such, AI-based preference prediction tools should be investigated as a part of improving TEL [54,55].

3.3. E-Learning Systems

Within a course, conditions related to the student are often overshadowed by those set by the instructor. E-learning systems have enhanced educational accessibility, providing diverse avenues for interaction with materials, which advances educational equity among a broad spectrum of students [56]. These systems meet the needs of various stakeholders through a framework that integrates pedagogical models, instructional strategies, and Information Communication Technology (ICT) [48]. In the context of COPES, the course conditions align with the e-learning services component, which controls both task and cognitive conditions, while the external evaluation aspect aligns with the people component, as it involves learner engagement and system evaluation [48,52]. By incorporating the conditions of e-learning systems, such as pedagogy and ICT quality, into the COPES framework, we can investigate most factors related to course modality [48] (Figure 1).

4. Research Design

4.1. Participants

The research was carried out among 140 instructors and 379 students from the engineering departments of seven Iranian universities, namely the University of Tehran, Sharif University of Technology, Isfahan University of Technology, Shiraz University, Sistan and Balouchestan University, Imam International University, and Ahvaz Shahid Chamran University. Participants were recruited via email from these seven universities. There were no restrictions based on age or other demographic factors, and participants did not receive any incentives or rewards for their involvement. For this study, two parallel surveys were developed and used to collect data from instructors and students. The surveys consisted of 50 and 49 questions, respectively, about the six dimensions of our theoretical framework of self-regulated learning and e-learning systems theory. Items were written to capture a variety of beliefs across six subscales, including theory and practice, motivation, pedagogy, knowledge, insight, and skills, working life orientation, quality of assessment, and information communication technology (Supplementary S17, Table S21). In addition, participants were asked to indicate the type of course in which they were enrolled or taught (i.e., theoretical–practical, theoretical, and practical) and assess their preferred situation in each type of course using Likert scales (Supplementary Table S1). The lack of an established standardized survey called for the generation of pertinent survey items corresponding to distinct components of the framework. A pilot study of the surveys was conducted with ten students and five instructors to identify potential issues with item clarity.

Prior to analysis, data cleaning procedures were implemented to ensure the suitability and accuracy of the collected data [22,47,57,58]. This procedure consists of identifying and addressing any inconsistencies, errors, outliers, or missing values in the dataset. For the student survey, the items of F2 and F3 for fewer than 50 students were empty, and we put the value of the response variable instead of them. Only 160 students responded to the I4 response variable, as they did not have experience with practical types of classes (R code and libraries are in Supplementary S17 and Table S21).

4.2. Validity and Reliability Analysis

To examine the reliability of the survey items, Cronbach’s alpha coefficient was used to assess the internal consistency of the items [59], while discriminant validity was assessed using the Heterotrait–Monotrait Ratio of Correlations (HTMT) method [60], which examines the correlations of indicators across constructs and within the same construct [61]. The HTMT correlation ratio is an approach to examine the extent to which latent constructs are independent of each other [62,63]. Acceptable values of composite reliability/Cronbach alpha range from 0.60 to 0.70, while the acceptable range of HTMT is less than 0.9 [62,63] (methodology in Supplementary S8; results in Tables S8 and S9).

Cronbach’s alpha for instructors’ surveys has a reliable value with a 0.95 confidence interval ($\alpha =0.767$). However, for students’ surveys, there are a couple of subscales with few items, which causes a reduction in Cronbach’s alpha. For the students’ survey, the 0.95 confidence interval has also a reliable value ($\alpha =0.598$). This is consistent with the findings of Rempel et al. [62,64], who found that Cronbach’s alpha can be reduced by subscales with few items. The result of Cronbach’s alpha for each subscale for both surveys is presented in Table S22 of Supplementary S18. Tables S8 and S9 show the HTMT values for all subscales of both surveys. The highest HTMT value observed for instructors is 0.6 and for students, 0.532, which indicates that there is sufficient distinction between any two subscales in both surveys [62,63].

5. Methodology

5.1. Machine Learning Process

The first step in the machine learning (ML) process is variable selection, followed by modeling the problem using ML models as illustrated in Figure 2. In this study, both surveys feature a relatively large number of questions (variables) and a relatively limited number of responses. Given our interest in the most relevant variables and the varied accuracy of different variable selection methods, considering the contribution to accurate predictions and imbalanced data (where one class in some variables has significantly fewer instances compared to another class), we employed four interpretable variable selection methods with a significance level of $\alpha =0.05$. To enhance the precision of variable selection and subsequent ML model implementation, a refined approach was adopted as outlined in [13,65,66]. This approach involved retaining questions that exhibited similarity or analogous distinctive response classes to those found in Likert scale questions, excluding binary (Yes/No) inquiries, and questions with unrestricted response options [67]. Additionally, questions related to a specific learning management system, socioeconomic indicators, or personal identifying details such as the name of the university and age, were excluded. Consequently, 38 questions were retained from the student survey, while 34 questions were preserved from the instructors’ survey for variable selection [66].

The chi-square test, a filter method, computes the chi-square statistic for each variable about the target item and compares it to a critical value or p-value determined by the chosen significance level, indicating a significant association between categorical variables [67,68,69]. Variables with chi-square statistics exceeding the critical value or p-values below the significance level are selected [68,70]. The Maximum Relevance Minimum Redundancy (MRMR) method, another filtering method, aims to identify a subset of variables that have high relevance to the target variable while minimizing redundancy among the selected variables [71]. Wrapper methods employ a specific ML model to assess subsets of variables and identify the optimal subset that maximizes the model’s performance [72,73,74,75]. Boruta, a wrapper method, employs a random forest model to evaluate variable importance, adding randomly generated variables and comparing their significance to original variables [76]. Recursive Feature Elimination (RFE), an embedded method, involves training a model on the full variable set, ranking variables based on their importance, eliminating the least important variables, and repeating this process [77,78].

Six response variables (Table S12) representing theoretical, practical, and theoretical–practical class types were analyzed using supervised ML models. The majority voting method was utilized to integrate outcomes and find the best sets of variables for each response variable [77,78]. This integration not only enhances the precision and reliability of the predictive model but also addresses challenges such as overfitting, noise, and the curse of dimensionality [77,78]. The REF method incorporates the population of the primary sample of variables entering the ML model. Moreover, the chi-square method lacks a definitive boundary for variable exclusion [36,65]. A threshold for the chi-square test was imposed, and the top variables according to REF criteria were selected from the chi-square test. The variable selection rule for majority voting to enter the ML stage is as follows: If an item has been eliminated by two or more variable selection methods, the item will be removed (Table S6). Tables S2–S5 in the Supplementary indicate the variables selected by each of the chi-square, MRMR, Boruta, and RFE methods, describing the situation of each variable concerning the response variables. For REF and Boruta, 70% of the data were used for training ML models and 30% for testing, with the DT (CRAT) model used for Boruta and RF for the REF with 500 trees [22,36,45,57,65]. Table S7 indicates the variables that remained for ML.

5.2. Machine Learning Models

After selecting the variables, the next step in the ML process is to model the problem using ML models. In this study, according to [9], three ML models were utilized, including DT, SVM, and RF)

The DT (i.e., the CRAN model) method constructs a tree-like model of decisions and their possible consequences. Each variable in each tree predicts the response variable, and the leaf of DT represents the prediction. The construction of the tree involves recursively partitioning the data based on the most informative variable, aiming to maximize the separation of the target variable. This process continues until a stopping criterion is met, such as reaching a maximum depth in variable searching or a minimum number of samples of data for each variable per leaf [79] (methodology in Supplementary S13; results in Tables S15 and S16).

The RF model constructs an ensemble of decision trees, where each tree is trained on a different subset of the data using a random selection of variables. During prediction, the model aggregates the predictions of individual trees to make the final prediction, resulting in a robust and reliable model. Additionally, RF identifies the key factors influencing the outcome. The cross-validation procedure for RF involved partitioning the dataset into 70% for training and 30% for testing, adhering to a significance level of $\alpha =0.05$ [80] (methodology in Supplementary S12; results in Tables S13 and S14).

As the response variable has multiple dimensions, the multiclass SVM was used to find an optimal hyperplane that separates different classes. Train data lie closest to the hyperplane that separates the classes in a binary classification problem. This hyperplane is determined by identifying a subset of data according to each variable called support vectors. The SVM model identifies these support vectors during the training phase and uses them to compute the optimal hyperplane. After we selected the variables of the ML model, by separating the test and train starts, we needed to learn them.In this study, 70% of the data were used for training the ML models, and 30% of the data were used to test the models [81] (methodology in Supplementary S14; results in Tables S17 and S18).

5.3. Accuracy of ML Models

Accuracy in machine learning refers to the measure of how often a classification model correctly predicts the true class of an instance, typically expressed as a ratio of correct predictions to the total number of predictions:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$$

(1)

Equation (1) shows the formulation of accuracy in ML.

Precision is the ratio of correctly predicted positive observations to the total predicted positives. It indicates how precise the model’s positive predictions are, providing a measure of the relevancy of its positive results:

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(2)

Equation (2) shows the formulation of precision in ML.

Recall (also known as sensitivity or the True Positive Rate) is the ratio of correctly predicted positive observations to all the actual positives. It indicates how well the model can capture all the positive instances:

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(3)

Equation (3) shows the formulation of recall in ML.

The F1-score is the harmonic mean of precision and recall. It provides a single metric that balances both concerns of false positives and false negatives, making it particularly useful when the class distribution is imbalanced:

$$\mathrm{F}1-\mathrm{score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(4)

Equation (4) shows the formulation of the F1-score in ML.

The explication of True Positives (TP) as the quantification of instances accurately designated as positive, True Negatives (TN) as the quantification of instances accurately designated as negative, False Positives (FP) as the quantification of instances inaccurately designated as positive, and False Negatives (FN) as the quantification of instances inaccurately designated as negative, contained within the framework, provides a lucid exposition of the pivotal constituents that underlie the accuracy metric [22,57].

6. Results

6.1. Prediction Using Classification Techniques

The outcomes derived from the ML prediction methodologies concerning the application of five-point Likert scale responses to anticipate the six designated response categories reveal an elevated level of precision attributed to random forest (RF) models (

Table 1. The accuracy results of ML models based on test results for a five-point Likert spectrum for students and instructors (the results summary of Tables S14, S16 and S18). Table S22 indicates the F1-score, recall, and precision.

Models	Students			Instructors
	I2 Theoretical	I4 Practical	I6 Theoretical–Practical	H2 Theoretical	H4 Practical	H6 Theoretical–Practical
SVM	0.45	0.43	0.56	0.48	0.53	0.42
DT	0.39	0.40	0.45	0.41	0.53	0.35
RF	0.45	0.43	0.58	0.49	0.62	0.37

: the results summary of Tables S13, S15 and S17; Table S22 indicates the F1-score, recall, and precision), which is consistent with prior scholarly investigations [82]. According to Table 1, SVM and DT have less accuracy in comparison to RF. However, in all of them, the accuracy is not at an acceptable level.

The F1-scores for the “Strongly Agree” (SA) and “Strongly Disagree” (SD) classes exhibit significant variability across models and items, indicating inconsistent classification performance. For the SD class, certain models, such as RF and DT, achieve high recall (perfect in some cases) but low precision, resulting in moderate F1-scores, reflecting the models’ tendency to over-predict this class. In contrast, the SA class often has very low or zero F1-scores, particularly for items like I4, H4, and H6, suggesting a severe difficulty in accurately identifying instances of “Strongly Agree” (Table S22).

We address class imbalance using the Random Over-Sampling Examples (ROSE) technique, which generates synthetic samples from minority classes to balance the dataset with 70% train and 30% test. ROSE applies a smoothed bootstrap resampling process, creating new data points that reflect the distributional characteristics of minority instances by drawing from a kernel density estimate, rather than simply duplicating existing samples. This method is particularly effective for datasets with ordinal data, as it enhances the diversity of training data and improves the generalization capabilities of machine learning models [83,84,85]. ROSE maintains the structural integrity of the data while improving the model’s ability to differentiate between categories, especially in tasks involving ranking or classification [86,87]. The results demonstrate that while RF continues to perform well, showing improved accuracy in items such as I4 (practical) for both students and instructors, SVM and DT still face challenges. Instructors’ practical responses (H4) particularly illustrate how ROSE reshapes the dataset, allowing SVM and RF to achieve slightly higher accuracy compared to the non-ROSE sampling. However, the data size remains a limiting factor, as ROSE-driven improvements in accuracy, while notable, do not lead to a significant leap in predictive power across all items. This suggests that while ROSE helps mitigate class imbalance, further adjustments, such as merging Likert categories, might be needed to enhance model robustness.

The F1-score, recall, and precision metrics (

Table 2. The accuracy results of ML models based on test results for a five-point Likert spectrum for students and instructors via the ROSE method of sampling (F1-score, recall, and precision are in Table S24).

Models	Students			Instructors
	I2 Theoretical	I4 Practical	I6 Theoretical–Practical	H2 Theoretical	H4 Practical	H6 Theoretical–Practical
SVM	0.44	0.66	0.47	0.64	0.46	0.54
DT	0.38	0.62	0.41	0.61	0.21	0.32
RF	0.47	0.74	0.58	0.64	0.36	0.43

and Table S24) further illustrate the impact of ROSE sampling on model performance across various categories. For students’ theoretical and practical categories (I2 and I4), RF sees an improvement in precision and recall, particularly in categories with previously low representation, such as strongly disagree or agree, where synthetic samples boost prediction diversity. However, support for middle-range categories, such as neutral, remains less robust, highlighting the continued challenges faced by models like DT and SVM. Instructors’ responses, particularly for theoretical–practical categories (H6), demonstrate a noticeable increase in recall for SVM after applying ROSE but at the cost of reduced precision, signaling a tendency for over-prediction in some categories. This trade-off is seen across models, where the addition of synthetic samples helps increase recall but introduces more false positives, reducing precision and yielding modest improvements in F1-scores. This suggests that while ROSE helps balance the dataset, its effectiveness in improving predictive performance is nuanced and dependent on the class distribution.

Since the ROSE imbalance model did not sufficiently improve accuracy, a merging approach was employed to address the response imbalance between the “strongly agree” and “strongly disagree” categories. This approach consolidated these classes, reducing the number of categories in the Likert scale to enhance model performance. This approach reduces the students’ and instructors’ bias in responses, merges “strongly agree” and “agree” responses, and “strongly disagree” and “disagree” classes [88]. After merging the response classes, the implementation of ML changed to that of

Table 3. Accuracy averages of models for students and instructors. Table S23 indicates the F1-score, recall, and precision.

Models	Students			Instructors
	I2 Theoretical	I4 Practical	I6 Theoretical–Practical	H2 Theoretical	H4 Practical	H6 Theoretical–Practical
SVM	0.70	0.75	0.80	0.50	0.65	0.55
DT	0.71	0.72	0.71	0.53	0.65	0.48
RF	0.78	0.81	0.94	0.69	0.72	0.79

. And after merging these response classes, the performance of the machine learning models improved as shown in Table 3.

Results in Table 3 demonstrate a significant improvement in prediction accuracy (Table S11; Supplementary S10. Table S23 indicates the F1-score, recall, and precision), particularly for the RF model, which remained more accurate than the DT and SVM methods. The change from a 5-point Likert scale to a 3-point Likert scale was particularly beneficial for the analysis of instructors’ survey responses due to an increase in the accuracy of prediction for three response variables of the instructor. Unlike the RF model for the instructors, results indicated that DT and SVM could not make accurate predictions within the 3-point Likert scale for the instructors (specifically, the H4 and H6 response variables), which could be the result of the small sample, though this is not true for the students’ variables.

Despite the observed accuracy levels within the five-point Likert scale, discernible discrepancies surfaced in the “strongly agree”, “strongly disagree”, and “Neutral” classifications (Table 1 and Table 2). Regardless of the employed Likert scale, it is evident (Supplementary S17) that the “Neutral” class consistently manifested the highest error rates. This phenomenon can be attributed to the imbalance in the population of the “Neutral” class according to the population of this class compared to other classes. According to Tables S13–S18, the random forest method demonstrated superior accuracy levels concerning the “Neutral” category in comparison to the remaining classification methods, regardless of the employed Likert scale. While the DT model has acceptable accuracy in ranges of agree and disagree for 3-point Likert, their effectiveness decreases within the “Neutral” class of 3-point Likert in comparison with the 5-point Likert (Supplementary S17). Within the class reduction process from five-point Likert to three-point Likert, the “Neutral” class remained constant in quantity, unlike the other two categories that experienced augmentation which increased the imbalance of this class. This merging of imbalanced classes subsequently heightened the likelihood of errors within the Neutral class. Meanwhile, the SVM model demonstrated greater accuracy upon the transition from a five-point to a three-point Likert scale. This enhancement in accuracy can be the output of merging two imbalanced classes. Despite the smaller number of data records from instructors, their predictions demonstrated higher accuracy and a unanimous opinion in comparison to the students. This can be attributed to the defined theoretical framework and strong variable selection, which is aligned with instructors’ preferences and lets their responses be more predictable.

The models exhibit strong performance in classifying the “Agree” (A) class, as evidenced by consistently high F1-scores of 0.809756, with both precision and recall exceeding 0.80. This suggests that the models accurately identify instances of agreement with minimal misclassification. For the “Disagree” (D) class, moderate F1-scores of 0.657143 and closely aligned precision and recall indicate a balanced but less robust performance. However, the models demonstrate lower performance in classifying the “Neutral” (N) class, reflected in significantly lower F1-scores (Table S23).

6.2. Subscales’ Ranking and Framework

Given the results above, the determination of subscale rankings in predicting student and instructor preferences (RQ2) was undertaken by leveraging the advanced classification capabilities of the RF. The Mean Decrease in Impurity methodology, which assesses the significance of variables, was harnessed for this purpose (Supplementary S15; the most important variables of RF and DT are in Table S19). For both instructors and students, pedagogy and motivation are the most important subscales to increase the predictability of the response variables (course modality); however, the ‘theory and practice’ subscale indicated comparably diminished predictability. This diminished predictability might be related to the population of “Neutral” responses in related items in the survey as seen in the Supplementary Materials (S12–S14). The subscale of ‘Knowledge, insight, and skill’ also provides less information for predictability for the instructors when compared to the students. The subscale of ‘working-life orientation’ also is a prominent subscale for both students’ and instructors’ theoretical–practical response variables. Determinative subscales such as motivation, pedagogy, insight, and skills, working-life orientation, quality of assessment, and ICT assume pivotal roles in shaping preferences. It should be noted that the correlation of the theory and practice subscale with the responses is not high (Supplementary S12–S14).

7. Discussion

The results of the study indicate the superior predictive capabilities of the RF model in comparison to alternative models such as DT and SVM. Indeed, RF can handle high-dimensional response classes, imbalances, and multivariable situations. This study corroborates other studies where RF has been compared to other ML models, such as SVM and Logistic Regression [89,90]. A notable finding from this study is the predictive efficacy of RF models for “Neutral” responses, which is in an acceptable range of accuracy for both the 3-point and 5-point Likert scales. The SVM also provides accuracy in an acceptable range; however, DT has lower accuracy in comparison to SVM for most of the response variables. Consequently, the DT model tends towards divergent classifications with more clear distinct classes, thereby decreasing its suitability for high-level classification undertakings. This observation aligns with prior research [91,92]. However, the discriminatory boundaries between response classes like “agree” and “strongly agree” are not always an indicator of non-biased selection.

The COPES and e-learning systems theoretical frameworks were used to define the subscales and subsequently the variables. Utilizing four different variable selection methods helped to simplify the ML model and select the most relevant variables, which made the prediction more robust [91,92]. This means that some of the variables were fully aligned with the response variables and could help us to predict the course modality. Indeed, the defined subscales and variables from the theoretical framework with a high level of confidence can explain the course modality preferences for both instructors and students [91,92,93,94].

The subscales were defined according to three conditions in the theoretical framework, after which the ML ranked the subscales according to the responses (

Table 4. Ranking of different dimensions based on the average reduction in impurity of each group of items in Supplementary S15 and Table S19.

	Groups
	Students			Instructors
	I2 Theo.	I4 Prac.	I6 T-P	H2 Theo.	H4 Prac.	H6 T-P
Likert type (5-scale)
1	Ped.	Ped.	Ped.	Ped.	Ped.	Work-life Ori.
2	Motiv.	Motiv.	Work-life Ori.	Motiv.	Motiv.	Ped.
3	Qual. of Assess. & ICT	Know., Ins. & Skill	Motiv.	Work-life Ori.	Work-life Ori.	Motiv.
4	Know., Ins. & Skill	Theory & Pract.	Qual. of Assess. & ICT	Qual. of Assess. & ICT	Qual. of Assess. & ICT	Theory & Pract.
5	Work-life Ori.	Work-life Ori.	Know., Ins. & Skill	Know., Ins. & Skill	Theory & Pract.	Qual. of Assess. & ICT
6	Theory & Pract.	Qual. of Assess. & ICT	Theory & Pract.	Theory & Pract.	Know., Ins. & Skill	Know., Ins. & Skill
Likert type (3-scale)
1	Ped.	Ped.	Ped.	Ped.	Ped.	Motiv.
2	Motiv.	Motiv.	Motiv.	Motiv.	Motiv.	Ped.
3	Qual. of Assess. & ICT	Know., Ins. & Skill	Qual. of Assess. & ICT	Know., Ins. & Skill	Work-life Ori.	Work-life Ori.
4	Know., Ins. & Skill	Theory & Pract.	Know., Ins. & Skill	Work-life Ori.	Qual. of Assess. & ICT	Theory & Pract.
5	Work-life Ori.	Work-life Ori.	Work-life Ori.	Theory & Pract.	Theory & Pract.	Qual. of Assess. & ICT
6	Theory & Pract.	Qual. of Assess. & ICT	Theory & Pract.	Qual. of Assess. & ICT	Know., Ins. & Skill	Know., Ins. & Skill

Note: Abbreviations represent the following—Ped. indicates Pedagogy (blue); Motiv. indicates Motivation (orange); Work-life Ori. indicates Work-life Orientation (green); Qual. of Assess. & ICT indicates Quality of Assessment and ICT (lighter gray); Theory & Pract. indicates Theory and Practice (light gray).

and Figure 1). Motivation and pedagogy were categorized under course and cognition conditions. Alignment between pedagogy and motivation was the most prevalent amongst all other subscales. This result does not mean that other subscales like those related to task conditions are not important, but it means that for any decision-making about the course modality, the pedagogical approach of the instructor and how to motivate students should be prioritized and studied in any environment that is aligned with the result of studies like [91,92,93,94,95,96].

The findings from this study demonstrate the potential for ML to accurately classify student and instructor course modality preferences based on theoretical frameworks. So, the answer to the first research question is positive because the ML models with good accuracy can explain the multidimensional data and predict the preferences of students and instructors regarding the class modality.

The answer to the second research question is also affirmative. By considering the COPES and e-learning systems as the frameworks and studying the preferences about course modality in the context of course, task, and cognitive condition, we have a tool that predicts the preferences of students and instructors in higher education. Indeed, while these theories separately do not directly explain the course modality situation, the course modality is explainable by adding the course conditions that come from the e-learning framework to the COPES framework.

8. Conclusions

The necessity for adaptable instructional modalities is paramount to upholding educational standards and ensure accessibility in accurately predicting course modality preferences, which is critical for universities as they address the intricacies of education in the post-pandemic landscape. The alignment of pedagogical strategies with motivational factors has emerged as a significant predictor, emphasizing the importance of considering both instructional design and student engagement when selecting course modalities. This finding is instrumental for educational institutions aiming to optimize learning experiences, where the flexibility and adaptability of course delivery methods are crucial. Aligning educational strategies and course designs with the preferences revealed through ML analysis promises to create more stimulating and efficacious learning environments, thereby enhancing student engagement and achieving superior educational outcomes [91,92,94,95,97]. The results of this study are applicable to universities and educational institutions that have large amounts of high-dimensional data with lots of different subscales. As the type of data in access universities varies, in situations where survey data are not available, behavior data or historical preference records can be used to implement machine learning, specifically RF as the most robust model according to our result, to make decisions about the educational policies and activities like course modality based on students’ and instructors’ preferences [91,92,94,95]. The research further elucidates the significance of incorporating psychological constructs related to self-regulated learning within the decision-making paradigm. This underscores the merit of a data-informed approach in crafting educational experiences that are both personalized and attuned to the evolving dynamics of student and instructor needs. Such an approach is instrumental in fostering educational strategies that are responsive and tailored, reflecting the contemporary demands of the academic community. The investigation showcases the pivotal role of AI in transforming higher education by facilitating a harmonious balance between the quality of instruction and the multifaceted needs of the academic populace. This endeavor is crucial not only for augmenting the flexibility and robustness of educational frameworks but also for ensuring their continued relevance and efficacy in addressing the exigencies of forthcoming educational paradigms.

9. Future Research Directions

The development of predictive models for student and faculty preferences through this research marks a pivotal advancement in comprehending and accommodating the needs of the academic community, thereby providing a critical instrument for administrators striving to deliver educational experiences that are high in quality, equity, and effectiveness. Future research endeavors should aim to refine these predictive models, with a particular focus on elucidating the roles of motivation and pedagogy or exploring alternative models that more acutely incorporate these dimensions. Such efforts necessitate comprehensive data collection strategies within educational settings, encompassing a variety of sources to amass data conducive to the application of advanced AI Big Data methodologies. This approach will facilitate educational institutions in making more informed, confident, and precise decisions regarding course modality selections, leveraging the rich tapestry of data at their disposal.