SMART: Selection Model for Assessment Resources and Techniques

Abstract

The European Higher Education Area has ushered in a significant shift in university teaching, aiming to engage students more actively in classes. Professors have leveraged virtual platforms and external tools to introduce interactive tasks. With the proliferation of technology, educators face a challenge in choosing the most suitable approach. This paper presents SMART (Selection Model for Assessment Resources and Techniques), a methodology that determines the optimal assessment activities for university-level education. The methodology employs multicriteria decision-making techniques, specifically AHP and TOPSIS methods, to optimize activities based on various subject-, lecturer-, activity-, and student-related criteria. According to SMART, the top five assessment tasks are group and individual report submissions, workshops, complex H5P activities, and questionnaires. Therefore, it is advisable to prioritize these activities based on the methodology’s results, emphasizing their importance over other assessment methods.

1. Introduction

The first attempt to create the European Higher Education Area (EHEA) was in 1999, when 29 education ministers signed the “Bologna Declaration”. Currently, the EHEA consists of 48 countries, and its implementation has involved a major change in most universities, moving from a traditional approach (teacher-focused) to a more student-centred approach [1]. All this has led to more dynamic classes, usually through interactive tasks based on information and communication technologies (ICTs) [2]. In fact, some authors consider ICTs to be an essential element in 21st-century education [3,4]. Thus, universities usually have virtual platforms, called learning management systems, where all the necessary elements of the subjects are included. In Spain, the most common one is based on Moodle [5]. According to [6], Moodle is characterized by a series of functionalities grouped into two classes: resources (which include teaching materials: web pages, documents, presentations, etc.) and modules (which provide interaction between students and teachers: databases, assignments, forums, questionnaires, wikis, activities based on the HTML5 package (H5P), etc.) [7,8,9]. These modules, in turn, are related to different types of activities: creation, organization, delivery, communication, collaboration, and assessment [10]. In [11] the different Moodle modules are grouped into the activity classes previously indicated. Along with the variety of Moodle modules, in recent years, numerous virtual tools have also appeared (such as Kahoot! [12], Socrative [13], Quizizz [14] or Genially [15]), which allow for the gamifying of classes [16,17]. Gamification is defined as a methodology used to increase motivation, competitiveness, and people’s effort by using typical game techniques [18,19]. Gamification can be conducted individually (each student competes against the rest of their classmates) or cooperatively (in groups); in the latter case, in addition to the fun and motivating dynamics created by gamification, participation and interpersonal relationships are also encouraged, creating a more suitable environment for learning [20]. This change in educational philosophy implies a transformation not only in the teaching methodology, but also in the way we evaluate learning. The transition towards student-centred teaching involves a more active and participatory approach, where students are seen as active agents in their own learning process. This shift requires assessments that go beyond simple memorization tests and truly reflect the deep understanding, practical application, and critical skills that students acquire throughout their education [21]. Furthermore, the transition to learner-centred education demands that students are actively involved in evaluating their own learning progress. In fact, students should have opportunities to assess their acquisition of knowledge and skills, as well as provide peer feedback, throughout a course. Involving students directly in these assessment processes, whether through self-evaluations, peer reviews, or other methods, can promote higher-order thinking, improved metacognition, and collaborative learning [22]. Some alternatives analysed in the methodology proposed in this article involve students in self-assessment and the evaluation of their peers.

For the lecturer, having all these alternatives often makes it difficult to decide which of them are the most appropriate, thus becoming a decision problem on which one to use. It should be remembered that any activity carried out in class must make sense for the subject, that is, encompass competencies and learning outcomes, and encourage collaborative learning or the use of new technologies [23]. In addition, from the lecturer’s point of view, it is also important to assess other aspects, such as the complexity of preparing and/or grading it. Multicriteria decision-making (MCDM) methodologies are a branch of operational research that deals with finding the optimal solution in complex scenarios (which include conflicting objectives and factors), allowing one to objectively assess the different alternatives and order them according to the criteria analysed [24]. These methodologies have become popular in recent years, and have been used in very diverse fields, such as the selection of materials for optimal design [25], the parametric optimization of machining processes [26], the selection of green technologies for the rehabilitation of existing buildings [27], the optimal selection of wind power plant locations [28], and many other applications [29].

However, as far as the authors of this work know, MCDM has not been used in educational contexts. Thus, here, SMART (Selection Model for Assessment Resources and Techniques) is proposed, a methodology that aims to determine the best activities to perform in class, using the following set of MCDM methods: the analytic hierarchy process (AHP) and the technique for order of preference by similarity to ideal solution (TOPSIS) to optimize the different activity alternatives based on different defined criteria. The rest of the document is organized as follows: Section 2 explains the methodology of the proposed model; the model evaluation is presented in Section 3, analysing the results in Section 4 and Section 5; finally, Section 6 summarizes the main conclusions obtained after this study.

2. Methodology

SMART aims to answer the following question: what assessment activities should be implemented in a class? Apparently, it seems simple to solve, but it is really a complex decision. To solve this problem, an optimized methodology is proposed that obtains the most appropriate assessment activities according to some defined criteria. Figure 1 shows an overview of SMART, including the following phases: “Data”, “Analysis”, and “Results”. While the “Data” phase is the basis of the model, the “Analysis” and “Results” phases make up its optimization process. They are described in the following sections.

2.1. Data Phase

This phase is the structure of the problem to be solved, where all the information is collected and prepared to execute the optimization process. In general, it is necessary to know the education level, modality, whether a virtual platform is used as the main medium for classes or as support, etc. Based on these data, criteria and alternatives are defined.

• The criteria are the most important indicators that are considered in the evaluation process of the alternatives. They can be quantitative or qualitative and can be organized into main criteria (categories), subcriteria, etc. For example, for the problem analysed in this paper, several categories are involved: students, teachers, subject, and the activity in question. These categories deploy a series of indicators to be evaluated.
• The alternatives are the different options involved in decision making. In this model, the alternatives correspond to the different assessment activities being analysed.

To effectively determine and define the criteria and alternatives, the authors recommend asking different people related to the educational context (e.g., teachers, counsellors, etc.) with an extensive experience in the field. In this way, it is possible to consider a more comprehensive set of criteria and alternatives.

2.2. Analysis Phase

First, the weight of the criteria is determined according to the AHP model and then the decision matrix is created. The authors recommend using AHP whenever the group of experts has extensive experience in the topic addressed (in this case, online teaching experience).

2.2.1. AHP

In 1980, Professor Saaty formulated the AHP [30], used in the decision-making procedure. The hierarchy of the model characterizes the method so that the upper level represents the goal to be achieved, the intermediate level incorporates all levels of criteria and subcriteria that link to the model, and at the base are the alternatives that will be evaluated. The fundamentals of the method are mathematical and psychological, and it can be used to evaluate alternatives. However, in this work, it is used to calculate the weight of the criteria, as detailed below:

1.

Design the hierarchical model of the problem. In this step, the problem is modelled with a three-level hierarchical structure, encompassing the goal or objective, criteria, and alternatives; see Figure 2.

2.

Assignment and assessment of priorities. The objective of this step is to obtain the weight of the criteria based on the evaluation of the criteria: it can be performed with a scale directly or indirectly through the comparison between pairs of criteria where the different priorities are compared in a matrix R. Each piece of data is a positive numeric value that determines the relative priority between the row criterion compared to the column criterion; see

Table 1. Fundamental paired comparison scale proposed by Saaty. Source: own elaboration based on [30].

Scale	Verbal Scale	Explanation
1	Equal importance	Two criteria contribute equally to the objective
3	Moderate importance	Experience and judgement favour one criterion over another
5	Strong importance	One criterion is strongly favoured
7	Very strong importance	One criterion is very dominant
9	Extreme importance	One criterion is favoured by at least one order of magnitude of difference

.

The complete mathematical procedure is as follows. A priority vector is intended to be determined as shown in Equation (1). For this, Equation (2) is proposed, from which matrix W is obtained by assigning the weights ($w_j$) associated with the comparison of criteria $C_j$ ($j=1,2,\dots ,n$). The elements of the matrix are positive numbers.

$$\overrightarrow{w}=[w_1,w_2,\dots w_n]$$

(1)

$$\left(\begin{array}{ccc}{\displaystyle \frac{w_1}{w_1}}& \dots & {\displaystyle \frac{w_1}{w_n}}\\ ⋮& \ddots & ⋮\\ {\displaystyle \frac{w_n}{w_1}}& \dots & {\displaystyle \frac{w_n}{w_n}}\end{array}\right)\left(\begin{array}{c}{w}_1\\ ⋮\\ w_n\end{array}\right)=\mu \left(\begin{array}{c}{w}_1\\ ⋮\\ w_n\end{array}\right)$$

(2)

A simplified way to state the above equation is according to Equation (3):

$$W·\overrightarrow{w}=\mu ·\overrightarrow{w}$$

(3)

where for row i, the sum of the elements is $w_i·{\sum }_{j=1}^n\frac{1}{w_j}$, and for column j, the sum of the elements is $\frac{1}{w_j}·{\sum }_{i=1}^n{w}_i={\displaystyle \frac{1}{w_j}}$.

Once the matrix is normalized, the sum of the columns obtains the vector $\overrightarrow{w}$. This method demonstrates the mathematical strengths of reciprocity, homogeneity, and consistency [24]. Specifically, consistency is demonstrated using the consistency ratio ($CR$) indicator according to Equation (4), where $RI$ is the random consistency index (simulation of 100,000 random reciprocal matrices [31]) and $CI$ is the consistency index:

$$CR=\frac{CI}{RI};CI=\frac{{\lambda }_{\max }-n}{n-1}$$

(4)

Due to the dimensions of the matrix being considered, the weights of the criteria will be valid if $CR\le 0.1$. This threshold varies if the matrix has other dimensions.

2.3. Decision Matrix

The decision matrix database for the TOPSIS method is the set of evaluations of each alternative with respect to the criteria; see

Table 2. Structure of the decision matrix. Source: own elaboration based on [24].

	${\mathit{w}}_1$	${\mathit{w}}_2$	⋯	${\mathit{w}}_{\mathit{j}}$	⋯	${\mathit{w}}_{\mathit{n}}$
	${\mathit{C}}_{\mathbf{1}}$	${\mathit{C}}_{\mathbf{2}}$	⋯	${\mathit{C}}_{\mathit{j}}$	⋯	${\mathit{C}}_{\mathit{n}}$
$A_1$	$v_{11}$	$v_{12}$	⋯	$v_{1j}$	⋯	$v_{1n}$
$A_2$	$v_{21}$	$v_{22}$	⋯	$v_{2j}$	⋯	$v_{2n}$
⋮	⋮	⋮	⋱	⋮	⋱	⋮
$A_i$	⋮	⋮	⋱	⋮	⋱	⋮
⋮	⋮	⋮	⋱	⋮	⋱	⋮
$A_m$	$v_{m1}$	$v_{m2}$	⋯	$v_{mj}$	⋯	$v_{mn}$

, where:

A_i 

alternatives, $i=1,\dots ,m$;

C_j 

criteria, $j=1,\dots ,n$;

v_ij 

evaluation of alternative $A_i$ with reference to criterion $C_j$;

W 

vector of weights associated with the criteria, obtained according to Section 2.2.1.

2.4. Results Phase: TOPSIS

Finally, the TOPSIS method is applied in order to obtain a ranking of alternatives. With the classification of alternatives provided by SMART, the most appropriate assessment activities to perform in class are determined. The TOPSIS method is based on the definition of the ideal and anti-ideal for the selection of alternatives, and was created by [32]. It states that the finally selected alternatives minimize the distance to the positive ideal solution and maximize the distance to the negative ideal solution.

Figure 3 shows a graphical representation of the method with five alternatives ($A_1,\dots ,A_5$), two criteria ($C_1$ and $C_2$), and the plot of the ideal and anti-ideal points. Alternative $A_3$ is the closest to the ideal and $A_2$ and $A_4$ are the farthest from the anti-ideal. TOPSIS solves the problem by calculating the weighted distances to the ideal and anti-ideal for each alternative, through a multivariate data analysis [33].

The TOPSIS method algorithm is as follows [24]:

• Construction of the decision matrix.
• Normalization of the decision matrix.
• Construction of the normalized weighted matrix.
• Determination of the positive and negative ideal solution.
• Calculation of the positive and negative ideal solution.
• Calculation of the relative proximity of each alternative to the positive ideal solution.
• Ordering of the alternatives according to their relative proximity.

3. Model Evaluation

As stated in Section 2, SMART consists of three phases: “Data”, “Analysis”, and “Results”. Each of these phases is evaluated following the case study.

3.1. Data

The education level is university, specifically in a technical subject of an official master’s degree, designed under Royal Decree 861/2010 of 2 July [34], which determines the basic competencies that an official degree must develop according to its level (bachelor’s, master’s, or doctorate).

The educational modality is virtual, using the Moodle learning platform, 4.2 version, based on free software under a General Public Licence.

3.1.1. Criteria

Table 3. Criteria organized by category. Case study. Source: own elaboration.

Category		Criterion
Subject	$C_1$	General and transversal competencies
	$C_2$	Specific competencies
	$C_3$	Learning outcomes
Lecturer	$C_4$	Complexity in preparing the activity
	$C_5$	Grading of the activity
Activity	$C_6$	Use of new technologies
	$C_7$	Encourages collaborative learning
	$C_8$	Integration with the platform
Student	$C_9$	Degree of difficulty to perform the activity
	$C_{10}$	Feedback from the teaching team

shows the selected criteria organized by category.

With respect to the subject category, criteria $C_1$–$C_3$ are linked to the classification used by the Ministry of Education in the University Register [35]. Criterion $C_1$ (general and transversal competencies) refers to personal and interpersonal competencies; specific competencies ($C_2$) are related to competencies of a training nature and achievement of knowledge related to the Master’s degree; the learning outcomes criterion ($C_3$) encompasses a set of indicators that students are expected to understand, know, and be able to perform at the end of the subject [36]. Therefore, it will be very positive if the number of learning outcomes and competencies (specific, general, and transversal) encompassed by an activity is maximum.

In the lecturer category, the criteria correspond to the complexity when designing the activity from a technological point of view for criterion $C_4$ and the complexity in grading the activity for criterion $C_5$. That does not mean that simple design activities are better; quite the opposite. The aim is for teachers to be prepared to develop robust and interactive activities and have the necessary tools for grading them, whether directly from the Moodle platform itself or by implementing new applications [37] capable of extracting and analysing the results. The objective will always be to promote student learning.

The activity category includes criteria related to the inclusion of innovative new technology activities ($C_6$), as can be the case with H5P activities [38], encouraging collaborative learning ($C_7$) with group activities in which students not only share knowledge, but also participate in the assessment process [39], and criterion $C_8$ is related to the integration of the activity within the platform where the course is developed (Moodle, in this case).

Finally, the student category includes the degree of difficulty to perform the activity ($C_9$) from a technological point of view, and the feedback from the teaching team ($C_{10}$) received by the students; it is very important to highlight the importance of feedback for students, since mistakes are a valuable source of learning [40].

3.1.2. Alternatives

Ten assessment activities are designed that make up the proposed alternatives ($A_1$–$A_{10}$), see

Table 4. Alternatives. Case study. Source: own elaboration.

	Activities
$A_1$	Assignment: Report (Individual)
$A_2$	Assignment: Report (Group)
$A_3$	Questionnaire
$A_4$	Lesson
$A_5$	Workshop
$A_6$	Complex H5P Activity
$A_7$	Forums
$A_8$	Glossary
$A_9$	Databases
$A_{10}$	Genially: Escape Room case

.

The different case study alternatives are described below:

A₁ 

Moodle assignment where students, based on the instructions specified in the assignment itself (activity content, format, structure, etc.), will submit a report they have prepared individually. The activity has associated submission dates. To carry out the activity, they need advanced use of different programs: a word processor, a spreadsheet for statistical analysis and graph insertion, and stipulated technical programs of the subject. The activity is graded with a points rubric, with individual feedback by section and general feedback for the entire activity.

A₂ 

Same type as the previous one, but carried out in groups. The lecturer previously forms the different work groups, the activity content is much more extensive, and grading is performed by groups also with a rubric and partial and total feedback.

A₃ 

Moodle questionnaire. It consists of blocks of questions of different types (multiple choice, true/false, single choice, match options, small calculations, fill in texts, etc.). Once completed, students can view their grade with brief feedback, as well as view their responses. The teaching staff prepares the questionnaire based on the random selection of a question database.

A₄ 

Moodle lesson. A group of pages with different types of information and associated questions of different modalities (essays, multiple choice, true/false, single choice, etc.). Movement between pages can have different itineraries, depending on student responses. At the end of the lesson, the student will see brief feedback and their grade.

A₅ 

Moodle workshop. As in alternative $A_1$, students submit a report according to the professor’s specifications. Grading will be performed by the students themselves, through a rubric designed by the teacher. Self-assessment will weigh 20% and peer assessment 80%.

A₆ 

H5P activity. The H5P plugin is installed on the Moodle platform; therefore, the activity and grading are contained in the classroom. Its composition is a multicolumn object made up of different types of activities: fill-in texts, drag images or texts, select in an image, etc.; see Figure 4. Students receive the grade immediately after finishing the activity; the questions include a review and editing before ending the activity. The design of the activity by the teacher has required prior learning of the different objects to be used.

A₇ 

Moodle forum. Discussion on a subject topic; students can start a thread and the rest intervene positively or not in the different threads. At the end of the activity, the teacher will grade both types of interventions. The grade will be reflected in the grade book using a rubric.

A₈ 

Moodle glossary. In a collaborative way, students will prepare a list of definitions of a subject specified by the teacher. The teacher will reflect the grade in the grade book.

A₉ 

Moodle database. Students will create a technical data sheet of a certain technology and with the fields defined by the teacher. The teacher will reflect the grade in the grade book.

A₁₀ 

Activity with Genially. The activity is not integrated into the Moodle platform, but is an external web. It consists of an interactive Escape Room-type activity, carried out by groups. It consists of different tests, each of which provides a number, see Figure 5. When they have the sequence of numbers, the Escape Room ends. The grade will be reflected by the teacher in the grade book for the different groups, according to the time it takes the groups to solve the activity.

4. Analysis

Continuing the methodology, the “Analysis” phase is entered, where the weights of the previously defined criteria are determined, and the decision matrix is created.

4.1. Criteria Weights

Using the described sequence of the AHP method, the weight of the criteria is determined. A group of three experts ($E{x}_1$, $E{x}_2$, and $E{x}_3$) participate in the decision process. They independently carry out the pairwise assessment of each criterion. The expert matrices are shown in

Table 5. Expert 1 ($E{x}_1$) matrix. Source: own elaboration.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$
$C_1$	1	1/5	1/3	3	3	3	1/3	3	3	1/5
$C_2$	5	1	3	9	9	7	7	9	9	3
$C_3$	3	1/3	1	9	9	5	5	5	7	3
$C_4$	1/3	1/9	1/9	1	3	1/5	1/3	1/3	1/3	1/7
$C_5$	1/3	1/9	1/9	1/3	1	1/3	1/5	1/5	1/5	1/7
$C_6$	1/3	1/7	1/5	5	3	1	1/3	3	3	1/3
$C_7$	3	1/7	1/5	3	5	3	1	3	3	1/3
$C_8$	1/3	1/9	1/5	3	5	1/3	1/3	1	3	1/3
$C_9$	1/3	1/9	1/7	3	5	1/3	1/3	1/3	1	1/5
$C_{10}$	5	1/3	1/3	7	7	3	3	3	5	1

,

Table 6. Expert 2 ($E{x}_2$) matrix. Source: own elaboration.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$
$C_1$	1	1/7	1/7	3	3	1/3	1/3	3	1/3	1/3
$C_2$	7	1	3	9	9	5	5	7	3	3
$C_3$	7	1/3	1	9	9	5	5	7	3	3
$C_4$	1/3	1/9	1/9	1	3	1/5	1/5	1/3	1/5	1/5
$C_5$	1/3	1/9	1/9	1/3	1	1/5	1/3	1/3	1/5	1/3
$C_6$	3	1/5	1/5	5	5	1	1/3	3	1/3	1/5
$C_7$	3	1/5	1/5	5	3	3	1	3	1/3	3
$C_8$	1/3	1/7	1/7	3	3	1/3	1/3	1	1/3	1/3
$C_9$	3	1/3	1/3	5	5	3	3	3	1	3
$C_{10}$	3	1/3	1/3	5	3	5	1/3	3	1/3	1

and

Table 7. Expert 3 ($E{x}_3$) matrix. Source: own elaboration.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$
$C_1$	1	1/3	1/3	9	9	5	3	3	5	5
$C_2$	3	1	3	9	9	9	7	7	9	3
$C_3$	3	1/3	1	9	9	7	5	5	7	5
$C_4$	1/9	1/9	1/9	1	3	1/5	1/3	1/3	1/3	1/3
$C_5$	1/9	1/9	1/9	1/3	1	1/5	1/3	1/3	1/3	1/3
$C_6$	1/5	1/9	1/7	5	5	1	3	3	5	3
$C_7$	1/3	1/7	1/5	3	3	1/3	1	3	3	3
$C_8$	1/3	1/7	1/5	3	3	1/3	1/3	1	3	3
$C_9$	1/5	1/9	1/7	3	3	1/5	1/3	1/3	1	3
$C_{10}$	1/5	1/5	1/5	3	3	1/3	1/3	1/3	1/3	1

. An example of this assessment process is now detailed for expert 1 ($E{x}_1$) and the specific competencies criterion ($C_2$); see Figure 6. It is moderately important compared to learning outcomes ($C_3$) and degree of difficulty to perform the activity ($C_9$); strongly important compared to general and transversal competencies ($C_1$); much stronger than use of new technologies ($C_6$) and encourages collaborative learning ($C_7$); and extremely important compared to complexity in preparing the activity ($C_4$), grading the activity ($C_5$), integration with the platform ($C_8$), and feedback from the teaching team ($C_{10}$).

The $CR$ indicator is calculated, resulting in the following: $C{R}_{E{x}_1}=0.0922$, $C{R}_{E{x}_2}=0.0928$, and $C{R}_{E{x}_3}=0.0972$, all $<0.10$. Therefore, the matrices are valid and the weights of the criteria are obtained; see

Table 8. Criteria weight. Source: own elaboration.

Criterion	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$
Weight	0.0747	0.3127	0.2295	0.0204	0.0164	0.0622	0.0764	0.0405	0.0509	0.0722

.

The individual weight of the criteria that stand out the most are the specific competencies ($C_2$) and learning outcomes ($C_3$), representing 33% and 24%, respectively, of the total criteria; see Figure 7.

4.2. Decision Matrix

The decision matrix, which is the cross-reference between the attributes of the criteria and alternatives, is created according to the objective function and unit of each criterion, as shown in

Table 9. Units and objective function of decision matrix attributes. Maximize (↑). Minimize (↓). Source: own elaboration.

Criterion	Unit	Objective Function
$C_1$ General and transversal competencies	%	Maximize (↑)
$C_2$ Specific competencies	%	Maximize (↑)
$C_3$ Learning outcomes	%	Maximize (↑)
$C_4$ Complexity in preparing the activity	Likert	Minimize (↓)
$C_5$ Grading of the activity	Likert	Minimize (↓)
$C_6$ Use of new technologies	Likert	Maximize (↑)
$C_7$ Encourages collaborative learning	Likert	Maximize (↑)
$C_8$ Integration with the platform	Likert	Maximize (↑)
$C_9$ Degree of difficulty to perform the activity	Likert	Minimize (↓)
$C_{10}$ Feedback from the teaching team	Likert	Maximize (↑)

. Criteria $C_1$–$C_3$ are expressed as percentages of the total competencies ($C_1$–$C_2$) or learning outcomes ($C_3$). The remaining criteria are evaluated according to a Likert scale [41], as shown in

Table 10. Likert scale. Source: own elaboration based on [41].

Value	Description
1	Very little
2	Little
3	Moderately sufficient
4	Sufficient
5	A lot

.

Table 11. Decision matrix. Source: own elaboration.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$
$A_1$	45	70	80	4	5	2	1	5	2	5
$A_2$	48	80	90	5	5	2	5	5	3	5
$A_3$	35	50	95	5	1	3	1	5	2	4
$A_4$	30	45	60	5	1	4	1	5	3	4
$A_5$	48	55	70	5	4	3	5	5	4	5
$A_6$	40	60	65	5	1	5	1	5	4	4
$A_7$	20	25	35	2	3	2	5	5	2	3
$A_8$	18	20	20	2	1	2	1	5	2	2
$A_9$	15	20	18	2	1	2	1	5	2	2
$A_{10}$	25	45	45	5	4	5	3	1	4	4

finally shows the decision matrix that will be used in the following phase (“Results”).

5. Results

In the “Results” phase, the alternatives are evaluated according to the TOPSIS method described in the Methodology section (Section 2.4). Based on the criteria weights (Table 8) and the decision matrix (Table 11) from the previous step (“Analysis”; see Section 4), the matrix is normalized (

Table 12. Normalized matrix. Source: own elaboration.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$
$A_1$	0.427	0.457	0.411	0.328	0.562	0.231	0.112	0.353	0.246	0.429
$A_2$	0.456	0.522	0.462	0.410	0.562	0.231	0.559	0.353	0.369	0.429
$A_3$	0.332	0.326	0.488	0.410	0.113	0.346	0.112	0.353	0.246	0.343
$A_4$	0.285	0.294	0.308	0.410	0.113	0.462	0.112	0.353	0.369	0.343
$A_5$	0.456	0.359	0.359	0.410	0.450	0.346	0.559	0.353	0.492	0.429
$A_6$	0.380	0.392	0.334	0.410	0.113	0.577	0.112	0.353	0.492	0.343
$A_7$	0.190	0.164	0.180	0.164	0.337	0.231	0.559	0.353	0.246	0.257
$A_8$	0.171	0.131	0.103	0.164	0.113	0.231	0.112	0.353	0.246	0.171
$A_9$	0.142	0.131	0.093	0.164	0.113	0.231	0.112	0.353	0.246	0.171
$A_{10}$	0.237	0.294	0.231	0.410	0.450	0.577	0.335	0.070	0.492	0.343

) and the normalized weighted matrix is obtained. The relative proximity of each alternative to the positive ideal solution ($P{R}_i$) is calculated, as shown in

Table 13. Normalized weighted matrix. Source: own elaboration.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$	${\mathit{C}}_8$	${\mathit{C}}_9$	${\mathit{C}}_{10}$	${\mathit{d}}_{\mathit{i}}^{+}$	${\mathit{d}}_{\mathit{i}}^{-}$	${\mathit{PR}}_{\mathit{i}}$
$A_1$	0.032	0.143	0.094	0.007	0.009	0.014	0.009	0.014	0.013	0.031	0.0493	0.1298	0.7248
$A_2$	0.034	0.163	0.106	0.008	0.009	0.014	0.043	0.014	0.019	0.031	0.0249	0.1563	0.8628
$A_3$	0.025	0.102	0.112	0.008	0.002	0.022	0.008	0.014	0.013	0.025	0.0726	0.1129	0.6086
$A_4$	0.021	0.092	0.071	0.008	0.002	0.029	0.008	0.014	0.019	0.025	0.0911	0.0759	0.4545
$A_5$	0.034	0.112	0.083	0.008	0.007	0.022	0.043	0.014	0.025	0.031	0.0624	0.1054	0.6282
$A_6$	0.028	0.123	0.077	0.008	0.002	0.036	0.008	0.014	0.025	0.025	0.0659	0.1042	0.6127
$A_7$	0.014	0.051	0.041	0.003	0.006	0.014	0.043	0.014	0.013	0.019	0.1365	0.0453	0.2491
$A_8$	0.013	0.041	0.024	0.003	0.002	0.014	0.008	0.014	0.013	0.013	0.1589	0.0194	0.1090
$A_9$	0.011	0.041	0.021	0.003	0.002	0.014	0.008	0.014	0.013	0.013	0.1605	0.0192	0.1067
$A_{10}$	0.018	0.092	0.053	0.008	0.007	0.036	0.026	0.003	0.025	0.023	0.0976	0.0677	0.4096

.

The top five alternatives from the ranking obtained by SMART correspond to the following alternatives: Assignment. Report (Group)—$A_2$; Assignment. Report (Individual)—$A_1$; Workshop—$A_5$; Complex H5P Activity—$A_6$; and Questionnaire—$A_3$, as shown in Figure 8. These five alternatives represent 72% of the ideal solution, therefore, it can be confirmed that they are the best assessment activities for the purpose of fulfilling the objective functions of each criterion, within a technical subject of an official master’s degree.

5.1. Discussion

The SMART methodology is useful both for educators and institutions.

The key implications of this methodology for educators are as follows:

• Understanding which activities are optimal for assessment based on the specific criteria and educational context. In this particular case study, assignments, reports, workshops, complex H5P activities, and questionnaires were prioritized. However, depending on the criteria and educational context, traditional methods could also be suitable.
• Understanding tradeoffs between different activity features based on the criteria evaluations.
• Potentially saving preparation and grading time by replacing less optimal activities identified by the model.

The key implications of this methodology for institutions are the following:

• Using ranked activity data to provide teachers/lecturers with standardized recommendations or resources for assessments.
• Allocating educational technology budgets based on the activity ranking given by the model (e.g., tools for creating simulations).
• Establishing faculty training priorities around highly ranked activities, if skill gaps exist.

5.2. Limitations of the Study and Future Works

The focus of this study is to support teachers/lecturers to choose what assessment activity to prepare. However, there are also some points that could be considered in the future:

• Even though all the activities under analysis have been carried out throughout the described course, it is also interesting to confirm if by conducting the top five alternative ranking, the students’ qualifications improve.
• Only instructor and activity factors were evaluated. Students’ preferences and perspectives could be incorporated into the SMART methodology by including student-related criteria in the criteria set used for evaluation, or by conducting a student survey to identify highly rated or preferred activities and using these as the activities (alternatives) for the SMART method.
• Analysing the results using other combinations of MCDM techniques. For instance, to weight the criteria, entropy (objective method), the analytic network process (ANP) or the best worst method (BWM) can be used. Moreover, the VIseKriterijumska Optimizacija I Kompromisno Resenje (multicriteria optimization and compromise solution, VIKOR) and more could rank the alternatives and then compare the results with the ÉLimination Et Choix Traduisant la REalité (elimination and choice translating reality, ELECTRE) to eliminate the least favourable options.

6. Conclusions

The European Higher Education Area has promoted major changes in university teaching so that classes follow a more student-centred approach. Thus, teaching staff have begun to generate interactive content, taking advantage of new technologies, either through the modules included in virtual platforms (like Moodle, in the Spanish case) or other external tools. Therefore, the difficulty encountered by the lecturer lies in determining which activity is most appropriate or preferable to use. SMART aims to solve this problem by analysing different university-level activities according to various criteria (related to the subject, lecturer, activity, and student) using the AHP and TOPSIS multicriteria decision-making techniques. According to SMART, and with the activities and criteria considered, the five best activities are submission tasks such as reports (group and individual), workshops, complex H5P activities, and questionnaires. While SMART has originally been designed for a technical master’s subject, it is crucial to acknowledge that its applicability extends to diverse modalities and educational levels. However, it is essential to emphasize that while the methodology offers valuable insights into activity selection, the specific findings may not be universally applicable. The ranking process should be independently conducted for each educational scenario to ensure optimized, context-specific results, considering the unique criteria and needs of each case.