An Expert-Opinion-Based Evaluation Framework for Sustainable Technology-Enhanced Learning Using Z-Numbers and Fuzzy Logarithm Methodology of Additive Weights

Abstract

As technology continues to shape the landscape of education, the need for effective evaluation frameworks for sustainable technology-enhanced learning (TEL) becomes increasingly vital. This study presents an expert-opinion-based evaluation framework, utilizing Z-numbers and the fuzzy logarithm methodology of additive weights (LMAW), to assess the sustainability of TEL approaches. This framework focuses on four main criteria: cloud services compliance, cloud M-Learning essentials, system and technological advancement, and organizations management readiness. Additionally, it incorporates 17 sub-criteria to provide a comprehensive evaluation of the system. Drawing on the expertise of subject matter specialists, the evaluation framework utilizes Z-numbers to account for the inherent uncertainty and imprecision in expert judgments. The fuzzy LMAW is applied to calculate the overall scores for each criterion and sub-criterion, enabling a quantitative measure of their importance in the evaluation process. The findings of this study will contribute to the development of a robust and scientifically rigorous evaluation framework for sustainable TEL. By incorporating expert opinions and employing Z-LMAW, decision-makers and stakeholders can objectively assess the sustainability of TEL systems. This framework holds promise for informing the design and implementation of strategies to enhance the quality, compliance, and technological advancements in TEL environments.

1. Introduction

Technology-enhanced learning (TEL) has revolutionized the field of education, offering new avenues for engaging and personalized learning experiences [1]. As the adoption of digital technologies continues to grow, it is crucial to assess the sustainability of TEL systems to ensure their long-term effectiveness and societal impact [2,3,4,5]. Sustainability in this context encompasses not only environmental considerations but also economic viability, social equity, and pedagogical effectiveness [6,7]. Educational institutions are increasingly recognizing the need to align their learning practices with sustainable development goals, aiming to create educational experiences that empower learners to become responsible global citizens who can address complex challenges [8,9,10]. Evaluating the sustainability of such systems requires a comprehensive framework that captures the multifaceted nature of their impact and provides actionable insights for educators, policymakers, and technology developers.

In line with this pressing need, this study introduces an expert-opinion-based evaluation framework, supported by the integration of Z-numbers and the fuzzy logarithm methodology of additive weights (Z-LMAW), for assessing the sustainability of TEL systems. By leveraging the collective wisdom and expertise of subject matter specialists, this framework combines qualitative insights with quantitative analysis, facilitating a comprehensive and rigorous evaluation process.

The evaluation framework centers around four key criteria identified as fundamental to sustainable TEL: cloud services compliance, cloud M-Learning essentials, system and technological advancement, and organizations management readiness [11]. Each criterion represents critical aspects of the TEL ecosystem, including compliance with cloud service standards, essential features for mobile learning in cloud environments, technological advancements, and organizational readiness for managing and sustaining TEL initiatives [12,13]. To further enhance the assessment, a set of 17 sub-criteria were identified to provide granular insights into the specific dimensions of sustainability.

The incorporation of Z-numbers and the fuzzy LMAW empowers decision-makers to systematically evaluate the sustainability of TEL systems. Z-numbers account for the inherent uncertainty and imprecision of expert judgments, ensuring a robust consideration of diverse perspectives [14]. The fuzzy LMAW synthesizes expert opinions into comprehensive scores, enabling a quantitative measure of the importance assigned to each criterion and sub-criterion [15].

By employing this evaluation framework, stakeholders in the education sector can make informed decisions and drive improvements in the sustainability of TEL systems. The results of the assessment provide valuable insights into the strengths and weaknesses of existing systems, enabling targeted interventions to enhance their sustainability and overall educational impact. Furthermore, this framework can serve as a stepping stone for the development of evidence-based policies, practices, and investments in TEL that align with the principles of sustainability. Through the application of the Z-LMAW method, it will be established which criteria are key to the success of TEL systems in sustainable education.

Since this study addresses the critical need for evaluating the sustainability of TEL systems through an expert-opinion-based Z-LMAW evaluation framework by combining qualitative and quantitative approaches, this framework provides a comprehensive understanding of the sustainability landscape and offers valuable insights for advancing TEL in a sustainable manner.

The aim of this paper is to evaluate and determine the importance of criteria for the implementation of TEL systems in educational institutions. Based on this, important information were collected on which criteria are most important to experts and which are not. Thanks to these results, it is possible to improve TEL systems in order to be more efficient and better achieve sustainability goals. The present research is focused on the College of Tourism with the primary goal of improving the system at this higher education institution. Based on these results, other higher education institutions in the region and beyond can improve their TEL systems so that they are better and better achieve the goals of sustainability in education for the future. In the future, it is necessary to use TEL systems to influence sustainability and overall efficiency in education, especially in specialist studies where the focus of the study is narrow and specialist, such as in the study of tourism. In order to achieve sustainability in education using TEL systems, this research needs to answer the question: what are the key criteria for the success of a sustainable TEL system?

The subsequent sections of this paper provide a comprehensive exploration of the expert-opinion-based Z-LMAW evaluation framework for sustainable TEL. In Section 2, the current literature is reviewed. In Section 3—Preliminaries, the methodology employed in this study, including the incorporation of Z-numbers and the fuzzy LMAW, is presented, offering insights into the evaluation process. Section 4 focuses on the case study conducted at the College of Tourism, and Section 5—Results highlights the practical implications and recommendations for educators, policymakers, and researchers based on the findings derived from the evaluation framework. In Section 6—Discussion, the results are critically analyzed, and their implications for the field of TEL are discussed, addressing key themes and areas of further investigation. Finally, Section 7—Conclusions summarizes the main contributions of this study, provides a synthesis of the findings, and outlines avenues for future research, underscoring the importance of sustainable TEL in the educational landscape.

2. Literature Review

Sustainable TEL has emerged as a significant area of research and practice in the field of education. The concept of sustainability in TEL encompasses multiple dimensions, including environmental, economic, and social aspects. Scholars have emphasized the importance of aligning educational practices with sustainable development goals [16], fostering a culture of environmental responsibility, and promoting social equity through TEL initiatives. Several studies have highlighted the potential of technology to minimize environmental impacts, such as reducing paper consumption [17,18,19], energy usage [20,21], and carbon emissions [22,23] through virtual learning environments and online resources.

Pedagogical approaches in sustainable TEL focus on learner-centered strategies, fostering active engagement, collaboration, and critical thinking [24,25]. Research has explored the integration of digital technologies, such as mobile devices, online platforms, and virtual reality, to enhance learning experiences and promote self-directed learning [26,27,28]. The use of gamification, adaptive learning systems, and personalized learning pathways has also been investigated to optimize engagement and knowledge retention in sustainable TEL environments [29,30,31].

Technological advancements play a crucial role in sustainable technology-enhanced learning [32]. Emerging technologies, such as artificial intelligence (AI), machine learning (ML), and the Internet of Things (IoT), have the potential to transform educational practices and improve the efficiency and effectiveness of learning processes [33,34]. Studies have explored the integration of these technologies to facilitate personalized learning experiences, adaptive assessment, and intelligent feedback mechanisms [35,36,37,38,39]. Additionally, the development of cloud-based learning platforms and mobile applications has expanded access to education, enabling learners to engage in learning anytime and anywhere [40,41,42].

Organizational considerations are vital for the successful implementation and sustainability of TEL initiatives. Educational institutions need to establish supportive policies, infrastructure, and resources to foster a culture of innovation and technology integration [43]. Research [44,45] has highlighted the importance of leadership, professional development programs, and collaborative partnerships between educators, policymakers, and technology developers to ensure effective implementation and long-term sustainability of TEL initiatives.

Since modern students are part of a mobile “homuter” society [46], their mobile phones, tablets, computers, and other information communication technology devices are considered not only integral parts of themselves but also the primary means of communication and acquiring knowledge. The rapid proliferation of technology has led to virtualization [46,47], wherein individuals’ fundamental social needs are fulfilled through computers and/or mobile devices. This has given rise to a new phenomenon known as “cyber-socialization” [48]. Consequently, traditional didactic lectures quickly become outdated teaching methods. To cater to the needs of modern students, teachers should incorporate a range of recent, cutting-edge pedagogical approaches, educational technologies, and teaching strategies into their toolkit [49]. These approaches can either supplement existing lectures or be used as alternatives to the conventional lecture format.

It is believed that technologies-in-practice are influenced by students’ knowledge, practices, and contextual factors [50]. However, the use of technology itself may lead to changes in its nature [51]. Additionally, previous studies [51] suggest that students tend to prefer their own mobile technologies while utilizing various distance learning platforms [52]. Research findings [53] indicate that video conferencing platforms and learning management systems (LMSs) are the two primary categories of online tools and resources employed by most institutions. Commonly used online tools include Canvas, Blackboard, Google Classroom, and Microsoft Teams, while platforms like Moodle, Zoom, WhatsApp, and Viber are also widely utilized.

While the literature on sustainable TEL is extensive, several gaps and challenges remain. Scholars have called for further research on the assessment and evaluation of the sustainability impact of TEL systems. The development of comprehensive evaluation frameworks, like the one presented in this study, can provide valuable insights into the strengths and weaknesses of existing systems and guide decision-making processes. Furthermore, the integration of ethical considerations, digital citizenship, and social responsibility in TEL practices requires continued exploration [54,55].

The literature on sustainable TEL demonstrates the increasing importance of aligning educational practices such as e-learning, m-learning, and u-learning [13] with principles of sustainability. By integrating environmental, economic, and social dimensions into TEL initiatives, educational institutions can create effective and environmentally conscious learning environments.

3. Preliminaries

Decision making is a pervasive phenomenon across various domains [56]. Diverse approaches have been employed to facilitate the decision-making process [57,58]. In 1965, Zadeh introduced fuzzy numbers as a means of accommodating human reasoning in decision making [59]. He posited that decision making could be performed even in the absence of precise data, wherein imprecise judgments come into play. Fuzzy logic serves as an extension of classical logic in decision making, particularly when assessments lack well-defined boundaries [60,61]. This arises due to the existence of real-world situations that defy clear definition, making it arduous to establish crisp set boundaries [62,63]. Consequently, evaluations are expressed as linguistic values, tailored to align with human cognition [64,65]. To utilize these values effectively, a fuzzy number membership function is formulated to assign fuzzy numbers to specific linguistic values [66]. In fuzzy numbers, the boundaries are not precisely delineated, leading to overlapping boundaries [67,68]. Over time, advancements have been made to enhance these fuzzy numbers, resulting in novel applications [69]. To address decision making under conditions of uncertainty, the concept of Z-numbers was introduced [70]. A Z-number, denoted as Z = (A, B), consists of an A fuzzy number representing the expert’s assessment value and a B fuzzy number representing the reliability of the A fuzzy number.

Considering an A fuzzy number $A=\left(a_1.a_2.a_3\right)$ and a B fuzzy number $B=(b_1.b_2.b_3)$, the Z-number can be expressed as:

$$\tilde{Z}=\left\{\left(a_1.a_2.a_3;w_A\right).(b_1.b_2.b_3;w_B)\right\}$$

(1)

To employ Z-numbers, a transformation from Z-numbers to classical fuzzy numbers is necessary. This transformation follows a series of steps [71,72]:

Step 1: Transforming the B fuzzy number into a crisp number using defuzzification. This step converts the fuzzy number into a crisp number to allow operations on the A fuzzy number.

$$\alpha =\frac{b_1+b_2+b_3}{3}$$

(2)

Step 2: Adding the weight of the second part (B) of the Z-number to the first part (A) of the fuzzy number. This step can be represented as follows:

$${\tilde{Z}}^{\propto }=\left\{⟨\left.x.{\mu }_{A^{\alpha }}\left(x\right)⟩\right|{\mu }_{A^{\alpha }}\left(x\right)=\alpha {\mu }_A\left(x\right)\right\}$$

(3)

Step 3: Converting the Z-number into a regular fuzzy number. This is achieved by taking the square root of the defuzzification value of the B fuzzy number and adding it to the A fuzzy number.

$${\tilde{Z}}^{\prime }=\sqrt{\alpha }·\tilde{A}=(\sqrt{\alpha }·a_1.\sqrt{\alpha }·a_2.\sqrt{\alpha }·a_3)$$

(4)

This transformation converts the Z-number into a classical fuzzy number by incorporating the degree of uncertainty represented by the B fuzzy number into the decision-making process.

In this study, the Z-LMAW method was used to determine the criteria weights. This method extends the fuzzy LMAW method with Z-numbers. The weight assignment is accomplished by considering the significance of each criterion, as represented by the A fuzzy number, along with the expert’s uncertainty in their evaluations, expressed by the B fuzzy number.

The LMAW method was initially introduced by Pamučar et al. [73]. In addition to weight determination, this method also allows for the ranking of alternatives. In this paper, the Z-LMAW method was employed to determine the criterion weights, thereby identifying the most critical criterion for sustainable TEL. The steps of the Z-LMAW method are as follows:

Step 1: Prioritization of criteria. In this step, experts evaluate the criteria ($C=\left\{C_1.C_2.C_3.\dots .C_n\right\}$) and determine the individual significance of each criterion based on their level of agreement according to the defined linguistic scale for the A fuzzy number. They also consider the degree of certainty in their assessment using the defined linguistic scale for the B fuzzy number (

Table 1. Linguistic scales for assessing criterion significance.

Linguistic Value	Fuzzy Number A	Linguistic Value	Fuzzy Number B
Absolutely low (AL)	(1. 1. 1)	Very small (VS)	(0. 0. 0.2)
Very low (VL)	(1. 1.5. 2)	Small (S)	(0.1. 0.25. 0.4)
Low (L)	(1.5. 2. 2.5)	Medium (M)	(0.3. 0.5. 0.7)
Medium low (ML)	(2. 2.5. 3)	High (H)	(0.55. 0.75. 0.95)
Equal (E)	(2.5. 3. 3.5)	Very high (VH)	(0.8. 1. 1)
Medium high (MH)	(3. 3.5. 4)
High (H)	(3.5. 4. 4.5)
Very high (VH)	(4. 4.5. 5)
Absolutely high (AH)	(4.5. 5. 5)

).

Based on these assessments, the priority vectors ${\tilde{A}}^e=({\tilde{a}}_{C1}^e.{\tilde{a}}_{C2}^e.\dots {\tilde{a}}_{Cn}^e)$ and confidence degree vectors ${\tilde{B}}^e=({\tilde{b}}_{C1}^e.{\tilde{b}}_{C2}^e.\dots {\tilde{b}}_{Cn}^e)$ are defined, considering the evaluations for the A fuzzy number and the B fuzzy number.

Step 2: The Z-number is transformed into an ordinary fuzzy number through the operations outlined in Equations (1)–(4). This transformation results in new priority vectors with integrated degrees of agreement with the given evaluations ${\tilde{p}}^e=({\tilde{\gamma }}_{C1}^e.{\tilde{\gamma }}_{C2}^e.\dots {\tilde{\gamma }}_{Cn}^e)$. After the transformation from Z-numbers to ordinary fuzzy numbers, the conventional steps of the fuzzy LMWA method are followed.

Step 3: Determination of the absolute fuzzy anti-ideal point (${\tilde{\gamma }}_{AIP}$). This value is smaller than the minimum value in the set of priorities.

Step 4: Calculation of the values of the fuzzy ratio vector ${\tilde{R}}^e=({\tilde{\eta }}_{C1}^e.{\tilde{\eta }}_{C2}^e.\dots {\tilde{\eta }}_{Cn}^e)$. This ratio vector is obtained by dividing the transformed Z-numbers by the value of the absolute fuzzy anti-ideal point.

$${\tilde{\mu }}_{Cn}^e=\left(\frac{{\tilde{\gamma }}_{Cn}^e}{{\tilde{\gamma }}_{AIP}}\right)=\left(\frac{{\gamma }_{Cn}^{\left(l\right)e}}{{\gamma }_{AIP}^{\left(r\right)}}.\frac{{\gamma }_{Cn}^{\left(m\right)e}}{{\gamma }_{AIP}^{\left(m\right)}}.\frac{{\gamma }_{Cn}^{\left(r\right)e}}{{\gamma }_{AIP}^{\left(l\right)}}\right)$$

(5)

Step 5: Calculation of the criterion weight vectors individually for each expert. In this step, the natural logarithm (ln) of the fuzzy ratio vector values is computed and divided by the natural logarithm of the product of these values.

$${\tilde{\omega }}_j^e=\left(\frac{\mathrm{l}\mathrm{n}\left({\tilde{\mu }}_{Cn}^e\right)}{\mathrm{l}\mathrm{n}\left(\prod _{j=1}^n{\tilde{\mu }}_{Cn}^e\right)}\right)=\left(\frac{\mathrm{l}\mathrm{n}\left({\mu }_{Cn}^{\left(l\right)e}\right)}{\mathrm{l}\mathrm{n}\left(\prod _{j=1}^n{\mu }_{Cn}^{\left(r\right)e}\right)}.\frac{\mathrm{l}\mathrm{n}\left({\mu }_{Cn}^{\left(m\right)e}\right)}{\mathrm{l}\mathrm{n}\left(\prod _{j=1}^n{\mu }_{Cn}^{\left(m\right)e}\right)}.\frac{\mathrm{l}\mathrm{n}\left({\mu }_{Cn}^{\left(r\right)e}\right)}{\mathrm{l}\mathrm{n}\left(\prod _{j=1}^n{\mu }_{Cn}^{\left(l\right)e}\right)}\right)$$

(6)

Step 6: Determination of the final criterion weights using the Bonferroni aggregator. This aggregator is employed to harmonize the individual criterion weights obtained from the experts.

$$\begin{array}{l}{\left(\frac{1}{k\left(k-1\right)}{\displaystyle \sum _{\begin{array}{c}i.j=1\\ i\ne j\end{array}}^k}{\tilde{\omega }}_i^{\left(e\right)p}{\tilde{\omega }}_i^{\left(e\right)q}\right)}^{\frac{1}{p+q}}\\ =\left\{{\left(\frac{1}{k\left(k-1\right)}{\displaystyle \sum _{\begin{array}{c}i.j=1\\ i\ne j\end{array}}^k}{\omega }_i^{\left(l_e\right)p}{\omega }_i^{\left(l_e\right)q}\right)}^{\frac{1}{p+q}}.{\left(\frac{1}{k\left(k-1\right)}{\displaystyle \sum _{\begin{array}{c}i.j=1\\ i\ne j\end{array}}^k}{\omega }_i^{\left(m_e\right)p}{\omega }_i^{\left(m_e\right)q}\right)}^{\frac{1}{p+q}}.{\left(\frac{1}{k\left(k-1\right)}{\displaystyle \sum _{\begin{array}{c}i.j=1\\ i\ne j\end{array}}^k}{\omega }_i^{\left(r_e\right)p}{\omega }_i^{\left(r_e\right)q}\right)}^{\frac{1}{p+q}}\right\}\end{array}$$

(7)

Step 7: Defuzzification of the fuzzy criterion weights into crisp values.

$$w_j=\frac{w_j^l+4\times w_j^m+w_j^r}{6}$$

(8)

4. Case Study

In an era marked by globalization and technology-driven industries, educational institutions such as the College of Tourism must adapt and embrace innovative approaches to meet the evolving needs of students and the industry. TEL presents opportunities to enhance the educational experience, engage students actively, and equip them with the necessary skills for success in the digital age. Established in 1967, the College of Tourism has a distinguished history as a leading institution in the field of tourism education. Over the years, the college has witnessed significant transformations in both the tourism industry and the educational sector. The rapid advancements in technology and evolving student expectations have necessitated a proactive approach to ensure that the college’s educational offerings remain relevant and effective. In response to the changing educational landscape and the transformative potential of technology, the college has undertaken an initiative to integrate TEL approaches into its curriculum.

TEL plays a crucial role in preparing future tourism professionals. By leveraging technology tools and platforms, the College of Tourism can achieve several significant benefits. Firstly, TEL facilitates interactive and experiential learning, enabling students to engage with real-world scenarios and simulations, thereby enhancing their critical thinking and problem-solving skills [74,75]. Secondly, TEL ensures industry relevance by keeping the college updated with industry advancements, thus providing students with relevant knowledge and skills [44]. Thirdly, TEL promotes global collaboration by transcending geographical barriers, allowing students to interact with peers from diverse cultural backgrounds [76,77]. Fourthly, TEL equips students with essential digital competencies, such as data analysis and online marketing, which are highly valued by employers in the tourism sector [78]. Lastly, TEL fosters a culture of lifelong learning by providing access to educational resources beyond traditional classroom settings [79].

To evaluate the sustainability of TEL at the College of Tourism, an expert panel comprising individuals with diverse perspectives and expertise was assembled. In this way, the panel included 5 experts who actively use TEL systems in teaching. The panel included a technology expert, an education specialist, a business manager, a tourism industry professional, and an educational researcher. The technology expert possesses comprehensive knowledge of educational technology trends, emerging tools, and practical implementations. They assessed the integration of technology platforms, software applications, and hardware devices to ensure seamless implementation and usability of TEL approaches at the College of Tourism. The education specialist is an expert in the field of education, focusing on pedagogy, instructional design, and curriculum development. Their evaluation encompassed how TEL approaches align with the college’s educational goals, promote student engagement, and enhance learning outcomes, ensuring that TEL strategies are founded on sound educational principles. The business manager provides insights into the organizational and managerial aspects of integrating TEL at the College of Tourism. They evaluated the financial implications, resource allocation, and long-term sustainability of TEL initiatives, ensuring that the college’s strategic goals and objectives were met. The tourism industry professional brings industry-specific expertise and insights. They evaluated the relevance of TEL approaches in addressing the needs and demands of the tourism industry, ensuring that TEL strategies align with industry trends, enhance employability, and prepare students for successful careers in tourism. The educational researcher contributes a research-based perspective, examining the impact and effectiveness of TEL in educational settings. Their assessment drew upon the existing literature, data analysis, and evidence-based recommendations to inform decision making and further enhance the sustainability of TEL at the College of Tourism.

To assess and rank the critical success criteria for sustainable TEL, the expert panel followed a meticulous selection process based on the work of Naveed et al. [11]. The framework encompassed main criteria such as cloud services compliance (CSC), cloud M-Learning essentials (CLE), system and technological advancement (STA), and organizations management readiness (OMR), which were further supplemented by relevant sub-criteria (Figure 1). By relying on the research-driven criteria and sub-criteria from Naveed et al. [11], the expert panel established a solid foundation for the subsequent evaluation and ranking of these identified factors. This selection process enabled the College of Tourism to make informed decisions and enhance the sustainability of its TEL initiatives. At the end of the panel discussion, survey questionnaires were distributed to the experts, with which the experts evaluated which factors most affect the efficiency of TEL systems at this college.

In the evaluation process, the expert panel underwent a two-step procedure. Firstly, they assessed the criteria using linguistic values for the A fuzzy number, as outlined in Table 1. This initial evaluation enabled them to determine the relative importance and significance of each main and sub-criterion. Subsequently, they also considered the degree of certainty in their assessment by referring to the defined linguistic scale for the B fuzzy number (Table 1).

The methodology used in this research is presented in Figure 2. The first step was the selection of experts, then the selection of criteria. Then, linguistic evaluations were used to determine how important each criterion was to a certain expert, with the expert having to determine how confident he was in his decision. These responses were collected and the linguistic evaluations were first transformed into fuzzy numbers, so Z-numbers were used to include uncertainty in decision making. Finally, the weight of the criteria was determined using the LMAW method.

5. Results

When determining the weights for the critical criterion for sustainable TEL, a questionnaire survey was conducted where the experts were initially required to provide ratings for the main criteria and subsequently for the sub-criteria. The process of weight formation for all other criteria was explained using the example of the main criteria. The experts were tasked with assessing the importance of specific criteria and determining their level of confidence in that decision. The linguistic values presented in Table 1 were utilized to establish these assessments. As evident from the responses, the experts exhibited a high degree of confidence in their criterion evaluations, resulting in predominantly high and very high values for the B fuzzy number (

Table 2. Main criteria evaluation.

	C1		C2		C3		C4
	A	B	A	B	A	B	A	B
Expert 1	AH	H	AH	H	EH	H	AH	H
Expert 2	E	H	AH	VH	EH	H	AH	VH
Expert 3	EH	H	H	VH	EH	H	EH	H
Expert 4	ML	H	EH	H	AH	H	H	H
Expert 5	ML	H	MH	H	ML	H	H	H

).

In order to obtain the weights for the main criteria, it was necessary to first transform the Z-numbers into fuzzy numbers. The linguistic values were initially converted into fuzzy numbers, with fuzzy numbers A and B determined using the membership function (Table 1). Subsequently, the defuzzification of fuzzy number B was performed, followed by taking the square root of this value (

Table 3. Transformation of fuzzy number B.

	C1		C2		C3		C4
	A	B	A	B	A	B	A	B
Expert 1	4.5, 5, 5	0.55, 0.75, 0.95	4.5, 5, 5	0.55, 0.75, 0.95	4, 4.5, 5	0.55, 0.75, 0.95	4.5, 5, 5	0.55, 0.75, 0.95
Expert 2	2.5, 3, 3.5	0.55, 0.75, 0.95	4.5, 5, 5	0.8, 1, 1	4, 4.5, 5	0.55, 0.75, 0.95	4.5, 5, 5	0.8, 1, 1
Expert 3	4, 4.5, 5	0.55, 0.75, 0.95	3.5, 4, 4.5	0.8, 1, 1	4, 4.5, 5	0.55, 0.75, 0.95	4, 4.5, 5	0.55, 0.75, 0.95
Expert 4	2, 2.5, 3	0.55, 0.75, 0.95	4, 4.5, 5	0.55, 0.75, 0.95	4.5, 5, 5	0.55, 0.75, 0.95	3, 3.5, 4	0.55, 0.75, 0.95
Expert 5	2, 2.5, 3	0.55, 0.75, 0.95	3, 3.5, 4	0.55, 0.75, 0.95	2, 2.5, 3	0.55, 0.75, 0.95	3, 3.5, 4	0.55, 0.75, 0.95
	C1		C2		C3		C4
	A	${Def}_{\tilde{B}}$	A	${Def}_{\tilde{B}}$	A	${Def}_{\tilde{B}}$	A	${Def}_{\tilde{B}}$
Expert 1	4.5, 5, 5	0.75	4.5, 5, 5	0.75	4, 4.5, 5	0.75	4.5, 5, 5	0.75
Expert 2	2.5, 3, 3.5	0.75	4.5, 5, 5	0.93	4, 4.5, 5	0.75	4.5, 5, 5	0.93
Expert 3	4, 4.5, 5	0.75	3.5, 4, 4.5	0.93	4, 4.5, 5	0.75	4, 4.5, 5	0.75
Expert 4	2, 2.5, 3	0.75	4, 4.5, 5	0.75	4.5, 5, 5	0.75	3, 3.5, 4	0.75
Expert 5	2, 2.5, 3	0.75	3, 3.5, 4	0.75	2, 2.5, 3	0.75	3, 3.5, 4	0.75
	C1		C2		C3		C4
	A	$\sqrt{{Def}_{\tilde{B}}}$	A	$\sqrt{{Def}_{\tilde{B}}}$	A	$\sqrt{{Def}_{\tilde{B}}}$	A	$\sqrt{{Def}_{\tilde{B}}}$
Expert 1	4.5, 5, 5	0.866	4.5, 5, 5	0.866	4, 4.5, 5	0.866	4.5, 5, 5	0.866
Expert 2	2.5, 3, 3.5	0.866	4.5, 5, 5	0.966	4, 4.5, 5	0.866	4.5, 5, 5	0.966
Expert 3	4, 4.5, 5	0.866	3.5, 4, 4.5	0.966	4, 4.5, 5	0.866	4, 4.5, 5	0.866
Expert 4	2, 2.5, 3	0.866	4, 4.5, 5	0.866	4.5, 5, 5	0.866	3, 3.5, 4	0.866
Expert 5	2, 2.5, 3	0.866	3, 3.5, 4	0.866	2, 2.5, 3	0.866	3, 3.5, 4	0.866
	C1		C2		C3		C4
	${\tilde{Z}}^{\prime }$		${\tilde{Z}}^{\prime }$		${\tilde{Z}}^{\prime }$		${\tilde{Z}}^{\prime }$
Expert 1	3.90, 4.33, 4.33		3.90, 4.33, 4.33		3.46, 3.90, 4.33		3.90, 4.33, 4.33
Expert 2	2.17, 2.60, 3.03		4.35, 4.83, 4.83		3.46, 3.90, 4.33		4.35, 4.83, 4.83
Expert 3	3.46, 3.90, 4.33		3.38, 3.86, 4.35		3.46, 3.90, 4.33		3.46, 3.90, 4.33
Expert 4	1.73, 2.17, 2.60		3.46, 3.90, 4.33		3.90, 4.33, 4.33		3.03, 3.46, 3.90
Expert 5	1.73, 2.17, 2.60		2.60, 3.03, 3.46		1.73, 2.17, 2.60		3.03, 3.46, 3.90

). Finally, the resulting value was multiplied by fuzzy number A. For example, the calculation for criterion C1 for the first expert is as follows:

$${\tilde{Z}}^{\prime }=\sqrt{\alpha }·\tilde{A}=\left(0.866\times 4.5. 0.866\times 5. 0.866\times 5\right)=(3.90. 4.33. 4.33)$$

This procedure was applied to the remaining criteria and for all experts. It transforms the Z-number into an ordinary fuzzy number. Table 3 details the step-by-step transformation into final Z-numbers.

Once the Z-number was transformed, the steps of the fuzzy LMAW method were applied. The first step involves dividing the fuzzy numbers by the fuzzy anti-ideal point. Since the minimum value of the transformed Z-number was 1.73, a value of 1.7 was chosen as the fuzzy anti-ideal point ${\tilde{\gamma }}_{AIP}$ = (1.7, 1.7, 1.7). Afterward, the natural logarithm was applied, calculating the natural logarithm for all obtained values. This value was then divided by the natural logarithm obtained from the product of the individual fuzzy numbers for all criteria.

For the first criterion and the first expert, the calculation is as follows:

$${\tilde{\omega }}_1^e=\left(\frac{\mathrm{l}\mathrm{n}\left(2.29\right)}{\mathrm{l}\mathrm{n}(2.29\times 2.29\times 2.04\times 2.29)}, \frac{\mathrm{l}\mathrm{n}\left(2.55\right)}{\mathrm{l}\mathrm{n}(2.55\times 2.55\times 2.29\times 2.55)},\frac{\mathrm{l}\mathrm{n}\left(2.55\right)}{\mathrm{l}\mathrm{n}(2.55\times 2.55\times 2.55\times 2.55)}\right)=(0.22, 0.26, 0.29)$$

The same calculation was performed for all criteria and all experts (

Table 4. Criteria vector weights for each criterion and each expert.

	C1			C2			C3			C4
Expert 1	0.22	0.26	0.29	0.22	0.26	0.29	0.19	0.23	0.29	0.22	0.26	0.29
Expert 2	0.07	0.13	0.20	0.26	0.31	0.37	0.20	0.25	0.33	0.26	0.31	0.37
Expert 3	0.19	0.25	0.33	0.18	0.25	0.33	0.19	0.25	0.33	0.19	0.25	0.33
Expert 4	0.01	0.09	0.20	0.23	0.31	0.44	0.27	0.34	0.44	0.19	0.26	0.39
Expert 5	0.01	0.14	0.41	0.18	0.33	0.68	0.01	0.14	0.41	0.24	0.40	0.80

).

Subsequently, the Bonferroni aggregator was applied to harmonize the weights assigned by individual experts. By applying the Bonferroni aggregator, the fuzzy weights for the main criteria were obtained. These weights were then defuzzified to obtain the final results (

Table 5. Main criteria weights.

Fuzzy Criteria Weight												Criteria Weight
C1			C2			C3			C4			C1	C2	C3	C4
0.09	0.17	0.28	0.21	0.29	0.42	0.16	0.24	0.36	0.22	0.30	0.43	0.174	0.298	0.247	0.304

). Based on these results, criterion C4 received the highest weight, while criterion C1 received the lowest weight (Table 5).

The same steps were applied to calculate the weights for the sub-criteria. Only the linguistic values (

Table 6. Initial decision matrix with linguistic values.

	C11	C11	C12	C12	C13	C13	C14	C14
	A	B	A	B	A	B	A	B
Expert 1	EH	H	EH	H	AH	H	AH	H
Expert 2	E	M	E	M	E	M	AH	VH
Expert 3	AH	M	E	M	H	H	AH	H
Expert 4	EH	H	H	H	AH	H	MH	H
Expert 5	H	H	MH	H	EH	H	ML	H
	C21	C21	C22	C22	C23	C23	C24	C24	C25	C25
	A	B	A	B	A	B	A	B	A	B
Expert 1	AH	H	AH	H	AH	H	AH	H	AH	H
Expert 2	AH	VH	AH	VH	AH	VH	AH	VH	AH	VH
Expert 3	E	H	AH	VH	H	VH	AH	VH	H	VH
Expert 4	EH	H	H	H	AH	H	MH	H	H	H
Expert 5	MH	H	H	H	MH	H	H	H	MH	H
	C31	C31	C32	C32	C33	C33	C34	C34
	A	B	A	B	A	B	A	B
Expert 1	EH	H	EH	H	EH	M	AH	VH
Expert 2	EH	H	H	H	H	M	AH	VH
Expert 3	AH	M	E	H	AH	VH	EH	VH
Expert 4	EH	H	AH	H	H	H	MH	H
Expert 5	H	H	MH	H	H	H	ML	H
	C41	C41	C42	C42	C43	C43	C44	C44
	A	B	A	B	A	B	A	B
Expert 1	AH	VH	AH	VH	AH	VH	AH	VH
Expert 2	AH	VH	AH	VH	AH	VH	AH	VH
Expert 3	MH	H	H	M	EH	VH	AH	VH
Expert 4	H	H	EH	H	ML	H	AH	H
Expert 5	H	H	MH	H	H	H	EH	H

) and the final weights for these criteria are presented in the following paragraphs, as the procedure is identical to calculating the weights of the main criteria.

Table 6 represents the linguistic ratings provided by the experts and serves as the initial decision matrix for determining the weights of the sub-criteria. The same steps were followed. First, the Z-numbers were transformed into fuzzy numbers, and then the weights of the sub-criteria were obtained using the fuzzy LMAW method. To obtain the final weights of these criteria, the weights of the main criteria were multiplied by the corresponding weights of the sub-criteria. Thus, the weight of C1 was multiplied by the weights of the corresponding sub-criteria (C11, C12, C13, and C14). The same procedure was applied to the other criteria. The weights obtained through the Z-LMAW method are presented in

Table 7. Critical criterion for sustainable TEL weights.

Criteria	Local Value	Rank	Global Value	Rank
C1	0.174	4
C11	0.250	3	0.047	16
C12	0.191	4	0.037	17
C13	0.270	2	0.051	15
C14	0.301	1	0.056	14
C2	0.298	2
C21	0.183	5	0.070	9
C22	0.215	1	0.082	2
C23	0.208	3	0.080	4
C24	0.208	2	0.080	3
C25	0.197	4	0.076	6
C3	0.247	3
C31	0.273	1	0.070	10
C32	0.239	4	0.062	13
C33	0.250	2	0.064	11
C34	0.249	3	0.064	12
C4	0.304	1
C41	0.244	2	0.076	5
C42	0.237	3	0.075	7
C43	0.229	4	0.072	8
C44	0.297	1	0.092	1

. To ensure the comparability of the criteria weights, adjustments were made to the weights of the sub-criteria for criterion C2. This was necessary because criterion C2 had five sub-criteria, while the other criteria had four sub-criteria. Therefore, the sub-criteria for criterion C2 were placed in an unfavorable position, resulting in their values being 25% lower than the other values. As a result, the weights of these sub-criteria were increased by 25% to maintain the same proportion.

Based on the obtained weights, the following conclusions can be drawn. When considering the main criteria, criterion C4 received the highest weight, followed by criterion C2, and criterion C1 received the lowest weight. When considering the sub-criteria, the best-ranked sub-criterion of the main criterion C1 is C14, followed by C13, while the lowest weight was assigned to C12. When considering the sub-criteria of the main criterion C2, the highest weight was assigned to C22, followed by C24, while the lowest weight was assigned to C21. When considering the sub-criteria of the main criterion C3, the highest weight was assigned to C31, followed by C33, while the lowest weight was assigned to C32. When considering the sub-criteria of the main criterion C4, the highest weight was assigned to C44, followed by C41, while the lowest weight was assigned to C43.

By examining only the weights of the sub-criteria, it can be concluded that the best-ranked sub-criterion is C44, followed by the four sub-criteria of the main criterion C2, namely C22, C23, and C24 (Table 7). The lowest weight was assigned to the sub-criterion C12. Based on these results, the main criterion C4 has the best weights and also received the highest weight compared to other main criteria, while the sub-criteria of the main criterion C1 have the lowest ranks and received the lowest weights among all main criteria. This research thus demonstrates the significant role of main criterion weights in determining the final weights of the sub-criteria.

6. Discussion

The primary goal of any educational system is to facilitate the effective transfer of knowledge from teachers or professors to students. Throughout history, this process has undergone significant changes as various teaching aids and methodologies have been employed to enhance the learning experience. In recent years, the rapid advancements in information and communication technology (ICT) have had a profound impact on the way knowledge is imparted [80]. These technological innovations have paved the way for the implementation of TEL systems, which aim to optimize the teaching and learning process. In their research, Yang et al. [81] showed that TEL systems are one of the key systems for transferring knowledge from professor to student. By leveraging the potential of ICT, educational institutions can embrace TEL approaches that contribute to the improvement of socio-economic growth and overall societal development.

The special application of TEL technology is reflected in the possibility of providing faster and sustained access to higher quality education, where online training is the most effective way of providing sustainable higher education [82]. In this way, future teachers can be prepared to prepare for lifelong learning for sustainable education [30] and use TEL technology. In addition, TEL technology can also be used by professors at universities in order to transfer their knowledge to students. In this way, the evolution of higher education is carried out, which integrates TEL and sustainability in education and enables the connection of higher education institutions with society [83]. In this way, it was important to determine what the key factors are in TEL systems at the College of Tourism, according to the experts.

The College of Tourism’s adoption of TEL serves as a noteworthy example of its commitment to sustainable educational practices. The case study presented in this research provides valuable insights into the college’s efforts to incorporate TEL approaches that align with its goals and objectives. The evaluation framework, developed with the input of an expert panel comprising individuals with diverse perspectives and expertise, ensures a comprehensive assessment of the TEL approaches employed by the college. This holistic evaluation approach considers the expertise and opinions of these specialists to ensure the effectiveness, relevance, and long-term sustainability of TEL practices at the College of Tourism. By combining the insights and knowledge of experts from various fields, the college can create an educational environment that prepares students to thrive in the rapidly evolving tourism industry.

The introduction of Z-numbers in the evaluation process introduces a level of uncertainty in assessing the criteria. This is due to the fact that when making decisions, no decision can be made with certainty, especially if fuzzy numbers and linguistic values are used [15]. The experts not only evaluated the importance of specific criteria but also assesed their level of confidence in their evaluations. This approach contributes to the decision-making process by considering the uncertainty and ambiguity inherent in decision making. The experts who were more confident in their judgments had a greater impact on the decision-making process compared to those who were less certain in their evaluations. This aspect of the approach enhanced the evaluation of the TEL criteria.

By employing the Z-number-based approach, the evaluation process incorporated the notion of uncertainty in the assessment of criteria. In addition to evaluating the importance of specific criteria, the experts were also encouraged to reflect on their level of confidence in their evaluations. This approach brings an additional layer of security and reliability to the decision-making process, as it considers and incorporates the inherent uncertainties and ambiguities that accompany decision-making. The experts who had a higher level of confidence in their judgments played a more influential role in the decision-making process compared to those who exhibited lower levels of confidence. This aspect of the evaluation approach enhances the overall robustness and validity of the TEL criteria evaluation. It ensures that the criteria deemed most significant by experts were given appropriate weightage reflecting the experts’ confidence and understanding of the importance of those criteria.

By employing this expert-based decision-making approach, the results reveal that the experts considered the criteria C4—Organizations Management Readiness and C2—Cloud M-Learning Essentials as the most significant. These two criteria received the highest weights in the evaluation of the main criteria. This is because cloud technologies [84] are increasingly being applied in the education sector, facilitating the connection between teachers and students in an online environment. Additionally, organizations management readiness plays a significant role in the implementation of online learning [11]. Due to these weights, their respective sub-criteria achieved the best results among all other sub-criteria. Consequently, sub-criterion C44—Commitment toward M-learning) obtained the highest weight, followed by criteria related to the main criterion Cloud M-Learning Essentials.

The findings of this study significantly contribute to the development of a robust and scientifically rigorous evaluation framework for sustainable TEL at the College of Tourism. By utilizing Z-LMAW, decision-makers and stakeholders can make informed decisions based on a comprehensive evaluation that accounts for inherent uncertainty and provides a quantitative measure of importance. This evaluation framework enhances the reliability and validity of the decision-making process, ensuring the selection of sustainable TEL strategies that align with the college’s goals and requirements.

However, it is important to acknowledge certain limitations in this research. Firstly, the evaluation framework relies heavily on expert opinions, which may introduce subjective biases. To mitigate this, future studies could consider incorporating diverse perspectives from a larger pool of experts or exploring alternative approaches such as crowdsourcing. Secondly, the framework primarily focuses on the identified main criteria and sub-criteria, but other factors may also influence the sustainability of TEL approaches. Future research could investigate additional dimensions, such as pedagogical effectiveness, user satisfaction, and cost-effectiveness, to provide a more comprehensive evaluation. Furthermore, future directions for research in this field could involve validating the evaluation framework through empirical studies and case studies in real-world TEL implementations. This would provide practical insights into the effectiveness and applicability of the framework in different educational contexts. Additionally, continuous refinement and improvement of the framework are necessary to adapt to the evolving landscape of technology and education.

7. Conclusions

Advancements in information and communication technology (ICT) and mobile technologies have facilitated the implementation of TEL in educational systems. This research presents an expert-opinion-based evaluation framework for assessing the sustainability of TEL approaches. To achieve this, an expert panel comprising individuals with diverse perspectives and expertise was assembled. The panel included a technology expert, an education specialist, a business manager, a tourism industry professional, and an educational researcher. These experts were assigned the task of evaluating the TEL approach criteria using linguistic values. The framework utilizes Z-LMAW to provide a comprehensive evaluation of TEL systems. Four main criteria, namely cloud services compliance, cloud M-Learning essentials, system and technological advancement, and organizations management readiness, along with 17 sub-criteria, form the basis of the evaluation framework. Among the main criteria, only the “Cloud M-Learning Essentials” criterion consisted of five sub-criteria, while the remaining main criteria comprised four sub-criteria each. During the evaluation of the criteria, the experts were also required to assess the level of confidence in their judgments.

By drawing on the expertise of subject matter specialists, the evaluation framework incorporates Z-numbers to account for the inherent uncertainty and imprecision in expert judgments. This approach enhances the scientific rigor and robustness of the evaluation process. The fuzzy LMAW method was applied to calculate the overall scores for each criterion and sub-criterion, enabling a quantitative measure of their importance in the evaluation process. The aim of this paper was to determine the key factors for the success of TEL systems. The research was conducted to determine the importance of TEL system factors, and in-depth interviews with experts were not conducted. This represents a limit of this research, so in future research it would be necessary to conduct in-depth interviews with experts in order to obtain some key data that would additionally affect the efficiency of TEL systems.

The findings derived from this approach revealed that the main criteria C4—Organizations management readiness and C2—Cloud M-learning essentials were considered the most significant by the experts. Consequently, these weights influenced the sub-criteria within these main criteria, attributing them to the highest importance in the implementation of the TEL approach in the case of the College of Tourism. By utilizing the Z-LMAW method, this research contributes to enhancing the decision-making process by accounting for uncertainty levels, a critical factor in determining the most significant criterion for the TEL approach.

The findings of this study contribute to the development of a scientifically rigorous evaluation framework for sustainable TEL. Thanks to the evaluations of the experts, important information was obtained about the factors that influence the application of TEL at the College of Tourism. Based on these results, it is possible to train employees to use other options within TEL systems and to achieve better effects from using TEL systems in teaching environments. By incorporating expert opinions and employing the Z-LMAW approach, decision-makers and stakeholders can objectively assess the sustainability of TEL systems. This framework holds promise for informing the design and implementation of strategies to enhance the quality, compliance, and technological advancements in TEL environments.