Acquiring, Analyzing and Interpreting Knowledge Data for Sustainable Engineering Education: An Experimental Study Using YouTube

Abstract

With the immersion of a plethora of technological tools in the early post-COVID-19 era in university education, instructors around the world have been at the forefront of implementing hybrid learning spaces for knowledge delivery. The purpose of this experimental study is not only to divert the primary use of a YouTube channel into a tool to support asynchronous teaching; it also aims to provide feedback to instructors and suggest steps and actions to implement in their teaching modules to ensure students’ access to new knowledge while promoting their engagement and satisfaction, regardless of the learning environment, i.e., face-to-face, distance and hybrid. Learners’ viewing habits were analyzed in depth from the channel’s 37 instructional videos, all of which were related to the completion of a computer-aided mechanical design course. By analyzing and interpreting data directly from YouTube channel reports, six variables were identified and tested to quantify the lack of statistically significant changes in learners’ viewing habits. Two time periods were specifically studied: 2020–2021, when instruction was delivered exclusively via distance education, and 2021–2022, in a hybrid learning mode. The results of both parametric and non-parametric statistical tests showed that “Number of views” and “Number of unique viewers” are the two variables that behave the same regardless of the two time periods studied, demonstrating the relevance of the proposed concept for asynchronous instructional support regardless of the learning environment. Finally, a forthcoming instructor’s manual for learning CAD has been developed, integrating the proposed methodology into a sustainable academic educational process.

1. Introduction

Today, many higher education institutions are integrating learning activity data analytics into their operations [1]. Universities are recognizing the benefits of information solutions that not only better students at all stages of their education, even the most challenging, but also implement ever more effective educational resources to enhance the learning experience for students and their instructors [2]. Teaching module organizers and instructors focus on analytics results and the use of algorithms to improve their content and flexibility to identify students at risk of academic failure as early as possible and then provide them with more targeted learning solutions [3].

Long before the global health crisis due to the SARS-CoV-2 virus, nearly a decade of student education data could be used to understand key aspects of learner characteristics that would differentiate those who are capable of graduating from those at risk of dropping out. With all educational procedures going online, the acquisition of electronic data from a variety of online sources has led to an increase in all analytical procedures, and their management is a concern for the academic community [4].

It has been more than two years since the learning experience in universities was totally disrupted by the consequences of the COVID-19 pandemic, resulting in a significant decline in student retention and academic performance. Sustainable measures were then gradually put in place to ensure students’ virtual presence and support them in their asynchronous tasks. Today, after almost a year and a half of exclusive and mandatory distance learning imposed by the pandemic, the learning processes of higher education have been tested in terms of applicability, feasibility and long-term sustainability. Although the health situation is not yet fully stabilized, the educational models deployed during the most critical period must be evaluated [5,6,7,8].

The first semester of the 2021–2022 academic year is undoubtedly a defining moment in the educational community. Although the post-COVID-19 period has not yet arrived, it is now appropriate to speak of the beginning of the “meta-COVID-19 period,” certainly defining a period of disruption, but offering many exciting opportunities in the academic world. Specifically, in the era of digital transformation of the educational procedure [9], the authors of [10] applied the Greek term “meta”, meaning beyond, to describe a promising future for academia, followed by a global health phenomenon. With the increasing implementation of digital learning systems during the health crisis, it is important to ensure that the system design can motivate and support active student engagement to achieve the required educational goals. Therefore, socially oriented technology tools can be incorporated into the design of online learning systems to increase student engagement and improve student performance during the learning process [11]. By adjusting learning tactics and introducing technological features applied during the epidemic into meta-COVID-19 instruction, academic institutions will be able to progress and thrive in sustainable educational models.

The work presented here consists of an experimental study that focuses on instructor monitoring of learner behaviors and engagement throughout the teaching module to assess the effectiveness and sustainability of the applied learning tactic. The methodology applied is based on video analysis and, more precisely, on the processing and interpretation of data coming from the consultation by students of online digital content directly from the reports of a YouTube channel made available as part of a computer-aided mechanical design (CAD) module [12]. The analysis of the experimental data collected, analyzed and interpreted should make it possible to evaluate the benefit of this YouTube channel dedicated to asynchronous pedagogical support in online or hybrid teaching environments. To convince the university community of the sustainable integration of the YouTube channel into the educational process, learners’ viewing habits were analyzed in depth from the channel’s 37 instructional videos, all in conjunction with the execution of the CAD course. To develop the results of this work, two distinct learning periods are considered: the first refers to exclusively distance learning spaces (i.e., during the most critical period of the health crisis), and the second reflects both face-to-face and mixed learning environments (i.e., mixing face-to-face and distance).

The study proposed here covers a wide range of skills: from the creation of a complete asynchronous educational and social environment based on a YouTube channel where digital observations can be acquired, to the extraction and exploitation of visualization data. The novelty of this study lies in the fact that the use of social media channels in online and hybrid learning spaces has not yet been analyzed in depth, for its sustainable integration in the academic educational procedure.

The development of this work will be articulated as follows: Section 2 will first present a review of the literature related to the objectives of this work. The methodological and organizational aspects will be presented in Section 3, starting with the creation of the educational YouTube channel, the sources of data exploration, and proceeding to the statistical analysis of the acquired data where the results will be presented. The main results obtained will be presented in Section 4, and a discussion, based on these results, will be conducted in Section 5. Finally, the conclusions and research perspectives will be analyzed in Section 6.

2. Related Work

Long before the emergence of distance learning imposed by the COVID-19 pandemic, institutions around the world incorporated learning management systems (LMS) into their instructional schemes, whether in online or blended learning environments [13]. During the health crisis, several learning tools were used, individually or in combination. At the University of West Attica (Greece), as well as at the University of Tours (France), the Microsoft (MS) Teams learning platform was, for example, widely used for synchronous transmissions, in combination with the E-Class (or Moodle) LMS for asynchronous support. In [14], researchers proved that using a single learning platform (MS Teams), supported by a social media channel such as YouTube for asynchronous support, limited the dropout rate of learners, compared to MS Teams supported by the LMS Moodle.

Just as considering customer preferences is critical to the development of a business, in education, analyzing students’ preferences and taking steps to provide them with learning materials in innovative learning spaces could be the key to improving their academic performance [15,16].

Educational data mining (EDM) seems to have a major effect in the field of education [17]. EDM and data analytics promise a better understanding of student learning, as well as new insights into the hidden aspects that influence learner performance [18]. Learning analytics (LA) is an emerging area of learning management systems that tracks and records student activities in online and virtual learning environments [19]. The role of LA is to use the data generated by students as they interact with new technology features to improve the teaching–learning process (TLP) and enable instructors to make better decisions in terms of structuring teaching modules [20]. It is generally accepted that the more data you collect, the more information you get. Therefore, the more information one acquires, the more accurate predictions, forecasts and estimates can be obtained. Data quality is very important, as it is affected by the number of variables and the amount of data acquired, which can lead to information sparsity, especially in cases where the quality of the data appears to be poor [21]. In addition, process analysis allows for the observation of unusual activities and behaviors, which can lead to the detection of “outliers”, alarm objects, and calls for intervention [22]. Given the power of the method, LA can therefore be a major feedback tool for educators and instructional designers to improve the learning experience [23].

Researchers in [24] developed a technology acceptance model (TAM) to examine which factors of social networking sites such as YouTube and TikTok can support and facilitate online knowledge acquisition. In this study, data collection was conducted using an online questionnaire on four external factors: content richness, innovativeness, satisfaction, and enjoyment. The results showed that both social networking sites contribute to knowledge sharing and acquisition. Although reports from YouTube channels were not retrieved and considered in this research, the authors concluded that to increase acceptance, the focus should be on uploaded video content.

Although the authors of [25] conducted a systematic review of the literature describing the sources and use of educational data, data analyses from the YouTube channel were unfortunately not considered. Additionally, in [26], the researchers examined the influence of instructor-generated video content on student engagement and participation in a course using the number of posts per week and the number of characters per post as parameters. They conducted an independent-sample t-test to compare student evaluation of the course by dividing the population into two groups: those who had been exposed to instructor-generated videos and those who had not. The test showed statistical significance between the two groups.

In [27], the authors discussed the use of YouTube analytics both to assess student attendance during lectures and to measure the impact of lectures on the student learning experience. Going further, the authors in [28] planned campaigns by processing analytics data from the tools offered by social media channels.

The remarkable popularity of social media applications can be attributed to the encryption technology used, which ensures user privacy and limits access to personal information [28]. Social media is thought to offer benefits such as enhancing human interaction through the use of electronic media, increasing creativity, creating a sense of affiliation and acceptance, encouraging engagement and cooperative learning, reducing restrictions in terms of space and social or economic position within a community, increasing interaction and communication among members, and improving users’ technological expertise [29]. With technological tools now widely available to people under the age of 20, it is possible to access social media sites with one hand, thanks to mobile technology and the use of inexpensive electronic devices such as tablets and smartphones [30,31].

Sharing videos or their URL links has become easier for instructors with the adoption of virtual learning environments (VLE). Content in teaching modules, discussion forums, and targeted sequences in online courses can be shared via YouTube video links embedded in the assessment features of learning platforms [32]. The analytics of YouTube channels can provide valuable information about learners’ viewing habits, as well as measures of how students engage in the learning process with videos [33]. With YouTube analytics, instructors can thus track video viewing behaviors on supportive tasks to better understand their usefulness [34]. Previous studies in this area have shown that students do not follow a video in its entirety. They play, stop, rewind, and replay the educational content in order to review segments of the video recordings. This specific learner behavior is intended to recall the part of the newly acquired knowledge that was not clearly defined [33,35]. In [35], the authors investigated the use of pausing and searching in videos of course recordings provided by the channel interface, and how these two features relate to students’ learning tactics and performance in a specific curriculum. Before the COVID-19 pandemic imposed restrictions on academia, researchers studied how learners in traditional learning environments and flipped classrooms interacted with the videos, as well as the nature of those interactions through analysis of processing data [32]. Instructors recorded short or long videos of portions of their lectures and uploaded them to VLE [32].

In transforming the traditional learning space into a virtual one, one of the most significant problems has been the loss of contact with the engineering environment itself, which is a key aspect of engineering education [36,37]. In addition, the authors of [10] associated educational data mining, processing, and analysis with the term “sustainability”, which is present and promoted in most aspects of everyday life, from business operations to manufacturing and the environment [38]. Data mining, the processing of data and eventual interpretation of the results, is a fundamental process that allows researchers to establish the relationship between raw digital data and the assessment of real world conditions [39]. With digital data now available by tracking activities, analysts can get lost in unnecessary information. To facilitate the process, researchers should be able to set their boundaries, creating controlled environments for data production, i.e., targeted to the goals of their research area.

In this study, the data collected and analyzed provides insight into students’ video content viewing behaviors from a dedicated YouTube channel. These behaviors were analyzed over two distinct time periods: the first in 2020–2021, i.e., during the most critical period of restrictions due to the COVID-19 pandemic, and the second in 2021–2022, during the early hours of the meta-COVID-19 period. The objective is to study the similarities between the delivery of engineering training modules in exclusively online mode and in mixed mode. This objective had already been set well before the implementation of the new pedagogical environments, whether online or hybrid; the social communication channel was then used as an asynchronous pedagogical support allowing the collection of information directly related to students’ behaviors, while avoiding the “noise” due to irrelevant details.

Given the results already available and the gaps identified by the literature review, this work is guided by the following five research questions:

• Can a YouTube channel provide adequate and quality asynchronous support on student tasks in online learning environments?
• Can we provide a guide for future instructors and organizers of CAD modules, willing to implement pedagogical methods supported by technology and social media sites such as YouTube, which would benefit the learning process in online and hybrid spaces?
• Can learners’ viewing habits be revealed in online and hybrid learning spaces through information provided by YouTube?
• Do learners’ behaviors and visualization patterns follow the flow of the module?
• Can educational data mining (EDM) from social media sites like YouTube provide a solid foundation for addressing student needs?

The answers to the above research questions should help demonstrate the relevance of educational data from social media channels such as YouTube to the academic community, and help institutions implement their strategies for sustainable digital transformation in higher education.

The research objective of this study is not limited to evaluating a specific YouTube channel as a tool to support asynchronous teaching. It also aims to provide feedback to instructors and suggest steps and actions to implement in their teaching modules to ensure students’ access to new knowledge while promoting their engagement and satisfaction, regardless of the learning environment, i.e., face-to-face, fully remote, and hybrid.

3. Methodological and Organizational Aspects

3.1. Foreword

The methodology described in this section was deployed in a 12-week “computer-aided mechanical design (CAD I)” module, a teaching module within the Department of Mechanical Engineering of the University of West Attica (Greece). Prior to the health crisis, this module was divided between traditional mechanical design in a room equipped with drawing boards and computer-aided design with Autodesk Inventor software in a computer lab.

During the most critical hours of the COVID-19 pandemic, videos (directly downloadable into the MS Teams environment available to students) were integrated into flipped classrooms to provide asynchronous instructional support to complement the online courses delivered synchronously via the MS Teams platform. All of this work was done, including the integration of the following three tasks:

• Provide asynchronous support to students and incorporate it into assessments of individual tasks.
• Guarantee students the robustness of the learning process, including avoiding any disruption caused by multiple platforms or LMS.
• Create a virtual lab to connect distance learning spaces with real engineering environments related to students’ future work [36,37,40].

During the period when pandemic restrictions were relaxed, the 12-week CAD module was divided into two main stages. For the first 4 weeks, all classes were conducted face-to-face in the classroom equipped with drawing boards. The objective of this phase was to teach students to represent views of a three-dimensional object by freehand drawings (sketches). In the second stage, conducted in the computer lab, students enrolled in activities to create three-dimensional mechanical objects in different views (top, side, and cross-section) using CAD tools. To complete these activities, learners had the option of participating face-to-face or online (the class was streamed live by the instructor via MS Teams). In both stages, asynchronous support videos were attached to the students’ assigned task. The YouTube channel “MCAD I UNIWA” was created to provide learners with all the video support needed to complete their learning independently. This YouTube channel, and all of its video content (see Appendix A), is managed by a CAD instructor with over twenty years of CAD experience. The administrator of the YouTube channel was also responsible for coordinating all activities of the various instructors in the teaching module. The MCAD I UNIWA’s YouTube channel policy is public. In order to target specific tasks and associate them with their asynchronous support video links, the tasks were signed as “assignments” on the MS Teams communication platform. Each task was announced in the students’ MS Teams dashboard, where instructions were provided in text form [36]. The YouTube channel links for each task were uploaded as reference material targeting the specific task to avoid confusion when searching the 37 videos to find the one relevant to the task, as shown in Figure 1. The instructor was motivated to attach the URLs of the videos to each task after considering that most viewers of today’s social media channels do not easily subscribe to the channels. In this way, we were able to reach both non-subscribing and subscribing students, with the latter being immediately notified of new videos.

What we seek to highlight in this manuscript is the presence of a significant relationship between learners’ listening habits and their behaviors, particularly in the acquisition of knowledge and skills necessary for graduation. To do so, we draw on feedback from the CAD I module, looking in depth at data collected from the “MCAD I UNIWA” YouTube channel. To achieve this objective, we implemented the methodology described in Figure 1. The first step was the creation of a YouTube channel in which most of the instructional video content was created from screen recordings and audio recordings intended exclusively for student use. We did not use the raw recording of the full laboratory lecture because we wanted to target specific tasks, i.e., fundamental to the mechanical engineering profession, to help students perform them outside of class [41].

The uploaded videos were divided into four categories, based on the learning objectives to be achieved at each stage of the teaching process. Video categories 1 and 3 were devoid of audio, primarily to invite the instructor to explain the content presented at their own pace and in their own style. Specifically, the sketch videos showed the instructor’s sketchbook, complete with pencil, as he or she drew freehand views of the object. The videos showing the model of the object being studied in three dimensions were intended to help students design the geometric shapes in all views around the object. The third and fourth categories were generated by a combination of screen recordings and audio recordings from the software modeling environment.

The YouTube channel administrator examined viewing patterns since the first video was uploaded to progressively interpret learners’ needs and determine if these screen and audio recordings met the demand for asynchronous support. During learning phases conducted exclusively online, visualization patterns could be identified by the end of the first semester. When the health regulations were relaxed, i.e., when university educational spaces were allowed to transform into hybrid learning environments (i.e., combining online and face-to-face learning), a challenge arose: to analyze the visualization patterns of learners in hybrid educational spaces and correlate the results with those of spaces conducted exclusively at a distance. Once all the data was collected, the method then consisted of filtering the data from the social media channel reports corresponding to the two distinct time periods: the first corresponding to the first semester of the 2020–2021 academic year in the exclusively distance learning environments and the second, the first semester of the 2021–2022 year in the face-to-face, virtual, and mixed learning environments. Thus, two years were necessary to collect sufficient data and to ensure the results that will be discussed in the rest of the manuscript.

The final step in the methodology is to perform statistical tests on the defined variables. First, normality tests were performed to assess the normality of the distribution. In the case where the distribution is Gaussian, parametric tests were chosen. In the opposite case, non-parametric tests were performed [42]. For determining if the variances analyzed for the two separate academic years’ time periods are equal or unequal, a series of F-tests were performed. These tests can eventually be completed by t-tests depending on the equality or inequality of the variances generated by the F-tests.

3.2. Participant Demographics

Although not such an easy task, analysis of YouTube channel reports provides detailed information about the demographics of the participants and gives a “typical user profile” while revealing repetitive viewing behaviors. In our case, the term “user” refers to engineering students attending the computer-aided mechanical design module. Since one of the research questions focuses on developing a manual for future instructors, it is essential that module coordinators observe student activities outside the classroom and ultimately develop a profile of the typical student, which each instructor must consider, before taking steps to improve module delivery. For this reason, information about the status of the YouTube channel subscription, the type of viewing device used, as well as the preferred operating system, is collected, in addition to the standard demographic data. Note that in the YouTube studio, this type of data is available in separate tabs for each of the variables, after filtering for the two observation periods (i.e., 2020–2021 and 2021–2022). The goal of this strategy is to provide a clear picture with enough numerical data to allow for the most accurate comparisons and conclusions possible.

Accumulating demographic characteristics directly from the source, as opposed to self-reported responses in questionnaires, can increase the validity of the acquired data. It should be noted that many studies, including [43], have relied exclusively on demographic attributes to assess and even predict students’ academic performance with high accuracy.

Table 1. Demographic information of mechanical engineering students who participated in the study.

Demographics	2020–2021 (n = 212)	Viewing Time (Hours)	2021–2022 (n = 230)	Viewing Time (h)
	Gender of the participant and age distribution
Female	10.0%		10.0%
Male	90.0%		90.0%
Age of the participant
18–24	60.9%		97.6%
25–34	37.7%		2.4%
35–44	0.9%		0.0%
45–54	0.5%		0.0%
Geography
National	97.8%	261.6	100%	174.6
International	2.2%	0.1	0%	0.0
	Subscription status
Not subscribed	75.9%	193.6	82.1%	151.4
Subscriber	24.1%	70.9	17.9%	30.2
	Type of viewing device used
Computer	75.0%	210.39	70.7%	137.4
Mobile phone	23.6%	52.91	27.7%	38.0
Tablet	0.9%	0.81	1.3%	5.6
TV	0.5%	0.32	0.3%	0.2
	Operating system
Windows	78.1%	209.4	67.2%	133.4
Android	17.7%	46.0	22.2%	31.1
iOS	3.7%	7.8	6.4%	12.5
Macintosh	0.4%	1.0	2.6%	4.4
Smart TV	0.1%	0.02	0.0%	0.0
PlayStation	0.0%	0.0	0.4%	0.2
Linux	0.0%	0.0	0.4%	0.1
WebOS	0.0%	0.0	0.4%	0.02
Xbox	0.0%	0.0	0.4%	0.01

provides a summary of the key demographic characteristics of the mechanical engineering students who participated in the study during the two time periods noted above. This summary was compiled from data directly extracted from YouTube channel reports, content available in YouTube Studio’s advanced analysis mode, and by specifying a custom time period. Each metric (gender, age, geography, etc.) was exported from the YouTube Studio view to Google Sheets and converted to an MS Excel file [44].

The number of students who took the online module was 212 in the winter semester of the 2020–2021 year. In 2021–2022, Table 1 shows a slightly higher number of students (i.e., 230) who took the module in a blended learning mode. Regardless of the two time periods considered, 90% of the learners were male. This percentage reflects the true majority of male students enrolled in the Department of Mechanical Engineering at the University of West Attica, as verified by the student registry. In the hybrid learning environments, 97.6% of the students were between the ages of 18 and 24, and the total number of participants was located in Greece. Considering that, in the exclusive online learning environments, 60.9% of the students were between the ages of 18 and 24 and 37.7% were between the ages of 25 and 34, it can be inferred that the online learning spaces offered a unique opportunity for older students to attend their courses without their physical presence. In addition, the exclusive online learning spaces allowed a small number of international resident students to take the module from another country.

The registration status of viewers is already an interesting factor: Only 24.1% of viewers are registered on the educational channel, indicating an initial trend that most students watch the videos repeatedly, without finding a reason to subscribe to the YouTube channel. This suggests that students have a similar attitude toward the educational channel, as probably with other social media sites.

Although personal computers were widely used in both periods studied, they are losing ground to mobile devices each year. Tablet use increased in the second period. Finally, television as a viewing device decreased in 2021–2022. Finally, the most used operating system was Microsoft Windows, but there is a 10.9% decline between 2020–2021 and 2021–2022, which is explained by the increasing use of tablets and the doubling of iOS android devices.

3.3. Reports on Students’ Views, Comparative Analysis and Discussion

A total of thirty-seven videos were uploaded to the YouTube channel. Since there was no preparation time available before the universities closed, the timing of the posting of each video was scheduled in parallel with the running of the laboratory module. The name of each video corresponds to the title of the assigned task. All videos can be distinguished by content and learning objective into the following four categories, as shown in

Table 2. Analysis of data from the “MCAD I UNIWA” YouTube channel.

	Number of Videos	Number of Views	Average Viewing Time (s)	Average Percentage of Viewing
Video category no. 1	11	3758	727	37.4
Video category no. 2	5	2107	444	28.5
Video category no. 3	9	4777	383	67.1
Video category no. 4	12	5576	1644	29.9
Total	37	16,218

:

• Sketch and make freehand drawings in two dimensions of views of given objects.
• Assist the CAD software with the basic commands needed to accomplish the assigned tasks.
• Preview of the object to be studied that was modeled in the three-dimensional CAD modeler. This specific type of video was created not only to help students perceive shapes and geometric entities, generate multiple representations of them (views), but also to develop their optimized design principle [45].
• Support on tasks by screen and sound recordings of the drawing procedure in the CAD environment.

In order to understand the actual size of the learners, the unique users’ metric was filtered from the reports provided by the YouTube channel. This specific metric provides a clearer picture of the estimated number of views during the two different time periods (i.e., during the pandemic period and during the start-up period of meta-COVID-19). The metrics for specific aspects related to viewing by video categories are presented in Figure 2 and Figure 3. Specifically, the fourth category, relating to the methodology of carrying out computer-aided design tasks, is the most viewed and, as expected, has the longest viewing time. The videos with the highest average viewing percentage also belong to this category. The third category of videos, showing an overview of filmed objects, comes next in the students’ preferences. It should be noted that this type of video has the shortest duration, limited to ten to fifty seconds. Furthermore, three of the nine videos of this type were generated in the second period, i.e., in hybrid learning environments, after taking into account the students’ requests. The second category of videos contains the smallest number of videos due to the fact that most of the software support tools were integrated in the fourth category. This type of video has a considerable number of views in relation to the number of views. This suggests that these videos are aimed at a specific audience, who will persist in watching the content to better understand the use of the software. Finally, the first category, drawing without sound, has a considerable number of views, duration of viewing and average percentage of viewing that mainly reflect the first period, when the educational process was carried out exclusively online.

4. Main Results

4.1. The Number of Views and Unique Viewers: Two Major Variables in the Foreground

Table 3. Variation of the number of views per week for two time periods per module step.

	Number of Weeks
	Sketching (First Stage)				CAD Drawings (Second Stage)							Final Exam (Third Stage)
Year	1	2	3	4	5	6	7	8	9	10	11	12	13
2020–2021	54	137	137	84	154	184	224	186	116	72	110	190	202
2021–2022	27	72	109	134	120	154	177	161	143	96	47	97	143
Variation	27	65	28	−50	34	30	47	25	−27	−24	63	93	59

Separate colors were applied in the two periods according to Figure 4.

, whose data is best depicted in Figure 4, summarizes the number of views per week for the two time periods studied (in 2020–2021, when distance learning was exclusive, and in 2021–2022, when hybrid learning was the norm) based on the sequence of course modules. The interest of Table 3 and Figure 4 is also to examine the variations between the two periods studied. It is important to note that the number of views decreased in 2021–2022, which may be due to the fact that students were attending classes in face-to-face mode and asynchronous support was not as necessary as in distance learning environments.

Figure 4 shows two trend curves for the number of unique viewers as a function of CAD module length (in number of weeks) for the two time periods studied (2020–2021 and 2021–2022). Both curves show similar trend dynamics, but only from the fourth course onward. From the beginning of the teaching until the fourth laboratory lecture, the trends are slightly different and can be explained by the fact that the module was delivered exclusively online in 2020–2021 and face-to-face in the first semester of 2021–2022. Higher values of unique viewers in both time periods, can be noticed in week seven. At this point in the educational process, students enroll in the third stage of the CAD module from the second stage of the module workflow. The seventh laboratory lecture is where the most difficult part of the new knowledge has been delivered, referring to the sectional views. It should be noted that at this stage, students who have not yet assimilated the layout of the plans, have difficulties in completing their weekly tasks. Finally, new knowledge was accumulated in the first seven lessons, combining the use of software, the rules of mechanical drawing and the perception of object views, therefore asynchronous task support was necessary in this specific period. From the 8th to the 10th week, the Christmas vacations took place and a clear downward trend can be observed in both lines of the time series. Upward trends are again observed from week 10 onwards, when students resume module attendance after the Christmas break. High values of unique viewers can also be distinguished in week 13, which can be interpreted as the fact that the laboratory lectures have reached their final point and the exams are only one week away.

These first graphically illustrated results must now be rigorously demonstrated by statistical data, which will be analyzed in the following sections.

4.2. Selected Statistical Variables

The statistical study proposed in this section is conducted on the following six variables: “Impressions”, “Unique viewers“, “Viewing time”, “Impressions Click-Through Rate (CTR)”, “Number of views” and “Average viewing time”. In particular, we will use YouTube Analytics terminology to describe these variables. The term “Impressions” is used to express the number of times one of the video thumbnails from the YouTube channel appears on the participant’s screen. Therefore, the “Impressions Click-Through Rate” (CTR) indicates how many impressions were converted into views. Its calculation (expressed as a percentage) is based on the ratio between the number of clicks and the number of impressions. This metric aims to reveal how many people saw the thumbnail, found it interesting and clicked on it. The “Views” metric indicates the number of times a video has been viewed. The term “viewers” refers to the number of individuals who watch a certain piece of content. A viewer may click more than once on a specific piece of content. This is why the “Unique viewers” metric is used, which is a more accurate and relevant variable since it only counts one person, even if they click on the same video content multiple times or use multiple devices or browsers. “Viewing time” is expressed in YouTube terminology as “watch time”. This audience retention metric, expressed here as an average value, counts the hours that a specific video is watched by individual viewers. For each of the six variables defined above, two pairs of datasets were compared:

• Dataset 1: data from the academic period 2020–2021;
• Dataset 2: data from the academic period 2021–2022.

4.3. Normality Test Results

First, for each variable, it was necessary to verify the normality of the data distribution. This normality test is very important insofar as it determines the choice to carry out a test, either parametric or non-parametric. In the remainder of this subsection, we will only discuss the main results of the procedure. The basis and steps of the Shapiro–Wilk test, along with detailed results, are presented in Appendix B.

The results in Appendix B show that, regardless of the two time periods studied (i.e., 2020–2021 and 2021–2022), only the two variables “Impressions” and “Number of views” have p-values greater than the 5% threshold, which does not allow us to reject the null hypothesis and thus to consider that each sample in question is normally distributed. For these variables, an F-test was performed to evaluate whether the variances of the variables (whose distributions can be considered as normal according to the Shapiro–Wilk test) are equal or not. The final objective is to reject or not the null hypothesis of the existence of statistically significant differences between the academic years 2020–2021 and 2021–2022 [36]. These F-tests will be complemented by Student’s t-tests to compare the means of the two data distributions considered.

For the other four variables (i.e., “Unique viewers”; “Watch time”; “Impressions CTR”; and “Average viewing time”) with non-normal distributions, a Mann–Whitney–Wilcoxon test (which is a non-parametric method), was performed to assess whether there is a statistically significant difference between the two periods of 2020–2021 and 2021–2022.

4.4. Hypothesis Testing Results

For each of the two normally distributed variables mentioned above (i.e., “Impressions” and “Number of views”), the Fisher–Snedecor test or F-test was performed. As with the results of the normality tests presented earlier, here we analyze only the results obtained (see

Table 4. Results of the F-test for the two normally distributed variables “Impressions” and “Number of views”.

	2020–2021 “Impressions”	2021–2022 “Impressions”	2020–2021 “Number of Views”	2021–2022 “Number of Views”
p-value	0.01	0.01	0.12	0.12

). However, the foundations and main steps of the method are recalled in Appendix C. The results in Table 4 show that for the “Impressions” variable, there is no equality of variances between the 2020–2021 and 2021–2022 data, since the p-value is below the 5% threshold. As for the variable “Number of views”, there is equality of variances as long as the p-value is greater than the 5% threshold. For this variable in particular, this allows us to conclude that for the two periods considered, since the variances are equal, learners’ need for assistance in completing their tasks from viewing asynchronous educational content does not depend on the learning environment.

The F-tests were supplemented with t-tests, as shown in

Table 5. Results of the t-test for the two normally distributed variables “Impressions” and “Number of views”.

	“Impressions”	“Number of Views”
Assumption	Unequal variance	Equal variance
p-value	0.055	0.08

. The results of these tests reflect the comparison of the means of the data sets of the variables “Number of Views” and “Impressions”. As the p-values in Table 5 are above the 5% threshold, the null hypothesis cannot be rejected. Therefore, no statistically significant difference was observed between 2020–2021 and 2021–2022.

For each of the four remaining variables (i.e., “Unique viewers”; “Viewing time or Watch time”; “Impressions CTR”; and “Average viewing time”) whose distributions are not Gaussian for the two periods studied (i.e., 2020–2021 and 2021–2022), a non-parametric Mann–Whitney–Wilcoxon test was performed.

The results in

Table 6. Mann–Whitney–Wilcoxon test results for the four variables “Unique viewers”; “Watch time”; “Impressions CTR”; and “Average viewing time”.

	Unique Viewers (10 Weeks)	Viewing Time or Watch Time (h)	Impressions CTR (%)	Average Viewing Time (s)
p-value	0.443	0.122	0.000	0.000
Null hypothesis rejected or not rejected	Null hypothesis not rejected	Null hypothesis not rejected	Null hypothesis rejected	Null hypothesis rejected

show that there is no statistically significant difference between the two periods studied for the variables “Viewing time” and “Unique viewers” because their respective p-values are above the defined risk of 5%. However, this is not the case for the other variables (i.e., “Impressions CTR” and “Average viewing time”). Indeed, their respective p-values are below the 5% risk, confirming the statistically significant difference between the 2020–2021 and 2021–2022 data.

All the statistical results described above, whether parametric or non-parametric, confirm the preliminary results established in Section 4.1. in that the two variables “Number of views” and “Unique viewers” do not depend on the learning environment (i.e., exclusively remote or hybrid environment).

5. Discussion

Beyond the analysis of the statistical tests proposed above, Figure 4 allows us to draw some major conclusions that we will review and discuss in this section.

The total number of unique viewers in the first semester of the 2020–2021 academic year was 719, and in the same period of the 2021–2022 year was 570. By calculating the percentage of unique viewers for each of the two periods compared to the total number, a percentage difference of 11.55% was calculated. The two curves in Figure 4 representing the data series for both time periods show similar trends in student viewing habits for both time periods, leading us to conclude that learners’ need for assistance in completing their tasks is not dependent on the learning environment. This observation is made when comparing online and hybrid modes of instruction. When comparing the visual behavior of students during the first four laboratory lectures taught in the face-to-face mode in 2021–2022 with the same period in 2020–2021 in the online spaces, the curve does not show similar trends. This specific observation can be explained by the fact that when teaching the online module, all twelve lectures were delivered exclusively at a distance, whereas in the hybrid learning spaces, the first four courses were delivered exclusively in face-to-face mode. In addition, the theme of the first four lectures was “sketching”, which involves freehand drawings. For the non-computer tasks, videos were only used for object representation, and their contribution was limited to categories 1 and 3, with tasks related to the first category being repeated face-to-face.

The variable “Impressions CTR” was one of the viewing measures that showed statistically significant differences between the two time periods in the non-parametric tests conducted. By comparing the proportion of reduction between the “Number of views” (11.86%) and “Impressions CTR” (1.76%), we can conclude that even though the “Number of views” decreased in the hybrid learning spaces, the “Impressions CTR” variable showed a very low percentage of reduction, which proves the positive attitude of students clicking on the thumbnails of the videos to watch them.

Although the results of this study revealed viewing patterns for both time periods examined, there are still some limitations, based on the circumstances in which the module was delivered during each time period. The learning experience at universities was affected by the consequences of the pandemic, resulting in a significant decline in student retention and academic performance. Sustainable measures, such as those implemented in this study, had to be taken to first ensure students’ virtual presence, as well as support for asynchronous tasks.

In the exclusive online instruction, the lack of physical contact between learners and their educators allowed the former to engage in the asynchronous support channel, which allowed instructors to track their viewing activities and analyze their viewing behaviors through data analysis by retrieving high-precision information. In the hybrid learning modes, specifically in the face-to-face delivered modules, learners had the opportunity to physically communicate with their instructors and get support for their tasks.

In synthesis of the above, the YouTube channel created and used in this work as an asynchronous tutoring tool has been integrated into the educational process. It provides quality asynchronous support when needed and is part of a long-term viability and sustainability approach. These new tools for supporting individual tasks outside of the classroom can benefit pedagogical practices and ultimately the learning process in very implicit ways and primarily by being “masked” by popular social media sites like YouTube. The channel analysis provided valuable information about learners’ visualization habits that can serve as guidelines for future instructors and developers of instructional module structure. The log of visualization measures revealed viewing patterns indicating that students’ visualization behaviors follow the flow of the module, and especially regardless of their mode of attendance and teaching space. The ability to follow the content of a module through educational videos at one’s own pace and preference contributes to the development of senses of quality and equity [19]. The EDM and its statistical analysis showed that the foundation of the YouTube channel met the needs of students regardless of the learning environment on which this tool was applied.

Moving forward, the methodology applied in this study provides direct feedback for future CAD instructors and instructional module developers. Our proposed recommendations and action plan are as follows:

• Recognize module gaps throughout the course. Recognize the needs of learners, for the promotion of sustainable engineering education. Reorganize the flow of tasks according to the learning objectives of the modules and determine the critical points of difficulty according to the knowledge to be acquired.
• Do not focus on a specific task, but on the unit by creating categories of units. Each unit may include several tasks, but the focus of the knowledge introduced is not the technical instruction itself, but the concept of the learning methodology. At the stage where the new knowledge is accumulated, call it the “pick of the curve”. Determine the points at which the new knowledge needs to be performed, as well as its nature in terms of asynchronous support. Do not be afraid to combine new knowledge with entertainment by using user-friendly digital tools, such as YouTube channels.
• Create direct access to certain parts of the course to redirect learners if necessary. Make access clear to avoid confusion.
• Analyze, evaluate and reconstruct the course based on the results. This means questioning the actions taken, modifying them if they fail, and adapting them according to the nature of the learning environments.
• Never neglect the social aspects: learners must be prepared for their professional future. Technical aspects can be learned through training, but methodology is the expertise of higher education instructors.

6. Conclusions and Future Work

In 2020–2021, during the most critical hours of the COVID-19 pandemic, higher education instructors, specifically those at the University of West Attica (Greece), created a social media channel (in this study, using YouTube), as part of a mechanical engineering CAD module, to provide students with asynchronous support for their teaching tasks. To provide learners with the most direct access, links to the 37 videos were attached to each assessed task. One year later, at the beginning of the meta-COVID-19 period, the same asynchronous task support technique was applied, but this time in face-to-face and blended learning spaces.

The experimental analysis proposed in this manuscript is based on the process of processing and interpreting acquired knowledge data extracted directly from the reports of a YouTube channel; this YouTube channel having been created and used by an instructor and containing educational videos of different categories based on the learning objectives of a CAD module. The main challenge here was to investigate the potential of the data as acquired from the YouTube channel and whether the raw material downloaded in the form of previews could be processed and reveal valuable information about how students use this form of asynchronous digital educational material; YouTube having been hijacked from its primary function, i.e., entertainment.

The shift from exclusively online to hybrid learning environments first showed that the use of asynchronous task support decreased. YouTube analytics were the most appropriate tool in terms of efficiency and accuracy for expressing student retention beyond physical, online, or hybrid engineering labs, as they not only recorded the number of times a video was viewed, but also differentiated between users who watched the same video multiple times. Specifically, we defined and used variables and measures that are very common on social media sites, expressing the level of audience acceptance, to examine the contribution of video to the learning process. The following six statistical variables were selected as primary measures expressing audience retention in YouTube channels: “Impressions”, “Unique viewers“, “Viewing time or watch time”, “Impressions Click-Through Rate (CTR)”, “Number of views” and “Average viewing time”.

The comparative analysis of the YouTube reports showed similar trends in student viewing habits over the two time periods studied (i.e., at the most critical time of the global health crisis and at the beginning of the meta-COVID-19 period), with a slight decrease of nearly 12% in viewing due to a return to traditional learning environments, where most students solved their tasks in class and did not require asynchronous support. In contrast to face-to-face and 100% distance learning, the analysis showed that trends in learner viewing habits are similar during online and hybrid learning spaces. In particular, reports from the social media channel showed that educational videos followed the weekly stream trend, resulting in an increase in active viewers, which was directly related to the increase in workload and workload accumulation.

After testing the normality of each of the above six variables, a series of hypothesis tests (i.e., parametric and non-parametric based on normality tests) were performed to accept or reject the null hypothesis of this study, which concerns the absence of statistically significant changes in learners’ listening habits over the two periods studied (i.e., 2020–2021 and 2021–2022). Of the six variables analyzed, only two—“Impressions CTR” and “Average viewing time”—show display metrics with statistically significant differences between the two periods studied. For both of these statistically significant variables, it is appropriate to focus on impressions CTR, which expresses the number of times viewers click to watch a video after seeing its thumbnail. Although there was a decrease in views and impressions CTR between the two time periods, the percentage reduction was not proportional for either variable.

Although we have answered the five research questions listed in Section 2, the current study has some limitations. Although many universities have begun to use data and analytics, there is still a long way to go before these tools can fully prove their potential in terms of improving the learning experience. This is especially true today, due to the unstable health conditions caused by the COVID-19 outbreak, although overall they seem to be gradually normalizing.

Future work will involve processing analyses of a second semester CAD module Computer Aided Mechanical Design (CAD II) and performing similar statistical tests to improve the reliability of the results. Due to the increase in the number of students and institutions participating in online learning and using digital tools over the past two years, there is now a plethora of data available that may not have been available before. Institutions of higher education may want to start using this data with an eye toward serving students ever better in the years to come.