What Does the Data Say about Effective University Online Internships? The Universitat Politècnica de València Experience Using MOOC during COVID-19 Lockdown

Abstract

Universitat Politècnica de València’s students can take in-company internships during their bachelor’s degrees, and, with the COVID-19 lockdown, 224 students had their internships cancelled. EdX launched a free certificate initiative for its partners, and UPV gave the possibility of using MOOCs to cover for the credit needed to graduate. We have tried to answer the question, “Is it possible to use MOOCs to replace an in-company internship in an emergency?” using Learning Analytics; 179 students chose this possibility. More than 90% of the students got their academic credit, and their satisfaction with the initiative was 4.6/5. They scored MOOCs’ quality with 4/5 and the contribution of MOOCs to their careers with 3.6/5; 95% will take a MOOC, and 69.3% think it is worth paying for the certificate. The answers to the question evaluating if MOOCs had given them the same knowledge as a company internship are positive but much less conclusive, with an average of 2.87/5. We conclude that MOOCs achieved the pursued goal during the emergency. With more time for planning and extra resources for remote support, they can be a good solution in environments where online is the only choice, and they can even be used as a tool to reinforce some of the knowledge needed to be successful in a traditional internship.

1. Introduction

In 2020, countries started enforcing massive lockdowns to stop the spreading of the COVID-19 pandemic. Most of these confinements included school and university closures [1] that created many academic problems for students and teachers to follow and complete their educational programs. MOOCs (Massive Open Online Courses) were used as an option to address the need for quality online material for higher education institutions [2,3].

This paper shows the experience of Universitat Politècnica de València (UPV) using MOOCs to cover students’ academic internships during COVID-19 in the context of the RAP (Remote Access Program), a program of free MOOC certificates offered by edX during the first lockdown.

This paper aims to study an emergency case in which face-to-face education was not an option and an online education option had to be adopted, to see if some findings can be applied to other situations in which learning online is the only possibility. It also tries to see if the knowledge acquired with MOOCs can be helpful in any way for traditional internships.

The paper proposes two research questions:

Research Question 1: Can MOOCs cover the knowledge acquired in a company internship in a forced online environment, like the one found in a strict lockdown?

Research Question 2: Were the students satisfied with the outcomes of the solution provided by UPV?

It is critical to review the history of MOOCs and their peculiarities to understand how UPV arrived at this solution.

Using MOOCs in higher education started almost at the beginning of the movement, growing slowly but accelerating in 2018, a year which saw a significant increment in the number of MOOC-based degrees offered [4]. Many things have happened since 2008 when David Cormier and Bryan Alexander first mentioned the term MOOC to refer to the course created by Stephen Downes and George Siemens, ‘Connectivism and Connective Knowledge/2008′ (CCK8), in which 2200 students learned actively among peers [5].

The term MOOC was definitively popularized at the end of 2011 to refer to courses launched by Stanford University that had over 100,000 enrolments each [6], in a format more similar to the traditional Higher Education model that was later called x-MOOC, to differentiate these courses from the original MOOC format called c-MOOC [7]. The big numbers created a media hype about a new educational revolution, so big that 2012 was called by some media “the year of the MOOC” [8], and some articles proclaimed that MOOCs were the most important educational technology in 200 years [9].

This made many Higher Educational Institutions (HEIs) worldwide experiment with these courses [10]. At the end of 2020, there were over 16,300 courses from 950 universities worldwide with over 180 million enrollments [4], with thousands of platforms offering MOOCs [11], ranging from country platforms to small niche ones [12].

In 2021, MOOCs are used in one of the most successful online graduate degree programs in the United States of America (USA), the Online Master of Science in Computer Science (OMSCS) from the Georgia Institute of Technology, with over 5000 students graduated since it started in 2013 [13]. At first, very few masters of this type were created. Still, in 2018, there was an acceleration in the number of programs developed, and now, over 70 MOOC-based online graduate programs have been created by 27 universities [14].

Soon after the media explosion of attention regarding MOOCs, critics pointed that the dominant x-MOOC model introduced no pedagogical innovation at all [15]. Four barriers to their widespread adoption were detected: there was a need to develop revenue models that made the concept self-sustaining, MOOCs should deliver valuable signifiers of completion such as credentials, badges, or acceptance into accredited programs, course completion was very low, and, finally, it is challenging to authenticate students so that it satisfies accrediting institutions or hiring companies [16].

MOOC dropout rates and low completion figures are a significant concern that has been extensively addressed in the literature [17,18,19,20,21]. However, there is no consensus that these figures are really a sign of failure [22], and some authors state that learners’ intent should be taken into account when defining the success of MOOCs [23]. In fact, the completion rates of students pursuing a certificate (as is the case of this study) are much higher [24].

Despite these problems, in 2016, edX platform started offering Micromasters that gave access to academic credit in HEIs, and Coursera followed in 2018 with Mastertracks [25]. These two initiatives opened the possibility to use MOOCs to expand the internal HE academic offer and integrate other external offers into their own HEIs’ ones. In 2021, all major platforms will offer these academic credit-bearing micro-credentials [26]. Consequently, there is an increasing number of experiences of universities granting academic credit for external MOOCs, either integrating them in blended courses [27,28], accepting micromaster credentials as part of one of their masters [29,30], using them in interuniversity networks as the virtual exchange program [31] or letting the students create a module of their bachelor studies using MOOCs [32].

Big MOOC platforms, such as edX, Coursera, and so on, generate large amounts of data, which not only offer users enormous possibilities of interoperability and access but also allow educational institutions the possibility of making a certain level of monitoring compatible with the certification needs [33].

Therefore, MOOCs are also being used in several initiatives to include migrants and refugees in Europe [34]. Some, as Kiron NGOs, use them to create study tracks that give them access to higher education through partnerships with German universities but are finding low completion rates, probably due to a combination of several factors: recognition issues (as the procedures of recognition of certificates are complex and unclear), different expectations of students (that have very different backgrounds and situations), the problematic conditions of refugees, a low threshold approach (as the offer is open to any refugee with interest in Higher Education) and the preference for the ‘warm support’ that blended learning offers [35].

The 2020 COVID-19 lockdowns were an unexpected but global litmus test for online learning around the world. They revealed strengths and weaknesses of long-standing initiatives, offering an extraordinary opportunity to test the possibilities of MOOCs that were hitherto unthinkable but that became the only option under the circumstances. During these confinements, one of the biggest problems for covering the academic offer of universities was related to the covering of students’ academic internships in external institutions. Some HEIs used MOOCs to cover them, from different approaches and with various results [36].

Learning Analytics, defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” [37,38], could help us evaluate the effectiveness of those adaptations beyond students’ satisfaction or their perceived quality [39].

The key value of this experience lies in its double vision. On the one hand, it is about validating the possibility of offering an online alternative for university students to carry out internships—based on the use of online resources external to the university—when face-to-face is not an alternative. On the other hand, it offers the possibility of evaluating the efficiency of this alternative using the Learning Analytics provided by the system to complement the participants’ vision in the activity.

2. Materials and Methods

This paper is structured as a case study. It is a research approach to generate an in-depth, multi-faceted understanding of a complex issue in its real-life context [40] that helps the institution to answer the research questions.

2.1. Case Study Context

Universitat Politècnica de València (UPV) is a middle-sized public university from Spain that in 2020 offered 46 bachelor’s degrees from 13 schools in which 20,200 students were enrolled [41]. In most of these degrees, the students can do an internship in a company as part of their studies during a variable amount of time that usually can be as long as a third of one of the four years of the degree.

In March 2020, the students of the last course that had chosen the internship option were already taking their internship or would begin it when a massive lockdown started to be enforced in Spain to contain the spread of the COVID-19 pandemic. No one could go out of their homes, a situation that lasted until June 2020, so most were cancelled, all but the few that could be continued started working from home.

Most of these students needed to complete the academic credits assigned to their internships to receive their degrees on time. It was uncertain for how long we would be confined, and they could not wait until they could retake their internships, so the University had to look for a solution for them.

As soon as 11 March 2020, edX, the global MOOC platform, mobilized to help its university partners support their students during lockdowns and maintain continuity in their activities, therefore, created a RAP initiative (Remote Access Program) [42]. The initiative saw the creation of an edX partner community group, where members could access the courses and programs of any other member at no cost. This collaborative effort allowed universities to offer their communities over 800 courses, with immediate access to online learning opportunities in a broad array of subject areas. In a few days, over 60 edX partners from all over the world joined the program to deliver blended and independent learning opportunities for students—as well as professional development for faculty and staff—through 30 June 2020 [43], and UPV was one of them, offering all its MOOCs to be included in the initiative. The initiative worked by distributing promotional codes to the different institutions that waived the certificate fees for the included courses.

The internship department of UPV saw that this initiative was a good opportunity to give the students an option to cover the remaining credits of their internships and created a protocol to do so.

Each bachelor’s and master’s degree from the University has an academic governing commission called ERT, so these commissions were instructed to select a list of courses from the initiative related to their degree that covered the competencies to be developed in the internships. Then students had to prepare a plan selecting courses from this list to cover for their remaining credits and present it to their tutors in the internship as well as the member of the ERT in charge of internships who had to approve it. Once the courses were finished, the student had to present a finishing report and the certificates of the courses to the same two faculty, who had to validate it and send a report to the internship department.

2.2. Materials and Data Instruments

The main sources of information have been the Learning Analytics provided by the MOOC platform and the academic system.

In addition, and as a complement, the institution has sent a satisfaction survey to people who had used MOOCs to cover their internship.

In the anonymous survey, the information gathered was: demographic information (age, gender, residence place, job status and level of studies), information about the initiative (level of satisfaction from 0 to 5, number of codes requested, number of codes used, number of certificates obtained), information about the motivations to participate (reasons to participate), previous knowledge about MOOCs, future use of MOOCs (intention to take MOOCs in the future, willingness to pay for MOOC certificates, reasons to pay for MOOC certificates), information about every MOOC taken (institution, name of MOOC, quality of the MOOC from 0 to 5, contribution to the professional career from 0 to 5, finalization of the MOOC), an open field for comments and a specific question asking if MOOCs had given them the knowledge they would have acquired from the internship from 0 to 5.

The structure and content of the survey were validated using a content and face validation [44] process that guarantees the appropriateness and relevance of the items as they appear to the persons answering the survey [45].

2.3. Data Analysis

For the study, a table from UPV’s internship department database has been used. All data from the applications of the students that were doing an internship (or going to start one when the lockdown was enforced) and wanted to change the remaining academic credits of their internships were recorded. In the table obtained from the internship database, each record has several fields of which the most important ones are: the school, the name and surname of the student, the gender of the student, the state of the evaluation (finished or not), the number of academic credits to be covered and an unstructured text field that describes the educational project proposed by the student and what they finally did in the end.

The unstructured text field was processed to extract the different MOOCs used by students to a different table, with one record per MOOC.

For analyzing survey data, averages and standard deviation of every quantitative answer have been obtained. Considering the sample is a convenience sample (only those that participated answered the survey), statistical inference values (as p-values or size effect) have been avoided due to the “inference becomes tricky or outright impossible” [46].

2.4. Participants

Of the 3500 students in the course 2020–2021 that were in the last year of their bachelor studies, 227 needed a solution to cover the credits of the internship they had scheduled. Of these 227,179 students (78.9%) chose to make a learning plan using MOOCs to cover the academic credits not completed. These students are considered the population of our study.

The students came from 10 of the 13 university schools with the distribution that can be seen in Figure 1 (the names of the schools are listed in

Table 1. Universitat Politècnica de València Schools.

ETSIAM FADE EPSG ETSID	Agronomical engineering school Business management school Gandía city polytechnical school Design engineering school
ETSInf	Computer Science school
ETSII	Industrial engineering school
EPSA	Alcoi city polytechnical school
ETSICCP	Civil engineering school
ETSIGCT	Geodesical engineering school
ETSIT	Telecommunication engineering school

).

There were 109 (60.9%) male and 70 (39.1%) female students in the group, which is similar to the distribution of gender at UPV (38.5% female) and the students’ gender distribution of the technical universities from the region [47].

The satisfaction survey about the initiative was sent to every person who participated in the RAP initiative from the UPV community. We received 1515 answers, of which 101 were from students who had used MOOCs to cover their internship, 56.7% of the 179 participants in the internship project.

As it has been stated, this paper analyzes the student’s records in the internship department, the platform databases and the student’s answers to the survey to look for an answer to the research questions, as they can shed light about using MOOCs to obtain the knowledge in an activity that requires in-person attendance, as a company internship, where there is a forced online environment.

3. Results

Reviewing the data from the internship department and the platform databases of the 179 students that chose the MOOC option:

• 162 (90.5%) have a “passing” grade in the status field of the database;
• 17 (9.5%) have a note that says that their internships were cancelled.

Those data mean that, for 90.5% of people that participated in the project, the academic commissions of the different schools considered that MOOCs gave students enough knowledge to cover the objectives of the internships and officially validate them.

The students chose 636 MOOCs from 193 MOOCs offered by 40 institutions, with an average of 3.55 per student.

With the courses, students covered 1594 credits (ECTS, European Credit Transfer and Accumulation System), 1569.5 curricular credits (needed to receive their degree) and 24.5 extracurricular credits (that enrich the student record but are not counted for the total credits the student needs to get the degree). In total, only 8 students took MOOCs to receive non-curricular credits (5 mixed with curricular credits and 3 only for non-curricular credits). The maximum credits students covered with these MOOCs were 18, and the mode was 6 curricular ECTS per student, as seen in

Table 2. ECTS. Distribution for curricular and non-curricular credits.

	Curricular ECTS	Non-Curricular ECTS	Total, ECTS
Max	18	5.5	18
Mode	6	0	6
Average	8.57	0.19	8.76

.

Of the courses chosen by students, 518 (81.4%) were in Spanish and 118 (18.6%) in English. As shown in Figure 2, most (55.5%) are from the own institution.

Regarding what courses students preferred, data show that 26 courses concentrate more than half the enrollments; 10 of them (those included in

Table 3. Number of enrollments per course in the 10 most popular courses.

Course	Number of Enrollments
Excel: gestión de datos (Excel: Data management)	43
Excel: fundamentos y herramientas (Excel: Fundamentals)	32
ODS en la Agenda 2030 de las Naciones Unidas: Retos de los Objetivos de Desarrollo Sostenible (SDG in UN 2030 Agenda: Challenges for SDGs)	28
Excel avanzado: importación y análisis de datos (Advanced Excel)	23
Introducción a la gestión de proyectos (Intro to Project management)	22
Gestión participativa: motivación y liderazgo organizacional (High involvement work practices: motivation and organizational leadership)	18
Liderazgo para mandos intermedios (Leadership for middle managers)	16
Diseña presentaciones eficaces con Powerpoint (Powerpoint Presentations)	14
Finanzas personales (Personal finance)	13
Buscar en internet (Search Internet)	12

) reunite more than a third of enrollments. The most popular courses are focused on practical skills and tools which are very in demand in the business world, with the ones devoted to Excel as the most popular (first, second and fourth position). One course that explains Sustainable Development Goals (ODS in Spanish), another about project management, and a couple about leadership are also high on the list.

The students using MOOCs for the internship were tagged before sending the general survey about the initiative so that a specific question could be added; 101 (56.7%) students of the 179 who used MOOCs to cover their internship answered the survey.

The average satisfaction with the initiative of these 101 students is 4.57/5 with an sd (standard deviation) of 0.93, slightly smaller than the general satisfaction 4.7/5 (sd 0.7) calculated with the answers of the total 1515 members of the UPV community that answered this question. They score the quality of the MOOCs they have taken with 4/5 with sd 1.3 (slightly smaller than the one obtained from all 1515 answers with 4.1/5 and sd 1.04). Their score of how much the MOOCs they have taken can contribute to their professional career is 3.7/5 with sd 1.39 (a figure slightly higher than the one obtained from all answers (3.6/5 with sd 1.26).

The 162 students with a “passing grade” received 574 MOOC certificates, with an average of 3.54 certificates per student.

Of the 101 students who answered the survey, 96 (95%) responded that they would enroll in other MOOCs to acquire new knowledge, and 70 (69.3%) considered that it is worth paying for the certificates.

However, how did these courses cover the students’ expectations to fulfill their academic internships in this emergency? To the question “rate from 0 to 5 if MOOCs have given you the knowledge that you would have obtained from a company internship, where 0 is “I haven’t acquired any knowledge related with the internship” and 5 is “MOOCs have given me the same knowledge”, the results, 2.87/5 with sd of 1.7, are still positive but much less conclusive, as 40 of the 62 students that answered the question gave a rating of 3 or more, as seen in

Table 4. Number of students per score points in the question related to the internship.

Score	Number of Students
0	9
1	7
2	6
3	13
4	15
5	12

.

Looking at the data from the 16 students that answered 0 or 1 to this question, their satisfaction with the initiative is 4.3/5 (sd 0.92), they score the quality of the MOOCs they have taken with 3.67/5 (sd 1.48), and they score that the MOOCs they have taken can contribute to their professional career with 3.3/5 (sd 1.48); these figures are 10% lower than the averages for all survey responders, but still positive.

Analyzing the 21 comments that the students using MOOCs to cover for the internship left in the open response field, we see that 16 are positive, 2 neutral and 3 negative.

In the positive comments, the students were very happy and grateful for the experience and the opportunity, stressing the importance of continuous training, especially in times of confinement:

“Excellent initiative; I have nothing more to add. I only offer my thanks for having participated in this great learning process.”

Comment No. 6.

“Above all, the Excel courses have been very useful for me, as I’m studying 4th year of Telecommunications and they haven’t taught us anything about Excel. And I think it is very important and necessary to know how to use it when working in a company. It could be improved by adding more content. Since the possibilities offered by the program are very broad, and in 3 courses you can’t cover them all.”

Comment No. 2.

“I consider it extremely opportune to continue with the release of codes that encourage the continuous training of students, preparing us better and complementing our classroom knowledge.”

Comment No. 20.

In the case of neutral comments, they remarked on the positive character of the initiative, but with some suggestions for future editions:

“The courses, in general, are good; what I regret is not having had enough time to do them and having to do the courses “in a hurry” to be able to homologate them to my internship.”

Comment No. 9.

Negative comments are very negative, pointing out that they think that MOOCs are not a good alternative for the internships:

“I don’t think that the edX courses validate an internship in a company. I know it was only because of the extraordinary situation due to COVID, but it doesn’t even compare.”

Comment No. 12.

“It is one of the worst solutions I have seen to solve the problem I had when I didn’t do an internship. I couldn’t do an internship and I was left without that professional experience that is required of every student coming out of engineering.”

Comment No. 19.

Seven comments asked for the continuation of the free certificate initiative.

4. Conclusions and Discussion

The closure of our educational institutions and companies during the COVID-19 confinements confronted us with unprecedented situations that forced governments to take emergency measures for the universities’ academic activity to continue and avoid further damage to the careers of our students.

The results explored in this work answer positively to the first research question: 90.4% of students using MOOCs to substitute the internships have a “passing” grade in the status field of their internship in the academic database of the University. Hence, the academic commissions of the different schools considered that MOOCs gave students enough knowledge to cover the objectives of the internships.

The average satisfaction with the initiative of students using MOOCs for internships that answered the survey is 4.57/5, and they scored the quality of the MOOCs they took with 4/5. These figures are similar to the 4.7/5 average course rating reported by Coursera in the 2021 Coursera Impact report [47].

Students’ score of how much the MOOCs they have taken can contribute to their professional career is 3.7/5; 95% answered that they would enroll in other MOOCs to acquire new knowledge and 69.3% considered it worth paying for the certificates.

All of the above indicates that the answer to the second research question is yes, students were quite happy with the initiative and the quality of the courses they took, and that they think that MOOCs are a very good tool for learning.

Knowing that the academic commissions of the different schools considered that the knowledge acquired was enough and that the students are happy with the initiative and the quality of the courses, there is only one more thing to look at to answer the first research question and see if MOOCs can cover for the knowledge acquired in a company internship in a forced online environment: the answers to the question of the survey specifically devoted to this subject. Most students (64.5%) think they have obtained knowledge similar to the one they would with the internship. Still, the agreement about this statement is much lower, as 14.5% think that MOOCs have not given them at all the knowledge they would have acquired in an internship, and 11.3% believe they have obtained only a small part of the knowledge. If we look at the negative comments of the open response field, 3 students are furious at substituting the internship with MOOCs. However, there are 16 positive comments and 2 neutral ones from students using MOOCs for their internship, so the comments are mostly positive, and the answer to the first research question is also yes.

The results are similar to those observed in an experience in India [36], where 75.5% of students were satisfied with the contents and delivery of MOOCs to substitute their internship, and 91.8% answered that they would continue to enhance their knowledge and skills through MOOCs. In this experience, the perception of students regarding if MOOCs gave them equivalent learning to the internship is much higher (95.5%).

Looking at the courses chosen to cover for the internship, having over 800 courses to choose from, offered by some of the most prestigious universities in the world, students chose courses in their language from their university and focused on practical skills and tools very in demand in the business world.

We have used Learning Analytics to answer the research questions, and we can conclude that, given the passing grades granted by the academic commissions and the opinions of the students about the initiative, the courses and the coverage of the knowledge they expected to get from the internships, MOOCs have been a good tool to provide an important part of the knowledge by company internships. Students are very happy with the initiative of providing them free certificates for the MOOCs. They think that the quality of the MOOCs they have used is very good and that they will be useful to improve their professional careers.

Most students think MOOCs have given them an equivalent knowledge which they would have acquired with a company internship, but there is room for improvement. The initiative was an emergency solution that had to be deployed on the fly with no planning or extra support. The student proposed the MOOC itineraries they wanted to follow, and the schools’ academic boards approved them.

With a little more time and resources, the MOOCs to be taken by each student could have been chosen with the school support, some extra projects could have been assigned to each student with remote support from a pool of teachers, and remote teams could have been organized to give them the possibility to work their teamwork competence. This is in line with the findings of [36] that indicate that positive social influence and better facilitating conditions improve perceived ease of use and perceived usefulness leading to better user satisfaction in a similar experience of using MOOCs to cover for internships during the lockdown.

In addition, we have confirmed in this experience that the Learning Analytics usually accessible in the xMOOC classic model—that is, the one generalized in the most popular platforms—do not give participants and institutions much information about what is happening inside the courses and do not enrich the almost entirely “black box” perspective of courses. The ambition of enriching the type of LA collected by platforms and managing those data by participants would be interesting to enhance these initiatives in the future [25].

In conclusion, this case study shows that, when a remote environment is compulsory and face-to-face is not an option, MOOCs can be a good tool to provide an important share of the knowledge that a student can obtain by a company internship. Looking at the answers of the discontented students, we think that, in an “out-of-emergency” context, careful planning and the addition of some extra activities and resources to support students remotely could make the MOOCs an excellent tool to gain some of the skills developed in company internships.

Looking at the valuation that students gave to the quality and the usefulness of the MOOCs they took, and the answers to the question about if they had obtained a knowledge similar to the one they would with the internship, we think that MOOCs can also be used to reinforce traditional internships, introducing itineraries of previous mandatory courses on basic skills (Excel, teamwork, leadership, project management, Python programming, etc.) adapted to the specific internship the student is going to take.

The conclusions obtained are based on the grades granted by the academic commissions and the opinion of the students about the knowledge they were going to obtain with the internships. One limitation of the study is that these are indirect means of checking that the knowledge acquired in a company internship has been obtained. Another limitation is that students had no prior experience of what knowledge they were going to receive with the internships, only expectations about it, so their evaluation is based on these expectations.