The Vision of University Students from the Educational Field in the Integration of ChatGPT

Abstract

ChatGPT has significantly increased in popularity in recent months because of its capacity to generate novel content and provide genuine responses to questions. Nevertheless, like all technologies, it is crucial to assess its limitations and features prior to implementing it into an educational setting. A major obstacle associated with ChatGPT is its tendency to produce consistent yet occasionally unreliable and inaccurate responses. Our study provides students with training in this area, and its objective was to analyse the opinion of those same university students studying education-related degrees regarding the efficacy of the usefulness of ChatGPT for their learning. We used a mixed methodology and two instruments for data collection: questionnaires and discussion groups. The sample comprised 150 university students pursuing degrees in teaching and social education. The results show that the majority of students are familiar with the technology but have not had any formal training in a university. They use this tool to complete academic assignments outside the classroom, and they emphasise the need for training in it. Furthermore, following the training, the students highlight an increase in motivation and a positive impact on the development of generic skills, such as information analysis, synthesis and management, problem solving, and learning how to learn. Ultimately, this study provides an opportunity to consider the implementation of educational training of this tool at the university level in order to ensure its appropriate use.

1. Introduction

1.1. ChatGPT and Development of Generic Competencies in Higher Education

In recent decades, there has been a significant transformation in the teaching–learning process employed in the higher education setting. The traditional method of teaching, which relied on teachers and lectures to transmit knowledge, has been replaced by a more student-centred approach, emphasising active and participatory learning methods, which heavily rely on the use of information technology (IT).

IT has been evolving and adapting to the needs of education, although it might be more accurate to state that education has had to respond to the changes and requirements of IT, albeit not necessarily at the same pace of progress.

Artificial intelligence (AI) has developed as a powerful IT technology that is widely used in various areas and fields. This term refers to computer technology that emulates human intelligence. It employs self-learning algorithms and analyses large amounts of data to execute tasks, predict and solve problems, and make decisions autonomously. AI is founded on the replication and emulation of human reason, the training and development of cognitive functions, the organisation of thought processes, and the capacity of behaviour [1].

This training and automatic learning, or machine learning, is carried out through algorithms. These algorithms are a set of rules or instructions for processing large data sets (big data) and identifying patterns within them, enabling the computer to execute various actions [2].

Artificial intelligence has made significant progress in recent decades and has been used in fields such as industry, finance, medicine, and particularly education [3]. Education has become one of the favourite application areas for AI researchers [4]. In their systematic review study on the use of AI in education, [5] identified its application for various purposes. These include the early detection of students’ academic performance, predicting school dropout rates, improving student academic and career guidance, facilitating learning for students with disabilities, fostering the development of critical thinking, enhancing teachers’ educational skills, and generating educational content. The authors of [6] noted that AI can be used in a university setting to facilitate teaching tasks, such as content recommendation, virtual tutoring, and supporting teaching staff; research tasks, such as identifying and generating ideas, analysing data, and assisting scientific dissemination; and management tasks, such as processing documents, planning and programming, and providing administrative support to students.

Generative artificial intelligence, or GAI, is the most popular and well-known form of AI. It is unique because it is able to generate new and original content in different formats, such as text, image, audio, video, computer code, etc. Within this type of AI, we find ChatGPT, which is defined by the tool itself as follows:

It is an artificial intelligence application developed by OpenAI that is based on language models, such as GPT (Generative Pre-trained Transformer), to interact with users through text. What sets ChatGPT apart is its ability to generate contextually relevant responses from prompts, which are text inputs provided by users that guide the direction of the conversation. By interpreting and responding to these prompts, ChatGPT can imitate human interactions, deliver information, solve problems, and generate creative content, making it a useful tool for a variety of applications, from virtual assistants to text generation [7].

Its ability to provide accurate and relevant responses relies on the users’ ability to ask clear, appropriate, and specific queries or provide clear instructions.

ChatGPT has experienced unprecedented user growth, surpassing 100 million users in January 2023 [8]. This growth has expanded its use to both the general public and the academic community despite it being relatively new [9]. University students possess a comprehensive understanding of its features and use it at different times during their academic stage for various purposes [10], such as searching for information, composing written works (essays, articles, scripts, etc.), coding, summarising content, translating texts, etc.

The advantages that university students highlight about ChatGPT include its speed and efficiency in information retrieval; time optimisation; clear and easily accessible information and knowledge; user-friendly interface; and personalised assistance and solutions for tasks and problem solving [9].

However, they are also aware of and concerned about the limitations and disadvantages of the tool, emphasising the risk of being given erroneous, false, or very generic information due to the possible inaccuracy, imprecision, or outdated information obtained as well as the unreliability of the sources used [9,11]. Other risks that the tool may pose to students are also important to note here, such as the lack of verification or contrast with other sources of the information given in addition to a lack of critical analysis of the answers obtained. Furthermore, excessive reliance on it can lead to reduced effort, diminished skills, limited creativity, and a decreased ability to think independently and deeply analyse topics. Nevertheless, despite these constraints, students acknowledge and emphasise its usefulness and positive influence on their learning and academic performance as well as the need to include it in the classroom [9,10].

Because of the educational and instructional value of ChatGPT and its positive impact on student learning, there is a need for universities to be open to it and to adapt it to the interests, motivations, and resources students access.

Thus, there are concerns from both academia and university teaching staff regarding the impact of ChatGPT on academic integrity and the challenges in detecting and preventing plagiarism as well as in designing appropriate and innovative assessments for this new AI-driven reality [12,13]. Despite mistrust, reluctance, or limitations regarding this technology, it is imperative for universities to adopt procedures, policies, and support and training strategies [12] to ensure its responsible and ethical use. Moreover, it is essential for teaching staff to be adequately prepared to integrate these tools into their teaching and evaluation practices [13]. This preparation is necessary to harness and develop the educational potential of AI, ultimately aiming to foster meaningful learning and enhance student competencies.

When discussing competencies, ref. [14] states that this term encompasses the total, combined, and reciprocal interaction of knowledge, skills, and attitudes. Similarly, competencies are integrative, dynamic, transferable, transversal, multidimensional, and multifunctional [15]: a competent professional not only possesses the necessary competencies for their profession but also applies them and updates them according to context [16].

The foundations of skill training were established by the Delors Report and the Tuning Project. The Education Report contains a valuable report called the Delors Report, published by UNESCO in 1996. It establishes four dimensions or pillars of competencies: learning to know (related to cognitive aspects), learning to do (procedural aspects), learning to be (attitudinal aspects), and learning to live with others (relational aspects) [14,17,18].

For its part, the Tuning Project divides professional skills into specific and generic ones. Specific competencies refer to those related to a specific area of knowledge, which strengthen the more technical profile of the students, while generic competencies are transversal, are common to all degrees, and are established as essential and important for the efficient practice of any occupation. Furthermore, they are related to personal growth and responsible social development [17,19,20].

As a result of technological and digital changes, education has had to incorporate not strictly cognitive skills as basic skills, such as creativity, critical thinking, and empathy. Artificial intelligence (AI) serves as a practical tool for developing these skills by decentralising and personalising the teaching–learning process [3].

Therefore, it is important to emphasise the significance of these generic or transversal competencies, considering their alternative: competencies that focus only on productivity and competitiveness. The aim is to foster students’ autonomy and self-management and the ability to learn how to learn by emphasising the importance of learning as well as its relationship and coherence with democratic values and the basic principles of human rights [21].

The teaching, pedagogy, and social education degrees’ white papers prepared by the National Agency for Quality Assessment and Accreditation [22,23] identified the following generic competencies for Spanish educational system degrees: the analysis, synthesis, and management of information; creativity; problem solving and decision making; interpersonal skills; teamwork; critical thinking; ethical commitment to the profession; leadership; entrepreneurship; and learning how to learn.

1.2. University Students’ Evaluation of ChatGPT: Background

While ChatGPT has demonstrated its usefulness in tasks such as synthesis, analysis, and information management (despite the previously mentioned risks), delving into the impact of this tool on the development of other generic competencies is essential. There are contrasting visions on how ChatGPT affects the improvement of skills such as creativity, critical thinking, or social skills: some research emphasises the positive impact of it, while others argue that it hinders their development [24].

Within this context, there are instances where students report that the use of ChatGPT fosters commitment, participation, and group collaboration [11], while others express concern about a decline in human interaction [25,26].

Similarly, there are studies that suggest the use of technologies like ChatGPT can enhance critical thinking skills by fostering curiosity and creativity [27]. In response to this idea, some students state that ChatGPT does not contribute to the development of critical thinking and may even have a negative impact. This is because the information obtained through this tool is not critically analysed and not compared with other sources or the students’ own ideas [10]. Additionally, it can hinder independent thinking [28]. Consistent with this perspective, [9] argued that the use of AI tools in classrooms will compel teachers to adapt their methodology to foster creativity and critical thinking in students.

The ethical aspect of the use of ChatGPT in education also generates controversy, both with regard to the academic training process and the future practice of the profession [28]. Students have identified concerns regarding the improper use of this tool, such as plagiarism, compromised testing security, disinformation, digital illiteracy, accessibility issues, negative social and economic impacts, job loss, etc. [25].

As reflected in other systematic reviews and meta-analyses, scholars agree that ChatGPT has a positive impact on motivation and academic achievement. One notable study conducted by [29] found that the adoption of AI-based chatbots had a beneficial effect on learning outcomes, manifesting in students’ interest, motivation, and academic performance. Another study conducted by [30] explicitly highlighted the positive effect of ChatGPT on student motivation, which in turn improved their academic performance.

Similarly, students themselves express a high level of satisfaction with the use of this tool in their teaching–learning process and experience an increase in motivation when using it, as [31] found in their study. This increased motivation facilitated by the use of ChatGPT was also associated with commitment, participation, and satisfaction in their learning experience [11].

1.3. Study Objectives

Based on the above, the main objective of this study is to analyse the opinion of university students studying education-related degrees on the efficacy of the usefulness of ChatGPT for their learning. Specifically, we set the following objectives:

-

Analyse students’ existing knowledge and experience with ChatGPT, including their familiarity with prompts and any previous training they have received in this area.

-

Determine the scope and frequency of its use by the student population.

-

Understand the advantages and limitations of the tool for students.

-

Analyse the impact of its use on fostering motivation towards student learning and on the development of generic skills.

2. Context and Process Phases

The study was carried out in two universities in the Valencian Community. The participating sample was made up of 150 university students, 67 students studying a social education degree and 83 students studying a teaching degree. The sample was 80% female and 20% male. The average age of the sample was 20.1 years. The research study phases were as follows:

The 1st phase used an initial questionnaire to gather data on the use of ChatGPT. Analysis of the results obtained from the initial questionnaire.

The 2nd phase involved training students in the use of ChatGPT and designing the prompts. The students received training on how to design prompts correctly based on correct formulation and elements they must contain (contextualisation, clear and specific instructions, clarity and conciseness, appropriate and coherent language, suitable length, required format for the answer, etc.). The subject teacher conducted this training over the course of three class sessions.

The 3rd phase was the design of prompts. Students individually designed their own prompts. The prompts had to relate to the subject matter. Social education students were required to create prompts by selecting a specific group and activities to work with. Teaching degree students were required to create a prompt that addressed topics they chose with other early childhood education students based on the principles of neuroeducation.

The 4th phase involved sharing the prompt and the ChatGPT response in the Padlet tool so that their peers can view it.

The 5th phase involved an analysis of strengths and weaknesses both in the design of the prompts and in the responses obtained from ChatGPT, as assessed by three peers. For this task, a rubric provided by the teacher was used to evaluate elements related to the formulation and content of the prompts.

The 6th phase involved a questionnaire to analyse the impact of ChatGPT on the learning process.

The 7th phase involved a selection of the students participating in the discussion group.

The 8th phase involved carrying out discussion groups.

The 9th phase involved an analysis of the results.

3. Method

This study employed a mixed approach [32], combining quantitative and qualitative methodologies so that we could use the benefits of each perspective and conduct a comprehensive analysis of the research subject. Intentional sampling was conducted because of the sample’s accessibility to the researchers.

3.1. Instruments

Prior to commencing the training, the students were given a seven-question questionnaire to ascertain the frequency of use of the tool and previous training. The questions were the following:

-

Are you familiar with ChatGPT?

-

If you answered YES, specify where you heard about ChatGPT.

-

Definition of the tool.

-

Frequency of use of the tool.

-

Definition of Prompt.

-

Training received at and outside university regarding the use of the tool.

-

Importance of implementing training on the tool at university.

Upon completion of the training, participants were given a questionnaire to assess the extent to which the use of ChatGPT contributes to the development of generic skills among university students in the field of education. The questionnaire comprised 17 items, each referring to a generic competence. A 7-point Likert scale was used (1 Strongly agree and 7 Strongly disagree). These were the questions included in the questionnaire:

-

Item 1. ChatGPT is an intuitive and easy-to-use tool.

-

Item 2. ChatGPT is a useful tool to enhance learning at university.

-

Item 3. The use of ChatGPT at a university level has more benefits than disadvantages.

-

Item 4. The use of ChatGPT in the classroom fosters motivation to learn.

-

Item 5. The use of ChatGPT should be incorporated into university classrooms.

-

Item 6. Using ChatGPT for learning is ethical.

-

Item 7. The use of ChatGPT favours the development of creativity.

-

Item 8. The use of ChatGPT favours problem solving.

-

Item 9. The use of ChatGPT favours decision making.

-

Item 10. The use of ChatGPT favours the development of interpersonal skills.

-

Item 11. The use of ChatGPT favours teamwork.

-

Item 12. The use of ChatGPT favours the development of critical thinking.

-

Item 13. The use of ChatGPT favours information analysis, synthesis and management.

-

Item 14. The use of ChatGPT favours leadership development.

-

Item 15. The use of ChatGPT favours ethical commitment to occupations.

-

Item 16. The use of ChatGPT favours the development of entrepreneurship.

-

Item 17. The use of ChatGPT favours the development of learning how to learn.

In the qualitative methodology, the discussion group technique was used for the purpose of conducting the qualitative phase. The group comprised 10 students, 5 from the social education degree and 5 from the teaching degree, both male and female. The discussion focused on two questions regarding the integration of ChatGPT in the teaching–learning process: Do you think you need training in this tool for your studies? What benefits and disadvantages does ChatGPT provide in your training?

3.2. Procedure

The two questionnaires were administered online using the Google Forms tool during the 23–24 academic year. Before being provided with the instruments, participants were asked to sign the informed consent. We ensured its anonymity and confidentiality of the data, and its use and dissemination were solely carried out for scientific purposes.

The discussion group took place during the same session as the questionnaires were distributed. The 10 students selected were chosen because of their communicative and listening skills, their ability to analyse different topics and contribute ideas clearly and concisely, and their ability to respect others’ turn to speak or their opinions. The sessions lasted 60 min and were conducted in Spanish and Valencian to accommodate bilingual participants and ensure the validity of the study. The sessions were recorded with the consent of all participants, and the data collected from each group were subsequently transcribed. Participants were informed on everything related to their participation during this session (informed consent, duration, time and day, etc.). The session was held on Teams and lasted one and a half hours. Recording was allowed with the authorisation of all the participants.

3.3. Data Analysis

The quantitative data were analysed using SPSS 24 (license from both participating universities) and the qualitative data through the researchers’ interpretation and analysis of participants’ discourse.

4. Results

This section outlines the information extracted from the two questionnaires by carrying out a descriptive analysis and presents the qualitative analysis.

The results obtained from the original questionnaire are as follows:

-

All students, without exception, confirmed being familiar with ChatGPT: half of them through personal acquaintances and the remaining half through the internet.

-

The students indicated that they had not received any training at the university and that they had relied solely on internet videos.

-

All students define ChatGPT as an artificial intelligence tool.

-

The students were unable to define a prompt, as they did not know what a prompt was.

-

All participants highlighted the importance of receiving training with this tool.

-

Regarding frequency of use, they used it to carry out activities and academic work, but outside the classroom, they did not use it in class or for their personal life.

Table 1. Descriptives of generic competencies.

Item	Mean	Std. Deviation
I1	5.1	1.2
I2	4.6	1.3
I3	4.4	1.2
I4	4.1	1.4
I5	4.4	1.3
I6	4.1	1.3
I7	3.4	1.4
I8	4.1	1.4
I9	3.9	1.3
I10	3.4	1.4
I11	3.5	1.3
I12	3.2	1.3
I13	4.2	1.4
I14	3.4	1.3
I15	3.5	1.3
I16	3.9	1.3
I17	4.1	1.5

shows the results extracted from the information collected from the questionnaire that was given to them (following the training they received) to verify whether ChatGPT contributed to the development of generic competencies as well as to ensure that students understood the advantages and limitations of the tool.

The highest averages were obtained in item 1 (ChatGPT is an intuitive and easy-to-use tool) and in item 2 (ChatGPT is a useful tool to enhance learning at university). Both questions pertain to the tool’s ease of use and usefulness. They contend that the tool has more benefits than disadvantages at a university level and advocate for its integration into university training as part of the degree programme (4.4). Similarly, they assert that the use of the tool fosters motivation for learning (4.1), and they consider it ethically acceptable (4.1).

Regarding generic skills, students assert that the following competencies are those that are most developed when using the tool: information analysis, synthesis and management (4.2), and problem solving and learning how to learn, with an average of 4.1 in the latter two.

The remaining competencies exhibit medium–low averages in terms of the correlation between students’ use of ChatGPT and the development of these generic competencies. These competencies are creative competence (3.4), decision making (3.9), interpersonal skills (3.4), leadership (3.4), teamwork (3.5), ethical commitment (3.5), and entrepreneurship (3.9).

The competency of critical thinking (3.9) scored the lowest average in terms of its development from using this tool.

In the student discussion group, the prioritised actions focused on the need to include training on ChatGPT into the existing training programme.

-

Students studying the social education degree and the teaching degree agreed that training in artificial intelligence and ChatGPT was essential.

-

The main benefits of the tool are personalised learning and creative thinking. The disadvantages identified by the group were the little training they have received to use this tool and not knowing how to filter the material generated by the programme.

-

They acknowledged they regularly used this tool to carry out academic work, but they were unaware of the need for prompts to obtain dependable and reliable data.

-

They understand the importance of designing appropriate prompts as well as the significance of sharing them with their peers and the co-evaluation they carried out.

-

The group suggested that this training should be implemented across all subjects.

-

They expressed a greater intrinsic motivation towards learning both during training and when designing and evaluating the prompts, as they find the topic intriguing, and consequently, their inclination to learn is heightened.

-

The integration of a subject on artificial intelligence into the study plan (teaching participants) was important.

5. Discussion

Since its introduction in 2022, ChatGPT has been the focus of various research studies assessing its positive and negative impact on the teaching–learning process [33]. The present study is no exception.

As shown by the results presented in the text, students use this tool for academic purposes outside the classroom, and while it has a significant influence in that context, it has not been used by them to the same extent on a personal or professional level. Consequently, its use is limited, particularly because of the absence of training in it. These results concur with the research carried out by [31] that found that university students were receptive to ChatGPT and acknowledged its potential in the teaching–learning process. However, a study conducted by [34] showed that only a minority of respondents (33%) had used ChatGPT in their educational practices and a large proportion (75%) found it inconvenient for their educational tasks owing to their lack of knowledge on how to use it. This directly relates to the urgent need for training in this area since equipping both students and teachers with the skills needed to effectively use artificial intelligence tools is essential, extending their use beyond writing papers or searching for information.

This need for training coincides with the findings of the current study and the research conducted by [35], which emphasises that there is little preparation in terms of the didactic management of this resource, which results in a negative view of the tool and scepticism surrounding its use. Similarly, according to other authors [36], there is a missed opportunity in the educational field due to the lack of training being offered in this tool, and there should be a focus on developing students’ digital skills, as they will be entering a workforce that heavily relies on artificial intelligence.

This study has found that students perceive ChatGPT to be an intuitive and easy-to-use tool, concurring with [37], and that it has the potential to improve learning. The findings align with the results obtained by [38], demonstrating that students enjoy using ChatGPT as a tool in their learning process since they consider it to be valuable tool that improves their educational experience. Furthermore, [39,40,41] argue that this tool is important as it will improve the teaching–learning processes because it is a powerful tool capable of transforming these processes.

Regarding the benefits of ChatGPT, the findings indicate that it is perceived to have more benefits than disadvantages since it improves the learning process and fosters motivation for learning, among other positive advantages. This motivation is related to the immediate responses given when asking questions and the efficiency in structuring or organising work. These findings align with [42], whose studies found that students believe using AI tools has a positive impact on the educational process since they reduce the differences between student performance and can improve academic performance. Likewise, [31] found that there is an increase in student motivation when they use ChatGPT. Research conducted by [43] shows that the use of this tool improves the class environment and reduces academic stress. Conversely, ref. [44] emphasise that an advantage of ChatGPT is the enhancement of cognitive capacity, leading to improved academic achievement.

It is important to emphasise the need to use this tool responsibly and ethically. Exploring the ethical implications that artificial intelligence can have in the educational context, as [40,45,46] did in their studies, is essential. All agree that its use should be appropriately regulated by advocating for the responsible use of AI, establishing appropriate policies to research it and creating an ethical observatory for artificial intelligence in education. However, the above is subject to dispute since said regulation can result in prohibition and censorship, and this would contradict the purpose for which ChatGPT has been designed.

In terms of general skills, this study found that the use of ChatGPT specifically influences the development of information analysis, synthesis and management skills, problem-solving skills, and learning how to learn. While limited research exists on this topic to directly compare our findings, it is worth noting that certain authors [47] have shown that ChatGPT has the ability to facilitate students in generating new knowledge and fostering curiosity in exploring new ideas. Moreover, they emphasise that because this tool generates seemingly credible answers, a detailed analysis of those answers is necessary, so that students can filter what is truthful and useful. All of this favours the capacity to select and analyse the information obtained through the prompts and the ability to learn how to learn. The authors of [48] suggest that students need to use a variety of cognitive skills to engage in all those tasks that filter the data generated by ChatGPT, such as evaluating, verifying, substantiating, and exploring the information and answers given, which will favour the development of those and other competencies and skills.

Creativity and critical thinking were the two generic competencies that obtained the lowest score in the present study. The study conducted by [49] revealed that assistance provided by AI in writing and verbal creativity tasks should be acknowledged as its use results in a significant improvement in students’ written productions. However, these authors also emphasise that AI cannot substitute human intelligence and creativity. Regarding critical thinking, the results obtained concur with those found by [24], arguing that ChatGPT should be used within ethical guidelines, emphasising the need to develop critical thinking within users. The authors of [48] note that to improve this skill, training in AI needs to be incorporated into curricula, although teachers are also responsible for this task. This will ensure ChatGPT users do not unquestioningly accept information without analysis.

6. Conclusions

As emphasised throughout this study, artificial intelligence tools, and specifically ChatGPT, have made a significant impact on the educational system. However, there are still many areas that need to be explored.

The purpose of this study aimed to analyse the students’ existing knowledge of ChatGPT and its influence on their university education learning as well as their past training in this area. We collected quantitative and qualitative data to determine whether the effectiveness of training and instructional design on the use of ChatGPT improved students’ competencies using the tool. The students were familiar with the tool, but they had not received any previous instruction on how to use it. The results of the study indicate that the proper use of the tool, following the training received, leads to an increase in motivation and a positive impact on the development of generic skills, such as information analysis, synthesis and management, problem solving, and learning how to learn. While it is still too early to specify its impact, there are already documented instances in education where the use of ChatGPT has shown improvements in the educational and training processes. However, despite the benefits, integrating changes into the educational model, the type of tasks that are designed and the type of skills that students need to attain, is essential.

As future lines of research, there are many uncertainties that have emerged with the introduction of ChatGPT in the educational field in 2022, but we will mention the areas where we believe research must focus if we want to gain a more comprehensive understanding of this tool and its potential. First of all, the specific training that should be given to teachers and students regarding the use of artificial intelligence tools needs to be identified. In this sense, further research into what basic concepts they should learn, what skills they should acquire, and the strategies that should be designed for adequate development would be of interest. Secondly, we must continue to analyse how it interferes with university students’ training: understanding its potential but also identifying its limitations. To do this, identifying aspects related to ethics and the responsible use of technology in general and AI in particular is essential. Thirdly and finally, we should explore the possibility of creating new educational contexts since the use of AI offers new possibilities in education.