Subject-Specialized Chatbot in Higher Education as a Tutor for Autonomous Exam Preparation: Analysis of the Impact on Academic Performance and Students’ Perception of Its Usefulness

Abstract

This study evaluates the impact of an AI chatbot as a support tool for second-year students in the Bachelor’s Degree in Early Childhood Education program during final exam preparation. Over 1-month, 42 students used the chatbot, generating 704 interactions across 186 conversations. The study aimed to assess the chatbot’s effectiveness in resolving specific questions, enhancing concept comprehension, and preparing for exams. Methods included surveys, in-depth interviews, and analysis of chatbot interactions. Results showed that the chatbot was highly effective in clarifying doubts (91.4%) and aiding concept understanding (95.7%), although its perceived usefulness was lower in content review (42.9%) and exam simulations (45.4%). Students with moderate chatbot use achieved better academic outcomes, while excessive use did not lead to further improvements. The study also identified challenges in students’ ability to formulate effective questions, limiting the chatbot’s potential in some areas. Overall, the chatbot was valued for fostering study autonomy, though improvements are needed in features supporting motivation and study organization. These findings highlight the potential of chatbots as complementary learning tools but underscore the need for better user training in “prompt engineering” to maximize their effectiveness.

1. Introduction

In recent years, the integration of digital technologies in higher education has grown exponentially. The COVID-19 health crisis accelerated the adoption of technological solutions in teaching and learning processes, highlighting the potential of online learning to facilitate self-directed and flexible environments. As a result, universities have been forced to develop the ability to quickly adapt to unforeseen circumstances (Cramarenco et al., 2023). Furthermore, the Fourth Industrial Revolution has driven a profound digital transformation in higher education institutions, linked to the need to align with labor market demands (Schwab, 2017).

The digitalization of educational institutions has promoted the expansion of online courses and hybrid or blended-learning teaching, allowing greater accessibility and educational flexibility (Alenezi, 2023). However, digital technologies have also been integrated into face-to-face teaching through virtual classrooms and learning platforms, complementing traditional education, and tailoring learning experiences to students’ individual needs. Among the benefits of this process, we can highlight more equitable access to education by making learning more flexible, personalized, and accessible (Dhawan, 2022; Valverde-Berrocoso et al., 2022).

One of the most recent advances in the use of digital technologies in higher education has been the integration of artificial intelligence (AI) in educational processes. In particular, AI-based chatbots have gained popularity due to their ability to interact dynamically with students, providing real-time feedback and fostering personalized learning. These systems allow students to access immediate answers to specific questions, which not only reduces exam-related anxiety but also enhances their ability to self-regulate the learning process (Ng & Leung, 2024; Su & Yang, 2023). Moreover, LLMs have demonstrated significant potential to enhance innovative teaching methods, such as the flipped classroom. By enabling the creation of interactive materials for students to review before face-to-face lessons, they promote active and personalized learning at home, fostering deeper engagement during class (Tan, 2023). In addition, automated quiz-generation tools powered by LLMs allow educators to provide personalized assessments aligned with individual learning progress (Hang et al., 2024).

The emergence of tools like ChatGPT, launched in 2022, along with other systems like Gemini, Perplexity, Claude, and others, has revolutionized the way we interact with information. These platforms automate complex cognitive tasks and encourage creativity, presenting new opportunities and challenges in education and research. Their widespread adoption among higher education students has been facilitated by the numerous advantages they offer, such as immediate and efficient access to knowledge and increased productivity in academic tasks. However, these tools are not always used appropriately, transparently, and ethically (Dabis & Csáki, 2024; Chen et al., 2024; Mironova et al., 2024).

The educational potential of chatbots is considerable. One of the most notable benefits is the personalization of learning (Abas et al., 2023; Agbong-Coates, 2024). Unlike traditional forms of teaching, which are often more static and generalized, chatbots enable direct and adaptive interaction with students, adjusting both the content and the pace of teaching to each individual’s specific needs. This personalization has been shown to have a positive impact on student motivation, as it provides a less stressful study environment more focused on their interests and learning pace (Holmes et al., 2019). Additionally, by offering immediate responses and allowing for content repetition, chatbots help students delve deeper into topics of greater interest or those they find more challenging, thereby improving knowledge retention (Deng & Yu, 2023).

Personalization also has significant implications for self-directed learning. By allowing students to manage their own study pace, chatbots reduce the pressure associated with traditional academic deadlines, fostering greater flexibility in the learning process. This not only contributes to improved academic performance but also strengthens autonomy and self-management, key skills in today’s educational environment. Recent studies indicate that students using chatbots tend to show greater satisfaction with their learning experiences and achieve better academic outcomes compared to those without access to these tools (Zhang et al., 2024; Guttierrez-Aguilar et al., 2024).

Self-regulation of learning is another aspect that chatbots can enhance. Students’ ability to plan, monitor, and adjust their own learning process is essential for academic success, especially in self-directed learning environments. Chatbots offer real-time feedback, allowing students to modify their study strategies as needed. This continuous and adaptive feedback fosters more efficient and self-directed learning, reducing dependence on teacher intervention and enabling students to take control of their own educational process (Ng & Leung, 2024; Labadze et al., 2023).

The adoption of chatbots in higher education largely depends on students’ perception of their usefulness and ease of use. Davis’ Technology Acceptance Model (TAM) (Davis, 1989) provides a theoretical foundation to evaluate these perceptions, emphasizing that the adoption of new technologies is driven by two main factors: perceived usefulness (the extent to which a tool enhances performance) and perceived ease of use (the effort required to use the tool). In the case of chatbots, research has shown that students find them intuitive and accessible, facilitating their adoption (Schei et al., 2024).

Despite the aforementioned benefits, the use of educational chatbots raises several important ethical challenges, particularly regarding student privacy. Chatbots collect large amounts of data on students’ study habits and academic performance, raising concerns about how such data is managed and protected (Kasneci et al., 2023). Therefore, educational institutions must develop clear and transparent policies to ensure the ethical and responsible use of student data, respecting their privacy rights (Holmes & Tuomi, 2022). This is crucial in a context of increasing AI use in educational settings, where the risk of data misuse is growing.

Another important concern is access to technology. Although chatbots can offer significant benefits for personalized learning, it is essential to ensure that all students, regardless of their socioeconomic status, have access to these tools. The digital divide remains a problem in many regions, and unequal access to advanced technologies can exacerbate existing inequalities in the education system (Holmes & Tuomi, 2022; Rahman & Watanobe, 2023). To mitigate this risk, universities must develop strategies to promote equitable access to technology and provide resources for students who may be at a technological disadvantage (Mat Dangi & Saat, 2021).

The impact of chatbots on learning outcomes has been extensively researched in recent years. Studies have shown that students using these technologies experience significant improvements in academic performance, both in content comprehension and information retention, as well as in learning self-regulation (Ng & Leung, 2024; Deng & Yu, 2023; Mungai et al., 2024; Martins et al., 2024; Yigci et al., 2024). These improvements are primarily due to the personalization of learning and the constant feedback that chatbots provide, allowing students to adjust their study strategies more effectively and resolve doubts immediately (Chang et al., 2023). Nonetheless, we are still in the early stages of integrating these technologies, and further studies are needed to support these apparent advantages.

In any case, to maximize the potential of chatbots, their implementation must be supported by a solid pedagogical framework. Educators must strategically integrate these tools into their teaching methodologies, promoting not only self-directed learning but also active learning and critical engagement (Deng & Yu, 2023; Halaweh, 2023; Chukwuere, 2024). A well-structured pedagogical approach that combines the use of chatbots with other teaching strategies can improve not only academic outcomes but also student motivation and engagement with their learning process (Lin & Chang, 2023).

This study evaluates the educational impact of a subject-specialized chatbot implemented as a support tool for second-year students enrolled in the course “The School of Early Childhood Education” in the Bachelor’s Degree in Early Childhood Education program at the University of La Laguna (Spain) during the 2023–2024 academic year. Through this implementation, the aim is to provide an accessible and effective resource for resolving specific doubts, contributing to more autonomous learning during the exam preparation period.

Research Questions:

• RQ1: How effective is the specialized chatbot in resolving students’ doubts and delivering clear and precise explanations to support exam preparation?
• RQ2: What is the frequency and patterns of chatbot use among students, and what types of questions do they typically ask?
• RQ3: How do students and teachers perceive the effectiveness and usefulness of the chatbot as a complementary learning tool?

2. Materials and Methods

This study evaluated the impact of using a specialized chatbot in the course “The School of Early Childhood Education” as a personal tutor during the study process and final exam preparation for second-year students in the Bachelor’s Degree in Early Childhood Education program during the June 2024 exam period. The research was conducted using a combination of surveys, in-depth interviews, and an analysis of the students’ interactions with the chatbot.

2.1. Participants

Participation in the use of the chatbot was entirely voluntary. All students enrolled in the course (92 in total) were invited to participate in the study, and the final sample consisted of the 42 students who agreed to take part. For 1 month, participants used the chatbot as a support tool for their final exam preparation. The course was led by a single instructor, who was also directly involved in the design and implementation of the research.

2.2. Research Procedure

2.2.1. Pre-Intervention Phase

The first step involved selecting and designing a chatbot. The Chatbase platform (https://chatbase.co, accessed on 17 October 2024) was chosen as the tool for creating and managing the educational conversational assistant. This selection was based on several key features that made Chatbase an appropriate choice for the study’s objectives:

• Ease of use: Chatbase provides an intuitive interface, allowing chatbot creation without advanced programming knowledge. This facilitated the quick setup and customization of the chatbot for the educational context.
• Compatibility with advanced AI: The tool uses natural language processing (NLP) technologies based on advanced models, such as GPT, which enabled the chatbot to answer complex questions coherently and accurately.
• Integration with knowledge bases: Chatbase allows the chatbot to be trained with specific course files and documents.
• Interaction recording: The platform logs and analyzes all user interactions with the chatbot, which proved valuable for subsequent data analysis.

The chatbot was designed and programmed to answer questions related to the course content, including definitions, detailed explanations, and practical examples. It also featured closed and open questions, using a knowledge base built from course materials and previous exams. To ensure pedagogical alignment, the chatbot design incorporated principles of active and self-directed learning. Specifically, it was programmed to guide students through incremental problem-solving tasks, encouraging independent inquiry through targeted questions to improve critical thinking skills. These pedagogical principles were further reinforced during training sessions, where students were taught strategies to maximize the chatbot’s educational potential. An evaluation framework was developed to assess the chatbot, focusing on key constructs such as accuracy, consistency, fluency, and perceived usefulness, all aligned with the goal of supporting self-directed learning. These constructs were assessed through tests conducted by the authors, including predefined questions to assess the chatbot’s ability to deliver correct answers, maintain consistent responses across similar queries, and demonstrate fluency across interactions. Findings from these evaluations informed iterative adjustments to the chatbot design, ensuring alignment with intended educational goals. Under these conditions, the chatbot’s performance was implemented and tested following a systematic procedure until optimal results were achieved before offering it to students (Sánchez-Vera, 2024). At the same time, an initial survey was conducted among participating students to collect demographic data, assess their familiarity with digital tools, and gather their expectations regarding the chatbot and its educational usefulness.

2.2.2. Intervention Phase

A training session was conducted to familiarize students with the use of the chatbot. During this session, the tool’s purpose, main features, and basic prompting techniques were explained, teaching students how to interact effectively with the chatbot to obtain useful responses (Knoth et al., 2024). Three prompting strategies were introduced: zero-shot prompting, few-shot prompting, and chain-of-thought prompting. Zero-shot prompting involved the chatbot responding to queries without prior examples, focusing on retrieving general knowledge effectively. Few-shot prompting included providing the chatbot with a small number of examples to guide its responses, enhancing its ability to address context-specific questions. Chain-of-thought prompting encouraged the chatbot to articulate its reasoning process step by step, making it particularly useful for solving complex or multi-step problems.

These strategies were illustrated through practical examples and live demonstrations, ensuring students could understand and apply them effectively. Additional guidance was offered to refine their ability to formulate clear, well-structured questions, maximizing the chatbot’s potential to provide relevant and useful responses.

For 1 month, students used the chatbot as a support tool for their final exam preparation. All interactions were recorded for later analysis to assess the tool’s impact on the learning process.

2.2.3. Post-Intervention Phase

Once the intervention period ended, several actions were taken to measure the results obtained. First, a final survey replicating the initial survey’s items was administered to identify changes in students’ perceptions. This survey also included specific questions to evaluate their experience with the chatbot. Additionally, six in-depth qualitative interviews were conducted with selected students to gain a more detailed understanding of their experiences. These interviews focused on exploring the chatbot’s perceived strengths and weaknesses and its usefulness as a study aid.

In terms of analysis, both quantitative and qualitative approaches were applied. Data from the pre- and post-intervention surveys were analyzed statistically to detect significant variations in students’ perceptions and skills. Additionally, a qualitative analysis of the interview responses and chatbot interactions was conducted, identifying key themes and patterns that provided a more comprehensive understanding of the chatbot’s impact on the learning process.

2.3. Data Collection Instruments

2.3.1. Instruments for Evaluating the Chatbot’s Quality

The chatbot’s performance was evaluated through an iterative process of testing and refinement. Adjustments were made to the chatbot’s prompts and knowledge base to ensure optimal performance in terms of accuracy, consistency, fluency, problem-solving ability, exam preparation effectiveness, and reliability. These constructs were selected for their relevance to both the pedagogical and technical goals of the chatbot. The selected indicators for evaluation reflect realistic standards suitable for a university environment. The evaluation was conducted through systematic tests in a controlled environment. While a 100% performance would be ideal, current technological limitations prevent such perfection. Therefore, the proposed values seek to balance pedagogical utility with the chatbot’s technical capabilities. These limits were established by consensus between the researchers and chatbot designers, representing conventional ranges designed to ensure that the chatbot is robust enough to meet educational needs without demanding impossible performance for current technology:

• Response Accuracy (≥90%): The chatbot must provide correct answers 90% of the time, ensuring that responses are largely accurate and contribute to quality learning.
• Response Consistency (≥95%): A 95% consistency rate ensures that responses are aligned with repeated concepts in different contexts, avoiding student confusion.
• Interaction Fluency (≥85%): Fluent conversation is critical for a positive experience, with 85% representing a comprehensible and natural interaction in most cases.
• Problem-Solving Ability (≥90%): The chatbot should effectively resolve complex questions at least 90% of the time, ensuring it can guide students in understanding difficult concepts.
• Exam Preparation Effectiveness (≥85%): To be perceived as useful, the chatbot must effectively help students prepare for exams, achieving an 85% user satisfaction rate.
• Reliability (100%): The chatbot is expected to have 100% availability, ensuring uninterrupted operation throughout the usage period.

The evaluation process included two sets of tests. The first set consisted of 60 questions taken from real exams, divided into six categories: concept definitions, true/false, multiple choice, concept association, short answer, and short essay. The second set evaluated the chatbot’s ability to adapt to open-ended and unstructured conversations, simulating more fluid and spontaneous interactions to measure its adaptability.

The chatbot was subjected to multiple rounds of these tests, and its responses were evaluated according to the criteria defined above. After each round of testing, adjustments were made to both the prompts and the chatbot’s knowledge base to improve the accuracy and adaptability of the responses. This continuous refinement cycle was maintained until the chatbot reached optimal performance levels according to the proposed indicators, ensuring its ability to meet pedagogical and technological expectations.

2.3.2. Pre- and Post-Intervention Surveys

Two surveys were designed and administered to assess the impact of the intervention. The constructs measured in the questionnaires were developed specifically for this study (ad hoc) to capture perceived usefulness, ease of use, interaction frequency, and overall satisfaction with the chatbot. These constructs were chosen to align with the study’s objectives of evaluating the chatbot’s role in supporting exam preparation and self-directed learning. The questions were created by the research team based on the specific context of the course and the chatbot’s intended functionalities. While the ad hoc nature of the survey limits the possibility of external validation, the questions were reviewed by three educational technology experts to ensure clarity, relevance, and alignment with the study’s goals. The first survey, conducted before the intervention, consisted of nine Likert-type questions addressing demographic data (age and gender), frequency of technology use, perceived technological proficiency, expectations regarding the chatbot, and prior experience with chatbots or AI in educational contexts. This set of questions provided baseline data on the participants.

The second survey, conducted after the intervention, included a total of 16 questions. Nine of these replicated items from the initial survey, allowing for direct comparisons between pre- and post-intervention results. The seven additional items focused on evaluating the chatbot experience, including interaction frequency, overall satisfaction, perceived usefulness in exam preparation, ease of use, areas where the chatbot was most helpful, interaction naturalness, and willingness to recommend the chatbot to other students.

The survey design, based on Likert scales, allowed for descriptive analysis of frequencies and percentages to identify changes in students’ perceptions and evaluate the intervention’s effectiveness, following validated methodologies in previous studies on educational interventions (Boone & Boone, 2012; Joshi et al., 2015).

2.3.3. In-Depth Interviews

In-depth interviews were conducted with six students after the intervention. These interviews provided a qualitative understanding of students’ experiences with the chatbot. This type of interview is particularly useful when seeking information about complex or subjective processes that cannot be captured by traditional quantitative methods. In-depth interviews are a valuable tool for exploring participants’ perceptions, experiences, and attitudes in detail, generating a richer, more contextualized understanding of the phenomenon under study (Kvale, 2007; Patton, 2015). The interviews focused on the perceived effectiveness of the chatbot, areas for improvement, and perceived challenges.

The six students were selected based on their level of interaction with the chatbot, determined by analyzing the number of conversations they had during the intervention phase. Participants were categorized into three groups: low interaction (1–10 interactions), moderate interaction (11–20 interactions), and high interaction (more than 20 interactions). Two students were chosen from each category to ensure a balanced representation of chatbot usage patterns. This selection strategy aimed to capture a diverse range of experiences, from minimal to intensive use of the chatbot. By including students from all interaction levels, the interviews offered a comprehensive understanding of how varying degrees of engagement influenced their perceptions of the chatbot’s usefulness, ease of use, and its impact on exam preparation.

Furthermore, during the analysis of the interviews, data saturation was achieved, as no new themes or insights emerged after conducting the six interviews. This suggests that the selected participants provided a sufficiently comprehensive representation of the experiences and perceptions within the study cohort.

2.3.4. Chatbot Interaction Analysis

The conversations between students and the chatbot were analyzed to identify the most frequent types of questions. The conversations were categorized thematically, allowing for an analysis of the main types of interaction and, consequently, how students used the chatbot. This approach provided a clear understanding of their tutoring needs and identified areas where gaps or deficiencies were observed. This method corresponds to what the literature calls human-machine interaction analysis, a technique that studies interactions between humans and automated systems, evaluating both the content of questions and usage patterns (Følstad et al., 2018; Diederich et al., 2021). This type of analysis is common in research on chatbots in educational settings, where conversations are examined to reflect user understanding and identify areas where technological interventions can improve the educational experience (Winkler & Söllner, 2018).

2.4. Ethical Considerations

This study was conducted in compliance with current regulations on data protection and ethics in educational research. Specifically, the principles established by the General Data Protection Regulation (GDPR) (Regulation EU 2016/679) (Winkler & Söllner, 2018) and the Spanish Organic Law 3/2018, of December 5, on Personal Data Protection and Guarantee of Digital Rights (LOPDGDD) (España, 2018), were followed to ensure the appropriate handling of participants’ personal information.

Informed Consent: All participants were fully informed about the study’s objectives, procedures, and scope before participating. They were told about the voluntary nature of the study and their right to withdraw at any time without consequences. Participants signed an informed consent form, which guaranteed their understanding and acceptance of the study’s terms.

Confidentiality and Anonymity: Participants’ personal and academic data were treated with strict confidentiality. Anonymization measures were applied by assigning unique codes to avoid the direct identification of participants. Data handling complied with GDPR and LOPDGDD provisions, ensuring that only the research team had access to the information, which was stored on secure, encrypted servers. It was also established that the data would be deleted after the study’s completion and the publication of the results.

Transparency and Results: It was ensured that the study’s results would be reported in aggregate form, without including information that could identify participants. Additionally, students were offered the option to receive a summary of the results, promoting transparency and the return of information to the participants.

In conclusion, the research rigorously adhered to data protection and digital rights regulations, ensuring the ethical and secure handling of participants’ personal information and respecting their privacy at all times.

3. Results

3.1. Overall Chatbot Performance

The research team conducted an iterative testing process in a controlled environment to evaluate and refine the chatbot’s performance before releasing it to students. These lab tests were based on predefined methodological indicators: accuracy, consistency, fluency, problem-solving ability, exam preparation effectiveness, and reliability.

Below are the results obtained in each category after successive adjustment cycles:

• Response Accuracy: The chatbot achieved 98% accuracy, surpassing the methodological threshold of ≥90%. This indicator assessed the system’s ability to provide correct answers to simple questions such as true/false or concept definitions. Zero-shot prompting was used for these types of questions, a technique where the chatbot generated answers without prior examples. This approach was effective for direct inquiries and required no additional adjustments to improve accuracy in this context.
• Response Consistency: Response consistency was measured based on the uniformity of responses in different contexts, especially for repeated questions or those involving related concepts. The chatbot achieved 96% consistency, exceeding the minimum standard of ≥95%. This result was attained through the application of few-shot prompting, where specific examples were provided to guide responses. This technique significantly improved response alignment in multiple-choice scenarios and questions that required concept matching.
• Interaction Fluency: The chatbot scored 89% in interaction fluency, meeting the methodological threshold of ≥85%. This metric was assessed based on the chatbot’s ability to maintain understandable and natural interactions, particularly in open-ended or unstructured conversations. Chain-of-thought prompting (CoT) was used in the evaluation, allowing the chatbot to break down its reasoning into clear steps, improving the structure and clarity of responses in essay-type or more complex questions.
• Problem-Solving Ability: The chatbot’s problem-solving ability, assessed in terms of its skill in answering complex questions and guiding the understanding of difficult concepts, reached 92%, surpassing the threshold of ≥90%. This result was achieved through a combination of few-shot prompting and CoT prompting, techniques that enabled the chatbot to provide more detailed and explanatory answers. These iterative adjustments to the chatbot’s configuration enhanced its ability to address deeper conceptual issues, aligning with the established pedagogical goals.
• Exam-Preparation Effectiveness: The research team also evaluated the chatbot’s effectiveness in exam preparation, achieving an 88% satisfaction rate, exceeding the ≥85% standard. This indicator measured the perceived usefulness of the chatbot in content review and exam simulations. Few-shot prompting and CoT prompting techniques were essential to ensure the chatbot provided precise and relevant feedback, helping students better prepare for their academic assessments.
• Reliability: The chatbot demonstrated 100% reliability during the lab testing period, meeting the proposed standard of full availability. This metric evaluated the system’s ability to remain operational without interruptions, even under high-demand conditions. The absence of failures or downtime during testing ensured that the system could respond continuously and stably, solidifying its technical robustness.

Under these conditions, the chatbot was deemed well-prepared for implementation in a real educational environment, meeting the minimum requirements established in the methodological framework.

3.2. Analysis of Chatbot Interactions: Most Consulted Content Types and Relationship with Learning Outcomes

During the exam preparation period, students interacted with the chatbot through 186 conversations, generating a total of 704 interactions. The average usage was 3.7 interactions per conversation and 16 interactions per student, although there was a high standard deviation of 11.35, indicating considerable variability in usage patterns. This suggests that some students made few inquiries, while others engaged more intensively, using the tool in various ways.

The interactions with the chatbot were categorized into six main types, reflecting the range of topics explored by students during their exam preparation. These categories closely align with the course content. For example, the Regulations and Legislation category includes inquiries related to educational laws, a component of the course that focuses on understanding regulatory frameworks governing various aspects of education.

Table 1. Most consulted content types and distribution of the interactions.

Category	Description	Number of Interactions	Percentage (%)
Definitions	Queries to clarify key terms and concepts	244	34.66%
Detailed Explanations	Requests for in-depth information on theories and frameworks	157	22.30%
Practical Applications	Questions about applying pedagogical theories in real contexts	105	14.91%
Comparisons and Differences	Queries to contrast concepts or methodologies	87	12.36%
Regulations and Legislation	Questions about educational laws and regulations	61	8.66%
Other Queries	Technical questions or requests for additional resources	50	7.10%

below presents a detailed breakdown of these categories, their frequency, and the corresponding percentage of total interactions:

The table shows that students primarily used the tool to clarify basic concepts and delve into theories, with the categories of Definitions (34.66%) and Detailed Explanations (22.30%) accounting for more than half of the total interactions. This was confirmed by several students in interviews, who indicated they turned to the chatbot when they needed to better understand a concept or theory, as “the chatbot explained things clearly and quickly, without needing to search through notes”.

The categories of Practical Applications (14.91%) and Comparisons and Differences (12.36%) were less frequent. These types of questions were relevant for connecting theory with practice and contrasting pedagogical approaches. However, students mentioned in interviews that they used the chatbot less for these topics because “they did not know how to formulate questions that would generate more applied or comparative responses.” Queries about Regulations and Legislation (8.66%) were limited, reflecting a lower interest in this area among students.

To analyze the relationship between chatbot usage and academic outcomes, students were grouped into four categories based on the number of interactions they conducted. This categorization was guided by pedagogical principles rather than statistical distribution, aiming to capture meaningful differences in how students engaged with the chatbot as a learning tool. The interaction ranges—” Low”, “Moderate”, “High”, and “Very High”—were defined based on observed patterns of usage, reflecting varying levels of engagement. For instance: The Low (1–10 interactions) category captures sporadic or minimal use, typical of students who relied on the chatbot occasionally as a supplementary resource. The Moderate (11–20 interactions) category represents balanced usage, where the chatbot complemented other study methods without dominating the learning process. The High (21–30 interactions) and Very High (>31 interactions) categories reflect increasingly intensive usage, with the latter potentially indicative of over-reliance on the tool (

Table 2. Relationship Between Chatbot Usage and Academic Performance.

Usage (Interaction Range)	Number of Students	Average Interactions	Final Exam Average Score (Range 1 to 10)
Low (1–10)	12	9.83	4.92
Moderate (11–20)	26	13.15	6.58
High (21–30)	2	22.00	8.00
Very High (>31)	4	50.00	6.00

).

This pedagogical framework prioritizes practical relevance, providing a nuanced understanding of how varying levels of engagement influence academic outcomes. Each group’s average final exam score was then compared to evaluate the potential impact of chatbot usage on academic performance.

Students who made moderate use of the chatbot (11–20 interactions) achieved an average score of 6.58, which was significantly higher than those with low use (1–10 interactions), whose average score was 4.92. Students with high use (21–30 interactions) reached the highest average score (8.00), although this group was small, making the results less generalizable. The group with very high use (>31 interactions) obtained an average score of 6.00, suggesting that excessive chatbot use does not guarantee better academic outcomes. This pattern indicates that moderate and strategic use of the chatbot aligns with optimal learning outcomes, while over-reliance may hinder independent study efforts.

Qualitative interviews provided deeper insights. Some students in the very high use group acknowledged that they relied too heavily on the chatbot for preparation, neglecting other forms of study and deeper engagement with materials. One student noted: “I think I relied too much on the chatbot to prepare, and in the end, I neglected other study methods, like reviewing my notes or digging deeper into the materials, which may have affected my results.” This excessive dependence on the chatbot seems to have limited their academic performance, as they failed to integrate it as a complement to more traditional study strategies.

The Pearson correlation coefficient analysis yielded a value of r = 0.186, p = 0.05, indicating a weak positive relationship between the number of interactions and exam scores. This analysis used continuous data for both variables (number of interactions and exam scores), which aligns with the assumptions of the Pearson correlation. Given the data’s normality and continuous nature, Pearson was deemed appropriate. Future studies with ordinal or non-normal data may consider the Spearman correlation as an alternative.

These findings suggest that while the chatbot can enhance learning when used strategically, its effectiveness diminishes with excessive reliance. Qualitative interviews reinforce this interpretation, as many students noted that although the chatbot was useful as a complementary tool, “it couldn’t replace reviewing notes or more in-depth study”. These insights emphasize the importance of integrating AI tools strategically to support rather than replace traditional study methods.

In summary, the quantitative and qualitative data show that the chatbot was mostly used to consolidate theoretical knowledge, particularly in the categories of definitions and detailed explanations. Moderate and balanced use of the tool appears to be associated with better academic outcomes, while excessive use offers no additional advantages and may reflect a reliance on the chatbot that limits other study methods. Overall, the chatbot is valuable as a complementary resource, but it cannot replace other study strategies.

3.3. Perceptions of the Chatbot’s Effectiveness and Usefulness

A key finding from the initial survey revealed that 51.2% of students had no prior experience using this technology in an educational setting. Therefore, it was crucial to analyze students’ perceptions after using the chatbot in light of their initial expectations. The expectations of a significant portion of users were based on a lack of familiarity, which may influence their final perception. This analysis helps identify possible discrepancies between what was expected and what was experienced, which in turn affects overall satisfaction with the tool. If expectations are met or exceeded, students are more likely to continue using it. On the other hand, a disconnect between expectations and perceived usefulness can lead to frustration, negatively impacting their evaluation.

Thus, we will first address the students’ initial expectations before the intervention and the perceived usefulness after using the chatbot. “Expectation” refers to students’ anticipated usefulness of the chatbot before the intervention phase, as measured through a pre-intervention survey. This included their thoughts on how the tool could support their learning and exam preparation. “Perceived usefulness”, in contrast, represents their evaluation of the chatbot’s effectiveness after the intervention, as captured in a post-intervention survey.

The impact of the chatbot on critical learning areas such as concept comprehension, content review, and exam preparation was analyzed. The following

Table 3. Comparison of students’ expectations and perceived usefulness of the chatbot.

Chatbot Usefulness Aspect	Expectations (%)	Perceived Usefulness (%)
Specific Question Resolution	84.2	91.4
Concept Comprehension	100	95.7
Practical Examples	42.1	61.4
Exam Training	52.6	45.4
Content Review	73.7	42.9
Motivation and Study Tips	10.5	0

summarizes the results of student opinions gathered before and after the intervention:

The area of specific question resolution received the highest ratings, with a perceived usefulness of 91.4%, surpassing initial expectations by 7.2%. This indicates that the chatbot was particularly effective when students asked clear and precise questions, allowing for direct and satisfactory responses. Comments such as “the chatbot provided the explanation I needed to understand a question and keep studying” highlight its ability to resolve specific issues and facilitate the study process.

In terms of concept comprehension, although there was a slight decrease from 100% to 95.7%, the chatbot still maintained a positive rating. Students praised the clarity of the responses, stating that the tool was helpful in defining key concepts. However, the slight drop suggests that some users expected a higher level of depth in the responses, likely influenced by the lack of precision in the questions asked, which impacted the interaction and the quality of the responses received.

The area of practical examples experienced a significant increase in perceived usefulness, rising from 42.1% to 61.4%. This increase reflects that when students formulated their questions correctly, the chatbot provided practical examples that helped consolidate learning. As one student explained: “The good thing about the chatbot is that if you have doubts about a concept, you can ask for examples, and it becomes much clearer.”

On the other hand, in exam simulation, the perceived usefulness decreased from 52.6% to 45.4%. This result contrasts with the technical tests, which showed an 88% effectiveness in exam simulations. Upon reviewing interactions, it was found that the issue lay in the low quality of the questions asked and the ineffective use of interaction strategies. The lack of precision in the queries prevented the chatbot from generating effective exam simulations, limiting its potential in this area. This highlights the importance of students’ proficiency in prompt engineering to maximize the chatbot’s performance.

In content review, perceived usefulness also saw a significant drop, from 73.7% to 42.9%. One student commented: “I expected the chatbot to help me review more, but the responses weren’t as useful.” Analyzing the interactions confirmed that the main obstacle was the inability to formulate questions that delved deeply into the topics, which affected the quality of the responses. Additionally, some students did not understand that the chatbot was a complementary tool to be used alongside more traditional study strategies. This lack of understanding may have limited the chatbot’s effectiveness in the review process.

Perceptions of motivation and study tips fell from an already low initial expectation of 10.5% to 0% after using the chatbot. This result shows that the tool did not meet expectations in this area, though it is important to note that the chatbot was not specifically designed to provide motivation or advice on study habits. This gap represents an area for improvement in educational chatbots. Incorporating features aimed at motivating and supporting study organization could significantly improve student engagement. By providing more comprehensive support, including both emotional and strategic guidance, chatbots could become even more valuable learning tools.

Overall, the results show that the chatbot was most effective in areas such as solving specific questions, understanding concepts, and exemplifications, as long as the questions were clear and well-structured. However, declines in areas such as content review and exam simulation reveal that a lack of precision in the questions negatively affected perceived usefulness. Furthermore, the lack of functionality related to motivation and study tips showed that the chatbot did not meet these important needs for some students, which could be improved in future versions.

In conclusion, the results highlight that students’ perceptions of the chatbot’s usefulness are heavily influenced by their ability to formulate effective questions, meaning their mastery of prompt engineering. Although students received a preparatory session, it was not enough to fully maximize the chatbot’s potential. The analysis of interactions reveals a high number of poorly formulated questions, which negatively impacted the quality of responses received and limited the tool’s effectiveness in several key areas.

Overall Perception of the Chatbot’s Usefulness

When asked whether they considered the chatbot to be an advantage for course preparation, the majority of students expressed a positive perception. Using a Likert scale (1 = not useful at all, 5 = extremely useful), 42.9% gave a score of 4, indicating that the chatbot was seen as a valuable, though not indispensable, resource. Another 35.7% rated the tool with the maximum score (5), suggesting that a considerable portion of the students found the chatbot very useful for their learning. However, 21.4% gave it an intermediate score (3), and 14.2% gave low scores (1 or 2), indicating that a minority did not find the chatbot particularly useful.

Interviews support these results. Some students highlighted the quick response time as one of the tool’s main strengths. One student noted: “It’s a very useful tool for resolving doubts quickly.” Another positive aspect mentioned was the autonomy the chatbot provided, allowing them to resolve doubts without relying directly on teachers. One student remarked: “The chatbot allows me to organize my study on my own without depending so much on the teachers and tutoring.” Additionally, several students said the chatbot enabled them to ask questions they wouldn’t feel comfortable asking in class due to embarrassment. One said: “Sometimes I don’t dare ask questions in class, but with the chatbot, I can ask anything without feeling bad”, indicating that the tool reduced emotional barriers in the learning process.

Another important finding is that both teachers and students observed a reduction in the need for personal tutoring and email queries. This highlights the chatbot’s role in fostering student autonomy, allowing them to manage their own learning and seek immediate answers to their questions without constant teacher intervention. This greater independence contributed to students feeling more confident in their exam preparation, allowing teachers to focus their tutoring on more complex topics or students who needed more personalized support.

However, some students pointed out limitations, such as the depth of responses on more complex topics. As one student stated: “The chatbot isn’t very useful for more difficult questions, where I need a deeper explanation.” This type of feedback reflects that while the chatbot was seen as a valuable tool, its usefulness was limited when it came to questions that required more detailed explanations or a more personalized pedagogical approach.

4. Discussion and Conclusions

This study evaluated the impact of a subject-specialized educational chatbot on exam preparation for second-year students in the Bachelor’s Degree in Early Childhood Education program. The results reveal that the chatbot was highly effective in resolving specific questions and improving concept comprehension, exceeding initial expectations in these areas. These findings align with previous studies that highlight the usefulness of chatbots in personalizing learning and providing immediate feedback, which reduces anxiety and improves academic performance (Ng & Leung, 2024; Holmes et al., 2019; Deng & Yu, 2023).

However, areas for improvement were identified, particularly in exam simulation and content review, where initial expectations were not met. The perceived usefulness in these areas was lower (42.9% for content review and 45.4% for exam simulation), suggesting a disconnect between what students expected and what the chatbot could offer. This limitation appears to be related to a lack of proficiency in prompt engineering, as students were unable to formulate questions effectively to obtain deeper and more applied responses. As Knoth et al. (2024) point out, success in interacting with AI systems like chatbots depends significantly on the user’s ability to generate clear and precise queries. In this educational intervention, the initial training in these skills was not sufficient to maximize the chatbot’s use. Future studies should provide more comprehensive training in prompting techniques and evaluate its impact on chatbot interactions.

Another important aspect is the relationship between chatbot use and academic performance. Students who made moderate use of the tool achieved better exam results (average score of 6.58) than those with low use (4.92). However, excessive use of the chatbot (> 31 interactions) did not lead to additional improvements, suggesting an over-reliance on the tool without a complementary focus on more traditional study strategies. These findings are consistent with research by Zhang et al. (2024), which shows that intensive use of educational technology does not always lead to better outcomes if not combined with traditional learning methods.

Additionally, the study highlighted the chatbot’s ability to foster study autonomy. Students valued the tool’s immediacy and accessibility, which reduced their dependence on tutoring and email queries. This increase in autonomy aligns with the principles of self-directed learning, as observed in previous studies on the use of chatbots in higher education (Labadze et al., 2023). However, the lack of functionalities that support motivation and study organization was noted as a significant limitation by students, suggesting that educational chatbots still have room for improvement in providing more comprehensive support.

The Technology Acceptance Model (TAM) serves as an insightful framework for analyzing these findings. According to TAM, the adoption of new technologies hinges on two key factors: perceived usefulness and perceived ease of use (Davis, 1989). In this study, students demonstrated a high level of satisfaction with the chatbot’s ability to address specific queries and enhance their understanding of concepts. However, the identified limitations highlight areas for enhancing perceived ease of use. Addressing this challenge could involve two complementary strategies: first, refining large language models (LLMs) to better interpret and respond to student requests with higher accuracy; and second, equipping users with improved skills in “prompt engineering” to effectively interact with the chatbot. These efforts can jointly contribute to a more effective and productive user experience.

5. Limitations and Future Research

While this study provides valuable insights into the impact of specialized educational chatbots, certain limitations should be acknowledged. The relatively small sample size, confined to a single academic context, may limit the generalizability of the findings to other courses, institutions, or educational levels. Furthermore, the students’ proficiency in question formulation (prompt engineering) significantly influenced the chatbot’s effectiveness, underscoring the need to explore more comprehensive training strategies in this area. Future research could focus on evaluating the impact of more extensive training programs in AI interaction, as well as developing chatbots with enhanced capabilities for interpreting less structured queries. Additionally, exploring the integration of features that support motivation, study organization, and students’ emotional well-being would be relevant. Finally, expanding the analysis to multicultural contexts and more diverse cohorts could provide a broader understanding of the educational applications of chatbots.