Custom-Trained Large Language Models as Open Educational Resources: An Exploratory Research of a Business Management Educational Chatbot in Croatia and Bosnia and Herzegovina

Abstract

This paper explores the contribution of custom-trained Large Language Models (LLMs) to developing Open Education Resources (OERs) in higher education. Our empirical analysis is based on the case of a custom LLM specialized for teaching business management in higher education. This custom LLM has been conceptualized as a virtual teaching companion, aimed to serve as an OER, and trained using the authors’ licensed educational materials. It has been designed without coding or specialized machine learning tools using the commercially available ChatGPT Plus tool and a third-party Artificial Intelligence (AI) chatbot delivery service. This new breed of AI tools has the potential for wide implementation, as they can be designed by faculty using only conventional LLM prompting techniques in plain English. This paper focuses on the opportunities for custom-trained LLMs to create Open Educational Resources (OERs) and democratize academic teaching and learning. Our approach to AI chatbot evaluation is based on a mixed-mode approach, combining a qualitative analysis of expert opinions with a subsequent (quantitative) student survey. We have collected and analyzed responses from four subject experts and 204 business students at the Faculty of Economics, Business and Tourism Split (Croatia) and Faculty of Economics Mostar (Bosnia and Herzegovina). We used thematic analysis in the qualitative segment of our research. In the quantitative segment of empirical research, we used statistical methods and the SPSS 25 software package to analyze student responses to the modified BUS-15 questionnaire. Research results show that students positively evaluate the business management learning chatbot and consider it useful and responsive. However, interviewed experts raised concerns about the adequacy of chatbot answers to complex queries. They suggested that the custom-trained LLM lags behind the generic LLMs (such as ChatGPT, Gemini, and others). These findings suggest that custom LLMs might be useful tools for developing OERs in higher education. However, their training data, conversational capabilities, technical execution, and response speed must be monitored and improved. Since this research presents a novelty in the extant literature on AI in education, it requires further research on custom GPTs in education, including their use in multiple academic disciplines and contexts.

1. Introduction

The motivation for this study can be found in the extremely quick development of Artificial Intelligence (AI) tools and their application in higher education, which especially applies to Large Language Models (LLMs). This study has followed the introduction of a highly anticipated LLM feature, enabling end-users of OpenAI tools, such as ChatGPT 4, to create custom-trained Generative Pretrained Transformer (GPT) chatbots. Its immediate objective is to determine how custom-trained LLMs could serve the role of Open Educational Resources (OERs) in higher education, specifically within e-learning. As far as we know, no similar studies in the extant literature might inform education staff and administrators in higher education of the potential strengths and challenges of this emerging technology.

Introducing commercial LLM functionality to train custom LLM models focused on a specialized field provides new opportunities for implementing Artificial Intelligence (AI) without coding or developing machine learning and algorithm competencies. Such developments make AI tools accessible to a wide range of educators at all levels of education, aiming to develop virtual teaching assistants with specialized training. Although such a proposition sounds attractive, it is still unknown how much training by field-specific data will contribute to the success of custom LLMs in educational settings [1].

It is important to note that the discipline-specific training of chatbots does not interfere with the generic capabilities of LLMs used by their underlying models. For example, ChatGPT provides the full functionality of its ChatGPT 4 model in the Custom GPT mode, which can be trained and used for a domain-specific purpose. At the time, when the paper had been conceptualized, and the initial version was written, opportunities for implementing specialized AI chatbots were relatively limited. Custom GPTs were not made available to free users of the ChatGPT service, who were restricted to the less powerful ChatGPT 3.5 model. Even with the relatively modest cost of the membership plan, such charges made it impossible for a broader audience of life-long learners to afford AI tools, especially in low-income countries. Other commercial LLMs (including Google Gemini, Anthropic Claude, and others) did not offer the functionality for custom training of LLM-based chatbots.

The other option available to education professionals was related to building a ChatGPT-powered assistant, which can be performed easily without programming skills. At the time of revising this paper, this option is still viable but requires a significant institutional budget. Namely, the educator is considered a software developer and charged per usage of their AI chatbot.

These requirements limited the implementation of custom-trained LLM chatbots without using one of the open-source AI models and datasets. Following such a route ensured that the specialized AI assistant served as a true Open Educational Resource (OER). However, a relatively high level of ICT competencies was required for its development. Therefore, when conceptualizing this paper, we decided to use a commercially available service (Botsonic) (a free test version of our chatbot is available at https://bot.writesonic.com/share/bot/f49db0d8-7e43-4d93-81fa-4e916fb88383, accessed on 7 April 2024), enabling individual content creators to create custom ChatGPT-based AI chatbots and freely test them before mass deployment.

AI tools and technologies are rapidly developing. The current OpenAI model ChatGPT 4 Omni (GPT-4o), launched to the public in May 2024, introduced significant changes to the relevant licensing and usage practices. While the public attention was focused on the GPT-4o multimodal capabilities (i.e., reasoning and communicating across video, audio, and text content), the custom GPT licensing also changed. Since GPT-4o became available to free users in May 2024 (although with a usage cap), custom GPTs can now be accessed by free users. This makes the popular Open AI GPT platform relevant for developing and implementing custom-GPT-based OERs without third-party tools. Since the latest changes have been announced, our free test version of the chatbot (in Croatian language) has been also made available through the ChatGPT platform at https://chat.openai.com/g/g-2NLU5DM8D-menadzerski-asistent, accessed on 6 June 2024).

However, the dynamic nature of the AI field might mean that the current licensing and usage patterns might change again soon.

This study aims to determine the initial student perceptions of a custom-trained LLM to answer the following research question:

• How efficiently can custom Large Language Models (LLMs) serve as Open Educational Resources (OERs) in higher education, specifically business management?

Since we have not found a single study on custom GPT evaluation, we believe our results will be useful to direct further research in the field. The novelty of this research topic and the rapidly evolving technological background of the AI field require that the research follows the AI developments as quickly as possible. This is the rationale for choosing a limited and applicative research question and using the exploratory approach in this study.

2. The Role of Large Language Models (LLMs) as (Open) Educational Resources

Using algorithms and statistical models to process and analyze extremely large datasets, LLMs can recognize patterns, make predictions, or generate responses based on the data they are trained on [2]. They enhance knowledge discovery by summarizing research papers, generating relevant research questions, and identifying relevant sources [3]. Unethical students can misuse this LLM capability to generate entire course assignments, which requires a new approach to academic assessment methods [4]. However, their output still requires human experts’ evaluation [5], as LLMs might ‘hallucinate’, i.e., provide coherently and convincingly sounding but rather misleading or inaccurate information [6]. Human experts also need to ensure that AI technologies and content do not reflect the biases and ethical concerns that potentially exist in their training materials and that their implementation aligns with the values of diversity, fairness, and inclusivity [7]. Other ethical issues related to data privacy and transparency also require the attention of faculty and other actors in educational settings [8,9].

A widely recognized LLM capability is related to the personalization of learning materials and delivery methods, which can be customized to reflect individual needs, learning styles, and the level of professional knowledge [10,11]. Even individual users could find LLMs useful to support them in life-long learning (LLL), as endorsed by policies on both the European Union and global scales [12]. These LLM capabilities, if adequately implemented, could empower educators to tailor educational resources to specific topics or course requirements and enable faculty to devote more time to other relevant tasks [13].

LLMs can enhance the learning experience by providing personalized feedback, positively impacting student understanding and engagement [14]. This capability can provide guidance and support, helping students overcome barriers to learning and achieving academic milestones [15]. By serving as virtual teaching assistants, LLMs provide real-time assistance, query resolution, and guidance, enriching students’ learning experiences, especially in online or hybrid learning environments [16]. They can enhance language skills and proficiency [17], enabling multicultural student groups to collaborate effectively [18]. Future custom LLMs may develop into comprehensive student support platforms, offering various services, from academic guidance to career counseling [19].

LLMs offer multiple opportunities to create Open Educational Resources (OERs), ultimately democratizing academic teaching and learning [20]. Due to their potential to generate high-quality educational content in various disciplines, LLMs have the potential to provide free access to educational materials to non-native speakers and students from different language and cultural environments. However, it seems that LLMs can only be used in a limited capacity, as the full automation of teaching and learning is still outside of their current level of development [21].

Educators may be able to adapt LLM-based OERs to fit their curriculum, teaching methodologies, and student needs, fostering a more personalized and engaging learning experience [22,23]. Custom training an LLM is the process of fine-tuning a pretrained LLM model with a specific corpus of training materials by using the following steps:

• Collection and preparation of data: A diverse corpus of training materials representing the intended learning outcomes’ scope, including textbooks, class notes, scholarly articles, and other training materials. The corpus should be cleaned and preprocessed to ensure relevance and quality. Training materials must also formally assess the ethical aspects to ensure diverse perspectives and arguments.
• Model training and fine-tuning: After collection, the dataset has to be used to fine-tune a generic LLM, usually a downloadable open-source model, to make it context-aware for specific academic teaching and learning scenarios.
• Continuous evaluation and updating: After deployment, the model has to be monitored continuously and evaluated for performance. The feedback needs to be used for iterative improvement of the model to remain helpful, accurate, and effective.

Creating and implementing an Open Educational Resource (OER) based on a custom GPT involved the following:

• Ensuring accessibility and scalability: The AI chatbot has to be accessible to students and educators, which can be achieved by integrating it into frequently used educational platforms and LMSs or providing a simple web interface.
• Customizing model to individual needs: Key aspects of the AI chatbot acceptance are adaptability and customization to different educational needs and cultural contexts to fit particular teaching goals or the needs of students.
• Addressing ethical concerns and transparency: Among ethical concerns are privacy, consent bias, and lack of transparency, which need to be addressed from the beginning of the process, i.e., the training material selection stage.

LLM-based OERs may foster a sense of community among educators and learners, encouraging knowledge sharing and co-creation [24]. In addition, by incorporating real-time data and insights, LLMs could ensure the OER’s relevance and effectiveness, promote knowledge exchange, encourage innovation, and reduce barriers to accessing high-quality educational resources [25]. By leveraging custom GPTs to generate OERs, educational institutions reduce the costs associated with content creation and acquisition and reallocate resources towards other educational priorities, such as student support services and infrastructure improvement [15].

3. Methods

We developed the analyzed chatbot in two subsequent steps. First, we developed a conceptual version of the chatbot using the OpenAI custom GPT functionality. We used the relevant OpenAI guidelines in formulating our prompting strategy [26], available in Appendix A. The chatbot was designed as an educational assistant in Croatian, aiming to assist 2nd-year students in the undergraduate programs of business schools in Croatian-speaking universities in Croatia and Bosnia and Herzegovina. Its domain-specific knowledge is based on the prescribed readings from business management authored by a group of regional scholars from Croatia and Bosnia and Herzegovina [27]. Since we could not test the chatbot using the ChatGPT environment due to the previously described limitations, we implemented it using the Botsonic platform. We distributed the link to its test implementation to four academic experts from Croatia and Bosnia and Herzegovina.

Our research approach used a mixed method design, combining qualitative and quantitative methods in two stages. This sequential exploratory method is especially valuable for emerging topics requiring in-depth exploration through qualitative methods before being more precisely assessed (using quantitative methods). The first stage involved collecting and reviewing interview information and informing the development of the following phase based on a previously verified instrument (survey).

In the first stage, we used nonrandom sampling to select four experts specialized in e-learning methods and their evaluation, business administration (management) teaching, and the role of technology in business marketing. The stratified purposeful approach used in expert selection (i.e., assuring that at least one local expert represents each of the fields considered relevant for the scope of the study) is considered appropriate for the exploratory nature of this study.

The experts were presented with five questions, developed using dimensions for evaluations of chatbots and other intelligent agents, previously suggested by Radziwill and Benton, i.e., Denecke et al. [28,29]. These were as follows:

• How can the quality and accuracy of the information provided by the chatbot be ensured?
• How does a chatbot handle complex inquiries?
• How adaptable is the chatbot to user requirements?
• How does the chatbot integrate with other tools for learning business management?
• How do we measure the success of a chatbot in improving business management learning?

After transcribing the interviews (available in Appendix B), we used the experts’ evaluation of our test chatbot to develop a summary report presented in the next part of the paper. We also used the results of this research stage to modify the items of the Bot Usability (BUS-15) Scale, originally developed and empirically verified for evaluation of Customer Relationship Management (CRM) AI chatbots [30,31].

BUS-15 Scale is grounded on the theoretical dimensions of perceived accessibility, quality of information, and functions provided by the chatbot, including conversational experience, perceived privacy and security, and response time. These consider Borsci et al.‘s [30] importance given to the relevance and accuracy of responses and the extent to which chatbot keeps the conversational context. These are essential attributes required to meet user requirements, as detailed in the BUS-15 scale validation. The scale contains specific items measuring chatbot interaction, including response time. They consider the accessibility dimension, ensuring effective interaction with users, regardless of specific user characteristics or the context in which a chatbot is used. In addition, the human–computer interaction (HCI) context, concerned with trust and security aspects [32], addresses the inclusion of privacy and security aspects into chatbot assessment.

The modified survey (see

Table 1. Adaptation of the BUS-15 scale.

Original Bus-15 Scale Items	Modified Items of the Scale for Business Management Chatbot Evaluation (Based on Input from Four Academic Experts)
Perceived accessibility to chatbot functions
1. The chatbot function was easily detectable.	Items are not relevant since the AI chatbot was directly presented to students.
2. It was easy to find the chatbot.
Perceived quality of chatbot functions
3. Communicating with the chatbot was clear.	Adopted directly.
4. I was immediately made aware of what information the chatbot can give me.	Adopted directly.
5. The interaction with the chatbot felt like an ongoing conversation.	Adopted directly.
6. The chatbot was able to keep track of context.	Adopted directly.
7. The chatbot was able to make references to the website or service when appropriate.	The chatbot was able to make references to respected websites or scholarly articles.
	The chatbot provides answers aligned with the teaching materials (textbook) and other courseware.
	The chatbot can comment on complex topics and connect complex topics and issues from the field of business management.
	Information provided by the chatbot is of better quality than the generic ChatGPT, Google Gemini, or the other Large Language Model I use regularly.
8. The chatbot could handle situations in which the line of conversation was not clear.	Adopted directly.
9. The chatbot’s responses were easy to understand.	Adopted directly.
Perceived quality of conversation and information provided
10. I find that the chatbot understands what I want and helps me achieve my goal.	Adopted directly.
11. The chatbot gives me the appropriate amount of information.	Adapted: The chatbot gives me the appropriate information to understand basic concepts from business management.
	The chatbot gives me the appropriate information to understand complex concepts from business management.
12. The chatbot only gives me the information I need.	Adopted directly.
13. I feel like the chatbot’s responses were accurate.	Adopted directly.
Perceived privacy and security
14. I believe the chatbot informs me of any possible privacy issues.	Items are irrelevant since the AI chatbot does not collect or use personal information.
Response time
15. My waiting time for a response from the chatbot was short.	Adopted directly.

Source: Adapted from [30,31], based on research results.

) was translated to Croatian (see Appendix C) by two experts in academic translation, specialized in business management, and uploaded to the online Qualtrics XM platform for electronic data collection. All items were measured on the standard 5-item Likert scale, with answers ranging from ‘1—Strongly disagree’ to ‘5—Strongly agree’, comparable to the original BUS-15 Scale. No personally identifiable data, including student names or other IDs, were collected.

This survey was used as a research instrument in the subsequent quantitative evaluation of our test chatbot, with the full-time undergraduate students enrolled at the Faculty of Economics, Business and Tourism at the University of Split, Croatia (EFST) and the Faculty of Economics at the University of Mostar, Bosnia and Herzegovina (EF SUM). The survey was implemented by using the licensed Qualtrics XM electronic survey system. We did not collect any data enabling the personal identification of survey participants. The anonymous data were collected with the permission of the institutional ethics board and with the students’ consent to use their anonymous data for academic research.

We distributed the links to the Qualitrics survey and the test implementation of the LLM chatbot for business management learning to the entire population of second undergraduate-year students at the two schools using the Moodle Learning Management System (LMS). All the students were previously enrolled in the Fundamentals of Management course, which did not involve using LLMs or other AI tools. There were no other previous formal attempts to use intelligent agents, LLMs, or other AI tools in academic teaching and learning at the two schools, which ensures that respondents could not be influenced by the success of similar initiatives in their academic environment.

Data collection lasted three weeks during the winter semester of the academic year 2023/2024. Our final sample size of 204 respondents was based on a nonrandom approach, using the specific method of snowball sampling, as we also specifically asked students to recommend filling in the survey to their colleagues at the two targeted business schools. As previously suggested in the research methods literature [33], this sampling approach is effective for studies limited by time and resources, as it builds upon the existing social connections and trust provided by the immediate social environment referrals. Weaknesses of the method are related to the sensitivity of research results to the availability and motivation of newly recruited respondents. The nonrepresentative nature of the sample design does not guarantee the representativeness or the results free from biases. However, it is a good starting point for applying quantitative methods in exploratory research designs.

It should also be noted that the nature of the sample limits our research, i.e., the population of young adults enrolled in the second and third year of undergraduate business studies at two regional business schools in Southeast Europe. We did not collect detailed demographic information about the survey participants.

Statistical analysis was conducted in IBM SPSS Statistics, version 25 (Armonk, NY, USA: IBM Corp.). Results are presented as mean and standard deviation. Cronbach’s alpha coefficient checked the internal reliability of the used instrument. Pearson’s correlation coefficient investigated the connection between the dimensions. The significance of the obtained results was analyzed at significance levels of 0.05 and 0.001.

4. Results

4.1. Qualitative Analysis of Expert Interviews

The four experts summarized perspectives on the effectiveness and quality of the chatbot as follows:

• Expert A expressed concerns about the quality and correctness of the LLM responses and referred to the domain-specific data quality used for training. They believed the chatbot was not up to the quality and accuracy standards when assessing complex concepts and their relationships.
• Expert B focused on the performance of the LLM and indicated that it seems a custom LLM model is underperforming compared with the generic LLM model (without custom training).
• Expert C agreed with Expert B on the quality and accuracy of responses and the depth of the management topics covered, which are the disadvantages of the custom LLM model.
• Expert D observed that though most of the answers are correct, they are more general. They suggested that the tool is more appropriate for explaining simple concepts than efficiently answering complex queries.

Considering the expert insights, the LLMs can be evaluated as generally possessing the potential to enhance the learning experiences in business management education and serve as a useful tool to build Open Educational Resources (OERs). Four main themes could be identified in the experts’ interviews: concerns about the quality and accuracy of chatbot output, chatbot adaptability, integration with educational tools, and measurement of chatbot success:

• Concerns regarding quality, accuracy, and adaptability emerged, with some experts expressing their reservations about the depth and specificity of the chatbot-generated output.
• A concern for adaptability was also noted, with varying views on how successful custom LLM models will serve users’ needs effectively.
• Integration with educational tools was highlighted, especially with the LMSs (Learning Management Systems) and other educational platforms, to increase accessibility and student engagement.
• Experts mentioned a variety of metrics and methods related to measuring chatbot success, focusing on user engagement, satisfaction, learning outcomes, etc. Methodological approaches suggested for evaluating such AI tools included measurement of learning outcomes and quasi-experimental studies.

4.2. Quantitative Analysis of Student Perceptions

In the second stage of the empirical research, we used the adapted research instrument. We distributed the link to the electronic version of the survey to the entire student body of the 2nd year of undergraduate studies of business and management at the Faculty of Economics, Business and Tourism at the University of Split, Croatia (EFST) and the Faculty of Economics at the University of Mostar, Bosnia and Herzegovina (EF SUM).

A total of 204 responses were obtained: 151 (74%) from EFST and 53 (26%) from EF SUM. The scale’s overall internal consistency proved adequate since Cronbach’s alpha value for the entire scale was 0.904. A relatively high value, much higher than the generally accepted threshold of 0.7, shows that the used research instrument can be internally consistent. Descriptive statistics for all questionnaire items provided in

Table 2. Descriptive statistics of chatbot evaluation items.

	N	Min	Max	Mean	St.Dev.
Perceived quality of chatbot functions
Communicating with the chatbot was clear.	204	1	5	4.22	0.647
I was immediately made aware of what information the chatbot can give me.	204	2	5	4.24	0.638
The interaction with the chatbot felt like an ongoing conversation.	204	1	5	3.69	0.887
The chatbot was able to keep track of context.	204	1	5	4.03	0.752
The chatbot was able to make references to the respected websites or scholarly articles.	204	1	5	3.67	0.862
The chatbot provides answers aligned with the teaching materials (textbook) and other courseware.	204	1	5	3.85	0.750
The chatbot is able to comment on complex topics and connect complex topics and issues from the field of business management.	204	1	5	3.89	0.780
Information provided by the chatbot have better quality, compared with the generic ChatGPT, Google Gemini, or the other Large Language Model I am using regularly.	204	1	5	3.51	0.985
The chatbot could handle situations in which the line of conversation was not clear.	204	1	5	3.50	0.879
The chatbot’s responses were easy to understand.	204	2	5	4.20	0.674
Perceived quality of conversation and information provided
I find that the chatbot understands what I want and helps me achieve my goal.	204	2	5	4.17	0.660
The chatbot gives me the appropriate information to understand basic concepts from business management.	204	1	5	4.12	0.753
The chatbot gives me the appropriate information to understand complex concepts from business management.	204	1	5	4.06	0.734
The chatbot only gives me the information I need.	204	1	5	3.71	0.889
I feel like the chatbot’s responses were accurate.	204	1	5	3.98	0.746
Response time
My waiting time for a response from the chatbot was short.	204	1	5	4.30	0.815

clearly show the following:

• Students’ feedback on the clarity of responses, their awareness of the information available from the chatbot, the immediacy of information delivery provided, and the conversational quality had high scores (with means scattered around the scale point of 4.0, on a five-level Likert scale).
• The chatbot’s ability to maintain the conversation’s context and provide relevant references was also highly regarded by students, although higher standard deviations show a higher level of variability.
• Unlike experts, students did not evaluate the chatbot responses as significantly worse than those of the generic LLMs. However, this could be attributed to their limited experience using LLMs and other AI tools in higher education settings.
• Users also appreciated the chatbot’s response accuracy and quick response time.

When compared across three dimensions (see

Table 3. Descriptive statistics of chatbot evaluation dimensions.

Chatbot Evaluation Dimensions	Mean	St.Dev.	Cronbach’s Alpha
Perceived quality of chatbot functions	3.88	0.51	0.84
Perceived quality of conversation and information provided	4.00	0.58	0.82
Response time	4.30	0.81	n/a

), there is a positive evaluation of the chatbot functions (with a mean value of 3.88), with a somewhat better evaluation of the quality of conversation and information provided (with a mean value of 4.00). Students were especially satisfied with the response time, achieving a mean value of 4.30. Simultaneously, standard deviations and mean scores for each of the previously mentioned evaluation dimensions increased, indicating varying expectations related to the quality of conversation, LLM output, and response time. Internal consistency, measured by Cronbach alpha values, is also acceptable for the two relevant dimensions, consisting of multiple items.

The analysis of linear correlations (see

Table 4. Pearson linear correlation coefficients among the chatbot evaluation dimensions.

Dimensions	Perceived Quality of Chatbot Functions	Perceived Quality of Conversation and Information Provided	Response Time
Perceived Quality of Chatbot Functions	1.00	0.76 **	0.58 **
Perceived Quality of Conversation and Information Provided		1.00	0.51 **
Response Time			1.00

** Significant at the 1% level.

) shows a high level of mutual dependencies among the dimensions of chatbot evaluation, including custom LLM functionalities, conversational quality, and response time, as demonstrated by a high level of statistical significance. There is a very strong correlation (0.76) between the perceived functionalities and the perceived quality of chatbot output and its conversational performance. According to the previous empirical results, improvements in the knowledge base used for training and the chatbot technical aspects are likely to impact students’ views on the quality of chatbot interactions. Students also expect that response times will enhance as chatbot capabilities improve (as indicated by a Pearson value of 0.58), although the perceived correlation is not as strong. The weakest correlation (0.51) exists between the response time (efficiency) and students’ perception of chatbot conversation and information quality. This empirical result suggests that response time significantly shapes the user experience with the chatbot.

According to these empirical findings, chatbot optimization must be approached holistically. Improving technical features, chatbot knowledge base, and its relevance, and reducing response times work together to enhance user satisfaction with custom LLM chatbots. A few students were providing an answer to an additional, open-ended question positioned at the very end of the research instrument. They expressed a generally positive evaluation of the chatbot. They suggested several potential improvements, including the need for faster responses, regular content updates, and the implementation of custom LLM digital assistants for other business school classes.

5. Discussion

This study is one of the first attempts to explore the potential of custom-trained LLMs in higher education. There is a pressing need to contribute to the research of widely available, specialized machine learning and its contribution to creating Open Educational Resources (OERs) since the extant literature does not cover this topic. We used the mixed-method approach to obtain qualitative feedback from domain experts (regional faculty with extensive experience in teaching business administration and implementing Information Technology in higher education) and quantitative feedback from undergraduate business and economics students.

While experts had concerns about the quality and contextualization of the chatbot output, this result of the empirical research can be attributed to their pedagogical experience and the perceived limitations of the AI tools compared with the competencies of experienced faculty. Simultaneously, students seem to appreciate the convenience of using a custom AI chatbot as an OER. Although this purpose has technical and commercial limitations, previously described in the introduction section, students’ initial reactions are very positive.

This study is the first exploratory research in the extant literature. Although its results are not generalizable across different subject fields and student populations, there is an interesting dichotomy between student evaluations and expert opinions due to limitations in the sample and a simplified approach to implementing the test AI chatbot. While the AI chatbot for business management learning seems to serve its purpose for the students, who might be looking for a convenient tool for test preparation and quick assessment of their knowledge, academic experts are looking at the pedagogical merits and the reasoning demonstrated by the AI tool. They are not completely acceptable to the experienced experts. At the same time, the depth of output and the quality of arguments are not comparable to the faculty with many years of teaching experience. However, even in a very early stage of development, a quickly developed custom LLM proved to serve as a useful OER to undergraduate students at two regional business schools in Southeast Europe.

Research results might also guide the faculty and other actors wishing to design and improve the OERs using the custom LLM tools. Within our limited sample, the functional and conversational capabilities of the test chatbot and its response speed proved to be highly correlated, which shows the need to simultaneously consider all aspects of students’ user experience and improve the chatbot by using a holistic approach.

6. Conclusions

This study explored a pilot project using the AI chatbot, which was implemented to assist business school students in business management. The piloted AI chatbot could also prove useful in improving higher education practices. This especially applies to sustainable educational practices that include nontraditional students and promote lifelong learning opportunities. AI tools enable flexible learning solutions and improve access to higher education, thus contributing to the United Nations Sustainable Development Goal (UN SDG4). According to our empirical results, such an approach creates value perceived by students. However, the consulted experts indicated several limitations concerning the quality of the generated content and how the chatbot handles complex queries. These limitations must be addressed when developing AI tools for higher education and lifelong learning at the academic level.

6.1. Theoretical and Practical Contributions

Our findings can be used to develop a comprehensive evaluation model of AI chatbots in education. We point out the theoretical dimensions of learner–AI interaction, including perceived quality, privacy/security, and accessibility. From practitioners’ point of view, we demonstrate the viability of commercial AI tools, enabling education (and other) professionals to train and deploy custom-trained LLMs as tools of choice for academic teaching and learning. We also identify current issues with AI tools to create open-access educational materials and approaches.

Our approaches contribute to UN SDG4, allowing higher education institutions to facilitate nontraditional and lifelong learning paths. Our results align with lifelong learning incentives [34,35] and practices in implementing advanced educational technologies [36]. In addition, compared with traditional (printed) learning resources, AI-based OERs have a minimal ecological footprint. This assures resource efficiency and environmental sustainability, both implied under SDG12 on sustainable consumption and production.

We also believe that integrating AI tools, such as the described business management chatbot, can help make educational programs more relevant and address the demands of the labor market. Custom-trained LLMs can be continuously updated to include emerging knowledge and skills required by the workforce, thus improving the employability of graduates and contributing to SDG8 on inclusive and sustainable economic growth, including full and productive employment for the current student generations [37].

6.2. Limitations and Future Research Directions

Limitations of this study are related to the specific educational context analyzed and the limited range of chatbot functionalities tested. Future studies should consider the use in multiple academic disciplines and additional educational contexts, including lifelong learning scenarios. More advanced AI features should be considered, including analysis of student sentiment and adaptive responses to the current level of students’ knowledge and competencies. Integrating multimedia elements and a multimodal learning approach (based on a voice or a video chat) could also enhance the interactivity and learners’ motivation to use the AI chatbot. All those functionalities would contribute to a completely personalized learning experience customized to the student’s context and educational goals.

Research questions, which future research could explore, are as follows:

• Are custom Large Language Models (LLMs) appropriate for developing Open Educational Resources (OERs) in higher education, regardless of the discipline and academic context?
• Can custom LLMs be used in the context of Lifelong Learning in higher education?
• What is the role of advanced custom LLM features, such as the analysis of student sentiment and adaptability to student requirements, in their role as effective teaching and learning tools?
• What is the role of advanced custom LLM technical features, such as reasoning and communicating across text, audio, and video, in their role as effective teaching and learning tools?

6.3. Implications for Educational Practice

The practical implications of these results are relevant for improving chatbot analysis and as potential guidelines in the design of AI chatbots within other academic disciplines. The most important takeaway lesson is that the chatbot development process has to be iterative and based on student feedback. Further training content has to be identified for use in custom training to meet the handling of complex student queries. Context understanding and interaction can be achieved through better cooperation with AI experts who may guide through better tools and approaches toward chatbot optimization.