Educational AI Chatbots for Content and Language Integrated Learning

Abstract

Using advanced artificial intelligence (AI) technology in learning environments is one of the latest challenges for educators and education policymakers. Conversational AI brings new possibilities for alternative and innovative Information and Communication Technologies (ICT) tools, such as AI chatbots. This paper reports on field experiments with an AI chatbot and provides insights into its contribution to Content and Language Integrated Learning (CLIL). More specifically, this paper presents an experimental use case of an educational AI chatbot called AsasaraBot, designed to teach high school students cultural content in a foreign language, i.e., English or French. The content is related to the Minoan Civilization, emphasizing the characteristic figurine of the Minoan Snake Goddess. The related chatbot-based educational program has been evaluated at public and private language schools in Greece. The findings from these experiments show that the use of AI chatbot technology for interactive ICT-based learning is suitable for learning foreign languages and cultural content at the same time. The AsasaraBot AI chatbot has been designed and implemented in the context of a postgraduate project using open-source and free software.

1. Introduction

In schools in developed countries, the traditional tutor-centered approach in which students are in a constant effort to keep up, retain information, and cope with the rapid pace of the modern educational process still dominates. However, such an approach cannot adequately address the challenges of the digital era, which requires special knowledge and skills from students and tutors to exploit the capabilities offered by Information and Communication Technologies (ICT). Therefore, tutors should use modern ICT tools to improve, and possibly optimize, the learning experience of their students.

In this context, the use of trending technologies has a new, significant direction potential. An important emerging educational technology is based on conversational AI, implemented as so-called AI chatbots. AI chatbots are intelligent systems/applications that are able to interact with humans in various aspects of daily life using natural language (NL). They can be found mainly as personal assistants in business customer support. In the educational context, AI chatbots can play the role of intelligent tutors by presenting educational material, stimulating dialogue, providing feedback to students, etc. In some cases, AI chatbots can also play a complementary/supportive role to human tutors by answering students’ questions and providing guidance in a 24/7 timespan, something that is evidently impossible or unprofitable to implement with human tutors.

This paper reports on recent research related to the design, implementation, and evaluation of an educational AI chatbot, aiming to support the modern methodological learning approach of Content and Language Integrated Learning (CLIL) [1], i.e., the simultaneous learning of both content (e.g., a cultural subject) and foreign languages (e.g., English and French). Specifically, in a CLIL approach, students are taught a specific subject in a foreign language instead of their mother tongue. Activities based on the CLIL method are mainly designed to allow language and content of a particular subject to be learned at the same time. More specifically, in this paper, a real use case of an educational AI chatbot called AsasaraBot is presented, which complements human tutors in teaching cultural content related to the subject of the Minoan Civilization, emphasizing the characteristic figurine of the Minoan Snake Goddess. The related chatbot-based educational program was evaluated in English and French as a foreign language at a public Lyceum and in two private language schools in Greece. Findings from the related experiments show that the use of AI chatbot technology for interactive ICT-based learning can effectively support the CLIL method for learning a foreign language and cultural subject at the same time. The AsasaraBot has been designed and implemented using free software, i.e., the Snatchbot platform.

The contribution of the work presented in this paper is the following: it introduces AI Chatbot technology to support the modern learning method of CLIL, mainly playing the role of an ICT-based tutor and secondarily the role of an ICT-based assistant of a human tutor. The proposed approach has been implemented and evaluated in English and French as foreign languages for Greek students, reporting on the effectiveness and potential of the approach. To this end, an educational program has been specifically designed and implemented according to the needs of distance ICT-based learning, due to the COVID-19 pandemic.

The structure of this paper is as follows: Section 2 provides background knowledge regarding the utilized technology, i.e., AI chatbots. Section 3 describes the proposed approach, and Section 4 presents its evaluation in a real setting, i.e., in foreign language classes. Section 5 provides a discussion of the research issues and challenges of the chatbot-based educational approach. Finally, Section 6 concludes the paper.

2. Preliminaries and Related Work

Conversational AI refers to AI techniques (e.g., natural language processing, machine learning, knowledge graphs) for engineering intelligent software agents that can be engaged in natural language conversations with humans or other agents. It includes software, such as voice assistants and chatbots. The first generation of this technology focused on short task-oriented dialogue [2], such as music playback (e.g., “Alexa, play music”) or information retrieval (e.g., “Cortana, what is the weather today in Athens?”). The current challenge in this area is a constant, coherent, and appealing dialogue, as current software is still far from being able to have natural everyday conversations with humans [3]. The chat approach is designed in such a way that it gives human users the impression that they are naturally chatting with another human.

Chatbots can play an important role in the field of education because they are an interactive mechanism compared to traditional e-learning systems [4]. Among other ICTs, chatbots are considered safe and accessible learning tools that can bring positive results in learning. One of the reasons that chatbots are popular in various businesses is the increased efficiency of the process. This argument is also used in education as it increases user–student satisfaction by speeding up response times and can be available 24 h a day to answer or clarify any question [5]. This allows teachers, for example, to avoid answering repetitive questions that can be easily answered through chatbots, while also supporting students who have missed one or more lessons [6].

In addition to simply providing the question–answer function or exchanging information between students and the system, a chatbot contributes to tackling the problem of supporting/managing several individual student cases. During a lesson, the teacher usually does not have enough time for formative feedback to each individual student [5]. However, formative feedback during learning processes is, according to Hattie [7], one of the most important factors in increasing students’ performance and motivation. An AI chatbot has the ability to assist each student individually, following different learning paces, to provide them with knowledge according to their individual cognitive level.

In addition, chatbots seem to allow students to receive individual support in a bug-friendly environment, giving them the opportunity to apply their cognitive skills anytime and anywhere. This reduces stress and increases the willingness to learn [8]. Furthermore, a chatbot can be considered an excellent data analysis tool. Shawar and Atwell [9] state that it is important to remember that the teacher is the backbone of the teaching process, and that learning technology can act as an enhancer rather than a substitute. For example, when the chatbot is used to answer students’ questions, the teacher can use the conversation logs and statistics reporting to see where the students are having problems and what their weaknesses are. Thus, the teacher can use this technology to look for problems, while students use it to solve them.

Despite the aforementioned advantages, there are still communication problems between chatbots and humans. For example, limitations are found in the ability to correctly recognize natural language (especially in languages beyond English, such as Greek), difficulties in the flow of dialogue [10], inability to control repetitive sentences, and difficulty in dealing with unknown sentences [11]. Another disadvantage is that chatbots are often unable to answer complex or unexpected questions, as a result of which, they misinterpret what the student says and produce inappropriate dialogue flows [12]. This can result in student frustration, who will then seek other means of learning. Finally, chatbots are considered predictable, redundant, and have a bad memory, as they find it difficult to memorize previous conversations, forcing users to ask the same question repeatedly [13]. This may upset the user due to the greater effort required.

The increasing use of AI chatbot technology has influenced the way modern ICT-based teaching and learning is designed and implemented. Freudbot (https://psych.athabascau.ca/html/Freudbot/Freudbot.html, last accessed 15 January 2022) is an example of a chatbot developed for the educational purposes of psychology students, and it is used as a teaching and learning tool in distance and online education. Through Freudbot, students are taught about the theories, concepts, and biographies of the famous psychologist Freud [14]. Another example is Ethnobot (https://www.designinformatics.org/research_output/the-ethnobot/, last accessed 15 January 2022), which plays the role of a human ethnographer who collects ethnographic data and asks users a series of conversational questions [15]. Chatbots have been developed for learning foreign languages, such as Cleverbot (https://www.cleverbot.com/, last accessed 15 January 2022) and CSIEC (http://www.csiec.com/, last accessed 15 January 2022) [16,17]. Autotutor [18] is a multidisciplinary chatbot aimed at encouraging learning through conversation with learners. Chatbot [19] is an educational chatbot aimed at promoting the study of computer science. StudyBuddy and SmarterChild [20] are examples of chatbots used in non-formal learning. The NDLtutor [21] has been developed to improve student involvement and reflection. Confucius [22] is used as a supplement in the classroom. The Xbot chatbot [23] aims to encourage high school students to improve their mathematical, programming, and logic skills.

A few chatbots have been created and designed for language learning. CLIVE is an intelligent chatbot for conversational language practice capable of language switching among any of the world’s major languages [24]. Gengobot is a chatbot-based grammar dictionary application for Japanese language learning that gives explanations and meanings in Japanese, Indonesian, and English [25]. The BookBuddy system [26] consists of three sub-chatbots assisting children in selecting the appropriate book for their English language level, supporting reading comprehension by answering vocabulary questions, conversing with them, and delivering individualized feedback. Mondly and Andy English Bot offer several possible responses so that learners can choose the most appropriate one [27]. The CSIEC system generates communicative responses according to user input, dialogue context, user and personality knowledge, common-sense knowledge, and inference knowledge [16].

Haristani [17] points out that related work on development and research on teaching and learning languages other than English is still in its infancy. Huang et al. [28], in their systematic review of chatbot-supported language learning, noted that little research evaluated chatbots compared to other equivalent tools and suggested research involving education levels other than university. AssassaraBot was built from the ground up to help students meet their content and language learning objectives in two languages: English and French. In this research, teaching through AssassaraBot was evaluated compared to a teacher teaching with other ICT tools in secondary education.

3. The Proposed Approach

Chatbots focus on dialogue/conversation, aiming for agent interaction that follows patterns similar to those followed by humans. Through conversation, chatbots must be able to analyze the context and suggest solutions to problems, interpret emotions, and act accordingly or contribute to the learning process [6]. Based on this aim, we have designed and developed an educational chatbot with the purpose of teaching secondary school students (ages between 12–18 years old) the history (cultural content) of the Goddess of Snakes, and at the same time, to teach them a foreign language. A chatbot named AsasaraBot is introduced to the students as the Goddess of Snakes, acting as a tutor who is playing a theatrical role to better engage his/her students. AsasaraBot is teaching without the intervention of a human teacher. Through the whole learning experience of students chatting with the AsasaraBot, the conversation is enriched with proactive verbal techniques that could be used by a teacher to address the student in a personalized and encouraging way. Examples of messages in this conversation form are:

• Questions facilitate learning in multiple ways. They motivate students to inquire and reflect and cultivate their cognitive and linguistic abilities. In this sense, AsasaraBot addresses certain questions to advance the flow of intercourse (e.g., “What do you see in this picture?” or “Qu’est-ce que tu vois sur la photo?”).
• Praise and encouragement support knowledge assimilation by motivating students, empowering their self-confidence, and strengthening their efforts toward learning. The AsasaraBot was designed to encourage and motivate with remarks such as, “Oh, why not?”/“Oh, pourquoi pas?” or “Super! You are really good!”/“Superbe! Τu as très bien compris le texte!”.

AsasaraBot is an original educational teaching software that implements the basic idea of learning cultural content and foreign languages (CLIL approach) in an interactive educational learning environment. This software is mainly based on the exchange of messages between a student and a virtual tutor–chatbot, which in our case takes the form (plays the role) of the Goddess of Snakes of the Minoan Civilization. From this role comes the name Asasara, which was chosen for the chatbot, a name that is encountered in inscriptions written in the Linear A script and often seems to be used for the Minoan Goddess of Snakes. AsasaraBot is deployed via different channels (Facebook Messenger, Viber) and an educational web site (https://eduasasara.wixsite.com/followasasara2020, last accessed 15 February 2022), aiming to guide students through dialogue to discover and learn cultural content that is available in two languages, English and French.

The purpose of AsasaraBot determines the functional requirements of the software, i.e., what functionality the software must provide, and how the system will behave while interacting with the users (students). Specifically, AsasaraBot meets the following main functional requirements:

• The user (student) communicates with AsasaraBot via guided and free (thus, hybrid) chat.
• The user is able to select the flow of the dialogue through available options (buttons, voicemail, etc.).
• The chatbot is available (online) through at least three messaging platforms (Webchat, Messenger, Viber).
• The chatbot has a start and restart button.
• The system provides the user with a simple and understandable dialogue flow.
• The system understands the natural language (English or French) of the user.
• The system informs the user in cases of misunderstanding of the message to take appropriate actions, e.g., repeat the message.
• The information provided to the user comes from reliable sources, e.g., Wikipedia.
• The system does not violate the privacy protection policy regarding the recording of data.
• Each user’s chat history is securely and anonymously stored in the system database.
• The input and output messages are exchanged via text or audio format.
• The content of the conversation combines text, image, video, and audio.

3.1. Design of Asasarabot

AsasaraBot is designed using the general architectural structure of a hybrid chatbot model (guided and free-text communication). Therefore, it includes the following components (as depicted in Figure 1): a natural language understanding engine, a dialog manager, and the information retrieval components along with a knowledge base.

• Natural Language Understanding (NLU) engine: The user’s message goes through the component of the NLU engine, where, through customized NLU models, the chatbot “uncovers” the meaning (intent) of the user’s message and provides the appropriate response.
• Dialog Manager: This component is responsible for handling/managing all the messages exchanged (questions/requests, answers/responses) in the form of text or audio messages. It is essentially the middleware of the chatbot connecting the NLU engine and the information retrieval components.
• Information Retrieval and Knowledge Base: In this component, the chatbot searches for the most appropriate response stored in the knowledge base (or externally in the Web) and sends the retrieved related information to the Dialog Manager. Basically, all the information and knowledge that constitutes the ‘brain’ of the chatbot is stored and handled here.

In Figure 1, the general architectural design of the implemented system is depicted.

3.2. Development of Asasarabot

AsasaraBot has been developed on Snatchbot (https://snatchbot.me/, accessed on 15 January 2022), a cloud-based platform that provides technological infrastructure for the design, development, and testing of chatbots, without requiring knowledge of software coding. Such a selection of platforms, among other platforms examined (e.g., DialogFlow, Rasa, OpenDialog), was made mainly due to its simplicity and codeless interface, as well as its functionality related to the design requirements of our approach. Thus, although a few limitations of the platform were identified (see later section), the advantages offered were determinant.

The knowledge base (KB) for AsasaraBot was built using a template-driven graphical interface of the platform. For each question designed in the dialogue, predefined answers were manually inserted into the KB. The KB is not automatically updated with new knowledge, since the chatbot does not have the ability to learn by itself. Such an extension/update of the KB must be performed manually by the developers who monitor the dialogue and collect the intents of users from the log files. Furthermore, since the interaction of the users with the chatbot is made anonymous, personalized customization of the dialogue is not supported. However, the customization of the dialogue (and consequently the improvement of interactions) can be performed at the class level by analyzing the interactions of students as a whole (e.g., how many understood the question or not).

During the development, the logic proposed by the Snatchbot platform for the creation of configurable components to be synthesized for the development of AsasaraBot was followed. Thus, effort was put to distinguish the necessary functionality and create components that would meet the requirements of each one of them. As a result, the main part of the chatbot application includes the following components:

3.2.1. Interactions

In the terminology of Snatchbot, interactions are the basic elements that define the behavior of a chatbot. The platform supports 13 types of interactions. The most basic type of interaction, that is, the basic building block of AsasaraBot’s structure, is where the chatbot makes a statement (bot statement) in plain text, and interacts with the user’s response. A total of 52 interactions were developed in AsasaraBot, defined according to the needs of the dialogue and the requirements of the educational scenario. Reformulations of these interactions were inserted between the basic interactions in cases where the user’s message was not understood by AsasaraBot, encouraging the user to try again (e.g., repeat the question).

3.2.2. Connections

The Snatchbot platform provides pre-trained natural language processing (NLP) models that are ready to be used at any time by the developers. The pre-trained “Detect Persons” NLP model already included in the Snatchbot platform was used, and additional custom models were developed from scratch in both languages (English and French), focusing on intents and entities that do not exist in pre-trained models. In particular, new models were developed to detect common phrases that users use when they are unable to answer a question, and models to detect negative and positive human reactions. For instance, the following custom entity-based NLP models have been developed to a) detect general positive/negative emotions, e.g., boredom, happiness, excitement, and b) detect disappointment or excitement from AsasaraBot performance (e.g., “you are not so helpful to me”, “I do not want to talk to you any more”, “you are so bad”). The need for developing new custom models appears when developers want to focus on knowledge (intents or entities) that does not exist in the pre-trained models. In custom models, the developer can manually insert knowledge (words or phrases) or automatically insert large numbers of samples at a time using the “Bulk insert” feature. Within the chatbot paradigm, the term “intent” refers to the goal (intention) the user has in mind when making/typing a question (what he/she really means), while the term “entity” refers to the modifier used by the user to describe an issue/concept. SnatchBot uses emoji and text analysis (NLU) to understand (interpret) emotions. NLP (models) is what allows Snatchbot chatbots to understand the meaning of a user’s statement (context in the model) and act accordingly. NLP models use machine-learning algorithms for training, i.e., to learn how to recognize the intent of users and related entities.

3.2.3. Chat Bubbles

This feature was primarily used to make the conversation more engaging. In particular, it allows the creator to split the dialogue into different bubbles so that a large amount of information is not displayed at once. Using the Instant Response or Delayed Response functions, the creator can specify how long the message bubble will be displayed to the user, offering a more natural conversation experience that mimics human conversation.

3.2.4. Text-to-Speech

Text-to-Speech functionality provides the option to listen to chatbot messages (instead or in addition to written ones). For each interaction selected to use the Text-to-Speech function, an audio file was generated per language.

3.2.5. Prior Responses

The bot’s dialogue reuses responses it has received earlier from the user who is chatting with it. For example, in the very first interaction with a student, AsasaraBot asks his/her name, and uses the response (the responded name) later in the conversation to personalize the interaction.

3.2.6. Quick Replies

This add-on feature offers users simple and guided ways to respond to messages. It has the form of ready-made buttons, where the user can tap as a response to a message. After the selection of a quick reply by a user, all other buttons are hidden, preventing users from tapping on buttons associated with previous interactions.

3.2.7. Custom Variables

Custom variables allow the collection of data/information from user interactions, especially for the execution of numerical functions during the conversation. For example, they were used to collect correct/wrong answers (as positive or negative points) to calculate the final test score for each user.

3.2.8. Arithmetic Operation

This functionality is responsible for performing numerical operations and was used to add or subtract points during the conversation and calculate the final test score. In particular, when a student answers a question correctly or incorrectly, a point is added or subtracted, respectively.

3.2.9. Embedding Media

This feature allows the integration of images, videos, audio, and files in the dialogue. These attachments can either be embedded directly by entering the URL in the chatbot message or via the Rich Cards option. This option is a set of cards, each of which can contain any combination of text, images, and buttons. Through this, it is possible to integrate video, video, and audio into the conversation for the needs of the educational scenario.

3.2.10. Connecting Channels

The AsasaraBot was initially attached to the official website of the proposed educational program. This process was performed simply by copying and embedding special code (provided by the Snatchbot platform) on a webpage. On the website, AsasaraBot appears as a pop-up window. A prerequisite for the integration of AsasaraBot in the Facebook Messenger channel was the creation of a Facebook page. Thus, two pages were developed to host the two versions, English and French, of AsasaraBot. Finally, AsasaraBot has also been deployed from a Viber channel.

3.3. The Dialogue

During the development of the AsasaraBot, a group of people was involved in the continuous testing of the overall process. In particular, these tests were performed by people with software engineering knowledge, by researchers familiar with chatbot technology (Conversational AI experts), by language teachers (Language Education Experts/Teachers), and by ordinary users. These tests significantly contributed to the elimination of bugs that occurred in the early stages of AsasaraBot development.

In addition, efforts were made to determine if AsasaraBot worked smoothly on the three messaging channels: Facebook Messenger, Viber, and the Webchat environment. A few problems were identified when switching platforms, for instance, the lack of STT (Speech-to-Text) functionality on Messenger and Viber platforms, a feature that is only provided in Webchat, the TTS (Text-to-Speech) functionality in Viber, where the users are required to download the voice message, and problems with the importing of external files. To overcome these problems, a number of actions have been tested, such as the use of Rich Cards, making the conversation more user-friendly and visually richer on all three messaging platforms. Finally, language teachers helped to improve the content of the dialogue with comments on detecting syntax and grammar errors, poor vocabulary, spelling mistakes, etc. in English and French dialogues.

After an extended number of experiments and evaluation cycles, the final flow of the dialogue implemented in AsasaraBot has been shaped in the form depicted in Figure 2. As can be seen, the dialogue is composed of a number of questions to be answered, and activities to be completed by the students. The specific flow is based on the general pedagogical aspects of educational programs that have been researched prior to the development of this dialogue.

The activities and questions implemented in the dialogue are depicted in Figure 3, Figure 4, Figure 5 and Figure 6.

3.4. The Platform

Based on the hands-on experience during the development and testing of the AsasaraBot, a number of advantages of the SnatchBot platform have been identified:

• It supports a large number of online messaging platforms, such as Webchat, Twitter, Facebook Messenger, Skype, WhatsApp, Viber, and Slack, while new ones are constantly being added.
• It facilitates the collection and analysis of the generated interaction data and provides statistics for each chatbot implementation.
• It supports natural language understanding (NLU), making a chatbot quite sophisticated.
• It supports text-to-speech (TTS) and speech-to-text (STT) conversion of messages.
• It provides a plethora of free chatbot templates that can be used to support developers at their early developing steps.

On the other hand, at the time of our experimentation with the platform, there were some issues and limitations depending on the communication channels chosen for interacting with AsasaraBot. In a few cases, the user was not allowed to send an audio message in the chatbot. Additionally, embedding files such as importing a video, or an audio clip did not work properly, e.g., as occurred while working with the Facebook Messenger channel.

4. AsasaraBot Evaluation

4.1. Experimental Educational Program for Cultural Content and English/French Language Learning

The experiments were conducted online (due to COVID-19 restrictions) on the 16 and 20 of December 2020 through the WebEx and Zoom platforms. The participating classes were a) English and French language classes of the 1st and 2nd grade of the 1st School of Excellence General Lyceum of Mytilene, Greece, and b) two private foreign language schools (French and English, respectively) also in Mytilene, Greece.

4.1.1. Participants

The selected students varied in terms of educational level, language level, gender, and age. A total of 61 students participated in the experiments. From those, 35 were selected from among the English and French language classes of the Lyceum (age 15 to 17 years old), 18 of them were students of English as a foreign language, and 7 were students of French as foreign language. Another group of 26 participants were selected from private foreign language schools. Of these, 10 students attended the school of English as a foreign language, while 16 attended the school of French as a foreign language. Two teaching methods were selected for the experiments: the proposed chatbot-based approach, and another with other ICT-based learning tools. In

Table 1. Experiments’ organization.

Department	Language	Language Level	Teaching Method	Number of Participants
Lyceum (Class B)	English	Basic (A1–A2), Intermediate (B1–B2), Proficiency (C1–C2)	Other ICT-based Learning	9
			Chatbot-Based Learning	9
Lyceum (Class A)	French	Basic (A1–A2), Intermediate (B1–B2), Proficiency (C1–C2)	Other ICT-based Learning	8
			Chatbot-Based Learning	9
English Private Foreign Language School	English	Intermediate (B1–B2), Proficiency (C1–C2)	Chatbot-Based Learning	10
French Private Foreign Language School	French	Intermediate (B1–B2), Proficiency (C1–C2)	Chatbot-Based Learning	16
				TOTAL: 61

, the identity of the experiments is summarized and organized per school, class, language, language level, and teaching method.

4.1.2. Evaluation Method

Two different evaluation methods were used in the experiments: chatbot-based learning and another ICT-based learning approach. The latter was developed using Google Forms, Padlet, and YouTube videos. Padlet (https://padlet.com/, accessed 15 January 2022) is a digital tool through which one or more “walls” can be created that can accommodate posts for sharing, including content such as videos, pictures, documents, audio, diagrams, and blueprints. It has a collaborative character, allowing for the collaborative engagement of students and teachers. This tool was used to construct a short diagram as an introduction to the educational program and, in particular, to the Minoan Civilization. As an introductory activity for the other-ICT-based learning method, a small diagram is designed using Padlet to answer two questions on the subject of Minoan Civilization, so that the educational process can start with a related discussion. The same questions were used in the chatbot, but this time they were placed by AsasaraBot.

A detailed task-based experimental setting was designed. A total of 35 students of the Lyceum were randomly assigned to two groups: 25 females and 10 males. An experimental group, consisting of 18 students, was involved in a learning scenario using the AsasaraBot learning method, while the members of the control group of 17 students were involved in the other-ICT-based learning method. Both methods involved the same educational content.

A pre-test and a post-test questionnaire-based approach were conducted to assess the content and foreign language knowledge gain of corresponding participants. The 26 students of the two private language schools also participated in the chatbot-based learning method. Overall, 44 students participated in the Chatbot evaluation group, that is, 18 students in the experimental group and another 26 students that used the Asasarabot without participating in the experiment (non-experimental group), as illustrated in Figure 7. All students in the chatbot evaluation group were given a questionnaire in order to evaluate their experience interacting with the chatbot. The students in the Lyceum were given the evaluation questionnaire after taking the post-test. Figure 7 illustrates the flow of the evaluation procedure.

4.1.3. Pre-Test

A pre-test (in the form of a questionnaire) was given to the students before the start of the educational program. This test consisted of six closed-ended question items (see Appendix A). Items A, B, and C assessed the student’s prior knowledge of the cultural content being examined, while question items D, E, and F assessed the knowledge of vocabulary in English and French. The items and the provided answers were collected via Google forms (due to privacy and language issues, they are available upon request).

4.1.4. Post-Test

A post-test (in the form of a questionnaire) was given to the students at the end of the educational program. This test consisted of a total of 11 closed-ended questions organized into two sections, similar to the pre-test. That is, content knowledge, as well as language learning were assessed, in alignment with the pre-test. The aim of this is to test the effectiveness of the educational program, which was taught with the two methods, i.e., with the chatbot-based and other-ICT-based methods. The items and the provided answers were collected via Google forms (due to privacy and language issues, they are available upon request).

4.1.5. AsasaraBot Evaluation Questionnaire

Finally, another questionnaire was given to the students who followed the chatbot-based educational program (see Chatbot evaluation Group in Figure 3) to evaluate the usability of the AsasaraBot environment, as well as the overall experience gained from the educational process. This questionnaire consisted of 18 questions and was structured into three sections. In the first section, information about the profile of the students was collected. In the second section, the questions aimed to evaluate students’ learning experiences in relation to the educational content of AsasaraBot, while the third section included questions for the evaluation of the students’ experiences in relation to the chatbot-as-software. The answers were collected using mainly a five-point Likert scale, ranging from strong agreement to strong disagreement. The items and the provided answers were collected via Google forms (due to privacy and language issues, they are available upon request).

4.2. Learner Experience Evaluation Results

While most participants were familiar with new ICT technologies, 23% (10 out of 44 students) reported that they had interacted with a chatbot in the past. Regarding their experience with AsasaraBot in terms of cultural content, most of the student participants reported that they had a positive impression. That is, 91% of the participants agreed that a student can learn cultural content with the assistance of a chatbot. On the other hand, only 48% of them were positive about using a chatbot for learning a foreign language. The majority of the participants (93%) reported that the conversation with the chatbot was engaging. Regarding their experience with AsasaraBot in terms of usability, 98% of the students reported that it was user-friendly, 54% of the students reported that it did not require any prerequisite knowledge for its use, and 91% of the students reported that the design and development of the dialogue was simple and understandable. These results were also confirmed by an interview that followed (qualitative analysis) at the end of the learning process, where the majority of participants stated that it was an interesting experience, that they liked the AsasaraBot environment quite a lot, and that it was enjoyable as a means for educational purposes. In fact, for many students this experience was unprecedented, and the use of chatbots in the context of teaching was an experience they wanted to have again in future courses.

4.3. Cultural Content and Language Knowledge Gain Evaluation

As mentioned earlier, the members of the experimental and control groups were given a pre-test and a post-test in order to assess knowledge gain when interacting with the AsasaraBot, compared to another technology-based educational environment. The experimental group had a mean performance (mean) of 3.61 and a standard deviation (SD) of 1.61 for N = 18, while the control group had a mean performance of 3.41, with a standard deviation of 1.8 for N = 17.

The Shapiro–Wilk test showed that the performance of both groups in the pre-test did not follow the normal distribution (W = 0.92633, p < 0.05). A non-parametric analysis based on the Kruskal–Wallis test (H(2) = 0.10244, p = 0.7489) showed no statistically significant difference between the performance of the two groups in the pre-test (a = 0.05).

The difference in the performance of the members of both groups in the post- and pre-tests was calculated. More specifically, the overall performance difference was calculated as well as the performance difference pertaining to both cultural content and language content knowledge gain. The results are shown in

Table 2. Experimental results for learning gain.

		Overall		Cultural Content		Language
Group	N	Mean	SD	Mean	SD	Mean	SD
AsasaraBot	18	1.44	1.25	0.778	1.00	0.667	0.907
Control	17	1.35	1.66	0.529	0.943	0.824	1.07

.

According to the Kruskal–Wallis non-parametric test, no statistically significant differences were found for the overall performance gain of the two groups (H(2) = 0.10344, p = 0.7477), the performance gain on cultural content learning (H(2) = 0.22496, p = 0.6353), or the gain in language learning (H(2) = 0.25127, p = 0.6162). Furthermore, no significant difference was found between the performance gains of male and female members in the experimental and control groups (H(2) = 1.7185, p = 0.1899).

Concerning descriptive statistics, the gain in performance appeared to be greater for the AssasaraBot group for cultural content learning, while the control group performed better in the question items for language learning. While this finding is not conclusive, it aligns with the previous finding from the evaluation questionnaire, where relatively few participants had a positive attitude toward using the chatbot for language learning, compared to cultural content learning.

5. Discussion

Based on the data collected from the evaluation of the presented approach, we discuss a number of issues that highlight our concerns in the domain of ICT-based learning and in the CLIL approach.

Initially, the use of chatbot technology, as part of the conversational AI paradigm, can enhance traditional teaching in many ways. A student has the opportunity to work at his or her own time, without the fear of failure and error, through a friendly environment, outside the classroom. Participants in the evaluated training program stated that the discussion with AsasaraBot was made in a language they understood in a quite interesting manner, and that they would like to use a similar type of training method again in the near future.

At the same time, a chatbot can be a useful tool for teachers by providing them with assistance during distance-learning courses. Due to the difficult situation of the COVID-19 pandemic, distance-learning courses were applied all over the world at different educational levels. One of the main goals of chatbot-based ICT methods is not to replace human teaching but to provide a support tool/method in the existing/traditional educational process. Based on the data collected from our experiments, it would be rather early to argue that chatbot-based ICT can fully replace traditional teaching, as there are still technical difficulties and obstacles, such as restrictions on natural language understanding, difficulties in the human-like flow of dialogue, and familiarization of teaching staff and students with such new technology.

Regarding the integrated/unified learning of foreign language and (cultural) content using a chatbot, the result is rather positive, although it is possible that the formulation of a small number of questions or the short time frame for completing the questionnaires have slightly affected the result. This argument is based on the student data collected, with a high percentage of correct answers. As the students also argued, the chatbot itself positively contributes to their educational process. In addition, the evaluation shows that in both languages, the data is positive, which proves that adapting the material to different foreign languages aims at similar learning outcomes. In the future, it would also be interesting to study the extent to which the level of language learning (e.g., basic or intermediate) contributes to the educational process, and how chatbot technology can contribute to the learning of language and cultural content at each of these levels.

6. Conclusions

This paper presented experiments with an AI chatbot that provides insights into the contribution of this technology (conversational AI) to the learning of cultural content and a foreign language at the same time (content and language integrated learning). The developed educational AI chatbot, called AsasaraBot, supports human tutors in teaching cultural content related to the Minoan Civilization and the figurne of the Minoan Snake Goddess. The related chatbot-based educational program has been evaluated in English and French at public and private schools in Greece, with encouraging preliminary findings. The use of AI chatbot technology for interactive ICT-based learning can efficiently support the learning of foreign languages and cultural content at the same time. Future plans for this research line include the use and evaluation of this technology as an assistive technology for human teaching, delivering part of a course curriculum with this method. The goal is to identify a gold balance between chatbot-based and other ICT or non-ICT learning methods for individual courses or for the entire teaching curriculum of a school. A future study with a greater number of participants will allow us to draw more conclusive results concerning the pedagogic effectiveness of chatbot-based teaching methodology.