The Development of an Emotional Embodied Conversational Agent and the Evaluation of the Effect of Response Delay on User Impression

Abstract

Embodied conversational agents (ECAs) are autonomous interaction interfaces designed to communicate with humans. This study investigates the impact of response delays and emotional facial expressions of ECAs on user perception and engagement. The motivation for this study stems from the growing integration of ECAs in various sectors, where their ability to mimic human-like interactions significantly enhances user experience. To this end, we developed an ECA with multimodal emotion recognition, both with voice and facial feature recognition and emotional facial expressions of the agent avatar. The system generates answers in real time based on media content. The development was supported by a case study of artwork images with the agent playing the role of a museum curator, where the user asks the agent for information on the artwork. We evaluated the developed system in two aspects. First, we investigated how the delay in an agent’s responses influences user satisfaction and perception. Secondly, we explored the role of emotion in an ECA’s face in shaping the user’s perception of responsiveness. The results showed that the longer response delay negatively impacted the user’s perception of responsiveness when the ECA did not express emotion, while the emotional expression improved the responsiveness perception.

1. Introduction

An embodied conversational agent (ECA) [1,2] is a kind of user interface that simulates human face-to-face conversation. While conversational agents, such as smart speakers, do not necessarily have human-like bodies [3], ECAs have human-like figures, such as computer-generated figures or humanoid robots. ECAs are used as an intelligent interface for various applications, such as home automation [4,5,6], education [7,8,9], healthcare [9,10,11], and tourism [12].

Since ECAs have human-like figures, users expect these ECAs to behave like humans, including nonverbal behaviors accompanying speech, such as facial expressions, gestures, and gaze [13]. However, because of the limitation of computational models, such as speech recognizers, the response delay of ECAs inevitably becomes longer than that of humans. In human–machine conversation, a larger response delay affects the cognitive performance of humans [14]. Therefore, a spoken dialogue system should respond to a user as quickly as a human would.

In human–human dialogue, an utterance often starts before the interlocutor’s utterance ends [15]. Humans can do this because of the ability to predict the transition relevance place (TRP) [16]. Several systems have tried to predict the timing of turn-taking before the previous utterance ends [17,18,19]. However, it is still difficult for spoken dialogue systems to predict the TRP and take turns properly in real time.

In human–human dialogue, it is known that pauses tend to be longer in cooperative (friendly) conversations than in arguments [20]. Devillars et al. found that in-utterance pauses in emotional utterances are longer than those in neutral utterances [21]. From these facts, we devised an idea to control the user’s impression induced by response delay by introducing emotional behavior into the ECA.

This paper is organized as follows: In Section 2, we describe the related work of ECAs. In Section 3, we describe the details of the development of the ECA with emotion recognition and facial expressions. Section 4 describes the experiment, including the experimental conditions, procedures, and results. We conclude the paper in Section 5.

2. Related Work

The term “embodied conversational agent” was coined by J. Cassell [1,2]. In [2], he listed the following properties an ECA should have:

• Recognizing and responding to verbal and nonverbal input;
• Generating verbal and nonverbal output;
• Dealing with conversational functions, such as turn-taking, feedback, and repair mechanisms;
• Giving signals that indicate the state of the conversation, as well as contributing new propositions to the discourse.

Nonverbal behaviors such as nodding or changing facial expressions are the special features of an ECA compared with conversational agents without bodies. Cassell et al. developed an ECA named Rea [22], which behaved as a real estate agent. Rea had most of the basic functionalities an ECA should have; however, most functions are realized using handwritten rules. Since then, many ECAs have been developed [23,24].

The emotional behavior of an ECA, including perception of the user’s emotion and expressing emotion using facial expression, is an important aspect when designing an ECA. The effect of introducing emotion in an ECA was first discussed in the late 1990s [25].

The main objective of introducing emotional behavior into ECAs is to enhance the user’s impression of the ECAs. Expressing positive emotion is especially expected to improve this impression [26]. Since then, many studies have been conducted to develop general models of emotional dynamics [27,28,29]. Not only are facial expressions and gestures effective but so is the paralinguistic property of speech in presenting specific emotions of ECAs [30,31]. Conventional models use rules to determine the ECAs’ emotional behavior, while recent studies use more sophisticated models based on deep learning. For example, Firdaus et al. proposed a model based on the multimodal-attention-based conditional variational autoencoder (M-CVAE) [32].

Since the aim of introducing emotional behavior into ECAs is to improve the user’s impression of the system, such behaviors are often used in ECAs for clinical or healthcare purposes [33,34,35,36].

As listed above, most studies have focused on how emotional behavior makes ECAs more human-like and improves the user’s impression. On the other hand, emotional behavior could interact with other factors of dialogue, such as response delay. This is the duration from the end of the user’s utterance to the beginning of the ECA’s utterance. As mentioned before, the response delay affects the user’s cognition performance [14]. Funk et al. reported that 0 to 250 ms of delay gave the best satisfaction and usability for in-vehicle conversational agents [37]. Gnewuch et al. reported an interesting result on the user’s preference for the response delay [38]; they proved that taking a long time to answer complicated questions improved the user’s impression of the system and perception of humanness. This result is considered a realization of response delay distribution according to different question types [15]. Heeman et al. showed that the response delay varies according to the dialogue act of the previous utterance [39].

As described, there are studies on how linguistic content affects the response delay; however, no research has been conducted on how emotional behavior and the response delay jointly affect the ECA’s impression. If we find any relationship between emotional behavior and response delay, it can be used to make the dialogue behavior of ECAs more natural.

3. Development of an Embodied Emotional Conversational Agent

To conduct the experiment, we first developed an ECA that converses with a user in real time (The system is available at https://github.com/SimonJolibois/Multimodal_Embodied_Conversational_Agent, accessed on 19 June 2024). Note that the technology used in the ECA is not novel; however, we tuned the system to work as quickly as possible so that we can control the response delay. The details of the system are described in our previous paper [40].

3.1. Overview

Figure 1 shows an overview of the system. The system observes the user’s face with a camera and captures the user’s voice with a microphone. The observed face is used to estimate the user’s emotion. The recorded voice is transcribed using the speech recognizer, and the transcription is used to select the system’s response and estimate the user’s emotion. The agent’s verbal response is selected from a predefined query–response database. The facial expression of the agent is controlled using both the emotion of the response sentence and the user’s emotion. When playing the response voice, lip movement is generated from the response voice.

3.2. Emotion Recognition from Facial Image

Using the OpenFace software toolkit (version 2.2.0) [41], the facial landmarks are collected as a CSV file. The facial expression is expressed as the Facial Action Coding System (FACS) [42]. It breaks down facial poses into distinct muscle actions, known as action units (AUs), allowing researchers to decode and classify emotions and expressions objectively.

We used the 17 AUs shown in

Table 1. The 17 AUs used for emotion recognition from a facial image.

No.	Description	No.	Description
01	Inner Brow Raiser	14	Dimpler
02	Outer Brow Raiser	15	Lip Corner Depressor
04	Brow Lowerer	17	Chin Raiser
05	Upper Lid Raiser	20	Lip Stretcher
06	Cheek Raiser	23	Lip Tightener
07	Lid Tightener	25	Lips Part
09	Nose Wrinkler	26	Jaw Drop
10	Upper Lip Raiser	45	Blink
12	Lip Corner Puller

for emotion classification. Then, we used a four-layer perceptron for the classification; the input dimension was 17; there were 15, 12, and 10 hidden units; and the output dimension was 8 for classifying eight emotions (neutral, anger, contempt, disgust, fear, happiness, sadness, surprise).

We utilized the CK+ dataset [43] for model training, which includes 593 images from 123 subjects, with 327 sequences explicitly labeled for emotions fitting stereotypical definitions. These sequences range from neutral to high-intensity expressions, classified into the same eight emotional categories. The model achieved an average accuracy of 85%.

3.3. Speech Recognition and Emotion Estimation

We transcribed the user’s speech using Microsoft Azure CognitiveService Speech-to-Text. Then, the user’s emotion was estimated from the transcription. We used the GoEmotions dataset [44], which comprises approximately 58,000 Reddit comments categorized among 27 emotions as neutral.

Because some of the emotions in the database are not useful when animating facial expressions (for example, love or remorse), a subset of the GoEmotions dataset containing 14 emotions was processed in the selection phase: anger, annoyance, confusion, curiosity, disgust, embarrassment, excitement, fear, grief, joy, nervousness, pride, sadness, surprise.

We developed the text emotion classification model based on the ‘base-uncased’ BERT model.

Table 2. Specifications of the BERT-based emotion classification model.

Number of layers/transformer blocks	12
Number of units in the hidden layers	768
Total number of parameters	110 M
Number of self-attention heads	12
Number of epochs	4
Batch size	16
Learning rate	$5\times {10}^{-5}$
Validation split	85/15

shows the specification of the model. After fine-tuning the model, we obtained an accuracy of 45% in emotion classification. Detailed results are shown in

Table 3. Text emotion classification results.

Emotion	Accuracy (%)
Anger	46
Annoyance	41
Confusion	35
Curiosity	45
Disgust	37
Embarrassment	37
Excitement	21
Fear	65
Grief	6
Joy	71
Nervousness	25
Pride	0
Sadness	59
Surprise	61
Weighted macro-average	45

. The model attained a weighted macro-average precision of 45%.

3.4. Dialogue Management

Dialogue management is based on a simple Q&A database [45]. In this dialogue management architecture, we prepare a database containing examples of users’ utterances and the system’s corresponding answers. When a user inputs an utterance, the dialogue system transcribes it using a speech recognizer. Then, the system chooses the example most similar to the input sentence from the database. Finally, the system synthesizes the speech using the corresponding answer text. Because of its simplicity and robustness, many spoken dialogue systems employ Q&A database architecture [5,45,46].

The domain of the dialogue was the question and answer of four artworks, shown in

Table 4. Artworks used in the dialogue system.

No.	Name of the Artwork	Painter
1	A Hughenot on St. Bartholomew’s Day	John Everett Millais
2	Liberty Leading the People	Eugéne Delacroix
3	The Roses of Heliogabalus	Lawrence Alma-Tadema
4	The Starry Night	Vincent van Gogh

. The system shows the image of the painting on the system, and then the user starts a conversation with the system by asking questions about the artwork.

To prepare the Q&A database, we first developed one that included information on the artworks and painters, as shown in

Table 5. Items in the artwork database.

No.	Item
1	Title in English and original language
2	Place and date of creation
3	Dimensions
4	Genre
5	Format
6	Subject matter
7	Cultural and historical context in which the artwork was born
8	The intent of the author for creating the piece
9	The current conservation place and methods
10	Previous exhibitions the artwork was part of
11	The ownership and sale history
12	Technical analysis done on the artwork
13	The painter’s name
14	The painter’s biography
15	The painter’s artistic movement(s)
16	The painter’s famous artworks
17	Inspirations/references used for the artwork

. All the information was gathered from Wikipedia and WikiArt. Next, we developed a set of questions and answer templates. For example, we prepare a question “When was the artwork painted?” and an answer template “It was painted on <DATE>”. Then, the Q&A database was generated from the template and items in the artwork database by substituting the tag such as <DATE> with the information in the artwork database.

When the transcription of the user’s utterance is given, we choose the nearest example sentence in the Q&A database by choosing the sentence with the largest cosine similarity.

3.5. Generation of Face Image and Facial Expressions

To select an appropriate facial expression, we consider both the emotion predicted from the user’s facial expression and the emotion prediction of the agent’s generated answer. The procedure is as follows:

• Prioritizing the answer’s emotion: The primary factor is the emotion predicted from the generated answer. If the prediction emotion is not neutral, the ECA’s face is rendered using that emotion.
• Maintaining Current Expression: In instances where the predicted emotion of the answer is neutral, the agent keeps the same facial expression.
• Reflecting the user’s emotion: If the agent’s facial expression is neutral twice, the agent mimics the user’s facial expression. It mimics the behavior of entrainment, where individuals can adjust their speaking rate, pitch, and gesture to match those of their conversation partner [47]. In the context of an ECA, entrainment can foster more engagement for the human user [48].

3.6. Text-to-Speech and Lip Syncing

Microsoft Azure’s Text-to-Speech service generates a response to the user’s speech, utilizing Speech Synthesis Markup Language (SSML) to define the language, voice, and pronunciation features, including emotional tones. The service outputs both the audio file and associated viseme data. While phonemes are crucial for audio speech synthesis, visemes are essential for creating accurate lip-syncing in visual representations, such as in animations or virtual avatars. In the context of an ECA, focusing on visemes is important because they directly impact the visual synchronization of the lips with the generated speech, enhancing the realism and coherence of the virtual character.

3.7. Facial Animation Rendering

The facial animation is rendered in the Unreal Engine 5 game engine. The animation is subdivided into four blueprints, each taking a specific role. One is the receiver/emitter, which communicates with the Python 3.10 script that controls the other parts of the system. Another takes the user’s input on the screen through the UI. A third blueprint stores all variables of the agent, and the last one handles the animation depending on these variables. The animation is carried out in layers, one after another, as shown in Figure 2. We first obtained the facial expression, the one we selected before. We added some looping animation for blinking, as it made the agent more aware-like and less robotic. We can also control the direction of the gaze and the head. Lastly, we added the lip-syncing animation.

The character is a Metahuman, a high-quality 3D character developed by Epic Games. It is available for free in Unreal Engine. Metahumans are notable for their high-fidelity textures and procedural skeleton system, which allows for the creation of a diverse range of realistic 3D characters suitable for various projects. These characters come equipped with pre-defined facial expressions and viseme poses, simplifying the process of animating them.

3.8. Controlling the Entire System

We divided the entire system into four threads. The first one captures the user’s face and extracts AUs. The second observes and recognizes the user’s speech using Azure CognitiveService Speech-to-Text. The third performs almost all tasks except animation rendering, including estimating emotion from face and text, dialogue management, and choosing the agent’s emotion. The final one renders the facial animation.

Figure 3 shows the user interface of the dialogue system. The ECA is displayed on the left side, and the artwork is on the right. The user talks with the ECA while looking at the artwork. An example conversation video is provided as Supplementary Materials (Video S1).

4. Experiment

4.1. Overview

As an evaluation of the developed system, we evaluated the users’ impressions of it related to emotional expression and response delay. The purpose of this experiment is to clarify how the presence or absence of emotional expression and the length of response delay in the ECAs developed here affect users’ impressions of the ECAs (e.g., satisfaction, responsiveness, interactivity, and information accuracy). As noted in Section 2, it is not clear how the response delay, which inevitably occurs in ECAs, interacts with the emotional expression of ECAs to alter various user impressions of ECAs. Therefore, in this experiment, we modified the emotional expression and response delay of ECAs to determine how they affect user impressions.

Based on existing research that highlights the response timing in human–human interaction, this research seeks to extend the research to ECAs [20,49,50]. In the experiment proposed by Funk et al. for automotive user interfaces [37], Satisfaction, Usability, and Context Awareness were evaluated depending on the response time of the user interface. This research aims to determine whether the agent response delay affects the user’s evaluation of the agent. Their work showed that the user’s evaluation declined with response delays exceeding 2 s or preemptive responses under 0 ms. Additionally, we aim to answer the research question of whether displaying emotions on the ECA’s face changes the user’s perception of the response delay.

4.2. Experimental Conditions

Participants in the experiment were international students enrolled at Tohoku University and were recruited through a bulletin board on campus. There were no restrictions on age range or English ability, but we explained that the dialogue would be conducted in English and asked only those participants who self-evaluated themselves as having sufficient English conversation skills to participate. For the definition of experimental parameters, we needed 24 participants to obtain a statistical power of 0.8 and an effect size of 0.8. Thus, we invited 25 participants (18 males and 7 females) to the experiment. Of these, 13 participants had experience using chatbots. Not all of the participants were native speakers of English; we asked the participants to choose their self-evaluation of English fluency among “beginner”, “intermediate”, “fluent”, and “advanced”. As a result, no participants were beginners, five were intermediate, five were fluent, and fifteen were advanced. The procedure of the experiment was as follows:

• We explained the purpose and procedure of the experiment to the participant.
• We asked the participant for a pre-dialogue questionnaire (

  Table 6. Questionnaire before the conversation.

  Q1	Gender
  Q2	Are you familiar with the concept of a chatbot? (No, I don’t know what a chatbot is./I don’t have much knowledge about it./Yes, I am familiar with the concept.)
  Q3	Have you already used a chatbot? (Never used before, Seldom use, Frequently use)
  Q4	How would you rate your English level? (Beginner, Intermediate, Advanced, Fluent)

  ).
• The participant held four rounds of dialogue with the agent. In each round, a participant conversed with the ECA about one particular artwork.
• After each conversation, the participant answered the questionnaire (

  Table 7. Questionnaire after each round of conversation (answered on 5-point Likert scale).

  Q1	Animation
  Q2	Conversation skill of the chatbot (Flow of the conversation, phrasing)
  Q3	Information accuracy (Does the agent’s answer correspond to your question?)
  Q4	Interactivity (Is the agent responsive to your inputs?)
  Q5	Interface (Layout, Visibility)
  Q6	Naturalness of the conversation (Does the agent look human-like when conversing?)
  Q7	Responsiveness (How fast does the agent answer back?)
  Q8	Overall satisfaction
  Q9	Usability (Ease of use)
  Q10	Visual aspect (Graphics)

  ) evaluating the current dialogue.
• After four rounds of dialogue, the participant answered the post-dialogue questionnaire (

  Table 8. Questionnaire after four rounds of conversation.

  Q1	Among the following parameters, what do you think hindered your conversation with the chatbot the most? (Visual aspect, Animation, Conversation skill of the agent, Conversation topic, Interactivity, Responsiveness, Interface)
  Q2	Please elaborate on your choice
  Q3	Which emotions/expressions were overrepresented? Underrepresented?
  Q4	Were you knowledgeable on the topic of artworks? Did the conversation topic hinder your ability to interact with the agent?
  Q5	Do you think one of the aspects of the system is not exploited enough?
  Q6	If you have any comments, please write them here

  ).

The order of the response delay and combination with the artwork are shown in

Table 9. The experimental design. In this table, “No.” is the participant number, and “Art”, “RD”, and “Emo” denote the artwork number, response delay, and existence of emotional expression, respectively.

	Round 1			Round 2			Round 3			Round 4
No.	Art	RD	Emo	Art	RD	Emo	Art	RD	Emo	Art	RD	Emo
1	1	500	Yes	2	1000	No	3	1500	Yes	4	2000	No
2	2	500	No	3	1000	Yes	4	2000	No	1	1500	Yes
3	3	500	Yes	4	1500	No	1	1000	Yes	2	2000	No
4	2	500	No	1	1500	Yes	4	2000	No	3	1000	Yes
5	1	500	Yes	2	2000	No	3	1000	Yes	4	1500	No
6	2	500	No	3	2000	Yes	4	1500	No	1	1000	Yes
7	3	1000	Yes	4	500	No	1	1500	Yes	2	2000	No
8	4	1000	No	1	500	Yes	2	2000	No	3	1500	Yes
9	1	1000	Yes	2	1500	No	3	2000	Yes	4	500	No
10	2	1000	No	3	1500	Yes	4	500	No	1	2000	Yes
11	3	1000	Yes	4	2000	No	1	500	Yes	2	1500	No
12	4	1000	No	1	2000	Yes	2	1500	No	3	500	Yes
13	1	1500	Yes	2	500	No	3	1000	Yes	4	2000	No
14	2	1500	No	3	500	Yes	4	2000	No	1	1000	Yes
15	3	1500	Yes	4	1000	No	1	500	Yes	2	2000	No
16	4	1500	No	1	1000	Yes	2	2000	No	3	500	Yes
17	1	1500	Yes	2	2000	No	3	1000	Yes	4	500	No
18	2	1500	No	3	2000	Yes	4	500	No	1	1000	Yes
19	3	2000	Yes	4	500	No	1	1000	Yes	2	1500	No
20	4	2000	No	1	500	Yes	2	1500	No	3	1000	Yes
21	1	2000	Yes	2	1000	No	3	1500	Yes	4	500	No
22	2	2000	No	3	1000	Yes	4	500	No	1	1500	Yes
23	3	1500	Yes	4	1000	No	1	500	Yes	2	2000	No
24	4	2000	No	1	1500	Yes	2	1000	No	3	500	Yes
25	2	2000	No	3	1000	Yes	4	500	No	1	1500	Yes

. The order was determined so that the different participants experienced different orders and combinations of artworks, response delays, and the presence of emotion. Overall, each parameter was balanced.

During the conversation, they asked the agent questions about the displayed artwork for as long as they wanted. The agent’s response delay was controlled with a minimum threshold so that the agent waited until the timer had finished. The response delay was set to 500, 1000, 1500, and 2000 ms because most of the response delay in a conversation is between 500 and 2000 ms [50]. The specific response delay was achieved by inserting an artificial delay after preparing the ECA’s response speech.

When the ECA was in the “with-emotion” condition, the ECA changed its facial expression according to the method explained in the previous section. In the “without-emotion” condition, the ECA’s face expressed a neutral emotion.

4.3. Questionnaire

We asked the participants three kinds of questionnaires: a demographic questionnaire before the conversation (Table 6), a questionnaire to evaluate each conversation (Table 7), and a questionnaire to evaluate the total impression of the ECA after all conversations (Table 8).

The questionnaire for evaluating the conversation was developed after the chatbot evaluation criteria by Peras [51], where the criteria for evaluating the ECA were separated into 14 categories: user experience [usability, performance, affect, satisfaction], information retrieval [accuracy, accessibility, efficiency], linguistic [quality, quantity, relation, manner, grammatical accuracy], technology [humanity], and business value. We removed the items related to the answer generation system (robustness of the agent, grammatical accuracy, accessibility, linguistic) because a set of answers was manually crafted. We also removed questions related to business value because they were irrelevant to this study. In addition, the naturalness aspect of the agent was divided into the visual and verbal aspects of the agent.

4.4. Number of Utterances in a Dialogue

As described above, we did not restrict the dialogue length; the participants spoke freely with the ECA. Therefore, the number of user utterances differs from dialogue to dialogue. Figure 4 shows the distributions of this number. Its average is 17.24, and its average duration is 287.66 s.

4.5. The Real Response Delay

As stated above, we controlled the response delay by inserting the artificial delay after preparing the response speech. However, if the preparation takes longer than the intended delay, the actual delay becomes longer. Figure 5 shows the jitter plots of the actual response delay. The red digits show the average. The figure shows that the average response delays were slightly longer for the 500 ms condition and almost as intended for the longer conditions.

4.6. Total Impression of the Conversation by the System

Figure 6 shows the mean values of the evaluation shown in Table 7, including all emotion and response delay conditions. The error bars show the standard errors. Values of the items “Interface” and “Visual aspect” were relatively high while “Naturalness” was relatively low. Most of the mean evaluation values were around 3, showing that the participants could use the system easily.

4.7. Effect of Response Delay and Emotional Expression

Next, we analyzed the evaluation of each conversation shown in Table 7 to determine whether their evaluation values were affected by the response delay and emotional expression.

Table 10. Results of MANOVA. The independent variables are emotions and different response delays; the response variables are the 10 items shown in Table 7.

Variable	Df	Pillai’s Trace	Approx. F	Df1	Df2	p
Response delay	3	0.23772	0.7315	30	255	0.84712
Emotion	1	0.16986	1.6983	10	83	0.09470
Delay × Emotion	3	0.45136	1.5053	30	255	0.04964

shows the result of the Multivariate Analysis of Variance (MANOVA) that investigates the effect of the emotional expression and the response delay upon all the evaluation values. As a result, we found a significant interaction between emotional expression and response delay (p = 0.050), but the independent factor did not show significance.

Table 11. Results (p-values) of the statistical test (ANOVA) for the evaluation of conversation. We used the response delay (500, 1000, 1500, and 2000 ms) and the emotional facial expression (yes or no) as factors. An asterisk (*) means 5% significance.

Item	Response Delay	Emotional Expression	Interaction
Animation	0.917	0.298	0.093
Conversation skill	0.794	0.192	0.270
Information accuracy	0.529	0.027 *	0.581
Interactivity	0.119	0.237	0.287
Interface	0.771	0.649	0.062
Naturalness	0.336	0.474	0.683
Responsiveness	0.026 *	0.069	0.038 *
Satisfaction	0.938	0.027 *	0.485
Usability	0.781	0.383	0.122
Visual aspect	0.834	0.273	0.258

shows the results of the Analysis of Variance (ANOVA) to evaluate differences among different response delay times and the presence of emotional facial expressions. In this analysis, the presence of emotional expression was treated as a categorical variable, and the response delay was regarded as a continuous variable. From these results, we found statistically significant differences in information accuracy (emotional expression), responsiveness (response delay), and satisfaction (emotional expression). We also found statistically significant differences in the interaction between the response delay and emotion for responsiveness.

Figure 7, Figure 8 and Figure 9 show the evaluation results of information accuracy, responsiveness, and overall satisfaction, respectively. As shown in Figure 7 and Figure 9, the participants’ perception of information accuracy and overall satisfaction increased when the agent changed facial expressions according to the emotion estimation; however, the response delay did not affect these evaluation results. The result of information accuracy seems to be related to the relationship between the speaker’s facial expressions and trustworthiness, where showing a happy face increases the speaker’s trustworthiness [52]. The overall satisfaction was not affected by the response delay, either. Chen et al. reported that the satisfaction of dialogue systems monotonically decreased when the response delay increased [53]. Their dialogue system used only speech and did not have facial images; this difference could cause the different responses of the users to the response delay.

On the other hand, the evaluation values of responsiveness decreased along with increasing response delay when the agent did not change facial expressions. This is a natural response of the participants because the responsiveness includes the speed of the agent’s answering behavior. Interestingly, the responsiveness value did not change for different response delays when the agent changed facial expressions according to emotion.

Asaka et al. reported that inserting a filler in parallel with the response generation process improved the users’ impression of the system’s responsiveness [54]. In their experiment, the physical response delay was actually shorter than the system without filler insertion; on the other hand, the response delay time of our system was constant regardless of the conditions. The reason the perception of the response delay changed is still unclear. One possible reason is that the users paid more attention to the agent’s facial expression, which made the perception of the response delay less sensitive.

4.8. Effect of Demographic Characteristics

We analyzed how the demographic characteristics of the participants affected the evaluation. Note that because the number of participants (twenty-five) is small and the distribution is not controlled, the inference from the result could be misleading. Therefore, we only show the results and discuss the possible reasons.

In this analysis, we excluded gender as a factor because the gender distribution was unbalanced (18 males and 7 females). We analyzed the effect of English proficiency (three levels: intermediate, fluent, and advanced) and the experience of having used chatbots (two levels: Yes or No; the answers ‘I don’t have much knowledge’ were merged to ‘No’). As a result of two-way ANOVA, we found statistically significant differences for Animation (interaction of English and chatbot use, p = 0.008), Naturalness (chatbot use, p = 0.045), and Visual aspect (English, p = 0.00012). As shown in

Table 12. Population of experience of using chatbot and English proficiency.

English	Use of Chatbot
	No	Yes
Advanced	7	8
Fluent	1	4
Intermediate	4	1

, English proficiency and chatbot use do not have a strong bias. Therefore, it is unclear why these factors affect the perception of visual presentation of the ECA.

4.9. Questionnaire Results After Four Rounds of Conversation

Finally, we summarize the results of the questionnaire after the four conversations. In Q1 of Table 8, the participant answered that the factors that hindered the conversation were the conversation skill (17 out of 25), responsiveness (4 out of 25), and interactivity (3 out of 25). The ECA sometimes gave inappropriate answers because the current system employs simple QA-database-based dialogue management. This problem could be solved by employing large language models (LLMs) for dialogue management.

5. Conclusions

In this study, we developed an ECA equipped with multimodal emotion recognition and emotional facial expressions. The agent and the user had a conversation about some artwork. The study examined how the agent’s response delay and facial expressions shape the user’s impression of the system. The results showed that short response delays did not significantly affect evaluations with the agent except for interactivity and responsiveness; emotional facial expressions positively impacted both satisfaction and information accuracy ratings.

Future research directions include implementing more sophisticated dialogue management systems using large-scale language models (LLMs). Current systems employ simple QA-database-based dialogue management, which may result in inappropriate responses; using LLMs will enable more natural and human-like conversations, further improving user satisfaction. It is also important to conduct a more detailed analysis of the interaction between emotional expressions and response delay. Understanding how emotional expressions change users’ perceptions of response delay will enable more effective agent design. Another possible topic for future research is investigating how users’ cultural backgrounds and personal characteristics affect their evaluations of agents’ emotional expressions.