Enhancing Task-Oriented Dialogue Systems through Synchronous Multi-Party Interaction and Multi-Group Virtual Simulation

Abstract

This paper presents two innovative approaches: a synchronous multi-party dialogue system that engages in simultaneous interactions with multiple users, and multi-group simulations involving virtual user groups to evaluate the resilience of this system. Unlike most other chatbots that communicate with each user independently, our system facilitates information gathering from multiple users and executes 17 administrative tasks for group requests adeptly by leveraging a state machine-based framework for complete control over dialogue flow and a large language model (LLM) for robust context understanding. Assessing such a unique dialogue system poses challenges, as it requires many groups of users to interact with the system concurrently for an extended duration. To address this, we simulate various virtual groups using an LLM, each comprising 10–30 users who may belong to multiple groups, in order to evaluate the efficacy of our system; each user is assigned a persona and allowed to interact freely without scripts. As a result, our system shows average success rates of 87% for task completion and 89% for natural language understanding. Comparatively, our virtual simulation, which has an average success rate of 80%, is juxtaposed with a group of 15 human users, depicting similar task diversity and error trends. To our knowledge, it is the first work to show the LLM’s potential in both task execution and the simulation of a synchronous dialogue system to fully automate administrative tasks.

1. Introduction

Administrative tasks play a pivotal role in upholding organizational efficacy. Though they are typically overseen by human assistants, the increasing demand for swift task execution poses ongoing strains. To tackle this issue, chatbots have emerged as invaluable assets, streamlining tasks such as appointment scheduling and message dissemination [1,2,3]. Many dialogue systems aiming to tackle knowledge-based inquiries have also emerged, proving to be valuable assets within organizations [4,5,6,7,8,9]. Nonetheless, most current chatbots assist each user independently, despite many organizational processes requiring the assistant to engage with multiple users simultaneously to complete tasks. As a result, these chatbots often find themselves merely managing personal to-do lists and answering knowledge-based questions, leaving the bulk of the workload still for humans to handle.

Large language models (LLMs) can enable us to develop chatbots that facilitate synchronous multi-party dialogues, where multiple users are simultaneously engaged with a chatbot to fulfill their tasks. With LLMs’ notable enhancements, there is a growing interest in training LLMs to build end-to-end dialogue systems as well as in augmenting them to improve upon dialogue systems [7,8,10,11]. However, inherent challenges from LLMs, such as hallucination and lack of controllability [12,13], make deploying pure LLM-based systems difficult in practice. To incorporate LLMs’ language capabilities into real systems, there is a need to couple them with more manageable frameworks to ensure in-control task completion while achieving robust language interactions.

Developing a synchronous multi-party dialogue system is challenging, but evaluating such a system presents yet another hurdle. While a human-based evaluation offers the closest representation of real-life environments, recruiting a large group of users to thoroughly assess these dialogue systems over an extended period is time-consuming and resource-intensive. To address this, automated methods for simulating virtual users have been proposed; however, most of them simulate single users at a time, not simultaneous engagements from multiple users [14,15,16,17]. Thus, devising innovative solutions for conducting virtual simulations to assess such workload-intensive systems is imperative to develop dialogue systems that are well-prepared for real-life environments at minimal cost.

We present a dialogue system adept at conducting synchronous dialogues among multiple users for 17 administrative tasks (Section 3). In tandem, we introduce a multi-group simulator to facilitate the evaluation of our system by simulating users among multiple groups (Section 4). Our system is tested across various virtual environments produced by the simulator, showcasing high confidence in handling those tasks, and the results are promising for the effectiveness of our simulator and its resemblance to human interaction (Section 5). To our knowledge, this is the first system that comprehensively manages administrative tasks via synchronous multi-party dialogues and the first simulator accommodating users shared across multiple virtual groups.

2. Related Work

2.1. LLMs in Task-Oriented Dialogue Systems

With the emergence of LLMs for various natural language processing (NLP) tasks [18,19,20], their applications in task-oriented dialogue systems have grown, particularly for intent classification and dialogue state tracking. For intent classification, Zhang et al. [21] proposed a few-shot model using contrastive pre-training and fine-tuning, outperforming previous transformer-based models. Parikh et al. [22] adapted GPT for zero-shot prompting, showing competitive performance to few-shot prompting. For dialogue state tracking, Heck et al. [23] assessed GPT accomplishment of this task on two popular datasets [24,25], which showed state-of-the-art results. Feng et al. [26] developed a framework that leveraged open-source LLMs, yielding results comparable to those achieved with GPT.

While LLMs have been evaluated for those tasks, the research that has explored their sequential execution in dialogue systems is limited. Bang et al. [27] and Chung et al. [28] adapted GPT to build end-to-end task-oriented dialogue systems, surpassing prior systems in task execution. While such end-to-end systems offer many benefits, achieving fine-level controllability using these systems—a crucial aspect for real applications—proves challenging.

LLMs have also been individually employed and augmented to create systems capable of managing administrative tasks. For instance, frameworks presented by Gebreab et al. [4], Hays and White [5], and Musumeci et al. [6] illustrate how LLM augmentations are applied to administrative tasks across various domains, including cybersecurity and healthcare. These frameworks typically focus on information retrieval and extraction from user input [29]. Works such as Jung et al. [7] and Pan et al. [8] also developed task-oriented dialogue systems in which LLMs were used for knowledge-grounded augmentation. These systems outperformed most systems built on pre-trained models in knowledge-grounded administrative tasks, such as answering frequently asked questions [7]. However, most existing dialogue systems are not equipped to manage interpersonal administrative tasks, which require more intricate frameworks in conjunction with LLMs. Thus, we adopt a state machine-based framework to regulate dialogue flows, while utilizing an LLM for language understanding and generation.

2.2. Synchronous Multi-Party Chatbots

The domain of synchronous multi-party dialogue systems, especially those for task-oriented dialogues, is underexplored. Toxtli et al. [30] introduced the TaskBot, allowing users to request that other users be reminded to complete tasks. The TaskBot effectively facilitated the transfer of reminders between users, fostering asynchronous communication among them. Mendoza et al. [9] presented another chatbot for interactions between students and teachers (STBot), showcasing various features, including scheduling events for teachers, exchanging files among users, and addressing frequently asked questions.

While both chatbots exhibit multi-party dialogue capabilities, they rely on commercial frameworks; TaskBot operates in the Microsoft Bot Framework (Bot Framework: https://dev.botframework.com), and STBot is built on Google Dialogflow (Dialogflow: https://cloud.google.com/dialogflow). Hence, their functions are limited by the features offered in those frameworks, hindering the adaptation of LLMs. Our system stands out, as it enables real-time synchronous dialogues between the chatbot and users for general tasks without targeting specific groups, and it does so by fully leveraging LLMs.

2.3. Multi-Group Virtual Simulations

Due to the inefficiency in time and labor associated with assessing dialogue systems, several automated evaluation methods have been suggested, such as self-talk, where a chatbot engages in internal conversations by taking both the roles of the bot and users, or interactions between multiple different chatbots [31,32,33,34]. The advent of LLMs and their widespread application in diverse dialogue systems has further diversified evaluation approaches, integrating human, automated, and LLM-based assessments to evaluate generated responses [35,36]. Recently, Svikhnushina and Pu [37] introduced a dialogue evaluation pipeline employing an LLM to interact with chatbots on behalf of human users, and it showed promising results, although the LLM interaction was limited to one chatbot at a time.

LLMs have proven to be capable in replicating human-like behavior, and there have been prior works exploring the simulation of virtual users with various personalities using LLMs [38,39]. As a result of this capability, the use of LLMs for user simulation has been studied in task-oriented dialogue systems as well. Lu and Wang [17] developed a framework that generated virtual students using LLMs to evaluate the quality of multiple choice questions. Sekulic et al. [14], Wan et al. [15] employed LLMs fine-tuned on augmented datasets and compared them with existing models such as Llama [18] and GPT. Liu et al. [16] suggested applying multiple simulators, with each simulator behaving as a distinct user, instead of employing a single simulator to represent multiple users when training task-oriented dialogue systems. Despite the advancements in using LLMs to simulate virtual personalities, however, synchronous multi-party simulation is still underexplored, particularly in a multiple group setting where the same user can belong to several groups, as introduced in our work.

3. Assistant Bot

Our dialogue system, the Assistant Bot, operates on the Emora State Transition Dialogue Manager (STDM) framework, which offers versatile ways of defining dialogue states and utilizing third-party APIs for the development of an end-to-end dialogue system. Upon user input, STDM determines a relevant state by interpreting the input context, moves to another state based on this interpretation, and generates an output using a template [40]. While STDM supports a rapid method of building a dialogue system with great controllability, it has several limitations. First, STDM relies on its own formal grammar, NATEX (NATural language EXpression), for language understanding, which often falls short in accommodating the diverse expressions encountered during conversations. Second, each user’s database is stored independently, thus, information extracted from one user cannot be shared across other users; although this adds an extra layer of security, it complicates the process of conducting synchronous multi-party dialogues. Third, it was originally designed for passive interactions, meaning that a chatbot built on STDM waits for users to initiate conversations without proactively engaging with them.

Section 3.1, Section 3.2 and Section 3.3 address these three limitations in STDM to develop a more robust dialogue system. Moreover, Section 3.4 outlines the administrative tasks that the Assistant Bot can handle. Figure 1 showcases the framework and execution process of the Assistant Bot, and Figure 2 demonstrates an example of synchronous multi-party dialogue between two mock users, showcasing the functionalities of the Assistant Bot.

3.1. Language Understanding and Generation

To enhance natural language understanding (NLU), our system uses GPT-3.5 Turbo (henceforth, GPT), known for its exceptional performance, instead of NATEX [41,42,43,44].

For each user input, a few-shot GPT prompt template is employed, pairing a question prompt with the user’s input to extract the necessary information. GPT is directed to respond in a JSON schema, which is then integrated into the system’s framework as a set of variables that the Assistant Bot accesses for subsequent tasks. The few-shot GPT prompt template is as follows:

{full_request} Respond only in the JSON schema such as {examples}. Analyze this quote: “{quote_to_analyze}”

In this template, full_request corresponds to the specific request GPT must fulfill in order to extract the relevant information, examples refers to example JSON schemas that the LLM should use as a response format, and quote_to_analyze refers to the user’s input. For instance, if the user provides the name of the recipient with whom they wish to schedule a meeting, the variables are populated with information pertinent to the task. The final GPT prompt is as follows:

The speaker wants to schedule a meeting with another person. What is that other personś name or ID? Ignore honorifics such as Dr, Mr, Mrs, Ms, or Mx. Respond only in the JSON schema such as {V.other_person_full_name.name: ‘Nora Allen’} or {V.other_person_full_name.name: ‘N/A’} if unavailable. Analyze this quote: “I need to meet with Dr. Smith”

Figure 3 illustrates how GPT is employed for NLU. For each user input, GPT infers the task (Make Appointment) and extracts relevant information (Professor Smith). If necessary, the extracted information is matched to database entries (Elizabeth Smith). When there is an ambiguity (e.g., multiple users with the last name Smith who are also professors), it enters an intermediate state for further clarification (e.g., Which of these people did you mean?). This information is then used to determine the next state, and an output is created by a template predefined for the state. Finally, the templated output is fed into GPT to diversify the final output. Should GPT fail to extract the necessary information, a combination of NATEX and regular expressions is employed as a backup to extract the information. Alternatively, the conversation flow is redirected to a state where the chatbot asks for clarification and reattempts to extract the required information.

3.2. Shared Database

For seamless information exchange and facilitation of synchronous communication and task execution, our system employs MongoDB, a NoSQL database system (MongoDB: https://www.mongodb.com). Each user record in the database contains a unique user ID, groups that the user belongs to, conversation history and relevant information interpreted by GPT (Section 3.1), and an inbox storing the user’s appointments, messages, feedback questions, and responses to groups. Access to the records of other users is strictly restricted for any user. When a user requests a task from another user (e.g., making an appointment), the user indicates their preferences (e.g., date and location), stored in the user’s record, and grants the system permission to share particular parts of the information with the other user. Only the permitted information is shared among the targeted users to accomplish the task, allowing our system to send requested messages, anonymously or with names attached, to individuals or specific groups.

3.3. Proactive Engagement

Many administrative tasks require the system to actively reach out to users in a timely manner, which is difficult to accomplish without a proper interface. For proactive engagement with our users, a mobile app is created using Flutter, an open-source framework for developing multi-platform applications (Flutter: https://flutter.dev). Figure 4 portrays the dashboard and chat interface of the Assistant Bot application. When a user requests a task, the app sends a push notification to the other user’s application on their mobile device, facilitating engagement with our system among human users (Section 5) (our app is available on both iOS App Store and Google Play).

3.4. Administrative Tasks

Our system manages four types of administrative tasks, as outlined in

Table 1. Description of tasks that Assistant Bot can execute.

Type	Task	Description	Figures
N&H	Set name	Establishes the names that the user wishes to be called by	Figure A1
	List roster	Lists the users who are registered in the database
APP	Make appointment	Sends an appointment request to a different user	Figure A2
	Confirm appointment	Confirms an appointment request between two users and adds to schedules
	Edit appointment	Edits an appointment request’s details
	Reschedule appointment	Reschedules appointment request to different date
	Cancel appointment	Cancels appointment request
MSG	Send message	Sends message to recipient	Figure A3
	Check inbox	Checks inbox for messages and appointment requests	Figure A4
GRP	Make group	Makes group of multiple people	Figure A5
	Delete group	Deletes a group that has been created (must be leader of the group to proceed)
	Leave group	Leaves a group that user has been added to (must be a group member to proceed)
	Check groups	Checks groups that users are in	Figure A6
	Send announcement	Sends an identical message to group members (must be leader of the group to proceed)
	Request feedback	Sends feedback questions to group members (must be leader of the group to proceed)	Figure A7
	Send feedback	Records responses to feedback question (must be a group member to proceed)	Figure A8
	Check feedback	Checks statistics of feedback question (must be leader of the group to proceed)	Figure A9

. Each type of task corresponds to an individual component within the Assistant Bot which the chatbot transitions to if the identified request corresponds to the task listed in Table 1.

3.4.1. Names and Hub

Upon the initial interaction, the Assistant Bot requests the user’s official name and preferred names. While the official name is stored in the database and matched for task requests from other users, our system dynamically incorporates preferred names into subsequent responses to enhance user engagement. Upon name confirmation, it moves to the hub state, where it allows the user to request any task, categorizes it into a task in Table 1, and directs the user to the corresponding state.

3.4.2. Appointments

The Assistant Bot streamlines appointment scheduling by managing details such as date, time, location, and purpose. Upon mutual confirmation, it updates users’ schedules and resolves any scheduling conflicts. Requesting users can edit or cancel pending appointment requests, while both requesting and requested users can reschedule any appointments with passed or conflicting dates.

3.4.3. Messages

A user can dispatch messages to any other users in the database via the Assistant Bot. Users can send messages either anonymously or with their names; however, anonymity is maintained externally to ensure user safety.

3.4.4. Groups

The Assistant Bot allows users to create groups, with the creator becoming the leader and being able to add other users as members. Groups are identified by unique IDs, and users can check the groups they belong to at any time. Leaders oversee group activities, including creation, deletion, and communication through announcements. They can also request feedback from group members and review statistics on member responses. Members can respond to feedback requests from the leader and leave groups at their discretion; however, rejoining requires the leader’s permission.

4. Multi-Group Virtual Simulator

It is challenging to coordinate with human users to assess the Assistant Bot’s ability to handle multiple users in different groups, especially for researchers lacking established user bases. Thus, we present a multi-user simulator harnessing GPT’s capabilities in persona impersonation and dialogue generation [45,46]. Each user is personalized with a distinct prompt and their past dialogue history with the chatbot. The rest of this section outlines our simulation process.

4.1. Preprocessing

Prior to simulation, personas for u users across g groups are first generated by GPT. Each persona contains the user’s unique ID, name, occupation, personality, and associated class IDs. Leveraging GPT for persona generation enhances scalability, as increasing u and g requires no extra human effort with a small additional cost.

4.2. Initialization

Each virtual simulation starts with initialization, when a user greets the Assistant Bot. The Assistant Bot then asks the user for their name. Once our system receives the name, it transitions the user to the hub state, so the user can commence a simulated conversation. This process is applied to designated users until they are all initialized.

4.3. Interactions

During interactions, the simulator dynamically loads the user’s dialogue history into a prompt template, which is then fed into GPT to generate an output tailored to the user’s persona.

The simulator presents this output as a user turn and generates a subsequent system turn according the corresponding state. The conversation history, including the last system output to that user, gets updated accordingly.

Upon each system turn, two additional functions are executed using GPT. The first monitors the ongoing conversation between the current user and the chatbot, checking for any pending notifications or tasks that have been resolved. The results, presented as a dictionary containing notification and task counts for each user, are used to update the relevant users’ notifications and tasks in the database. The second function ascertains whether or not the conversation is concluded. If not, it continues with the current user until it meets a natural conclusion.

A conversation concludes once all relevant states have been explored to address the requested task. When a conversation with one user concludes, the simulator selects a different user based on their task and notification status, as follows:

$$\underset{\forall i\in U^{\prime }}{argmax}({\displaystyle \frac{(\alpha ·n_i)+(\beta ·t_i)}{{\Sigma }_{\forall j\in U^{\prime }}(\alpha ·n_j)+(\beta ·t_j)}})$$

$n_i$ and $t_i$ are the number of notifications and tasks of the i’th user, respectively. $U^{\prime }$ is the set of all user numbers, excluding the one assigned to the current user. In our case, $\alpha =2.5$ and $\beta =1$ are used such that users with higher notification counts receive greater weight, simulating real-life behavior, where people tend to prioritize notifications. From users with the highest weights, the next user is selected via random sampling to continue the simulation.

The prompts for generating the simulated user’s response, as well as those for updating notifications and tasks and checking if a conversation has concluded, are illustrated in Appendix A.1.1.

4.4. Termination

Each simulation is constrained by a maximum number of turns m, encompassing both system and user turns. The simulation terminates either upon reaching the turn limit or resolving all tasks and notifications for the designated users.

5. Experiments

5.1. Virtual Simulation

To assess the efficacy of our system, various college class scenarios are simulated. Given $(u,g)$, where $u/g$ equates to the respective number of users/groups (Section 4), five rounds of simulation are conducted, each composing different combinations, as follows:

$$(u,g)\in \left\{\right(10,1),(20,1),(20,2),(20,3),(30,1\left)\right\}$$

For instance, in the scenario $(20,2)$, we simulate 20 students across 2 classes, ensuring every student takes at least one of those two classes. Additionally, one instructor is simulated for each class.

Table A1. Personas given to the virtual users in the simulation. The first group of rows lists the instructors of each course (course written in Class), and the second group of rows lists the students. CS: Introduction to Computer Science, PSY: Introduction to Psychology, HIS: United States History: From Colonial Period to Present.

Name	Class	Personality	Tendencies
Joseph Cho	CS	enthusiastic	organized and plans material meticulously
Amanda Johnson	PSY	approachable	interactive and encourages participation
Michael Brown	HIS	analytical	enjoys solving complex problems, often takes a methodical approach
Olivia Adams	CS	creative	approaches problems with unconventional thinking
Leila Ahmed	CS	creative	enjoys experimenting, thinks outside the box
Mohammed Ali	CS, PSY	analytical	enjoys solving challenging problems
Michael Brown	CS	analytical	enjoys solving complex problems, often takes a methodical approach
Valerie Chen	CS, HIS	enthusiastic	organized and plans study schedules meticulously
Diego Fernandez	CS	outgoing	prefers group study sessions, actively participates in class discussions
Amelia Garcia	CS, PSY, HIS	independent	prefers to work alone, excels in self-paced learning
Rohan Gupta	CS, PSY, HIS	analytical	approaches problems logically
Mary Johnson	CS	introverted	enjoys individual projects, excels in structured assignments
Fatima Khan	CS, PSY, HIS	creative	applies creative problem-solving techniques
Daniel Lee	CS	team-oriented	works well in groups, enjoys collaborative projects
Maria Lopez	CS, PSY, HIS	curious	likes exploring new concepts, often conducts independent research
Rachel Martinez	CS	outgoing	works best in groups, enjoys discussing ideas with classmates
Ethan Miller	CS	diligent	thrives in structured learning environments, maintains organized notes
Yuki Nakamura	CS	focused	attentive in lectures
Sophia Nguyen	CS	curious	explores additional resources beyond lecture material
Aisha Patel	CS, PSY	reserved	prefers self-study, takes detailed notes during lectures
Aaliyah Patel	CS, HIS	friendly	enjoys group discussions
Nadia Ramirez	CS, PSY, HIS	organized	creates detailed study guides, follows a structured study routine
Emily Rodriguez	CS	friendly	prefers group study sessions, good at explaining concepts to peers
Isabella Rodriguez	CS, HIS	enthusiastic	eager to learn, actively participates in class discussions
Elijah Scott	CS	persistent	consistently follows through on projects, enjoys mastering difficult concepts
Alexandre Silva	CS	introspective	prefers self-paced learning
Kumar Singh	CS, HIS	enthusiastic	eager learner, actively participates in group discussions
Ben Smith	CS, PSY	easygoing	prefers studying in a group, collaborates on group projects
Haruto Suzuki	CS, HIS	reserved	prefers studying alone, methodical in approach
Sato Tanaka	CS, PSY	friendly	prefers collaborative learning
Ryne Waters	CS	analytical	works well under pressure, completes assignments last minute
Chloe White	CS	curious	enjoys exploring new topics
James Wilson	CS	adventurous	enjoys hands-on projects and experimentation

in Appendix A.1 describes the list of personas generated by GPT during the preprocessing stage (Section 4). Moreover, the maximum number of turns for termination in each round is set to $m=500$ (Section 4).

5.2. Human Simulation

To compare the behaviors of the virtual classroom with those of humans, a group comprising one professor and 15 college and graduate students is recruited to simulate a classroom scenario, interacting with our Assistant Bot via the mobile app (Section 3.3). Each user is provided a persona description to role-play, in tandem with those assigned to the virtual users in the simulation (Table A1 in Appendix A.1). To ensure fair comparisons, the class details for both the virtual and human simulations are kept consistent. This simulation comprises a total of 1821 turns.

We acknowledge that the sample size in the experiments is relatively small, which may limit the generalizability of the results. Future work will involve conducting experiments with larger sample sizes to ensure broader generalizability and to further validate the effectiveness of the virtual simulation and Assistant Bot.

5.3. Evaluation Metrics

For each round of simulation, the Assistant Bot and the virtual simulator are evaluated and compared to the human simulation. The task completion of the Assistant Bot is tested by how successfully it handles each requested task using the metrics below:

• NLU: The failure rate due to NLU shortcomings (in %).
• STD: The failure rate due to inherent limitations in the STDM framework, hindering it from exiting the current dialogue state until it is fully resolved, even when the user requests another task (in %).
• SUC: The overall success rate; 100 − (NLU + STD).
• SWS: The success rate without accounting for STD, used to evaluate system performance excluding fails due to limitations of the STDM framework; 100 × SUC/(100 − STD).

Moreover, each turn generated by the Assistant Bot is assessed for its robustness in natural language understanding using the following metrics:

• INT: The error rate resulting from inaccurate intent classification by GPT (in %).
• DST: The error rate resulting from inaccurate dialogue state tracking by GPT (in %).
• STD: The error rate due to the STDM limitations, leading to NLU failures (in %).
• SUC: The overall success rate; 100 − (INT + DST + STD).
• SWS: The success rate without considering STD; 100 × SUC/(100 −STD).

Finally, the appropriateness rate (APP) of the simulator is assessed for each virtual user turn, which addresses various factors such as hallucinations by GPT, responses irrelevant to the Assistant Bot’s last response, and repetitive statements disregarding the user’s previous responses. If the repetition pertains to the requested task, it is still considered relevant.

5.4. Results

Table 2. Success and error rates for task completion, natural language understanding, and the virtual simulator across the 10 evaluation metrics (Section 5.3). Vir-Avg: the average rates across the five scenarios (Section 5.1). Hum-Avg: the average rates within the human simulation (Section 5.2). Bolded values represent the highest success rates.

Scenario	Task Completion				Natural Language Understanding					Simulator
$(\mathit{u},\mathit{g})$	NLU	STD	SUC	SWS	INT	DST	STD	SUC	SWS	APP
(10, 1)	9.3	36.7	54.0	85.3	7.1	1.8	22.4	68.7	88.7	69.8
(20, 1)	7.8	34.6	57.6	87.8	4.6	3.0	23.9	68.5	90.0	81.8
(20, 2)	6.8	38.3	54.9	88.7	4.5	2.9	22.5	70.1	90.4	79.6
(20, 3)	6.2	37.7	56.1	90.0	3.6	5.3	20.9	70.2	88.7	83.6
(30, 1)	9.7	41.0	49.3	83.5	5.5	4.6	20.4	69.4	87.2	77.0
Vir-Avg	8.0	37.6	54.4	87.1	5.1	3.5	22.0	69.4	89.0	80.3
Hum-Avg	10.5	43.1	46.4	81.6	5.2	4.2	26.0	64.6	87.3	-

presents the evaluation results for our Assistant Bot and Virtual Simulator. For task completion, performance is notably influenced by STDM; when discounting STDM errors, it achieves high success rates of 87.1% and 81.6% for the virtual and human simulations, respectively. Interestingly, comparisons of different class numbers, $(u=20,g\in \{1,2,3\left\}\right)$, show improved SWS with more classes; distributing a fixed number of students across a greater number of classes reduces the size of each class, which may lead to more successful task completion.

For natural language understanding, the average success rates rise by 19.6% and 22.7% for the virtual and human simulations, respectively, when discarding the STDM errors, suggesting that while STDM enables rapid prototyping of dialogue systems, a more robust framework is needed for real applications. Both SUC and SWS remain relatively stable across all scenarios, confirming the robustness of the NLU approach enabled by GPT in our system.

Comparing the virtual and human simulations reveals similar performance trends, underscoring the adaptability of our virtual simulation for evaluating synchronous dialogue system performance. Both simulations produce comparable numbers of tasks per turn; 44% and 40% of turns in the virtual and human simulations, respectively, are task requests. While the human simulation comprises a broader range of tasks (20) compared to the virtual one (14), the virtual simulation still elicits a sufficiently diverse task distribution, facilitating effective evaluation of the bot’s capabilities. It is important to note, however, that it is easy to design a virtual simulation that accommodates a dynamic range of tasks, which can result in a more varied task distribution, as users are completely controllable in this setting.

6. Discussion

6.1. Error Analysis on the Assistant Bot

For each erroneous turn of the Assistant Bot during the virtual simulation, the generated dialogues and preceding context are examined to understand its limitations.

Table 3. Analysis of errors in the Assistant Bot. STDM: errors caused by the STDM limitations; NLU-DST/IC: NLU errors caused by GPT’s misinterpretation of dialogue state tracking & intent classification, respectively.

Category	Error Type	%
STDM	State Lock	0.68
	NATEX Issues	0.20
	System Bugs	0.11
	Multiple Task Requests	0.01
NLU-DST	Multiple State Extraction	0.84
	Information Mis-extraction	0.16
NLU-IC	Context Mis-interpretation	0.62
	Task Mis-classification	0.38

highlights the common errors that resulted in incorrect responses from the Assistant Bot. Examples of these errors are in Appendix A.3.

6.1.1. STDM Limitations

The primary source of the Assistant Bot’s errors stems from limitations in STDM, leading to various issues in dialogue flow. Of these limitations, g errors emerge as prominent issues, as the Assistant Bot cannot move on to a new task as a result of the STDM locking it into the current state until fully resolved, even when the user wishes to abandon the current task and transition to the new one (Figure A10). Although our system mainly uses GPT for NLU, it still defaults to NATEX as a fallback NLU method in cases where GPT fails due to internet connection or server issues. NATEX errors arise when NATEX takes over GPT for NLU with incorrect interpretation (Figure A11). Certain errors are induced by unexpected system bugs in STDM (Figure A12), while a minor portion of errors are attributed to STDM’s inability to handle multiple task requests in a single turn, in which cases, STDM processes one of the requests and disregards the others (Figure A13).

6.1.2. NLU Errors in Dialogue State Tracking (DST)

Most of the NLU errors occur due to GPT shortcomings in dialogue state tracking (DST) and intent classification (IC) tasks. The primary cause of DST errors is GPT’s limited ability to robustly extract multiple dialogue states at once, as the GPT prompt becomes too complicated when many states are included (Figure A14). Additionally, GPT variability and inconsistency sometimes result in incorrect information extraction.

6.1.3. NLU Errors in Intent Classification (IC)

For IC, the main source of errors stems from context management issues, where user inputs are misinterpreted as certain tasks. Due to overlap among tasks within the same type, such as making and canceling appointments, context ambiguity can result in incorrect intent classification (Figure A15). In addition, GPT may wrongly classify the user’s intended task, causing the Assistant Bot to enter an incorrect state.

6.2. Error Analysis on the Virtual Simulation

Every erroneous turn from the virtual simulation is analyzed to assess its effectiveness in handling synchronous multi-party dialogues.

Table 4. Error analysis on the virtual simulation.

Error Type	%
Ignoring Context	0.39
Forgetting Conversation History	0.30
Forgetting User Information	0.19
Confusing Conversational Partner	0.06
Role Reverse	0.06

shows the causes of these inappropriate responses. The most frequent errors occur when the virtual simulation ignores contexts from the Assistant Bot’s previous response (Figure A16). While these errors disrupt dialogue flow, it is important to note that they are not unlike human behavior, as humans often respond in ways that overlook the chatbot’s previous response.

Another primary cause stems from the LLM’s inability to focus on the provided information to accurately simulate the user. The simulated user frequently forgets parts of the conversation history, leading to requests for tasks that have already been completed (Figure A17). Furthermore, it fails to recall the provided information, generating irrelevant or incorrect content in relation to the given context.

The third type of error occurs when the simulation forgets information about the user, resulting in content that is inconsistent with the simulated user’s information (Figure A18). A small portion of errors occurs when the simulation becomes confused about whom it is conversing with, leading to responses that address the Assistant Bot incorrectly. This often happens when the bot delivers a message from another (simulated) user, causing the simulation to address the user rather than the Assistant Bot (Figure A19). Finally, role reversal occurs when the simulation fails to act as the user, instead generating responses that resemble the characteristics of the Assistant Bot (Figure A20).

6.3. Limitations

The limitations encountered by the Assistant Bot are largely rooted in the inherent constraints of the STDM framework. One significant issue is the use of NATEX. While NATEX is effective for the rapid prototyping of conversation transitions, it falls short in terms of natural language understanding due to its manually constructed nature. Although NATEX can be expanded to handle a variety of user inputs, creating a comprehensive dictionary that captures all possible transitions is impractical. This challenge, combined with the inherent ambiguity of natural language, often leads to categorization and transition errors. Furthermore, state lock errors prevent the Assistant Bot from smoothly handling multiple tasks. This is particularly problematic when users request multiple tasks within the same utterance or when they wish to switch tasks mid-conversation.

Given these limitations, exploring an alternative framework to replace the STDM could provide significant improvements. Since the STDM is robust in generating relevant natural language, a framework that can replicate the STDM’s robust natural language generation while also dynamically navigating between different task-oriented conversation states is crucial for enhancing our chatbot’s performance.

Additionally, the limitations of GPT contribute to errors in both the Assistant Bot and the virtual simulation. For the Assistant Bot, GPT’s variability and its limited understanding of administrative task categorization can lead to incorrect information extraction, resulting in errors in dialogue state tracking and intent classification. For the virtual simulation, maintaining in-character responses that aligned with user profiles and conversation history proved challenging, especially when dealing with extensive context. Fine-tuning user prompts or the model specifically for simulation tasks could improve accuracy in these areas. Furthermore, as GPT-3.5 is currently used for both the Assistant Bot and the virtual simulation, utilizing more powerful models such as GPT-4 could enhance the performance of both the simulation responses and the Assistant Bot’s natural language understanding.

7. Conclusions

While addressing the complexities of synchronous multi-party dialogue and multi-group virtual simulation presents challenges, this work is the first to demonstrate promising potential of a task-oriented dialogue system in simultaneously handling multiple users and leveraging large language models (LLMs) to supplement human evaluation. Despite limitations in the STDM framework and variabilities in GPT, our system exhibited robust natural language understanding and task execution, proving its practicality as a real-world application. Our virtual simulation also demonstrated a diversity in task requests similar to human interactions, underscoring its reliability as an evaluative tool to overcome the challenge of recruiting human groups.

However, the limitations of the STDM framework, coupled with the inherent variability of GPT models, suggests several directions for future research. Future work should focus on refining context management to further improve simulation performance and reliability. Enhancing the Assistant Bot’s ability to understand situational nuances could also strengthen its natural language comprehension. Exploring alternative frameworks that offer stronger language generation capabilities, while addressing the constraints of the current STDM approach, could further advance the system’s effectiveness. Additionally, applying the virtual simulation methodology to a wider range of group settings beyond the context of college classrooms could help generalize its applicability and adaptability.

While the Assistant Bot and virtual simulation help make significant advancements in the automation of administrative tasks and multi-user dialogue management using LLMs, continued research and refinement are necessary to fully realize the potential of these innovative approaches in diverse and dynamic environments. We look forward to advancements in task-oriented dialogue systems that employ synchronous multi-user features, as well as in evaluation frameworks that leverage the robust capabilities of large language models.