Multi-Microworld Conversational Agent with RDF Knowledge Graph Integration

Round 1

Reviewer 1 Report (Previous Reviewer 2)

Paper was significantly revised, and authors also followed our recommendations in all points.

Author Response

Thank you kindly for all your feedback and for re-checking the paper, we highly appreciate it!

Reviewer 2 Report (Previous Reviewer 1)

Before reviewing this paper, first of all, it appears that the paper is still in editing and does not appear to be a final version. Therefore, it is impossible to understand the contribution claimed in this paper.

Of course, in this paper, agents such as chatbots suggest that NLU is possible through the knowledge graph. Still, judging whether NLU is possible only with the relationship between entities through the knowledge graph query statement is difficult.

Through the quantitative results of the syntactic mode proposed in this paper, it is necessary to provide a concrete basis for how much the performance of the proposed model improved NLU PIPELINE.

Also, the notation of the classes and properties used to build the knowledge should be consistent (refer to Table 1).

In addition, it is necessary to evaluate how much the NLU performance improved by interactive agents by dynamically inserting data into the knowledge base.

As a researcher and reader in the field of knowledge graphs, the topic of this paper is fascinating. Still, it isn't easy to pinpoint the academic contribution to the proposed model with this currently submitted version of the paper.

Author Response

Thank you kindly for your thorough review!

Before reviewing this paper, first of all, it appears that the paper is still in editing and does not appear to be a final version. Therefore, it is impossible to understand the contribution claimed in this paper.

  Response: We regret this confusion. The version you reviewed was a resubmission in which we wanted to highlight all performed changes. Now, we opted for a simplified color scheme in which new text is highlighted in blue.

Of course, in this paper, agents such as chatbots suggest that NLU is possible through the knowledge graph. Still, judging whether NLU is possible only with the relationship between entities through the knowledge graph query statement is difficult.

  Response: We clarified the scope of the article and the need for knowledge graphs.

Through the quantitative results of the syntactic mode proposed in this paper, it is necessary to provide a concrete basis for how much the performance of the proposed model improved NLU PIPELINE.

  Response: Table 2 presents the significant improvement from the baseline (F1= 75.9%) to the best model using RoBERT and syntactic features (F1=82.6%). We also clarified this in the text.

Also, the notation of the classes and properties used to build the knowledge should be consistent (refer to Table 1).

  Response: Entries have been updated. We are sorry that the current template does not support coloring the new values.

In addition, it is necessary to evaluate how much the NLU performance was improved by interactive agents by dynamically inserting data into the knowledge base.

  Response: This is a great suggestion, we introduced it in the conclusion as a future research path. It is too much for the scope of this paper, it can be an independent study by itself. Thank you very much for understanding.

As a researcher and reader in the field of knowledge graphs, the topic of this paper is fascinating. Still, it isn't easy to pinpoint the academic contribution to the proposed model with this currently submitted version of the paper.

  Response: Thank you kindly. We have clarified the research objectives and the main contributions.

Reviewer 3 Report (New Reviewer)

The topic of the paper is interesting and timely. The paper presents interesting work and contributes to the body of knowledge in the field. However, there is still room for improvement.

The main comments for improvements are: 

First, the aim and focus of the paper should be more clearly stated. The abstract raises huge expectations by stating a very significant and difficult problem. The introduction does not seems to include a clear statement of the objectives and scope of the study. The discussion section acknowledges the limitations  of the work which better indicate its scope. It is important that the aim but also scope of the research is clearly stated in the abstract and introduction. 

Second, an extensive and comprehensive literature review is missing. There are references in the introduction but a systematic literature review that provides a solid and comprehensive review of existing state of the art and also justifies the design decisions taken by the authors (when compared to alternatives from the literature) is missing. 

Finally, but also related to the first point, in the discussion section, there is a need for better explaining the value added of this research as compared to the literature. 

Regarding structure, the abstract needs re-writing to better adhere to the requirements of a scientific journal. The authors should also consider having a separate section on Literature Review after the Introduction section. 

Author Response

The topic of the paper is interesting and timely. The paper presents interesting work and contributes to the body of knowledge in the field. However, there is still room for improvement.

 Response: Thank you kindly for your thorough review and appreciation! All changes are marked in blue.

The main comments for improvements are: 

First, the aim and focus of the paper should be more clearly stated. The abstract raises huge expectations by stating a very significant and difficult problem. The introduction does not seems to include a clear statement of the objectives and scope of the study. The discussion section acknowledges the limitations  of the work which better indicate its scope. It is important that the aim but also scope of the research is clearly stated in the abstract and introduction. 

 Response: We rewrote much of the abstract to clarify our scope and we have also introduced the research objective in the introduction.

Second, an extensive and comprehensive literature review is missing. There are references in the introduction but a systematic literature review that provides a solid and comprehensive review of existing state of the art and also justifies the design decisions taken by the authors (when compared to alternatives from the literature) is missing. 

  Response: We have introduced additional references in the introduction and better clarified our goal.

Finally, but also related to the first point, in the discussion section, there is a need for better explaining the value added of this research as compared to the literature. 

  Response: Besides the changes in the first part of the paper, we have introduced a closing paragraph in the discussion section.

Regarding structure, the abstract needs re-writing to better adhere to the requirements of a scientific journal. The authors should also consider having a separate section on Literature Review after the Introduction section. 

  Response: The abstract was rewritten and we added the Literature Review section as suggested - indeed, the Introduction was too cluttered.

Round 2

Reviewer 3 Report (New Reviewer)

The authors have addressed most of the review comments and as a result the paper's presentation is improved.

However: 

1. There is still no systematic literature review performed (e.g. using PRISMA (https://www.prisma-statement.org/) and/or Webster and Watson [1]) but this is not a critical obstacle.

2. The adoption of Semantic Web / RDF is not sufficiently justified.

3. Since, chatbot/knowledge graph work in various domains is reviewed, [2] can be also mentioned.

4. Finally, the style of references should be checked to adhere to the journals guidelines (e.g. reference [11] starts with an "and"). 

---

[1] Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Quarterly, 26(2), xiii–xxiii.

[2] https://www.mdpi.com/2078-2489/13/5/225/htm  

Author Response

The authors have addressed most of the review comments and as a result the paper's presentation is improved.

Response: Thank you very much for your comments and feedback!

However: 

1. There is still no systematic literature review performed (e.g. using PRISMA (https://www.prisma-statement.org/) and/or Webster and Watson [1]) but this is not a critical obstacle.

Response: Thank you for your suggestion, the tool is indeed cool and we will consider it in other studies. Nevertheless, the main goal of this paper differs greatly from an extensive literature review on a domain, and all relevant studies have been included. Nevertheless, thank you for understanding that this is beyond the current goal of this paper.

2. The adoption of Semantic Web / RDF is not sufficiently justified.

Response: We have restructured the introduction to better follow our argumentation for selecting knowledge graphs and we have expanded the following paragraph:

“An agent must store and have access to knowledge to be useful in practical environments requiring contextualized interactions with the user. In contrast to relational databases that have a rigid structure, the agent should support a flexible and easily extensible representation model capable of making inferences on the underlying information. As such, we find it optimal to represent agent data in a knowledge graph that provides a unified form for storing information as nodes and properties, which can be later on queried starting from any given entity and can be used to make new inferences. Therefore, we consider a native Resource Description Framework (RDF) triplestore as a knowledge base for storing the various static and dynamic information the agent may harness and expand through user interactions. We describe a data representation model for various kinds of information and evaluate its viability by putting it into a practical personal assistant implementation with three microworlds.”

The argumentation is further expanded when considering additional knowledge graphs that may be explored: “In the future, this microworld might be used to handle general knowledge questions using an external data source, such as Wikidata or DBpedia, which can be queried through SPARQL, as well. This further emphasizes the adequacy of using knowledge graphs for presenting agent data.”

3. Since, chatbot/knowledge graph work in various domains is reviewed, [2] can be also mentioned.

Response: A good match, thanks! We have added the corresponding entry in the article.

4. Finally, the style of references should be checked to adhere to the journals guidelines (e.g. reference [11] starts with an "and"). 

Response: Great for pinpointing it out - we had an issue in the Bibtex entry. The paper considers the Overleaf template and we have checked all other references, everything seems in order now.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

This paper is an empirical study that expands the meaning of conversational agents using the RDF knowledge graph and shows the classification performance of language models through the syntactical function of sentences.

In this paper, the authors use the RASA, which includes various NLP functions, and RDF is applied to improve the extensibility of data modeling.

However, it is necessary to sufficiently explain the three contributions to be claimed in this paper more theoretically and objectively following the review comments:

1. The experimental results of table 1 show that when syntactic features are considered in the RoBERT large language model, which is a large-capacity corpus-based pre-learning model, the performance of intent classification is excellent in Precision, Recall, and F1-score. 

However, this result is not enough to explain how the syntactic features, which can be said to be the contribution of this paper, are applied to the LM model to bring about excellent performance. 

The quantitative results are needed to sufficiently explain the syntactic features' usefulness with diverse experimental datasets.

2. Also, to ensure the scalability of data modeling, the explanation of how RDF is applied to the conversational agent is insufficient. It is unclear in terms of quantitative results to claim that data modeling is extended only by deriving various relationships through the SPARQL queries.

3. To ensure the scalability of data modeling, how the proposed RDF modeling and what SPARQL queries can be applied for verifying intents. In other words, the authors should explain the advantages of RDF data modeling in detail.

4. In line 162, the GitHub[20] URL should be inserted into the reference page. Maybe, the URL for implementing conversational agents (line 338) should be mentioned in the reference.

Author Response

1. Thank you for your valuable feedback! We detailed the use of the syntactic features and provided more examples of how they were built and used in the pipeline.
2. Thank you for your observation! We addressed it by adding more details about the RDF knowledge graph both in the abstract and in the article’s text. We described the types of information currently stored using the RDF and the manner in which it can be extended.
3. In relation to the previous answer, we provided a more in-depth description of the RDF graphs, as well as additional examples of queries. We also added visual examples of SPARQL queries and examples of how the data is stored in the graph.
4. Great point, we introduced the URL in the reference, sorry for the Bibtex omission.

Reviewer 2 Report

In the paper the benefits of conversational agents in mediating the interaction in various context is explored. Authors enable plugging in different domains of knowledge, and to extend the conversational agent's area of expertise. The agent can learn new information dynamically from the user conversations by building a knowledge graph as a network of facts and information. In the abstract it is clearly given the results about intent classification task, but not directly what are the final experience and research conclusions regarding knowledge graph integration. Is this approach the only one available/novel regarding learning dynamically from the user conversations, or are also others, and if yes, how this methodology differs or relate to other approaches for this task. What are the limitations in case of RDF knowledge graphs for such task ? I would suggest that this is more clearly written in the abstract, and possibly in the paper, since the title contains RDF knowledge Graph, and readers would expect firstly new findings regarding using such graphs for conversational agents. 

Author Response

Thank you kindly for your feedback! We added a conclusion in the abstract about the RDF knowledge graph. We also expanded underlying details in the paper, where it was possible.