Building Trust in Conversational AI: A Review and Solution Architecture Using Large Language Models and Knowledge Graphs

Abstract

Conversational AI systems have emerged as key enablers of human-like interactions across diverse sectors. Nevertheless, the balance between linguistic nuance and factual accuracy has proven elusive. In this paper, we first introduce LLMXplorer, a comprehensive tool that provides an in-depth review of over 205 large language models (LLMs), elucidating their practical implications, ranging from social and ethical to regulatory, as well as their applicability across industries. Building on this foundation, we propose a novel functional architecture that seamlessly integrates the structured dynamics of knowledge graphs with the linguistic capabilities of LLMs. Validated using real-world AI news data, our architecture adeptly blends linguistic sophistication with factual rigor and further strengthens data security through role-based access control. This research provides insights into the evolving landscape of conversational AI, emphasizing the imperative for systems that are efficient, transparent, and trustworthy.

1. Introduction

The ability to comprehend and engage in natural language dialogue is an intrinsic aspect of human intelligence. In contrast, machines inherently lack this capability, necessitating methods to translate natural language into machine-readable formats and vice versa. Conversational AI, which has evolved since the mid-1960s, addresses this gap. Initial developments were based on pattern-matching, while more recent approaches leverage generative AI technologies for coherent, context-aware responses. This evolution has captured the interest of both researchers and the wider community, leading to applications across various domains such as healthcare [1,2,3], retail [4,5], journalism [6,7], finance [8,9], and education [10,11].

1.1. History and Evolution of Conversational AI

The journey of conversational AI can be traced back to the mid-1960s, marked by seminal technologies like Eliza [12], PARRY [13], and subsequent innovations such as Watson [14], A.L.I.C.E. [15], and GPT models [16]. Eliza’s keyword identification approach was succeeded by PARRY’s conceptual modeling of mental disorders. A.L.I.C.E. in 1995 marked a significant shift by utilizing heuristic pattern-matching with XML-based rules. The progression continued with SmarterChild’s [17] natural language processing in 2001, IBM Watson’s semantic analysis, and further personal assistants like Siri [18], Google Assistant [19], Cortana [20], and Alexa [21]. These developments culminated with OpenAI’s ChatGPT [22], grounded in the GPT-4o architecture, which harnessed vast datasets for more nuanced conversational capabilities, fueling a range of industry-wide use cases (Appendix B). Figure 1 provides a timeline encapsulating these critical advancements.

1.2. Development and Advancement of LLMs

Historically, language models were constrained by limited training data and specific tasks like classification. The introduction of generative pretrained transformers (GPT) marked a significant evolution, expanding the modeling capabilities to encompass zero-shot and few-shot learning [23,24]. LLMs now support a wide range of NLP tasks such as text summarization, machine translation, sentiment analysis, and named entity recognition.

Several types of LLMs have emerged, each with unique characteristics. Autoregressive models, like OpenAI’s GPT-3 and GPT-2, generate text by predicting the next word based on preceding words. Transformer-based models, including Google’s BERT [25], T5 [26], and PaLM [27], use self-attention mechanisms to manage dependencies in input data, enhancing performance on various tasks. Seq2Seq models, exemplified by the original Transformer model, are designed for tasks involving input and output sequences, such as translation. Additionally, multimodal models like OpenAI’s DALL-E and CLIP handle multiple data types simultaneously, showcasing advanced capabilities in processing and generating text and images.

The transformation began with BERT, a pioneer in bidirectional encoding [25], followed by OpenAI’s GPT-1, which laid the groundwork for generative AI. GPT-2 introduced “Prompt Engineering”, which significantly improved text generation [28]. T5 expanded the field further with an 11-billion-parameter model [26]. OpenAI’s GPT-3 became a landmark achievement with 175 billion parameters [29], while Google’s PaLM [27] and Meta’s LLaMA continued to push the boundaries with even larger models. OpenAI’s ChatGPT, fine-tuned from GPT-3.5, specializes in dialogue generation through reinforcement learning with human feedback (RLHF).

Recent innovations include OpenAI’s GPT-4o [30] and Google’s Gemini 1.5 Pro, which are multimodal models capable of processing text, audio, and images, reflecting rapid advancements that have paved the way for more robust and versatile natural language understanding and generation.

1.3. Key Challenges of LLMs

Despite the substantial advancements in LLMs and their growing prevalence in various applications, several inherent limitations and challenges remain. These include the following:

• Hallucination: In their quest for coherence, LLMs may inadvertently prioritize fluency, potentially at the expense of factual accuracy. Such behavior arises from the inherent nature of LLMs, which, while trained on expansive datasets, generate outputs based on recognized patterns rather than genuine comprehension. Consequently, while the generated content may appear plausible, it is susceptible to factual inaccuracies.
• Trustworthiness: The expansion in the deployment of LLMs in critical domains such as healthcare and finance accentuates the imperative for reliable output. Assessing the veracity of information, particularly when it originates from regions outside the model’s explicit training data, poses substantial challenges.
• Explainability: The complexity of deep learning models, LLMs included, often results in a phenomenon colloquially termed the “black box” problem. As these models evolve, incorporating an increasing number of parameters, their internal decision-making processes become less transparent. This opacity becomes particularly problematic in sectors where clear interpretability and rationale for outputs are mandated.
• Untraining ability: LLMs undergo meticulous fine-tuning based on vast datasets. Consequently, adjusting or rectifying specific knowledge components without embarking on a comprehensive retraining process remains a notable challenge. Such modifications become essential in cases where the previously acquired knowledge becomes outdated or is misaligned with evolving societal norms.
• Privacy and ethical awareness: Given the breadth of data on which LLMs are trained, there exists a non-trivial risk of inadvertently disclosing private or sensitive information during interactions. Moreover, there is an ethical dimension to consider, especially when models generate or amplify biased or potentially harmful content. Addressing these concerns mandates a holistic approach, encompassing not just model refinement but also infrastructural and oversight considerations.

To address some of the aforementioned challenges, retrieval-augmented generation (RAG) [31] has emerged as an effective NLP system that enhances LLMs by combining retrieval-based and generation-based models. In RAG, a retrieval model first extracts pertinent information from a vast database. A generation model then uses these retrieved data, along with the initial query, to formulate the final response. For the retrieval component, structured databases such as knowledge graphs or vector databases are crucial. However, for enterprise-scale applications that handle extensive datasets, vector databases often fall short compared to knowledge graphs. Knowledge graphs excel due to their ability to understand data semantically, represent data structurally, and their superior interpretability, scalability, and ease of integration with external knowledge. They also facilitate better reasoning and inference: capabilities that vector databases typically lack.

1.4. Main Contributions and Paper Organization

This paper proposes a novel architecture that combines the strengths of large language models (LLMs), knowledge graphs (KGs), and role-based access control (RBAC). By integrating LLMs’ natural language understanding capabilities with KGs’ structured representation of real-world knowledge and RBAC’s access control and privacy mechanisms, the proposed architecture aims to address the aforementioned limitations.

To enhance clarity, the paper is structured into two main parts. The first part provides an extensive review of the state-of-the-art of LLMs, highlighting their performance and limitations. This review sets the stage for understanding the critical gaps that need to be addressed. Building on these insights, the second part presents our proposed architecture, demonstrating how the integration of LLMs with KGs and RBAC can mitigate the identified challenges and improve overall system performance.

The main contributions of this paper include the following:

• A comprehensive review of large language models (LLMs): We present a Large Language Model Explorer (LLMXplorer), an Excel-based tool that encapsulates over 150 key open and closed source LLMs. It provides a systematic overview of 16 key features, including parameters, training hardware, applications, industry relevance, country of origin, and more, offering insights into the state and evolution of the field.
• Applied analysis of LLMs in various industries: An extensive investigation into practical use cases, limitations, and challenges in the technological adoption of LLMs across different sectors. This analysis contributes to a better understanding of real-world applications and the potential for trustworthy conversational AI.
• Functional solution architecture: The development of a state-of-the-art, privacy-aware, explainable, and trustworthy conversational AI architecture. This design uniquely integrates LLMs, knowledge graphs, and RBAC, and is empirically tested on the curated AI news dataset.

The rest of this paper is organized as follows: In Section 2, an exhaustive review of prominent LLMs is provided and LLMXplorer is introduced. Section 3 and Section 4 discuss the practical implications, technological impacts, market trends, and industry-specific applications of LLMs. Section 5 introduces a functional solution architecture integrating LLMs with knowledge graphs, aiming for a trustworthy conversational AI system. Section 6 presents the discussions and future implications. Finally, Section 7 concludes the paper.

2. Comprehensive Review of State-of-the-Art LLMs

The release of OpenAI’s GPT-3.5 in October 2022 [32] spurred significant advancements in LLMs, with major companies like Google and Meta launching new models to enhance accessibility and capabilities. Appendix A discusses the methodologies and high-level training processes tailored for LLMs, shedding light on key aspects such as architecture, training methodologies, and accessibility.

Recent surveys, such as [33,34,35], provide valuable insights into the development and capabilities of LLMs, focusing on architectural innovations, training methodologies, and reasoning capabilities. While these reviews offer substantial information, they often lack a dynamic and continuously updated tool for tracking ongoing developments. Our review addresses this gap with the introduction of LLMXplorer.

2.1. LLMXplorer: Large Language Model Explorer

With the proliferation of open-source LLMs, there are now over 36,000 distinct variations [36]. The Large Language Model Explorer, i.e., LLMXplorer, is freely available. Access requests can be sent via: https://forms.gle/TNUbqHiCBsinD4Bu8 (accessed on 2 June 2024) [37] is introduced as an Excel-based tool that provides insights into a concise catalog of over 205 LLMs based on strict inclusion criteria, such as public accessibility, detailed documentation, practical applications, recognizable releasing entities, and comprehensive training details. It classifies them based on capabilities, applications, industry sectors, and more. The LLMXplorer captures 17 key attributes for both open-source and proprietary models, including the following:

• Release Date: Official launch date of the LLM.
• Number of Parameters: Indication of model complexity.
• Base Model: Foundational pretrained model for the LLM.
• Training Data Size: Dataset volume used in training.
• Training Hardware: Infrastructure used in model training.
• Training Time: Duration of model training.
• Context Length: Length of the input text the model can process in one go.
• Target Application: Primary use case of the LLM.
• Model License Type: Governing license of the LLM.
• Modalities: Supported data types (text, image, video).
• Releasing Company: Entity releasing the LLM.
• Industry: Targeted domain or sector of the LLM.
• Origin Country: Base country of the releasing entity.
• Estimated User Base: Approximate adoption rate.
• Privacy Awareness: Model’s emphasis on data protection.
• Ethical Awareness: Model’s ethical design and use considerations.
• Reference Link: Source link for more LLM details.

2.2. Insights from LLMXplorer

Figure 2 illustrates the number of open-source and closed-source language models released each year from 2018 to April 2024, highlighting a significant increase in the release of open-source LLMs starting in 2020, with a peak in 2023 at over 60 models. In contrast, the number of closed-source LLMs also increased initially but plateaued at around 20 models per year from 2021 to 2023. This trend highlights the growing preference and focus on open-source development in the AI community, possibly driven by the benefits of collaborative improvement and transparency.

Figure 3 and Figure 4 provide detailed timelines of large language model (LLM) development for both open- and closed-source categories. From Figure 2, over the past seven years, open-source LLMs have seen an average of 19 releases per year with considerable variability (standard deviation of 21.6), peaking at 65 models in one year. In comparison, closed-source LLMs, over the last five years, have averaged 14 releases per year with a more consistent output (standard deviation of 9.87) and an annual range of 3 to 25 models. Both sectors demonstrate a growth trend in annual releases, underscoring a heightened interest in LLM development. Notably, the growth rate of closed-source models is slower, suggesting differences in community engagement or resource allocation.

Development frequencies by company, represented in Figure 5 and Figure 6, reveal that Meta dominates in the realm of open-source LLM development, while Google Research is the leader in developing closed-source LLMs, highlighting the significant roles these companies play in the field.

The geographical distribution of LLM development, as seen in Figure 7 and Figure 8, indicates that the USA is the leading country in developing both open- and closed-source models, followed by the UK, Canada, and China, showing a strong regional concentration in LLM advancements.

Figure 9 illustrates the parameter distributions for open- and closed-source models. Open-source LLMs show a parameter range from 0.035 billion to 198 billion, with an average size of 27.77 billion parameters. Models like CPM2, BLOOM, BLOOMZ, OPT, and BlenderBot3 exceed 175 billion parameters. Conversely, closed-source models range from 0.06 billion to 1200 billion parameters, with an average of 182.94 billion. Exceptionally, models like GLaM and PanGu-E have surpassed the trillion-parameter mark.

3. Applied and Technology Implications for LLMs

This section highlights the social, ethical, legal, privacy, regulatory, and physiological implications of LLMs.

3.1. Social and Ethical Implications

The open-source availability of large language models (LLMs) has democratized AI access, fostering innovation across various user groups and providing a competitive alternative to the offerings of tech giants [38]. Techniques such as LoRA [39] allow efficient internal data handling and quick model fine-tuning for personalized model development. However, the complexity of LLMs introduces risks of misuse and bias, highlighting the need for diverse development teams and comprehensive bias audits to ensure fairness. Moreover, disparities in LLM access potentially exacerbate societal digital divides, necessitating enhanced efforts in technology dissemination and education [40].

Responsible development of AI focuses on balancing the benefits of LLMs with potential risks, requiring collaborative efforts from multiple stakeholders to promote transparency, robust data governance, and educational outreach. The opaque nature of these models (“black box”) complicates trust and understanding, making research into the explainability and development of user-friendly interpretations crucial. Furthermore, addressing the propagation of harmful content involves implementing diverse training datasets and watermarking techniques, supported by both algorithmic and human oversight mechanisms to ensure safe and responsible LLM deployment [41].

3.2. Legal, Privacy, and Regulatory Perspective

LLMs must adhere to regulations like GDPR while preventing unintentional data disclosure. Recent regulatory shifts, like the “EU AI ACT” [42] and proposed Chinese regulations, emphasize responsible generative AI use. Furthermore, LLMs’ capacity to generate human-like text challenges copyright norms. Addressing this requires clear intellectual property guidelines and plagiarism detection tools [43].

3.3. Physiological Perspectives

LLM-based conversational agents carry potential risks, including undue trust, psychological vulnerabilities, and design biases. Promoting responsible design and prioritizing user privacy is paramount. Anthropomorphized systems may lead to user overreliance, risking unsafe outcomes or psychological harm, which underscores the need for diligent oversight and responsible usage practices. Additionally, users might unknowingly divulge private information to these agents, highlighting the importance of implementing robust safeguards and launching user education campaigns to prevent misuse. Furthermore, the design of conversational agents often unintentionally reinforces stereotypes, requiring sensitivity and adherence to unbiased design principles to mitigate such biases [44].

4. Market Analysis of LLMs and Cross-Industry Use Cases

4.1. Market Size of LLMs and Driving Factors

The global natural language processing (NLP) market is projected to surpass USD 91 billion by 2030, expanding at a CAGR of 27% [45]. A major driving force behind this growth is the influence of large language models (LLMs). While exact market figures for LLMs remain elusive, certain factors listed in

Table 1. Factors driving the LLM market growth.

Factors	Description
Enterprise Adoption	Organizations utilize LLMs to optimize customer service, automate operations, and enhance efficiency. Sectors such as e-commerce, healthcare, and finance are primary beneficiaries.
AI Industry Growth	LLMs, central to AI initiatives, benefit from the holistic expansion of the artificial intelligence sector, accentuated by deep learning advancements.
Language-as-a-Service (LaaS) Emergence	LaaS platforms provide ready-to-use language models and APIs, enabling enterprises to harness language capabilities without intensive model training.
R&D Momentum	Continuous advancements in NLP and machine learning propel the market. Innovations in model design, training techniques, and performance optimization catalyze industrial interest.

hint at their increasing dominance. Despite the absence of precise market data, it is evident that the sector demonstrates substantial potential and is set to continue expanding across various industries.

4.2. LLM Development Opportunities

The development of LLMs presents several applied challenges that are critical to address in order to mitigate potential risks associated with their deployment. One of the primary concerns is the ability of LLMs to generate disinformation, which can convincingly undermine the credibility of information and lead to harmful consequences. Additionally, these models have the capability to produce deepfakes and other forms of sophisticated manipulated media, which pose significant threats of deception and can be used to manipulate public opinion.

Privacy and data concerns are also paramount, as the handling of vast amounts of training data can introduce vulnerabilities, potentially leading to data breaches and the exposure of sensitive information. Moreover, there is the risk of bias amplification, where LLMs may inadvertently reflect and reinforce existing biases present in their training datasets, thus perpetuating societal prejudices.

The potential for malicious usage of LLMs to produce harmful, offensive, or abusive content is another significant challenge, which can facilitate hate speech or targeted assaults. The outputs from LLMs can sometimes lead to unintended outcomes or raise ethical concerns, highlighting the urgent need for stringent oversight and the establishment of comprehensive guidelines to ensure responsible development and application of these technologies. Recognizing the challenges associated with LLMs, there exists a spectrum of opportunities for refining and evolving these models, as shown in

Table 2. Opportunities in LLM evolution.

Opportunity	Description
Bias and Fairness	Develop techniques to detect and neutralize biases in LLMs, ensuring outputs that respect diverse perspectives.
Ethical Guidelines	Forge ethical standards and frameworks for the responsible creation and deployment of LLMs.
Explainability	Innovate methods to augment LLM transparency and interpretability, clarifying decision-making processes for users.
Contextual Accuracy	Enhance LLMs’ contextual comprehension to produce more relevant and precise responses.
Collaborative Efforts	Foster collaborations amongst stakeholders, like researchers and policymakers, to address LLM-related societal and ethical implications.

, with the potential to fuel cross-industry business use cases (see Appendix B).

5. Solution Architecture for Privacy-Aware and Trustworthy Conversational AI

With the advancement of conversational AI platforms powered by LLMs, there arises a need to address issues of explainability, data privacy, and the ethical use of data to ensure their safe usage in industry and practical applications. LLMs, despite their significant natural language understanding capabilities, often operate as black-box models, making their decisions difficult to interpret. Integrating knowledge graphs (KGs) [46] with LLMs offers a solution to these challenges by coupling the structured knowledge representation of KGs with the linguistic proficiency of LLMs. Furthermore, the incorporation of role-based access control (RBAC) into the architecture ensures that access to the AI system aligns with organizational policies and is granted only to authorized roles.

The integration of key components such as LLMs, KGs, and RBAC systems provides distinct advantages in the development and application of conversational AI platforms.

LLMs are foundational to conversational platforms, offering deep linguistic understanding that facilitates nuanced interactions. These models are capable of continuous adaptation, refining their responses based on ongoing user interactions, which enhances the relevance and accuracy of their output over time.

KGs deliver structured and validated domain-specific knowledge, significantly enhancing the system’s explainability. They do this by tracing the origins of information, which not only clarifies the decision-making process but also complements LLMs by filling gaps in domain-specific expertise. This synergy between LLMs and KGs is crucial for applications requiring a high degree of accuracy and reliability in specialized fields.

RBAC ensures data privacy and aligns AI system usage with organizational policies and compliance mandates through controlled access mechanisms. This methodical approach to managing data access significantly advances the security of AI systems, providing a structured framework that supports secure and efficient operations across various levels of user engagement.

Several architectural paradigms can merge KGs and LLMs. As detailed by Shirui Pan et al. in [47], the primary techniques include KG-enhanced LLMs, LLM-augmented KGs, and a synergistic LLMs+KGs approach. Our proposition of a closed-loop architecture is in line with this synergistic design, underscoring the advantages of combining LLMs and KGs.

5.1. System Components and Design Choices

To elucidate the practical implementation of our architectural design, we anchor our exposition using media and journalism, specifically leveraging news data aggregated from the AI NewsHub platform (https://ainewshub.ie, accessed on 2 June 2024).

5.1.1. AI NewsHub Dataset Description

The AI NewsHub dataset is a systematically aggregated collection of news articles, procured daily from web crawling and search engines, and stored within a relational database system. Each record within this dataset represents an article and encapsulates the following attributes: identifier (ID), title, article content, published date, publisher name, and associated country of origin. Upon acquisition, articles undergo an analytical phase wherein they are classified based on their relevant topics, affiliated industry sectors, and originating publishers’ categories.

5.1.2. Construction of the AI NewsHub-Based KG

The KGs derived from the AI NewsHub dataset comprise two primary node classes: articles and topics. The ”article” node includes attributes such as ID, content, sentiment analysis results, and a multi-dimensional vector array. The ID and content are extracted directly from the dataset, sentiment is derived using the Vader sentiment analysis tool [48], and the vector array is generated using the FastText library [49].

Each ”article” node establishes a directed relationship with a ”topic” node if it is categorized under a specific topic. The ”topic” node contains the topic ID and name, extracted from the topic table in the dataset. The KG is managed using Neo4j [50], with queries structured in Cypher [51].

The data processing involved several key steps:

• Extraction: Articles were parsed to extract relevant content using advanced natural language processing (NLP) techniques. The dataset included topic tags and industry sectors for AI news articles.
• Transformation: The extracted content was transformed into a structured format suitable for graph representation, involving tagging and categorizing key entities and topics.
• Data cleaning: Ensuring the quality and accuracy of the KG was crucial. This included the following:

  –

  Deduplication: Identifying and merging duplicate nodes and relationships to avoid redundancy and ensure uniqueness.

  –

  Normalization: Standardizing entity names and topics for consistency. Topics and industry sectors were mapped to each article, with relevant tags available on the AI NewsHub web portal at www.ainewshub.ie (accessed on 2 June 2024).

  –

  Validation: Cross-referencing data with reliable sources to verify accuracy before adding them to the KG, ensuring reliability.

• Loading: The structured data were loaded into the Neo4j database, creating nodes and relationships according to the defined schema.

By the end of the development phase, the KG comprised approximately 13,000 nodes and 26,000 relationships, categorized into articles, topics, entities, and relationships such as “HAS_TOPIC”. A visual representation of a KG node is also shown in Figure A3.

5.1.3. Llama-2 LLM

In July 2023, Meta introduced the Llama-2 series [52], an advancement in its large language model lineup. The models in this series are characterized by parameters ranging from 7 billion to 70 billion. When compared to the preceding Llama-1 models, Llama-2 variants have been trained on an expanded dataset with an increase in the token count of 40% and an extended context length of 4000 tokens. The 70B model incorporates the grouped-query attention mechanism [53], which is designed for efficient inference.

A specialized variant within the Llama-2 series is the Llama-2-Chat, which is fine-tuned for dialogue applications using reinforcement learning from human feedback (RLHF). Benchmark evaluations indicate that the Llama-2-Chat model offers improvements in areas such as user assistance and system security. Performance metrics suggest that this model’s capabilities align closely with those of ChatGPT, as per human evaluators. Additionally, the 70B version of Llama-2 has been recognized on HuggingFace’s OpenLLM leaderboard, exhibiting strong performance on several benchmarks, including but not limited to ARC [54], HellaSwag [55], MMLU [56], and TruthfulQA (MC) [57].

With the details of the three main components provided, we will now describe the proposed system architecture in the next section.

5.2. Architecture Workflow

Figure 10 and Figure 11 present the architectural workflow of our proposed system. The former offers a more general overview, elucidating the synergistic connection between these three components. The latter is a detailed view of the intricate interactions between knowledge graphs (KGs), large language models (LLMs), and role-based access control (RBAC).

The detailed sequence of operations in the system is described as follows:

• Step 1: The user (U) communicates a specific request to the RBAC service (S).
• Step 2: The RBAC service (S) evaluates the permissions associated with the user, forwarding the request to the access control (AC). The AC determines the user’s data access rights. Upon granting access, the process proceeds to Step 3. If denied, it transitions to Step 2.2.
• Step 3: The prompt analysis module refines and refactors the user’s prompt (if necessary) and identifies the key capabilities required to put together an appropriate response. In our case of journalism, the key capabilities include natural language understanding and generic output response or specialized capabilities such as similar article finder, sentiment analysis, fact-checking, and prediction for article topics and relevant industry sectors.
• Step 4: The Llama-2 LLM processes the user’s request based on the identified capabilities. If a generic response is required, the LLM responds to the user directly (Step 4.1). If specialized features from the KG are required, the process moves to Step 4.2.
• Step 5: Llama-2 generates or invokes relevant Cypher instructions for Neo4j based on the required capabilities.
• Step 6: A Cypher validation layer (CVL) ensures the integrity and safety of these instructions. Validated queries are executed on the Neo4j knowledge graph.
• Step 7: KG processes the queries, extracting the pertinent data. An error handling (EH) mechanism inspects the extracted insights for anomalies.
• Step 8: Error-free insights are compiled to be returned back to the LLM.
• Steps 9 and 10: LLM formats the insights for user-friendly presentation and provides a response to the user. The user (U) receives the curated data, ensuring only permitted information is accessed. Users have the option to offer feedback through a feedback loop (FB), which might guide the LLM’s subsequent interactions.

Based on the presented workflow, we now elucidate the decision-making process of the language learning model (LLM) based on two primary scenarios: one where the LLM interacts with the knowledge graph (KG) to deduce answers, and another where the LLM operates autonomously.

5.2.1. Interactions between LLM and KG

The LLM’s reliance on the KG is made manifest in the following application scenarios:

1. Similar article finder

For implementing recommendation systems in media and journalism services, it is imperative to identify articles akin to a given one. Using the cosine similarity metric, we ascertain the resemblance by evaluating the cosine value between the focal article and others. The topmost articles in terms of similarity scores are then selected, as represented in Algorithm 1.

Algorithm 1: Identify Top 5 Articles resembling Article 100

Computing the cosine similarity between two vectors typically has a complexity of $O\left(n\right)$, where n is the dimensionality of the vectors. Given that we are comparing a given article to m articles, then the complexity becomes $O(n\times m)$. Sorting the similarity scores to retrieve the top five has a complexity of $O(mlogm)$. Therefore, the overall complexity is $O(m\times n+mlogm)$.

2. Article sentiment analysis

In situations necessitating sentiment extraction from a specific article or specific articles with the same set of sentiments, the model retrieves the sentiment pre-recorded in the article node within the KG. This retrieval is achieved by executing a MATCH Cypher query based on the article’s unique identifier, as delineated in Algorithm 2.

Algorithm 2: Sentiment Extraction for Article with article_id 100

The complexity here is dominated by the search operation. Finding a node by its unique identifier in a well-indexed graph database like Neo4j is an $O\left(1\right)$ operation.

3. Article topic prediction

Articles devoid of any designated topic can be assigned one based on the topics of closely resembling articles. This is realized by associating the given article with others and then linking the most similar one’s topic. Algorithm 3 elucidates the related Cypher query.

Algorithm 3: Topic Inference for Article 100 from its analogous article

Similar to the “Similar Article Finder”, the complexity of computing the cosine similarity across all articles is $O(m\times n)$. However, in this case, we also have a filtering operation, which selects articles with a similarity score greater than 0.97. The complexity of filtering is $O\left(m\right)$. Returning only one result (using LIMIT 1) does not substantially change the computational complexity, but it significantly improves the real-world performance of the query since it halts once a match is found. Therefore, the overall complexity in this case is $O(m\times n)$.

5.2.2. Autonomous LLM Operations

LLMs extensively trained on diverse datasets demonstrate the ability to autonomously handle certain user queries without relying on external databases such as KGs. This autonomous capability is particularly evident in scenarios that require a deep understanding of text, summarization abilities, or responses that do not demand explicit verification against structured knowledge sources. For instance, in text summarization, the LLM applies its extensive training to distill and encapsulate essential points from articles independently of KG interactions. Additionally, the LLM efficiently addresses generic journalism queries, drawing on its comprehensive training across journalism-related content to provide insights on journalistic standards, writing styles, and media ethics. Moreover, the model excels in contextual interpretations, where it leverages its intrinsic understanding of relational data to interpret and respond to journalism-related inquiries based on the surrounding context without the need to extract explicit facts.

5.3. Architecture Evaluation for Selective Tasks

We performed an experimental evaluation of the proposed architecture using the AI NewsHub dataset to benchmark our solution against two mainstream approaches: Approach A (baseline Llama-2 without KG) and Approach B (LLM with vector databases) using proprietary Microsoft Azure GPT-3.5 with an off-the-shelf RAG pipeline employing hybrid search.

The accuracy of each approach was evaluated using specific applications: contextual information retrieval and query-based knowledge extraction. This metric was calculated by comparing the retrieved information against a manually curated ground truth dataset for 50 news articles.

Table 3. Architecture evaluation for information retrieval and contextual understanding.

Metric	Proposed Solution	Approach A	Approach B
Information Retrieval Accuracy (%)	95.4	60.3	72.3
Contextual Understanding Accuracy (%)	93.8	70.2	75.1

demonstrates the superior performance of the proposed architecture over existing approaches in terms of the accuracy of information retrieval and contextual understanding.

6. Discussions

Conversational AI, while advancing rapidly, faces hurdles in balancing linguistic depth with accurate information representation. As highlighted in our comprehensive review through the Large Language Model Explorer (LLMXplorer), which provides a systemic overview of numerous LLMs, linguistic proficiency often comes at the expense of transparency. Knowledge graphs, while offering factual precision, might fall short of mimicking human conversational fluidity.

The evolution and integration of LLMs in the digital environment necessitate a comprehensive understanding of their multifaceted implications. Open-source models promote AI democratization, paving the way for innovation. Concurrently, the challenges presented by algorithmic bias and the potential widening of digital divides underscore the importance of interdisciplinary collaboration among technologists, policy-makers, and end-users. The clarification of legal frameworks, especially in data protection and intellectual property, becomes essential given LLMs’ capabilities. Additionally, the anthropomorphic attributes of these systems elevate concerns about user over-reliance and misplaced trust. Achieving the optimal utility of LLMs requires a concerted effort to advance technology, which underscores their vast potential. Still, it also highlights the need for architectures that can bridge the aforementioned gaps and, in response, address ethical, legal, and physiological dimensions.

Through our exhaustive applied analysis of the practical use cases, challenges, and limitations of LLMs across industries, our work introduces a functional solution architecture that uniquely integrates KGs and LLMs. This system not only stands out in its linguistic capabilities but also ensures factual consistency. With the incorporation of RBAC, we further the cause of data security by restricting users to role-specific information and thereby fostering trust for real-world and industry-wide use cases.

The application domain of media and journalism, exemplified using rich data from the real-world product (AI NewsHub) platform, serves as both a case study and a validation point. It is a testament to the architecture’s robustness and efficiency. Notably, the adaptability of the proposed architecture means it can seamlessly cater to a myriad of use cases and applications, emphasizing its cross-industry relevance.

The proposed architecture underscores its significance through several pivotal design choices. It employs a specialized large language model (LLM), Llama-2-Chat, which is fine-tuned for dialogue, ensuring enhanced conversational accuracy and quality. The use of Neo4j for knowledge graph navigation leverages its open-source nature, widespread industry adoption, and proficiency in navigating knowledge structures, facilitated by the efficient Cypher query language. A Cypher validation layer is strategically positioned between the LLM and Neo4j to validate queries, mitigating potential unintended data access and enhancing system security. Additionally, the architecture incorporates a feedback mechanism that allows for the iterative refinement of system responses and continuous learning of the LLM. Moreover, the system adopts specific algorithms for tasks such as article similarity detection, sentiment analysis, and topic prediction, which contribute to its transparency and maintainability.

Our solution architecture presents critical advancements tailored to address contemporary challenges in conversational AI. Through the seamless amalgamation of KGs and LLMs, the design strikes an equilibrium between linguistic sophistication and factual veracity, obviating the shortcomings inherent to individual models. The employment of the Llama-2-Chat LLM enhances conversational fidelity, elevating the user interaction experience. By harnessing Neo4j for knowledge graph navigation, the architecture not only optimizes data retrieval but also exhibits adaptability to evolving data paradigms. The use of RBAC provides further guardrails for enhanced data security. Additionally, an iterative feedback mechanism ensures the system’s alignment with dynamic user needs. Our methodological stance, demonstrated through a tailored algorithmic approach, enhances transparency and mitigates the “black box” dilemma associated with deep learning-based models including LLMs. In essence, our architecture epitomizes a forward-thinking approach, synergizing linguistic depth with factual accuracy, all while underscoring user trust and systemic resilience.

6.1. Research Limitations

While our architecture has been rigorously tested and validated within the media and journalism context using the AI NewsHub platform, its performance and applicability in other industries remain to be explored. This could potentially limit the generalizability of our findings.

The employment of the Llama-2-Chat demonstrates promising conversational capabilities. However, its comprehensive adaptability and performance in diverse linguistic and cultural scenarios are yet to be thoroughly evaluated.

Furthermore, the iterative refinement enabled by our feedback loop is heavily dependent on user engagement and the quality of their input. This introduces the potential risk of bias or skewed learning if not carefully curated.

Lastly, our study focuses predominantly on the Neo4j database for navigating the KG. The exploration of benefits or challenges associated with alternative knowledge graph databases is beyond this research’s scope but warrants attention in future investigations.

6.2. Future Implications

The converging point of linguistic depth and factual accuracy, as epitomized by our system, is a promising indication of where conversational AI is headed. It sets a precedent for building architectures that are not only sophisticated in their response generation but also trustworthy in the information they convey. By addressing key challenges and providing tangible solutions, this work paves the way for future technologies that prioritize both efficiency and trustworthiness. Future endeavors could focus on expanding the architecture’s cross-domain adaptability, evaluating diverse LLMs, and refining feedback mechanisms in tandem with expanding knowledge graph databases.

7. Conclusions

This research elucidates the advancements and challenges in conversational AI, emphasizing the imperative balance between linguistic richness and factual accuracy. By introducing a novel architecture that synergistically integrates large language models and knowledge graphs, we provide a tangible solution to the current transparency and trust issues. Our comprehensive analysis of LLMs and their practical applications across various industries further contributes to the understanding and potential of trustworthy conversational AI. As we look ahead, this work not only addresses present challenges but also charts a promising trajectory for future conversational systems that prioritize efficiency, security, and user trust.