A Knowledge Graph Embedding Model Based on Cyclic Consistency—Cyclic_CKGE

Abstract

Most of the existing medical knowledge maps are incomplete and need to be completed/predicted to obtain a complete knowledge map. To solve this problem, we propose a knowledge graph embedding model (Cyclic_CKGE) based on cyclic consistency. The model first uses the “graph” constructed with the head entity and relationship to predict the tail entity, and then uses the “inverse graph” constructed with the tail entity and relationship to predict the head entity. Finally, the semantic space distance between the head entity and the original head entity should be very close, which solves the reversibility problem of the network. The Cyclic_CKGE model with a parameter of 0.46 M has the best results on FB15k-237, reaching 0.425 Hits@10. Compared with the best model R-GCN, its parameter exceeds 8 M and reaches 0.417 Hits@10. Overall, Cyclic_CKGE’s parametric efficiency is more than 17 times that of R-GCNs and more than 8 times that of DistMult. In order to better show the practical application of the model, we construct a visual medical information platform based on a medical knowledge map. The platform has three kinds of disease information retrieval methods: conditional query, path query and multi-symptom disease inference. This provides a theoretical method and a practical example for realizing knowledge graph visualization.

1. Introduction

Nowadays, human society has accumulated a great deal of knowledge [1,2]. Especially in recent years, with the promotion of a knowledge graph, a large number of online knowledge graphs compatible with artificial intelligence have appeared. A knowledge graph is essentially a semantic network [3], which expresses all types of entities, concepts and their semantic relationships. Compared with traditional knowledge representation forms, a knowledge graph has the advantages of high concept coverage, diverse semantic relations, friendly structure [2] and high quality, which makes a knowledge graph gradually become the most important knowledge representation form in the era of big data and artificial intelligence. In recent years, large knowledge bases have been developed to store structured information for common sense, such as Freebase [4]. KG is represented as a directed multigraph, a knowledge graph, where entities and relationships are represented as nodes and edges of different types [5], respectively. They usually consist of numerous facts in a triplet structure: i.e., (head entity, relation, tail entity) [5,6].

After the appearance of a knowledge graph, researchers have paid much attention to it. Google [7] first used relevant theories to revamp its search engine and build a knowledge graph [1] with 57 billion entities and more than 1.8 billion relationships. Therefore, it is no longer just a search engine based on string retrieval and matching, but gradually developed into an intelligent search platform that can truly understand the meaning of user input search strings, and can return the integrated facts as results more accurately. At the same time, Baidu also followed Google’s technology and launched the Chinese Knowledge Graph [8,9]. Subsequently, a knowledge graph began to rapidly extend from search engines to finance [10], agriculture [11], medical [12,13] and other industries, and became an indispensable technical means for integrating and analyzing data in the era of big data.

In recent years, due to the rapid development of information technology and the popularization of a medical information management system, massive medical knowledge and clinical diagnosis data have emerged. Reasonable use of these medical data to build a complete medical database is a prerequisite for promoting intelligent medical care. And a complete professional medical database for the promotion of medical knowledge retrieval, medical-assisted diagnoses and intelligent management of medical records also has a very great role in promoting. The use of a knowledge graph to mine and manage complex medical data, and simply represent them as nodes and relationships, has become a hot topic in academic research. In 2022, Gao Zhensen [14] adopted the nonlinear transformation method to solve the semantic resolution problem in representation learning. By selectively weighing the semantic elements of entity vectors, the representation and differentiation of different relational semantics can be realized. The proposed algorithm has good semantic distinction ability for complex relation types and related entities, and can effectively improve the relational inference accuracy on a knowledge graph. In order to solve the problem of asymmetry of relationships and make the model adapt to different types of network relationship reasoning tasks, in 2020, Baidu, Inc. [15], based on the existing entity relationship data in the knowledge graph, adopted recurrent neural networks to conduct semantic coding of symbol combinations, and then proposed a neural network model based on symbolic semantic mapping. The model is applied to the distributed representation of a learning graph, which makes the model perform well in a knowledge graph expansion task and graph-based multi-label classification task. To improve the overall recall rate of the system and avoid cascading errors, Mohamad Norhafizi [16] proposed a Simultaneous Neural Entity-Relation Linker (SNERL) model in 2021. Input text is encoded through an attention mechanism to obtain the context of each entity mentioned in the input. These contextualized representations are then used to predict the distribution of references to entities on the hierarchy and the distribution of references to relationships on the hierarchy. The probability of combining these predictions is then calculated for each mentioned pair and summarized at the document level to obtain the final probability of the predicted text tuple. The experimental results show that the SNER model has excellent results in the chemical disease relationship data set (CTD). Aiming at the problem of the large span of medical relationships, Sun Shufen [17] proposed a dynamic potential structure optimization strategy in 2020, which extracted document-level graphs in a node-to-node manner to infer relationships between sentences, so as to improve information aggregation in the entire document.

With the global outbreak of the new coronavirus, experts and scholars have also put medical mapping into the study of the new coronavirus. In 2022, Xiao Yu [18] used the Stanford University seven-step method and Protege and Ontofox tools to reorganize COVID-19-related knowledge by extracting existing ontologies, knowledge bases and diagnosis and treatment guidelines at home and abroad. A COVID-19 domain ontology covering disease, diagnosis, cause, virus, transmission, symptoms, treatment, medicine, prevention, etc., has been constructed. The results show that the ontology has good consistency and literature sampling shows that it has a high term coverage rate.

If we want to construct a medical knowledge map, we need to extract triples from a complex medical knowledge base first. In reality, there are often a large number of overlapping triplets in the medical knowledge base, such as the description in the medical database: “Osteoarthritis complicated by limb joint movement disorders, serious limb varus and flexion contracture deformity, and finally joint disability.” The triplets that can be extracted from this sentence are (osteoarthritis, complications, limb joint mobility disorder), (osteoarthritis, complications, limb varus), (osteoarthritis, complications, flexure contracture deformity) and (osteoarthritis, complications, joint disability). The four triples obviously have the same head entity and relationship, so all four triples are overlapping triples. These overlapping triples are a serious challenge for triple extraction tasks.

In response to this challenge, Meng Fanhua [19] proposed the CopyRE model. The model uses the Seq2Seq model and replication mechanism, but the disadvantage is that entities with multiple words cannot be extracted, and there may be an order for identifying misheaded and tail entities. Ishiyama Noriko [20] proposed the CasRel model to deal with the overlapping problem of multiple relations. Fang An [21] proposed a scheme to extract the relationship first and then the head and tail entities, respectively, and proved the effectiveness of the model through ablation experiments and a probabilistic analysis. Wang, Y [22] proposed the TPLinker model. Since the model first encodes the entity pairs and relations and forms a corresponding coding matrix, and then performs subsequent processing, this model can effectively deal with overlapping problems and is very suitable for triplet extraction of medical knowledge maps.

Since the knowledge graph in reality is mostly incomplete, how to correctly reason/predict missing links is another challenge. The TransE [23] model is an early classical model for missing link prediction. However, because it embeds the head, relation and tail entities into the same semantic space, it cannot predict the result well when dealing with overlapping problems. Since its model is very simple, there have been many subsequent variants of it, such as TransH [24]. ComplEx [25] was the first model to use complex space for encoding. RotatE [26] builds on this to reason specifically about symmetric/antisymmetric [27], inversion and composition relations. There are a lot of symmetric/antisymmetric [28], opposite and combination relationships in a medical knowledge base, so the RotatE [26] model is very suitable for medical knowledge graph [2] inference.

In order to solve the above problems, we built a disease information search platform based on a medical knowledge graph. This paper trained the TPLinker model and the RotatE [26] model, respectively, through the medical knowledge base, and adjusted the hyperparameters [29] through KGTuner [30]. The trained TPLinker model is used to collect the text data into triples, and the incomplete triples are supplemented with the RotatE [26] model to realize the prediction/inference function. In this way, a medical knowledge map is constructed, and the visualization of the medical knowledge map is realized through Neo4j. For the input of patients and their families, this paper again uses the TPLinker model to extract triples (or more accurately, entities) and normalize them into the standard input of the system. Then, based on the inference result of the RotatE [26] model, corresponding diseases and complications are searched as outputs.

The main contributions of this paper are as follows:

(1)

Compared with the existing research results, this paper is the first to construct a more real and comprehensive medical knowledge map by using the medical knowledge base.

(2)

The information platform developed in this paper extracts entities from the input of patient symptoms, and retrieves possible diseases and corresponding complications according to entities.

(3)

Aiming at the particularity of the medical knowledge base, this paper combines TPLinker and RotatE to realize a complete medical knowledge map and constructs it as a visual intelligent medical system.

The remaining sections of the paper are organized as follows: Section 2 reviews the work, Section 3 provides background knowledge on the knowledge graph embedding model, Section 4 introduces the proposed approach, Section 5 presents experimental results and finally, Section 6 summarizes the paper.

2. Related Work

In view of the problems existing in the above knowledge graph, such as incomplete triples, many experts and scholars have proposed many methods for completion or prediction. These methods are collectively known as knowledge graph embedding. The embedding methods can be roughly divided into four categories, namely, the method based on vector translation, the method based on a convolutional neural network, the method based on a graph neural network and the method of joint embedding a knowledge graph and logic rules.

2.1. Method Based on Vector Translation

TransE [23] is the first paper to use the translation model for knowledge graph embedding research. The model is simple, easy to train and has strong explanatory properties. The basic idea is to represent head entities, relations and tail entities as vectors, respectively. However, TransE [23] can only handle 1-1 relational patterns, and it is not effective for complex relationships such as 1-N, N-1 and N-N relational patterns. Based on the TransE [23] model, a number of subsequent research results were extended, including the use of projection vectors or matrices to transform the head and tail entities into relational vector Spaces. During this period, many models began with Trans, such as TransH [24], TransR [31] and TransD [32]. TransH treats relationships as hyperplanes and projects head and tail entities into the hyperplane of a particular relationship. Although the TransH [24] model allows each entity to have a different representation under different relations, it still constructs entities and relations in the same semantic space, which limits the representation ability of TransH to some extent. TransR projects entities and relationships into different Spaces and transforms them on the corresponding relational Spaces. Under the same relation, the head and tail entities share the same projection matrix. However, the types or attributes of the head and tail entities of the relationship may be very different. The projection from the entity space to the relation space is an interaction process between the entity and the relation, so it is unreasonable for TransR [31] to only associate the projection matrix with the relation. Compared with TransE [23] and TransH [24], due to the introduction of spatial projection, the TransR [31] model has significantly more parameters and higher computational complexity. TransD [32] further simplifies TransR [31] by decomposing the projection matrix into the product of two vectors.

2.2. Method Based on Convolutional Neural Network

While translation models use simple operations and limited parameters to learn embeddings, they produce lower-quality representations. Recent research has shown that models based on convolutional neural networks (CNNs) [33] can generate richer, more expressive feature embeddings and therefore also perform well in relation prediction. Dettmers et al. proposed ConvE [34], which was the first model to apply CNN to KG tasks. It is characterized by structuring head entities and relationships into “pictures” and using convolutional networks to calculate the features of the pictures. The tail entity is then viewed as a “label” for classification. This constitutes a data set for supervised learning. The whole process also did not use the translation model. It uses stacked 2D convolution to reshape entity and relational representations, thereby improving their expressiveness while maintaining parametric efficiency. ConvKB [35] is another convolution-based approach that applies width convolution to stacked entity, relationship and object embeddings.

2.3. Method Based on Convolutional Neural Network

A graph neural network (GNN) [36] is a method of structuring data into a directed graph, and the knowledge graph itself is a directed graph. Therefore, this network has an inherent advantage for knowledge graph embedding research. There are two popular variants of GNN, namely, the Graph Convolutional Network and the Graph Attention Network (GAT) [37]. Existing methods learn KG embeddings either by focusing only on entity features, or by considering entity features and relationships in a disjoint manner.

2.4. Logical Rule-Based Approach

The knowledge graph embedding model attempts to combine other types of available knowledge, such as relational paths, relational type constraints, entity types and entity descriptions, to learn better embedding. Logic rules, usually based on Markov logic networks [38], have been widely used in knowledge reasoning and knowledge acquisition. Recent studies have shown that including background knowledge, such as logic rules, can improve the performance of embeddings in downstream machine learning tasks. However, in their work, rules are modeled separately from embedding methods as a post-processing step, and thus do not help to achieve better embedding. Rocktaschel et al. proposed a new federated model called LR-KGE [39]. The model codifies the rules into the embed. However, their work focused on relationship extraction tasks and creating embeddings for entity pairs, so it was not possible to discover relationships between unpaired entities.

3. Background

Knowledge Graph Embedding Model Based on Cyclic Consistency

Cycle consistency (CC) was first proposed in the paper Cycle GAN (Cycle GAN) [40] published by Junyan Zhu et al., which is used to realize image-to-image transformation in two domains without paired data.

In this paper, we propose a knowledge graph embedding model based on cyclic consistency (Cyclic_CKGE), which uses embedded two-dimensional convolution to predict missing links in knowledge graphs. It consists of a single convolution layer, a projection layer for embedding dimensions and an inner stacking layer. First, the embeddings of entities and relationships are reshaped and spliced, and then the resulting matrix is used as the input of the convolutional layer. The generated feature mapping tensor is vectorized and projected into a K-dimensional space [41] and embedded with all candidate objects. This is shown in Figure 1.

In general, two-dimensional convolution is used instead of one-dimensional convolution, and the expressiveness of the model is improved by adding additional interaction points between the models.

4. Method

4.1. Cyclic Consistency Principle

First, in the forward channel, the input is passed through a complex nonlinear transformation $\mathrm{f}$, such as a neural network, to obtain the data $\tilde{\mathrm{y}}$ in the output domain. If the Euclidean space distance ${\mathrm{d}}_1$ between $\mathrm{y}$ and $\tilde{\mathrm{y}}$ is approximately 0, we have $\mathrm{y}=\mathrm{f}\left(\mathrm{x}\right)$. Then, in the reverse channel, the output data $\tilde{\mathrm{y}}$ is transformed by the inverse transformation $\mathrm{f}\mathrm{’}$ of the previous complex nonlinear transformation $\mathrm{f}$ to obtain the new input domain data. Then, if the Euclidean space distance ${\mathrm{d}}_2$ of x and $\tilde{\mathrm{x}}$ is approximately 0, we have x = ($\tilde{\mathrm{y}}$) = $\mathrm{f}\mathrm{’}\left(\mathrm{f}\right(\mathrm{x}\left)\right)$. So, f can be viewed as a transformation from mathematics to data. In the same way, $\mathrm{f}\mathrm{’}$ can also be viewed as the transformation of data $\mathrm{y}$ to data $\mathrm{x}$. This design directly solves the reversibility of the network, but also indirectly ensures that the network can learn a smooth deformation field, that is, helps the network to exclude those acyclic consistent transformations. In addition, this design does not increase the number of parameters in the model. The entire transformation process is shown in Figure 2.

4.2. The Loss Function Is Constructed Based on the Principle of Cyclic Consistency

Cyclic consistency loss comes from CycleGAN. The basic structure of a CycleGAN is composed of two Gans, forming a ring network. This is shown in Figure 3. $\mathrm{G}$ stands for generator. ${\mathrm{G}}_{\mathrm{X},\mathrm{Y}}$ indicates that data in domain X are transferred to domain Y, and ${\mathrm{G}}_{\mathrm{Y},\mathrm{X}}$ indicates that data in domain Y are transferred to domain X. DX and DY represent the discriminators of GXY and GYX, respectively. Circular consistency loss occurs to prevent the generator GXY or GYX from converting all the data into another domain, i.e., X and Y can be smoothly converted from the source domain to the destination domain and back again.

Its change process is shown in Formula (1). Then, the formula of cyclic consistency loss is shown in Formula (2). Among them, ${\mid \cdots \mid }_1$ is expressed as the ${\mathrm{L}}_1$ norm. ${\mathrm{E}}_{\mathrm{x}}(\cdots )$ is represented as “....” in x sample. The expected value of ${\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{x}\right)}$ is expressed as the probability distribution of x. Then, $\mathrm{x}\sim {\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{x}\right)}$ is represented as a sample of random variables subject to ${\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{x}\right)}$. Similarly, ${\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{y}\right)}$ is expressed as the probability distribution of y. Then, $\mathrm{y}\sim {\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{y}\right)}$ is represented as a sample of random variables subject to ${\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{y}\right)}$.

$$\mathrm{y}\to {\mathrm{G}}_{\mathrm{Y},\mathrm{X}}\left(\mathrm{y}\right)\to {\mathrm{G}}_{\mathrm{X},\mathrm{Y}}\left[{\mathrm{G}}^{{}_{\mathrm{Y},\mathrm{X}}}\right(\mathrm{y}\left)\right]\to \mathrm{y}\mathrm{’}$$

(1)

$$\mathrm{loss}{\mathrm{s}}_{\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}}({\mathrm{G}}_{\mathrm{X},\mathrm{Y}},{\mathrm{G}}_{\mathrm{Y},\mathrm{X}})={\mathrm{E}}_{\mathrm{x}-{\mathrm{P}}_{\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}}}\{\mid {\mathrm{G}}_{\mathrm{Y},\mathrm{X}}[{\mathrm{G}}_{\mathrm{X},\mathrm{Y}}\left(\mathrm{x}\right)]-\mathrm{x}{\mid }_1\}+{\mathrm{E}}_{\mathrm{y}-{\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{y}}}\{\mid {\mathrm{G}}_{\mathrm{X},\mathrm{Y}}[\mathrm{G}\mathrm{Y},\mathrm{X}\left(\mathrm{y}\right)]-\mathrm{y}{\mid }_1\}$$

(2)

The loss of cyclic consistency maintains a high degree of consistency between the x and y inputs and the x′ and y′ outputs, thus making the generator learning more targeted, and the final loss function is shown in Formula (3). $\mathsf{\lambda }$ is the control coefficient (generally 10), which is used to control the proportion of the adversarial network and the cyclic consistency loss in the final loss.

$$\mathrm{l}\mathrm{o}\mathrm{s}{\mathrm{s}}_{\mathrm{a}\mathrm{l}}\left({\mathrm{G}}_{\mathrm{X},\mathrm{Y}},{\mathrm{G}}_{\mathrm{Y},\mathrm{X}},{\mathrm{D}}_{\mathrm{x}},{\mathrm{D}}_{\mathrm{y}}\right)=\mathrm{l}\mathrm{o}\mathrm{s}{\mathrm{s}}_{\mathrm{C}\mathrm{A}\mathrm{N}1}({\mathsf{C}}_{\mathrm{X},\mathrm{Y}},{\mathrm{D}}_{\mathrm{y}},\mathrm{x},\mathrm{y})+\mathrm{l}\mathrm{o}\mathrm{s}{\mathrm{s}}_{\mathrm{G}\mathrm{A}\mathrm{W}2}\left({\mathsf{G}}_{\mathrm{Y},\mathrm{X}},{\mathsf{D}}_{\mathrm{x}},\mathsf{y},\mathsf{x}\right)+\mathsf{\lambda }\mathrm{l}\mathrm{o}\mathrm{v}\mathrm{e}{\mathrm{s}}_{\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}}({\mathsf{G}}_{\mathrm{X}},\mathsf{Y},{\mathsf{G}}_{\mathrm{Y}},\mathsf{X})$$

(3)

4.3. The Knowledge Graph Embedding Model Is Constructed

According to the cyclic consistency and ConvE model, a new convolutional neural network model is constructed in this paper. Specifically, for semantic matching tasks, the resulting entities should be consistent with one-to-one mapping constraints. The initial measure of similarity follows a rule from one to many. The correspondence with the highest score can be treated as an input value, and each feature can correspond to multiple features as shown in Figure 4a. Therefore, a circular consistency policy is introduced to encourage one-to-one matching. Similar to the process of closest mutual matching between images, the association relationship between the two feature representations is estimated to impose their one-to-one matching constraints, and the associations p→q and p←q are calculated, as shown in Figure 4b.

Each initial triplet in Figure 4 contains all pairs of one-way fractions from the original triplet to the target triplet. Feature representation p can match multiple features. The circular consistency constraint is used to encourage one-to-one matching in Figure 4b.

In this paper, a convolutional and fully coupled neural network connection prediction model is established. The main feature of the model is that the scores are determined by embedding two-dimensional convolution. Formally, the evaluation function is defined as shown in Formula (4).

${\mathrm{r}}_{\mathrm{r}}\in {\mathbb{R}}^{\mathrm{k}}$ is a relational parameter dependent on ${\mathrm{r}}_{}$, and $\overline{{\mathrm{e}}_{\mathrm{h}}}$ and $\overline{{\mathrm{r}}_{\mathrm{r}}}$ represent two-dimensional reconstructions of ${\mathrm{e}}_{\mathrm{h}}$ and ${\mathrm{r}}_{\mathrm{r}}$, respectively: if ${\mathrm{e}}_{\mathrm{h}},{\mathrm{r}}_{\mathrm{r}}\in {\mathbb{R}}^{\mathrm{k}}$, $\overline{{\mathrm{e}}_{\mathrm{h}}},\overline{{\mathrm{r}}_{\mathrm{r}}}\in {\mathbb{R}}^{{\mathrm{k}}_{\mathrm{w}}\times {\mathrm{k}}_{\mathrm{h}}}$, where ${\mathrm{k}=\mathrm{k}}_{\mathrm{w}}\times {\mathrm{k}}_{\mathrm{h}}$.

$${\mathsf{\Psi }}_{\mathrm{r}}({\mathrm{e}}_{\mathrm{h}},{\mathrm{e}}_{\mathrm{t}})=\mathrm{f}\left(\mathrm{v}\mathrm{e}\mathrm{c}\right(\mathrm{f}\left({[\overline{{\mathrm{e}}_{\mathrm{h}}},\overline{{\mathrm{r}}_{\mathrm{r}}}]}^{\ast }\mathsf{\omega }\right)\left)\mathrm{W}\right){\mathrm{e}}_{\mathrm{t}}$$

(4)

In the forward transfer process, the model performs a row vector search operation in two embedded matrices; one is the entity matrix, referred to as ${\mathrm{E}}^{\mid \mathsf{\epsilon }\mid \times \mathrm{k}}$, and the other is the relation matrix, denoted as ${\mathrm{R}}^{\mid \mathrm{R}\mid \times \mathrm{k}\mathrm{\prime }}$, where $\mathrm{k}$ and $\mathrm{k}\mathrm{\prime }$ are the embedded dimensions of entities and relationships, and $|\mathsf{\epsilon }\mid$ and $|\mathrm{R}\mid$ represent the number of entities and relationships. The model then connects $\overline{{\mathrm{e}}_{\mathrm{h}}}$ and $\overline{{\mathrm{r}}_{\mathrm{r}}}$ and feeds them as inputs into a two-dimensional convolution layer with filter $\mathsf{\omega }$. This layer returns a feature mapping tensor $\mathsf{\tau }\in {\mathbb{R}}^{\mathrm{c}\times \mathrm{m}\times \mathrm{n}}$, where $\mathrm{c}$ is the number of 2D feature mappings of dimension $\mathrm{m}$ and $\mathrm{n}$. The tensor $\mathsf{\tau }$ is then reshaped into the vector $\mathrm{v}\mathrm{e}\mathrm{c}\left(\mathsf{\tau }\right)\in {\mathbb{R}}^{\mathrm{c}\mathrm{m}\mathrm{n}}$, which is then projected into the K-dimensional space using a linear transformation with the matrix $\mathrm{W}\in {\mathbb{R}}^{\mathrm{c}\mathrm{m}\mathrm{n}\times \mathrm{k}}$ and matched to the embedded object ${\mathrm{e}}_{\mathrm{o}}$. With the inner product, the parameters and matrix $\mathrm{W}$ of the convolution filter are independent of the parameters of entity $\mathrm{s}$ and $\mathrm{o}$ and the $\mathrm{r}$ relationship.

To train the model parameters, the activation function applies the logarithmic sigmoid function $\mathsf{\sigma }(.)$, or $\mathrm{p}=\mathsf{\sigma }\left({\mathsf{\Psi }}_{\mathrm{r}}\right({\mathrm{e}}_{\mathrm{s}},{\mathrm{e}}_{\mathrm{o}}\left)\right)$, and minimizes the following binary cross entropy loss, as shown in Formula (5).

Where t is the label vector, and the dimension is ${\mathbb{R}}^{\left|\mathrm{x}\right|}$ for 1-1 scores and ${\mathbb{R}}^{1\times \mathrm{N}}$ for 1-N scores. The elements of vector t have relation 1; otherwise, they are 0.

$$\mathcal{L}\left(\mathrm{p},\mathrm{t}\right)=-\frac{1}{\mathrm{N}}{\sum }_{\mathrm{i}}({\mathrm{t}}_{\mathrm{i}}\cdot \mathrm{l}\mathrm{o}\mathrm{g}({\mathrm{p}}_{\mathrm{i}})+(1-{\mathrm{t}}_{\mathrm{i}})\cdot \mathrm{l}\mathrm{o}\mathrm{g}(1-{\mathrm{p}}_{\mathrm{i}}\left)\right)$$

(5)

The model uses rectified linear elements as nonlinear f for faster training, and batch normalization after each layer to stabilize, normalize and improve convergence. The built model is structured in several stages using dropout. In this paper, dropout embedding is used for aspects such as feature map operation after convolution, and the hidden unit layer after the fully connected layer. Adam is used as an optimizer and label smoothing is used to reduce overfitting due to label saturation.

Unlike other link prediction models, a pair of entities and a relationship are labeled as a triple (h,r,t) and labeled (score 1-1), and a pair of entities (h,r′) are labeled simultaneously for all entities or $\mathrm{t}\in \mathsf{\epsilon }$ (score 1-N). If you analyze the 1-1 score on a high-end GPU and use batch size and 128 embed size, the training channel and convolutional model evaluation on the FB15k take 2 min and 3 h. Using a 1-N score, where the corresponding numbers are 45 s and 35 s, there is a significant improvement of more than 300 times in terms of assessment time. In addition, this method can be extended to large knowledge graphs and improves the speed of convergence. For a single step forward or backward with a batch volume of 128, from N = 100,000 to N = 1,000,000 entities, the computation time only increases from 64 ms to 80 ms; in other words, the number of entities increases by 10 times, and the computation time only increases by 25%, proving the scalability of the method.

4.4. Constructing Medical Knowledge Map

The medical knowledge map constructed in this paper extracts a large amount of complex medical knowledge by triples. The medical knowledge is presented to the user in the form of a map, so that the user can clearly see the relationship between the disease and symptoms, and improve the visual experience. To construct a graph, the first problem to be considered is to define the entities and relationships in the graph. Based on the consideration of this problem, this paper defines six entity types (as shown in

Table 1. Entity type.

Entity Class	Give an Example
Food	Eat beef
Examinations	Blood routine examination
Departments	Internal medicine
Diseases	Diabetes
Symptoms	Fever, cold
Information	Lead to decreased immunity

) and eight relationships (as shown in

Table 2. Relationship type.

Relation Class	Give an Example
Departments and departments	Medicine and surgery
Illness and avoidance of food	Avoid spicy food when you have a cold
Sickness and proper food	You should pay attention to a light diet
Diseases and drugs	Take insulin for high blood sugar
Disease and examination	Take an X-ray of a broken bone
Diseases and symptoms	Flu and sneezing
Diseases and complications	Hypertension and kidney failure
Diseases and departments	Fracture and bone surgery

) according to the International Classification of Diseases (ICD-10 Beijing Clinical Edition v601) [42].

The whole system frame diagram is shown in Figure 5. In the multi-symptom disease inference, the whole process of the system is demonstrated by taking the patient’s input of “I have frequent eye pain recently, and I am afraid of light and accompanied by tears” as an example. First, the system needs to convert the unstructured text entered by the patient into a standard structured triplet (entity) through the TPLinker model and normalization operation. Standard structured entities are then fed directly into the smart healthcare system. At this time, the system can predict the corresponding disease and deduce the complication from the visual interface according to the knowledge graph constructed before.

4.5. Study on Analytic Processing of Patient Input Symptoms

When a patient enters their illness into the medical diagnosis system, the system needs to extract the keywords of the illness entered by the patient. In recent years, the research of “topic triples” extracted from unstructured texts has been very hot, but there are many challenges to carry out this work simultaneously. In 2020, Y Wang et al. proposed a single-stage joint extraction model [11] (TPLinker model), which can find the overlapping relationship of one or more entities. At the same time, it is not affected by exposure bias, and the joint extraction problem is reduced to the tag pair connection problem, which solves the entity nesting problem perfectly. Based on the TPLinker model, entity relationship extraction was carried out to extract structured information, so as to achieve the effect of analyzing and processing symptoms.

Given a sentence, the triples in it can be represented by a set of metadata: a given relationship and the boundaries of the corresponding head/tail entities. For example, in the sentence “The new type of [infectious disease] [Brucellosis] is serious”, the relation is given with “type”, the head entity is “infectious disease” and the tail entity is “Brucellosis”. In this way, the corresponding triplet can be obtained from the disease entered by the patient. To turn the above data into a callable sequence, we first design a link between three tokens:

• Entity Head to Entity Tail (EH to ET)
• Subject Head to Object Head (SH to OH)
• Subject Tail to Object Tail (ST to OT)

Then, embed the above three links into a matrix: Figure 6 shows this.

The purple number represents the EH-to-EH link, the red number represents the SH-to-SH link and the blue number represents the ST-to-OT link. The horizontal and vertical axes of the matrix represent the subscript of the corresponding token in the original sentence, respectively. If there is a link between the corresponding token pairs, the corresponding element of the matrix is set to 1 (otherwise, it is 0) and the three colors of 1 on the way correspond to three types of links, respectively. For the sake of explanation, the three matrices are superimposed together. In actual processing, the corresponding matrices of the three links are independent.

4.6. An Introduction to Algorithms for Reasoning about Diseases and Complications

Some relationships are symmetric in the medical knowledge graph (e.g., complications); some relations are antisymmetric (e.g., belonging). In the medical knowledge map constructed with this system, the connection pattern in the knowledge map is modeled according to the known medical knowledge facts, so as to predict the disease of patients and deduce the complications caused by the disease.

The RotateE model has achieved good results in various relationship categories, so this paper proposes to use the RotateE model (as shown in Figure 7) as knowledge graph embedding to represent the node relationship of the medical knowledge graph.

Any complex number in RotateE’s model can be viewed as a rotation vector in the complex plane. Specifically, in the medical knowledge graph, the model uses complex numbers to predict diseases and deduce complications. It maps the disease symptoms (entity head) and disease complications (relationship relation) entered by the patient into a complex vector space, and defines each disease complication as a rotation from the beginning entity to the end entity. Assuming that we know the symptoms of the disease (as h in the formula) and the complications of the disease (as r in the formula), for a complex triplet (h, r, t), the relationship is shown in Equation (3), from which t (name of the disease) can be derived.

$$\mathrm{h}·\mathrm{r}\approx \mathrm{t}$$

(6)

5. Experiment

5.1. Implementation Details

This section describes the related software, operating environment and hyperparameters.

Table 3. The operating environment required using the system.

Required Plug-In Environment	Versions
python	3.6
elasticsearch	7.10.1
numpy	1.20.3
jieba	0.39
pandas	1.1.5
rank-bm	0.2.1
scikit-learn	0.24.1
scipy	1.5.4

shows the system operating environment. The parameter configuration of the knowledge graph embedding model proposed by us is shown in

Table 4. Hyperparameter configuration of the model.

Hyperparameter	Disposition
#negative samples	2048
loss function	BCE_adv
gamma	3.10
adv.weight	1.93
regularizer	FRO
reg.weight	6.51 × 10−6
dropout rate	0.00
optimizer	Adam
learning rate	6.43 × 10−4
batch size	512
dimension size	1000
inverse relation	False

.

5.2. Evaluation Metrics

We carried out horizontal comparison experiments on different models in multiple knowledge graph data sets (WN18, FB15k, YAGO3-10, Countries). The evaluation index formula (Mean Reciprocal Ranking (MRR), hit@k) was used for the analysis. Experiments show that the proposed model not only has scalability, but can also infer and model several relational patterns well such as symmetry and antisymmetry. Therefore, complex medical relational reasoning is very suitable for the model proposed in this paper.

This paper mainly selects the following commonly used knowledge graph data sets. The data set details are as follows.

WN18 is a subset of WordNet, consisting of 18 relationships and 40,943 entities. Most of the 151,442 triples consist of conjunctions and superword relations, and for this reason, WN18 tends to follow a strict hierarchy.

FB15k is a subset of Freebase that contains about 14,951 entities with 1345 different relationships.

Yago3-10 is a subset of YAGO3, consisting of at least 10 relationships per entity. It has 123,182 entities and 37 relationships. Most triples involve descriptive attributes of people, such as citizenship, gender and occupation.

Countries is a benchmark data set that is useful for evaluating a model’s ability to learn long-term dependencies between entities and relationships. It consists of three subtasks that increase in difficulty in a step-by-step fashion, where the minimum path length to find a solution increases from 2 to 4.

In order to evaluate the performance of various models on triplet extraction of a knowledge graph, this paper uses the link prediction method to give the cross-sectional comparison results of the TransE model, DistMult model, ConvE model and CCO model on various performance indicators. The specific step is to predict the missing head or tail entity of the triplet in the data set. Specifically, MRR (Mean Reciprocal Ranking) and hit@10 were used to measure the model performance level. Specific definitions are as follows:

Mean Reciprocal Ranking: Mean Reciprocal Ranking (MRR), which is defined as the arithmetic mean of the reciprocal of all triples of the test set, is shown in Formula (7). MRR is an international mechanism used to evaluate search algorithms where the first result corresponds to a score of 1, the second matches to a score of 0.5 and the NTH matches to a score of 1/n. If there is no match, the sentence score is 0. The final score is the sum of all the scores. In the formula, rank represents the link prediction rank of the triplet.

$$\mathrm{M}\mathrm{R}\mathrm{R}=\frac{1}{{\left|{\mathrm{T}}_{\mathrm{test} }\right|}_{(\mathrm{h},\mathrm{r},\mathrm{t})\in {\mathrm{T}}_{\mathrm{test} }}}\sum _2(\frac{1}{\mathrm{r}\mathrm{a}\mathrm{n}{\mathrm{k}}_{\mathrm{r},\mathrm{t}}\left(\mathrm{h}\right)}+\frac{1}{\mathrm{r}\mathrm{a}\mathrm{n}{\mathrm{k}}_{\mathrm{h},\mathrm{r}}\left(\mathrm{t}\right)})$$

(7)

hit@k: hit@k represents the ratio of triples where a single sort is in the first k during the test, as shown in Formula (8). Regarding hits@k, the larger the value, the better the link prediction of the model. k is usually 1, 3 and 10. Again, arrange the f function values as described above, and then see if the correct answer for each test triplet is in the top ten of the sequence, and if so, count +1. The final top ten/total number is Hit@10.

$$\mathrm{h}\mathrm{i}\mathrm{t}@\mathrm{k}=\frac{1}{\left|{\mathrm{T}}_{\mathrm{test} }\right|}{\sum }_{(\mathrm{h},\mathrm{r},\mathrm{t})\in {\mathrm{T}}_{\mathrm{test} }}\frac{1}{2}(|\{(\mathrm{h},\mathrm{r},\mathrm{t})\mid \mathrm{r}\mathrm{a}\mathrm{n}{\mathrm{k}}_{\mathrm{r},\mathrm{t}}(\mathrm{h})⩽\mathrm{k}\}|+|\{(\mathrm{h},\mathrm{r},\mathrm{t})\mid \mathrm{r}\mathrm{a}\mathrm{n}{\mathrm{k}}_{\mathrm{h},\mathrm{r}}(\mathrm{t})⩽\mathrm{k}\}|)$$

(8)

For the Cyclic_CKGE (our) model parameter efficiency, as shown in

Table 5. DistMult model and Cyclic_CKGE model parameter comparison.

Model	Parametric	Embeddedness	hit@3	hit@10
DistMult	1.89 M	128	0.25	0.41
	0.95 M	64	0.25	0.39
	0.23 M	16	0.17	0.31
Cyclic_CKGE	5.05 M	200	0.35	0.49
	1.89 M	96	0.35	0.49
	0.95 M	54	0.33	0.46
	0.46 M	28	0.30	0.43
	0.23 M	14	0.28	0.40

, Cyclic_CKGE (parameter of 0.23 M) in FB15k-237 outperformed DistMult (parameter of 1.89 M) on three of the five indicators. However, the Cyclic_CKGE model with a parameter of only 0.23 M obtains the best result on FB15k-237, reaching 0.425Hits @10. Compared with the best model, R-GCN, its parameter exceeds 8 M and reaches 0.417 Hits@10. Overall, Cyclic_CKGE’s parametric efficiency is more than 17 times that of R-GCNs and more than 8 times that of DistMult. For Freebase as a whole, these models will be over 82 GB for R-GCNs, over 21 GB for DistMult and only 5.2 GB for Cyclic_CKGE.

On the basis of grid search, the hyperparameters of the Cyclic_CKGE model are selected according to the mean exchange rank (MRR). Grid search has the following hyperparameter ranges: embed cull {0.0, 0.1, 0.2}, feature map cull {0.0, 0.1, 0.2, 0.3}, projection layer cull {0.0, 0.1, 0.3, 0.5}, embed size {100, 200}, batch size {64, 128, 256}, learning rate {001, 003} and label smoothness {0.0, 0.1, 0.2, 0.3}.

On WN18, YAGO3-10 and FB15k, the combined effect of the following parameters is better: embedding cull of 0.2, feature map cull of 0.2, projection layer cull of 0.3, embedding size of 200, batch size of 128, learning rate of 0.001 and label smoothness of 0.1. For the Countries data set, we increased embedded cull to 0.3, implicit cull to 0.5 and set label smoothness to 0. Validation sets are evaluated every three cycles using early stops, based on common levels on validation sets (WN18, FB15k, YAGO3-10) and average statistics for OC-PR (Countries). For Countries, unlike other data sets, the variance of the results is high, so the average execution is 10 times to produce a 95% confidence interval. For distances and complex results after training 1-1, AdaGrad is optimized using the embedding size of 100 (Duchi, Hazan, and Singer, 2011) [43] by standardizing the model by forcing the L2 criterion for entity embedding to be 1.

By comparing the parameter scale with the experimental results, it is verified that the Cyclic_CKGE model can obtain better experimental results with fewer parameters, as shown in

Table 6. (a). WN18 link prediction results. (b). FB15k link prediction results.

(a)
	WN18
	MR	MRR	hit@3	hit@10
TransE	401	—	—	0.892
DistMult	902	0.822	0.914	0.936
Cyclic_CKGE	251	0.942	0.968	0.969
(b)
	FB15k
	MR	MRR	hit@3	hit@10
TransE	125	—	—	0.471
DistMult	97	0.654	0.733	0.824
Cyclic_CKGE	64	0.745	0.801	0.873

.

The specific link prediction results of the four types of models on the data sets WN18RR and FB15k-237 are shown in

Table 7. (a). Link prediction results for WN18RR. (b). Link prediction results for FB15k-237.

(a)
	WN18RR
	MR	MRR	hit@3	hit@10
TransE	4166	0.231	0.352	0.499
DistMult	5110	0.430	0.440	0.490
Cyclic_CKGE	3288	0.460	0.450	0.540
(b)
	FB15k-237
	MR	MRR	hit@3	hit@10
TransE	355	0.295	0.291	0.464
DistMult	254	0.241	0.263	0.419
Cyclic_CKGE	246	0.316	0.350	0.491

. The Cyclic_CKGE model has achieved state-of-the-art performance on all metrics, as well as on some metrics on the FB15k and WN18. On Countries, it solved the challenges of S1 and S2 and performed well on S3, scoring higher than other models such as DistMult, as shown in Table 7.

5.3. Intelligent Medical System Function Test

The intelligent medical diagnosis system constructed in this paper has three query methods, which are conditional query results, path query results and multi-symptom disease inference results. In the intelligent medical diagnosis system, relevant disease information can be searched through conditional search. Enter the name of the disease, location of the disease, infectivity, population of the disease and other relevant information in the search box, as shown in Figure 8, and the related diseases can be searched. For example, by searching the information of the disease name “Streptococcus pneumoniae septicemia”, the relevant disease information can be obtained as shown in Figure 9.

In the path search of the intelligent medical diagnosis system, the relationship between the two nodes can be inferred by selecting the start node and the end node, as shown in Figure 10. For example, in the start node, select diabetes, and in the end node, select diabetes, then the relationship between the two nodes with complications can be retrieved to form a closed loop, as shown in Figure 11.

In the multi-symptom disease inference, the patient’s input of “eye pain, fear of light, accompanied by tears” is taken as an example, as shown in Figure 12. At this time, the system can predict the corresponding diseases and deduce complications from the visual interface according to the knowledge graph constructed previously, as shown in Figure 13.

6. Conclusions

6.1. Intelligent Medical System Function Test

Due to the rapid development of the Internet and era, the development of a knowledge graph is very rapid, and it has a very good application in many fields, such as sky eye search, and intelligent question and answer of enterprise information. In this paper, intelligent medical treatment is realized by constructing a medical knowledge map, and satisfactory results are obtained. However, due to the limited technology and knowledge mastered at present, and there is no relevant professional doctor to give professional guidance to the constructed medical knowledge map, there are still some functions in the system that need to be improved and upgraded. It is hoped that the system developed in this paper can be put into use in the future to improve the current medical environment. The following is a summary of the technical problems and solutions in the construction of a medical knowledge map and intelligent medical system in this paper:

(1)

Entity nesting problem

The entity nesting problem is a situation in which a shorter entity is completely contained within another longer entity in a sentence. This kind of problem is very common in natural language processing. In addition to the TPLinker model used in this paper to perfectly solve such problems, in 2021, Ma et al. proposed an effective cascade double decoder method to extract overlapping relational triples [44], which includes a text-specific relational decoder and a relational corresponding entity decoder. A text-specific relational decoder detects relationships in sentences according to their text semantics and uses them as additional features to guide entity extraction. For each extracted relationship with trainable properties, a span labeling scheme is used to detect the corresponding head and tail entities in the relationship. The problem of word overlap is solved.

(2)

Accurately define entity relationship

Due to the large and complex medical data and the limited labeling data, it is necessary for domain experts to manually define entity relationships in the medical knowledge graph, but this is a very expensive project. In this paper, based on the RotateE model, the entities in the medical atlas are predicted when some relationships in the medical atlas are known, so as to build a relatively complete medical knowledge atlas. However, due to the lack of guidance from medical experts, there are still some uncertain relationships in the atlas. How to accurately define the entity relationship in the graph is also the focus of subsequent research.

(3)

The problem of making medical data sets

At present, the construction of a medical knowledge map in China is still in its infancy, and there is no professional medical data set in China. It is very difficult to construct a complete and accurate medical knowledge map. In the future, it is hoped that different medical institutions in China can conduct academic research and discussion in this respect, and provide accurate and high-quality medical data for the construction of a medical knowledge graph under the premise of ensuring patient privacy.

(4)

Privacy issues

Privacy is an important issue for hospitals to consider when releasing information about personal medical records. The purpose of constructing the medical knowledge graph is to solve the medical problems of the public without infringing on the privacy of personal data. At present, medical data are basically distributed separately in different medical institutions, and there is no mining work on the value of medical data among institutions, which has played a certain hindering role in the development and construction of a medical knowledge map. Therefore, it is very important to build a confidential medical database.

6.2. The Future Development Direction of Medical Knowledge Graph

The medical knowledge map constructed in this paper realizes the function of extracting medical knowledge from massive data. It is of great significance to the modern medical industry to manage, share and apply it reasonably and effectively, and it is also a hot issue in many enterprises and scientific research institutions. With the continuous improvement in related technologies in the field of a knowledge graph, it has become an inevitable trend to develop the combination of knowledge graph construction technology and medicine. The development direction of a medical knowledge map in the future should reflect the following aspects.

(1)

Construct multilingual medical knowledge map

Medical research between countries is an important condition for the progress of the world medical cause, and it is also an inherent requirement to promote the globalization and diversity of medical knowledge. Therefore, the multilingual medical knowledge map is very important to realize the interaction and sharing of medical knowledge among countries. In this era of deepening knowledge globalization, strengthening the exchange of medical knowledge and culture between China and foreign countries has become an indispensable link in China’s modernization. China and foreign countries can benefit each other. Therefore, the construction of a multilingual medical knowledge map may become an important trend in the future development of the medical field.

(2)

Construct a dynamic medical knowledge map

Regarding patients, from disease to curing this process, the clinical symptoms are not static. With the continuous advancement of the treatment process, the patient’s condition should be improved, or the treatment does not work, so the diagnosis and treatment is a dynamic process. However, the knowledge graph constructed in this paper is static. Building a dynamic knowledge graph is the way forward. By constructing a dynamic knowledge graph, medical guidance and drug recommendation for patients can be more accurate. This further improves the accuracy of the entire medical system and effectively improves the user experience.

(3)

Construct a multi-modal fusion medical knowledge base

At present, there is a problem of confidentiality of medical data, and the academic exchanges of various medical units are not transparent and not open. Due to the influence of many factors, the existing medical knowledge is limited in scale, which is mainly represented by text and graphic data. But the form of information is not single, for example, in the sound, image, photo and other aspects, there is also a considerable amount of medical information. So, the establishment of multi-modal fusion of a medical knowledge base can make full use of various forms of medical knowledge. This has a profound and broad impact on promoting the construction of a medical career.