Combining Semantic and Structural Features for Reasoning on Patent Knowledge Graphs

Abstract

To address the limitations in capturing complex semantic features between entities and the incomplete acquisition of entity and relationship information by existing patent knowledge graph reasoning algorithms, we propose a reasoning method that integrates semantic and structural features for patent knowledge graphs, denoted as SS-DSA. Initially, to facilitate the model representation of patent information, a directed graph representation model based on the patent knowledge graph is designed. Subsequently, structural information within the knowledge graph is mined using inductive learning, which is combined with the learning of graph structural features. Finally, an attention mechanism is employed to integrate the scoring results, enhancing the accuracy of reasoning outcomes such as patent classification, latent inter-entity relationships, and new knowledge inference. Experimental results demonstrate that the improved algorithm achieves an up to approximately 30% increase in the MRR index compared to the ComplEx model in the public Dataset 1; in Dataset 2, the MRR and Hits@n indexes, respectively, saw maximal improvements of nearly 30% and 112% when compared with MLMLM and ComplEx models; in Dataset 3, the MRR and Hits@n indexes realized maximal enhancements of nearly 200% and 40% in comparison with the MLMLM model. This effectively proves the efficacy of the refined model in the reasoning process. Compared to recently widely applied reasoning algorithms, it offers a superior capability in addressing complex structures within the datasets and accomplishing the completion of existing patent knowledge graphs.

1. Introduction

With the iterative growth of information data, the need for organizing patent knowledge has reached new heights. Given the vast number of patent examiners and the substantial burden of retrieval, there is a propensity for oversight and errors, making the construction of patent graphs significantly important to improve the level of business work in the patent field and the efficiency of data mining. As a knowledge-based system, a patent graph can extract and reflect the knowledge points of documents, assisting in the verification of patent novelty and creativity. However, existing patent knowledge graphs are not very complete and are rather sparse, affecting the accuracy of knowledge graph application systems. Constructing a comprehensive patent knowledge graph requires research based on traditional knowledge graphs. Therefore, completing traditional knowledge graphs has become an urgent issue. The existing knowledge graph reasoning methods mainly include knowledge graph reasoning based on traditional neural networks [1,2,3,4] and knowledge graph reasoning based on graph neural networks [5,6,7,8,9].

Mainstream knowledge graph reasoning methods extract semantic information features of entities and entity relationships from graphs for classification and reasoning of knowledge graph nodes. Miric et al. [10] compared and contrasted various machine learning methods with keyword-based approaches, highlighting the advantages of machine learning methods. However, their approach also led to the acquisition of a substantial amount of irrelevant information. Zhang et al. [11] utilized a multilayer Convolutional Neural Network (CNN) to design a classifier that effectively reduces the dependence on labeled data. However, the performance improvement is not significant in the presence of a large amount of unlabeled data. Yan et al. [12] proposed the DualRE framework to improve the prediction module. However, the issue of selecting high-quality samples from the unlabeled data remains unresolved. Hong et al. [13] introduced a competitive new method. Experiments have proven the effectiveness of AdvSRE, though the model’s training time was not considered. Santoro et al. [14] used deep neural networks to construct research models in visual question answering, relation prediction, and text question answering tasks, achieving good results, but the model lacked stability in learning-disordered information. Hildebrandt et al. [15], based on seq2seq [16], used LSTM to learn entity and entity relationship information in knowledge graphs, enhancing the stability of knowledge graph inference. However, the acquisition of relationships between entities remains incomplete. Kambhatla et al. [17] employed a maximum entropy model combined with various lexical, syntactic, and semantic features derived from text. This model achieved competitive results in the Automatic Content Extraction (ACE) evaluation, but further optimization is needed to improve inference accuracy.

With the development of graph neural networks and thanks to the compatibility of knowledge graph information with graph neural networks, more and more scholars are trying to integrate them into research on knowledge graph reasoning. Zhou et al. [18] proposed the Global Context-Enhanced Graph Convolutional Network (GCGCN) to capture rich global contextual information of entities within documents, demonstrating improved document-level relation extraction (RE) performance. However, it lacks inference capability in unknown domains. Schlichtkrull et al. [19] utilized Graph Convolutional Networks (GCNs) to obtain information from graph nodes, accelerating knowledge graph inference but not enhancing accuracy. Zhang et al. [20] integrated decay information theory into knowledge graph node information acquisition using Graph Neural Networks (GNNs). This algorithm achieved good results in single-framework tasks, but the collection of node information in GNNs was not comprehensive.

Constructing a knowledge graph containing all knowledge and information requires a massive workload. However, in patent knowledge graph inference algorithms, patent data mining provides substantial information support, significantly enhancing the accuracy and effectiveness of these algorithms. Yoon et al. [21] developed a technology intelligence system based on attributes and functions, utilizing patent data mining to identify technological trends. Daim et al. [22] combined patent analysis with bibliometric techniques to propose a method for constructing a knowledge graph of technological development, aiding in the identification and prediction of emerging technological trends. Lee et al. [23] proposed a keyword-based technology roadmap method to construct technology roadmaps, thereby guiding new product development and technological innovation.

Similarly, in the patent knowledge graph field, reasoning algorithms continually iterate to achieve higher performance standards. For example, Shan et al. [24] used extracted features to train a Support Vector Machine (SVM), effectively improving the F-score for relevance, but further optimization of inference capability is needed. Zhang et al. [25] proposed a method based on Semantic Maximum Divergence Measure (SMDM), demonstrating excellent improvements in F1 scores and external metrics, but it overlooked the extraction of structural information. Li et al. [26] introduced a Knowledge-Driven Convolutional Neural Network (K-CNN), and experiments showed that this model outperformed the current state-of-the-art relation extraction models, yet it performed poorly on large-scale datasets. Choudhary et al. [27] integrated a multi-head attention mechanism into Graph Neural Networks, proposing an attention-based knowledge graph inference GNN, which enhanced the accuracy of inference tasks but focused solely on the analysis of semantic information in the graph.

Although most of the inference models in the aforementioned studies have improved inference speed and accuracy to varying degrees through innovative approaches, they still struggle to fully utilize the information within and surrounding the nodes in knowledge graphs. Moreover, methods based on traditional neural networks and graph neural networks predominantly focus on enhancing the acquisition of semantic information from the graph while neglecting the structural information. Consequently, they lack the ability to infer knowledge in unlearned domains. To address the aforementioned issues, this paper first utilizes patent data mining techniques to automatically identify necessary content. This provides rich data support for knowledge graph inference algorithms. On this basis, a directed graph representation model of the patent knowledge graph is constructed, thus establishing the fundamental framework of the patent knowledge graph. Building on this, a reasoning method for patent knowledge graphs (SS-DSA) that integrates semantic and structural features is proposed. This method enhanced the application scope of the knowledge graph. By integrating semantic and structural features and incorporating a self-attention mechanism, the accuracy of the inference results is significantly improved. This supports complex reasoning tasks while markedly enhancing the inference outcomes, as illustrated in Figure 1. This process further refines the content and reasoning capabilities of the patent knowledge graph. Additionally, the improved patent knowledge graph inference methods can enhance the value and utility of patent knowledge graphs by uncovering potential connections between patents, identifying novel technological combinations and innovation points, pinpointing technological breakthroughs and development directions, and inferring relationships between different technological fields.

This study proposes a patent knowledge graph inference method that combines semantic and structural features (SS-DSA). First, a directed graph representation model of the patent knowledge graph is designed. Subsequently, by integrating inductive learning with graph structural feature learning and employing an attention mechanism to consolidate the scoring results, the accuracy of inference results for patent classification, potential relationships between entities, and new knowledge inference is improved. The main contributions of this research are summarized as follows:

(1)

A comprehensive foundational framework for a patent knowledge graph based on a directed graph model has been constructed, allowing for more precise structured representation and organization of the rich knowledge and information contained within patent documents.

(2)

An innovative patent knowledge graph reasoning method (SS-DSA) that combines semantic and structural features has been proposed, with the integration of a self-attention mechanism significantly enhancing the inference outcomes.

(3)

The effectiveness of the patent knowledge graph reasoning method, which combines semantic and structural features, has been validated through extensive ablation experiments.

2. Methodology

Based on the knowledge graph framework, this paper first establishes a directed graph representation model specifically for patent knowledge graphs. This model enables the patterned representation of patent information, providing a more intuitive display of patent structures and relationships, and offers comprehensive information integration to support subsequent knowledge graph inference. We propose a patent knowledge graph inference algorithm (SS-DSA) based on graph structural features. To avoid limitations, the model combines semantic information with graph structural information to construct a scoring function that captures a more comprehensive range of inference relationships, effectively enhancing the model’s reasoning capabilities. Finally, we introduce a multi-head attention mechanism to integrate the scoring functions for semantic information and structural information. This integration aids in understanding complex relationships between different entities and further improves the accuracy of the model’s inference results.

2.1. Directed Graph Representation Model for Patent Knowledge Graphs

To achieve a modeled representation of patent information, this paper incorporates the main elements of patents (such as inventors, applicants, patent classification numbers, cited documents, etc.) as nodes within a directed graph representation model. The relationships among these elements (such as co-inventors, same applicant, same classification number, etc.) are represented as directed edges, visually demonstrating the structure and relationships within patents.

Based on this model, when analyzing patent citation relationships, each patent is treated as a node, and the citation relationships between patents are represented with directed edges. This directed graph visually displays the citation structure of patents, further facilitating the analysis and forecasting of technological trends and research hotspots within patents. In the analysis of patent classification number relationships, each classification number is treated as a node, and relationships between different classification numbers are represented with directed edges, thereby understanding the connections and similarities between different classification numbers. Figure 2 provides an example of the directed graph representation model for a patent knowledge graph.

2.2. Patent Knowledge Graph Reasoning Method Based on Graph Structure Feature Learning

This paper addresses the limitations of traditional knowledge graph reasoning algorithms, which primarily focus on semantic information. By utilizing inductive learning to mine structural information within the knowledge graph, we propose a patent knowledge graph reasoning method based on graph structure feature learning, effectively enhancing the accuracy of the model’s reasoning. The structural design of the model is detailed as follows:

(1)

Graph Input of the Algorithm: To enable computers to process the entities and relationships in the patent knowledge graph, which exist in textual data form, it is necessary to convert them into vector format. This paper employs knowledge graph embedding techniques to store entities and relationships in vectors for retention.

(2)

Acquisition of Semantic Information from the Knowledge Graph: To overcome uncertainties in the reasoning process, the paper utilizes the concept of Graph Convolutional Networks (GCNs) on top of knowledge graph embedding techniques to capture information within the knowledge graph.

First, we establish a model of a knowledge graph that encompasses multiple relational types. The entire knowledge graph is denoted by $\mathrm{G}=(V,\epsilon ,R)$, within which the entity vectors are represented by ${\nu }_i\in V$, and $r\in R$ denotes the variety of relations $r$ contained within the relationship type $R$. $\epsilon$ represents the set of relationship instances, which corresponds to the edges of the graph. The triple relationship pair $\left(v_i,r,v_j\right)\in \epsilon$ includes two distinct entities and one type of relationship.

GCN, a differentiable information propagation model, has evolved from the principles of thermodynamic propagation. Typically utilized for knowledge graph link prediction, GCN is a variant of the foundational graph neural networks (GCNs) and is extensively applied in tasks such as knowledge graph link reasoning. It is adept at capturing node and edge information within the graph, enabling end-to-end learning and reasoning. Its message propagation model, as illustrated in Equation (1), delineates the process of information dissemination and feature representation updating among nodes within the graph structure:

$$h_i^{(l+1)}=\sigma \left({\displaystyle \sum _{m\in M_i}{\vartheta }_m\left(h_i^l,h_j^l\right)}\right)$$

(1)

Herein, $h_i^l$ represents the entity variable, $v_i$ is the value at the $l$-th layer of the hidden layer, $h_i^{l+1}$ denotes the value of the entity variable $v_i$ at the $l+1$-th layer of the hidden layer, $M_i$ represents the set of neighboring nodes of $v_i$, and $\sigma (.)$ employs the ReLU activation function. The hidden value of a certain layer, through the weight matrix coefficient $W^l$, undergoes a pairwise linear transformation to obtain an aggregated result of the neighbor nodes, denoted as ${\vartheta }_m$. Within this, ${\vartheta }_m$ aggregates these two feature representations into a new feature representation, $W^l$ signifies the weight matrix of the $l$-th layer, which is used for learning the transformation of node features, as shown in Equation (2):

$${\vartheta }_m\left(h_i^l,h_j^l\right)=W^l{h}_j^l$$

(2)

The definition of the information propagation model concerning the entity variable within a multi-relational graph is based on the GCN model, as shown in Equation (3):

$$h_i^{(l+1)}=\sigma \left({\displaystyle \sum _{r\in R}{\displaystyle \sum _{j\in N_i^{\tau }}\frac{1}{c_{i,\tau }}{W}_r^l{h}_j^l+W_o^l{h}_i^l}}\right)$$

(3)

To compute each entity in the knowledge graph related to the relationship, the parameter $N_i^{\tau }$ is denoted as the index set of neighboring nodes that have a relationship $r\in R$ with the $i$-th node in the knowledge graph. $c_{i,\tau }$ is the normalization constant for the relationship r and the node pair (i,j), $W_r$ is the weight matrix associated with the relationship type r, and $\sigma (.)$ employs the ReLU activation function. The detailed method of information aggregation is illustrated in Figure 3:

After embedding structural knowledge into the graph, a multi-layer neural network is added to the entire knowledge graph reasoning model, and the obtained feature information is used as the input for the decoder for reasoning purposes. By using negative sampling, feedback is provided to the information in the hidden layer for existing correct relationships during training. Negative and positive samples are processed using a random selection mode and negative sampling of $\omega$. The result of the loss function, optimized through cross-entropy, is shown in Equation (4), where $\omega$ acts as a parameter controlling the frequency of negative sampling. The normalization factor ${\displaystyle \frac{1}{\left(1+\omega \right)\left|\stackrel{\wedge }{\epsilon }\right|}}$ uses $\left|\stackrel{\wedge }{\epsilon }\right|$ to standardize the loss, where $\left|\stackrel{\wedge }{\epsilon }\right|$ represents the size of the positive sample set, $\left(s,r,o,y\right)$ for missing triples, $T$ for all sets of missing triples, $s$ as the subject entity, $r$ as the relation, $o$ as the object entity, $y$ as a label indicating the existence of the triple, $l$ as the optimized loss value, $f$ as the model’s predicted probability of the existence of the triple, and $sigmoid$ as the output processed through the activation function.

$$\begin{array}{l}l=-{\displaystyle \frac{1}{\left(1+\omega \right)\left|\stackrel{\wedge }{\epsilon }\right|}}{\displaystyle \sum _{(s,r,o,y)\in T}y\log \left(sigmoid\left(f\left(s,r,o\right)\right)\right)}+\\ (1-y)\log \left(1-sigmoid\left(f\left(s,r,o\right)\right)\right)\end{array}$$

(4)

As illustrated in Figure 4, the architecture of the graph convolutional network model for knowledge graph relation reasoning consists of a convolutional information propagation network and a scoring model.

(3)

Acquiring Structural Information in Graph Neural Networks: Utilizing the theory of isomorphic knowledge graph mapping, structural information from the knowledge graph can be extracted.

The process begins by identifying the target head and tail entities, incorporating their associated nodes, entities, and relationships as the structural information of the graph.

In detail, information from the K-hop neighborhood of the target nodes $u$ and $v$ is concurrently obtained, denoted as $N_{k(v)}$ and $N_{k(u)}$, respectively. The intersection of $N_{k(v)}$ and $N_{k(u)}$ forms the structure from which the corresponding structural information is derived.

After acquiring the graph structure, entities and triples within the structure surrounding $u$ and $v$ are labeled with information and the shortest path length is denoted as $\left(d\left(i,u\right),d\left(i,v\right)\right)$, where $d(i,u)$ represents the distance between nodes $i$ and $u$, and $d(i,v)$ signifies the relative positions of other nodes to the target node.

To extract semantic information from the image, a search is first conducted for the target head and tail entities within the patent knowledge graph. Subsequently, entity and relationship information of the target nodes is extracted using related algorithms, followed by an update of the model parameters.

For extracting structural information, an improved Graph SAGE network is employed to capture the structure of the surrounding nodes of the target in the knowledge graph. Past structural information is then utilized to predict information collection tasks for unknown relationships within the knowledge graph.

A feature of the knowledge graph is that the target node combined with its surrounding nodes forms a complete graph. Based on this, part of the structural information around the target node is filtered, and the number of hops and the distance between nodes are assessed. If the number of hops is excessive or the distance is too great, that part of the information is discarded.

Using isomorphic information theory, positive samples in the dataset are processed by combining the head entity, tail entity, and correct relationships to form an overall subgraph structure, which disregards spatial positioning. When dealing with negative samples, those that do not form an overall subgraph structure are excluded, meaning that the structural information of that part of the data is not learned. In the task of extracting structural information, only the features of relevant nodes are extracted, rather than the features of the entire graph, as represented in Equations (5) and (6):

$$a_u^k=AGGREGAT{E}^k\left(\left\{h_{u^{\prime }}^{k-1}:u^{\prime }\in N\left(u\right),h_v^{k-1}\right\}\right)$$

(5)

$$h_u^k=CONCA{T}^k\left(h_u^{k-1},a_u^k\right)$$

(6)

where $a_u^k$ represents the aggregated information from node $u$ at layer $k$, and $N\left(u\right)$ represents the set of neighbor nodes for node $u$. The nodes within this range are referred to as the L-hop neighborhood of node $u$, set to 3. $AGGREGAT{E}^k$ is the aggregation function that combines the features of neighboring nodes at layer $k$. $h_u^k$ denotes the hidden information at layer $k$, calculated by combining information from the previous layer’s hidden state with the current layer’s aggregated information, providing a new feature representation for node $u$ at layer $k$. $CONCA{T}^k$ is the concatenation function that merges the current feature representation of node $u$ with the aggregated information of its neighbors at layer $k$.

The AGGREGATE computation in Equation (5) is defined as shown in Equation (7):

$$a_u^k={\displaystyle \sum _{r=1}^R{\displaystyle \sum _{v\in N_r(u)}{\alpha }_{r{r}_{uv}}^k}}{W}_r^k{h}_v^{k-1}$$

(7)

where $R$ represents the total number of known relationships within the knowledge graph, and $N_r\left(u\right)$ denotes the set of neighbor nodes for node $u$, where these neighbors share a relationship $r$ with node $u$. $W_r^k$ is the transformation matrix for relationship $r$ at layer $k$, utilized for transforming node features. ${\alpha }_{r{r}_{uv}}^k$ indicates the weight matrix associated with the relationships surrounding the node, $W_r^k$ is the weight matrix for node $r$ at layer $k$, and $h_v^{k-1}$ represents the hidden information of node $\nu$ at layer $k-1$.

The CONCAT method referred to in Equation (6) is defined as shown in Equation (8):

$$h_u^k=\mathrm{Re}LU\left(W_{self}^k{h}_u^{k-1}+{\alpha }_u^k\right)$$

(8)

where $W_{self}^k$ denotes the self-loop weight matrix at layer $k$, which is used for the transformation of its own features, specifically the self-connection weight.

The global representation of structural information and its computation process is detailed in Equation (9):

$$z_{{\vartheta }_{(u,v,r_t)}}^L={\displaystyle \frac{1}{\left|V\right|}}{\displaystyle \sum _{i\in V}{h}_i^L}$$

(9)

where $L$ indicates the use of the node feature representation from the $L$-th layer of the graph network to calculate this global feature. $\left|V\right|$ represents the set of vertices in graph ${\vartheta }_{\left(u,v,r_t\right)}$, which is the total number of nodes in the graph. ${\displaystyle \frac{1}{\left|V\right|}}$ is a normalization factor used to calculate the average.

Based on the above description, the information extraction process is constructed as shown in Figure 5:

(4)

Semantic Information and Structural Information Scoring Function: Relation reasoning involves predicting information not originally present in the knowledge graph. Specifically, this involves ranking the predicted relationships by score and retaining the highest-scoring option as the final prediction result.

Equation (10) presents the scoring function used to predict relationship $\left(s,r,o\right)$, where $s$, $r$, and $o$ represent the vector representations of the subject entity, relationship, and object entity, respectively. $R_r$ is the matrix representation corresponding to relationship $r$, $e$ is the base of natural logarithm, $e_s^T$ is the transpose of the embedding vector of entity $s$, and $e_o$ is the embedding vector of entity $o$.

$$f_1\left(s,r,o\right)=e_s^T{R}_r{e}_o$$

(10)

As shown in Equation (11), the score is computed by integrating the latent feature representations $h_u^L$ and $h_v^L$ of target nodes $u$ and $v$ at layer $L$, the surrounding structural information $u$ and $v$, and the relationship embedding vectors of nodes $u$ and $v$. Here, $z_{{\vartheta }_{(u,v,r_t)}}^L$ denotes the structural information between nodes $u$ and $v$ under relationship type $r$, which is obtained from the $L$th layer of the graph neural network. $\oplus$ represents the concatenation of vectors, and $r_{r_t}$ corresponds to the feature representation of the relationship between specific temporal nodes at this step.

$$f_2\left(u,v,r_t\right)=W^T\left[z_{\vartheta \left(u,v,r_t\right)}^L\oplus h_u^L\oplus h_v^L\oplus r_{r_t}\right]$$

(11)

2.3. Multi-Head Attention Mechanism Fusion Block

In recent years, attention mechanisms based on adaptive features have proven effective in enhancing model performance and have been widely applied in knowledge graph reasoning. Due to the complexity of patent knowledge information analysis, and in order to further obtain comprehensive information from the knowledge graph and better integrate semantic and structural information to enhance the model’s precise semantic understanding and robust reasoning capabilities, the multi-head attention mechanism is applied at the score fusion stage.

The multi-head attention mechanism merges the scores derived from semantic and structural information, ensuring that the resultant scores reflect both the semantic and structural information of the knowledge graph. This mechanism allows the model to dynamically weight each element in a sequence, assign varying degrees of importance, and aggregate this information into a single output vector. This aggregation considers the content of each element in the sequence and their interactions. In natural language processing tasks, it aids models in understanding context, capturing long-distance dependencies, and within knowledge graphs, it facilitates the understanding of complex relationships between different entities.

The computational method of the multi-head attention mechanism for merging semantic and structural information from graphs is illustrated in Equation (12):

$$\{\begin{array}{l}MultiHead\left(Q,K,V\right)=Concat\left(hea{d}_1,hea{d}_2\right)W^o\\ hea{d}_i=a\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^k\right)\end{array}$$

(12)

where $MultiHead\left(Q,K,V\right)$ is the multi-head attention function, and $Q,K,V$ represent the inputs for queries, keys, and values, respectively. The outputs of two attention heads are $hea{d}_1$ and $hea{d}_2$, with $Concat$ denoting the concatenation of these two heads’ outputs. $W^o$ is the linear transformation matrix that integrates the outputs, with each head capturing different aspects of the information from the input sequence. Each head $hea{d}_i$ computes its own output of the attention function $a$, where $Q{W}_i^Q,K{W}_i^K,V{W}_i^k$ are the inputs $Q,K,V$ multiplied by their respective weight matrices.

$hea{d}_1$ and $hea{d}_2$ specifically represent the attention for semantic information and structural information parts, respectively. The calculation method is shown in Equation (13), where softmax is used as the activation function, and $\sqrt{d_k}$ is a scaling factor used to control the size of the dot product to prevent issues of gradient vanishing due to excessively large dot products.

$$a\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(13)

When computing the attention for semantic information, the input consists of entity vectors from the knowledge graph, denoted as $X=\left(x_1,x_2,\cdots x_n\right)$. The calculation of attention weights for the semantic information component $hea{d}_1$ is shown in Equation (14).

$$a_1=softmax\left({\displaystyle \frac{X{X}^T}{\sqrt{d_k}}}\right)X$$

(14)

In the process of calculating the attention vector for structural features, the relationship between the target node’s structural information and the final result is represented by the degree of the entity node, with the input for the weight coefficient calculation denoted as $D=\left(d_1,d_2,\cdots d_n\right)$, and the attention weights for the structural information component $hea{d}_2$ shown in Equation (15).

$$a_2=softmax\left({\displaystyle \frac{D{D}^T}{\sqrt{d_k}}}\right)D$$

(15)

Based on the output values from Equations (14) and (15), semantic and structural scoring information are merged. The fusion method is outlined in Equation (16).

$$f={\displaystyle \sum _{i=1}^n{\alpha }_i{f}_i}$$

(16)

The weight calculation parameters in Equation (16) are detailed in Equation (17), with the calculated ${\alpha }_i$ being normalized weights, where the sum of all weights equals 1.

$${\alpha }_i={\displaystyle \frac{\exp \left(a_i\right)}{{\displaystyle \sum _{i=1}^n\exp \left(a_i\right)}}}$$

(17)

Utilizing these steps, the final scoring result is obtained. The innovative combination of graph structural information to construct a scoring function has yielded a more comprehensive set of reasoning relationships. The model can model the importance of different parts of the input data and effectively use the self-attention mechanism to assist in knowledge graph reasoning. In the field of patent knowledge graphs, this mechanism can help the model determine the importance of different entities and relationships, providing more refined responses for complex queries.

3. Experiment Result and Analysis

3.1. Experimental Preparation

3.1.1. Experimental Environment

All experiments were conducted on the same server. The server’s operating system is Ubuntu 18.04, equipped with an RTX 3080 Ti GPU. The deep learning framework used is PyTorch 1.7.1, with Python as the programming language.

3.1.2. Dataset

This paper selected four different datasets from a patent library in the Shanghai area for comparative experiments. The specific details are shown in

Table 1. Summary of information on the three different datasets.

Dataset	Entity	Relation	Edge			Average Node Degree
			Training Set	Validation Set	Test Set
Dataset 1	40,943	11	86,835	3034	3134	2.12
Dataset 2	14,541	237	272,115	17,535	20,466	18.71
Dataset 3	75,492	200	149,678	543	3992	1.98
Dataset 4	10,094	251	368,868	46,302	46,159	36.54

.

Dataset 1 primarily includes information on hyponymy and hypernymy relationships, concept semantics, synonyms and antonyms, polysemy, and homonymy. This dataset aids the model in understanding and reasoning about complex semantic relationships between terms, enhancing both the accuracy and breadth of reasoning.

Dataset 2 comprises 14,541 entities and 237 types of relationships. It can be used to train and test the model’s extensive knowledge and reasoning capabilities, ensuring that the model can accurately reason across a diverse range of entities and relationships.

Dataset 3 contains over 3 million knowledge graph triplets. It helps the model learn and validate reasoning capabilities within a large-scale knowledge graph, ensuring that the model can handle and reason with massive amounts of information.

Dataset 4 consists of 368,868 data points available for training. It provides a balanced set of training and validation data, allowing for rapid evaluation of the model’s reasoning abilities and generalization performance while ensuring effective training.

These patent datasets primarily include relevant information such as patent publication numbers, inventors, applicants, IPC patent numbers, and domain keywords. By offering rich and diverse semantic and relational information, these datasets are well-suited for validating the capabilities of patent knowledge graph reasoning models. They cover aspects ranging from fine-grained semantic relationships to large-scale knowledge graphs, helping to comprehensively assess the model’s reasoning performance across different contexts and scales.

3.1.3. Evaluation Metrics

This paper selects three commonly used evaluation metrics for assessing the effectiveness of relation reasoning tasks: Mean Rank (MR), Mean Reciprocal Rank (MRR), and Hits at rank n (Hits@n). MR is used to measure the average position of the correct answers among all possible answers; MRR aims to calculate the average of the reciprocal ranks of the correct entities; and Hits@n is designed to measure the percentage of correct entities that appear in the top n ranks. By selecting Mean Rank (MR), Mean Reciprocal Rank (MRR), and Hits@n as evaluation metrics, we can comprehensively and accurately assess the performance of the patent knowledge graph inference algorithm. These metrics not only reflect the overall effectiveness of the algorithm but also take into account user experience in practical scenarios, ensuring that the algorithm performs excellently across different dimensions.

The calculation method for MR is as follows in Equation (18):

$$MR={\displaystyle \frac{1}{\left|N_t\right|}}{\displaystyle \sum _{i=1}^{\left|N_t\right|}ran{k}_i}$$

(18)

where $\left|N_t\right|$ represents the index of the triplet, and $ran{k}_i$ indicates the rank of the $i$-th triplet in the reasoning task. Additionally, the calculation method for the MRR metric is as follows in Equation (19):

$$MR={\displaystyle \frac{1}{\left|N_t\right|}}{\displaystyle \sum _{i=1}^{\left|N_t\right|}\frac{1}{ran{k}_i}}$$

(19)

The Hits@n (where n = 3) metric is shown in Equation (20):

$$Hit@n={\displaystyle \frac{1}{\left|N_t\right|}}{\displaystyle \sum _{i=1}^{\left|N_t\right|}1(ran{k}_i\le n)}$$

(20)

where $1(\cdot )$ is an indicator function, such that the value of $1(\cdot )$ is 1 when $ran{k}_i\le n$ is true; when $ran{k}_i\ge n$ is the case, the value of $1(\cdot )$ is 0.

3.1.4. Baseline Models

To ensure a fair comparison, this paper selects four baseline models: ComplEx [28] (Complex Embeddings), RotatE [29] (Rotation Embeddings), MLMLM [30] (Link Prediction with Mean Likelihood Masked Language Model), and DensE [31] (Dense Embeddings). Each method introduces innovations in semantic and structural reasoning, improves the quality of entity and relationship embeddings, captures more complex graph structural patterns, and thereby enhances reasoning performance.

3.2. Experimental Comparison and Analysis

In the experiments conducted across four datasets, the comparison involved baseline models ComplEx, RotatE, MLMLM, and DensE. Regarding the evaluation metrics, a lower MR value indicates that the correct answers are generally ranked higher, suggesting better model performance; a higher MRR value indicates that the model tends to rank correct answers near the top, showing better performance in the dataset; a higher Hits@n score indicates the model’s increased accuracy in predicting correct answers at the top ranks, thus demonstrating superior performance. In this paper, the metric $n$ is set to 3, providing the model with the ability to precisely match the top predictions, and the best scores in the evaluation parameters are highlighted in bold in the tables.

In the experiments, the SS-DGA model proposed in this paper is first compared against several benchmarks across three datasets. Subsequently, a set of ablation studies is conducted to investigate the impact of different model components on the reasoning performance.

3.2.1. Comparative Experiments with Baseline Models

The experiments conducted on three publicly available datasets compared the proposed SS-DGA model against baseline models.

Table 2. Comparison of SS-DGA and Baseline Models on Dataset 1.

Methods of Comparison	Evaluation Indicators
	MR	MRR	Hits@n
ComplEx	9021	0.397	0.612
RotatE	3245	0.412	0.486
MLMLM	1523	0.511	0.532
DensE	2865	0.481	0.512
SS-DGA	2834	0.524	0.501

,

Table 3. Comparison of SS-DGA and Baseline Models on Dataset 2.

Methods of Comparison	Evaluation Indicators
	MR	MRR	Hits@n
ComplEx	631	0.311	0.255
RotatE	142	0.322	0.351
MLMLM	451	0.287	0.275
DensE	158	0.321	0.371
SS-DGA	173	0.362	0.541

and

Table 4. Comparison of SS-DGA and Baseline Models on Dataset 3.

Methods of Comparison	Evaluation Indicators
	MR	MRR	Hits@n
ComplEx	6310	0.421	0.601
RotatE	5142	0.309	0.515
MLMLM	5452	0.187	0.491
DensE	1520	0.521	0.671
Ours	4721	0.542	0.682

, respectively, display the performance comparison results of the proposed SS-DGA and the baselines across the three datasets.

To better validate the performance of the proposed model, three datasets of varying complexity were introduced. In datasets of varying complexity, as shown in Table 2 and Figure 6, comparative experiments indicate that the SS-DGA model achieves the best results in terms of the MRR metric in Dataset 1. Compared to the ComplEx, RotatE, MLMLM, and DensE models, the SS-DGA model demonstrates improvements of approximately 32%, 27%, 3%, and 8%, respectively. It also achieved the second-best result in the MR metric, indicating that the model is more effective at identifying the relevance of the correct answers. The proposed model is capable of modeling the importance of different parts of the input data. Based on the semantic information, the integration of graph structural information has enabled the model to realize more comprehensive reasoning relationships, enhancing the model’s reasoning capabilities and depth of understanding of relationships. However, in the Hits@3 metric, there was only about a 3% improvement compared to the RotatE model, which might be due to the presence of a significant amount of polysemy and homonymy information in the dataset. Such conditions increase the complexity of model parsing and the application of knowledge graph structures, leading to less pronounced effects of structural information learning. This difficulty in precisely distinguishing and utilizing structural information indicates that there is still room for improvement in the model’s handling of polysemy and complex relationships.

As shown in Table 3 and Figure 7, it is evident that in Dataset 2, the SS-DGA model achieves the best results in both the MRR and Hits@3 metrics. Compared to the baseline models ComplEx, RotatE, MLMLM, and DensE, the SS-DGA model improves the MRR metric by 16%, 12%, 26%, and 13%, respectively. Notably, the Hits@3 metric shows an improvement of nearly 112% compared to the MLMLM model. This demonstrates that the model is highly effective at placing the correct answers at the forefront of the prediction list and significantly increases the probability of the correct answers appearing within the top three rankings. Consequently, this significantly reduces the length of the candidate list that users or downstream systems need to review to find the correct answer. It also markedly enhances the likelihood of further exploration by users. The model also achieved good results in the MR metric, indicating its overall ability to effectively process and rank candidate relationships. This suggests that the self-attention mechanism can significantly aid the model in reasoning within the knowledge graph, understanding the complex relationship between different entities, and more effectively integrating semantic and structural information. Thus, the effectiveness of the proposed model in the reasoning process is demonstrated.

As shown in Table 4 and Figure 8, in Dataset 3, the SS-DGA model achieves only the second-best results in the MR metric. However, it achieves the best results in both the MRR and Hits@3 metrics. Specifically, the MRR metric shows improvements of 29%, 75%, 187%, and 4%, respectively. The Hits@3 metric shows a maximum improvement of nearly 39% compared to the MLMLM model. A high MRR indicates that the model is very adept at quickly ranking the correct triples at the top, which directly impacts the efficiency with which users can find information. A high Hits@3 means that the probability of the correct answers appearing within the top three rankings has significantly increased, further demonstrating the model’s superiority in predictive accuracy.

The experimental results validate the model’s ability to discern the importance of different entities and relationships, providing more refined responses for complex queries, and effectively reasoning about triplet information. Particularly, it excels in rapidly and accurately identifying the correct triplets.

Through the comparative experiments conducted on datasets of varying complexity, the SS-DGA model has been validated as being capable of thoroughly mining structural information. The use of the self-attention mechanism effectively assists the model in knowledge graph reasoning.

3.2.2. Ablation Studies

To investigate the impact of different model components on the performance of SS-DGA, this section conducted a set of ablation experiments on Dataset 4. These experiments explored the impact of each enhancement by removing them one at a time. Initially, for the SS-DGA module, which builds upon the acquisition of semantic information, the directed graph representation model of the patent knowledge graph designed in this paper was removed, abbreviated as DS. Subsequently, the effects of integrating graph structural information and the attention mechanism were analyzed and were denoted as SI and AM, respectively. Furthermore, to further demonstrate the effects of different components within SS-DGA, the performance of the baseline and SS-DGA (with various model components removed) was compared.

Table 5. Results of SS-DGA Component Ablation Experiments.

Methods of Comparison	Evaluation Indicators
	MR	MRR	Hits@3
ComplEx	3234	0.398	0.401
RotatE	2142	0.349	0.315
MLMLM	3452	0.317	0.414
DensE	1780	0.421	0.421
SS-DGA (w/0_DS)	2970	0.518	0.496
SS-DGA (w/0_SI)	2120	0.488	0.486
SS-DGA (w/0_AM)	1805	0.509	0.501
SS-DGA	1521	0.542	0.512

displays the performance comparison results of the ablation experiments.

By conducting ablation studies on certain components of the SS-DGA, it was found that each component of the model impacts performance, demonstrating that all parts of the model are effective. Specifically, in Dataset 4, the removal of the directed graph representation model led to a decline in MRR performance by 2.4 percentage points and a decrease in Hits@3 by 1.6 percentage points. Removing the directed graph model means that, specifically within the patent domain, the model cannot intuitively display the structure and relationships of patents, which diminishes the convenience of information extraction. Without the help of a patent-specific directed graph model, it is impossible to obtain more direct neighbor information around entities, leading to an overall decrease in model performance. This proves that neighborhood information of entities can indeed better enrich entity representations and also provide more semantic representations for reasoning tasks. In experiments where the module integrating graph structure was removed, the MRR and Hits@3 indices decreased by 5.4 and 2.6 percentage points, respectively. The significant performance deterioration demonstrates the substantial impact of structural information on knowledge graph reasoning. Integrating structural information based on semantic information significantly enhances the overall performance of the model. Considering that the attention mechanism can adaptively enhance the expression of important features in the model, the effects of having or lacking this mechanism were also tested. The experiments showed that the removal of the attention mechanism resulted in decreases in MRR and Hits@3 by 3.3 and 1.1 percentage points, respectively. The attention mechanism allows the model to focus more intensively on analyzing and extracting features from critical areas, thereby better capturing key information to enhance the model’s reasoning capabilities. In summary, the ablation experiments verified that each part of the model contributes significantly to final performance, demonstrating that the components of the model are not redundant or unnecessary.

3.3. Experimental Conclusions

The experimental results indicate that the patent knowledge graph reasoning method proposed in this paper has achieved favorable outcomes across Datasets 1, 2, and 3. This success is attributed to the innovative incorporation of graph structural feature extraction, which offers a significant advantage in handling complex structures within the datasets. Ablation studies conducted on Dataset 4 demonstrate the effectiveness of each component of the model, confirming that all parts are functional. The method designed a directed graph representation model for the patent knowledge graph. Based on acquiring semantic information, it combines graph structural information to construct a more comprehensive scoring function for reasoning relationships. The introduction of a multi-head attention mechanism allows for the fusion of scores derived from semantic and structural information, ensuring that the resulting scores reflect both the semantic and structural information of the knowledge graph. Moreover, the model is capable of modeling the importance of different parts of the input data and uses the self-attention mechanism to further enhance the accuracy of the results. This approach effectively facilitates the inference and completion of missing relationships within the knowledge graph.

4. Conclusions

This paper addresses the issue of incomplete patent knowledge graphs, which impacts the accuracy of patent knowledge graph application systems, by establishing a directed graph representation model for patent knowledge graphs. It proposes a reasoning algorithm based on graph structural feature learning and integrates a self-attention mechanism. This algorithm significantly improves the accuracy of reasoning outcomes such as unknown patent classification, potential relationships between entities, and new knowledge, demonstrating the strong reasoning capabilities of the patent knowledge graph reasoning method that combines semantic and structural features when dealing with complex structural knowledge graphs. This method can effectively complete and refine the patent knowledge graph.

In the future, patent knowledge graph inference methods that combine semantic and structural features hold extensive potential for practical applications, including patent novelty assessment, patent retrieval systems, patent technology analysis systems, and technology transfer and licensing.

Patent Novelty Assessment: The proposed inference algorithm, which integrates graph neural networks and self-attention mechanisms, can uncover latent relationships between patents and identify novel technology combinations and innovations.

Patent Retrieval Systems: The enhanced inference algorithm can significantly improve the accuracy and efficiency of patent retrieval systems.

Patent Technology Analysis Systems: The inference algorithm proposed in this paper can more accurately infer patent classifications, relationships between inventors, and latent knowledge within patents.

Technology Transfer and Licensing: By employing the proposed inference method, potential opportunities for technology transfer and licensing can be better identified and analyzed, facilitating the commercialization of innovative results.

Additionally, we will explore strategies to better handle complex cases of polysemy and synonymy, further analyze the model’s performance across different types of queries or relationships, and enhance the extraction of structural feature information.

The following considerations are essential for subsequent research:

(1)

Introduce a context-aware embedding mechanism to improve handling of lexical polysemy.

(2)

Optimize graph structure parsing algorithms to enhance understanding of complex entities and relationships, especially in dealing with homonymy.

(3)

Integrate natural language processing technologies to enhance the semantic expressive capabilities of knowledge graphs.