IntME: Combined Improving Feature Interactions and Matrix Multiplication for Convolution-Based Knowledge Graph Embedding

Abstract

Knowledge graph embedding is a method to predict missing links in knowledge graphs by learning the interactions between embedded entities and relations in a continuous low-dimensional space. Current research on convolution-based models tends to provide sufficient interactions for extracting potential knowledge. However, sufficient interactions do not mean that they are reasonable. Our studies find that reasonable interactions can further stimulate knowledge extraction capability. Reasonable interactions need to ensure that the elements participating in interactions are disordered and in a reasonable number. To model reasonable interactions that cannot be specifically quantified, we propose a concise and effective model IntME to address this challenge. In detail, we utilize checked feature reshaping and disordered matrix multiplication to form two different types of feature maps to ensure the disorder of the interacting elements and control the number of elements before feature fusion by the shapes of the feature maps after channel scaling reshaping. In feature fusion, we employ large convolution filters and pointwise filters for the deep and shallow linear fusion of feature interactions, which can take into account both explicit and implicit knowledge extraction capability. The evaluations of four benchmark datasets show that IntME has a powerful performance in convolution-based models and a lower training cost, and also demonstrate that our proposed approaches based on reasonable interactions can effectively improve knowledge discovery capability.

1. Inroduction

Knowledge graph is a directed heterogeneous graph structure consisting of multiple triplets of relations and entities, each of which can be canonically represented as $(h,r,t)$, with a link start position, the head entity, a link itself, the relation, and a link end position, the tail entity. There are very classical large knowledge graphs, such as WordNet [1], YAGO [2], Freebase [3], DBpedia [4], and they have been used for many other tasks including social networks [5], question answering [6], and recommender systems [7]. In the real world, people often want to discover potential undiscovered knowledge through existing knowledge, and such needs are more urgent in knowledge systems. To resolve the issues caused by the uncertainty and incompleteness of knowledge graphs, link prediction is proposed to solve the problem of predicting missing links based on the existing triplets.

The most active approach for the link prediction task is knowledge graph embedding, which strives to learn the representational characteristics of all the entities and relations in the existing triplets. The majority of the models are distance-based models in initial research on knowledge graph embedding, such as TransE [8], TransR [9], TransH [10], TransO [11], which use distance relationships through low-dimensional spatial embedding between head entities, relations, and tail entities in triplets as training targets. Also, bilinear models, which capture the intrinsic interaction modes of the triad by tensor decomposition to predict missing links, include models such as Rescal [12], DistMult [13], and ComplEx [14]. With the continuous expansion of knowledge graphs and the increasing demand for potential knowledge discovery, the limitations of translation-based models, which find it difficult to handle nonlinear and deep features, come to the fore. Subsequent research has gradually shifted to neural networks with a better ability to fit nonlinear features. ConvE [15] is the first application of convolution networks in knowledge graph embedding models, using a vector reshaped into a matrix form and utilizing 2D convolution to extract interaction information.

However, ConvE simply stacks the feature maps, and there are few interactions of elements. As shown in Figure 1, ConvR [16] thinks of transforming relation embedding to internal convolution filters to achieve interactions, while JointE [17] extends the internal filters to entities and relations, and combines 1D convolution to increase the extraction ability of explicit knowledge, but the form of internal convolution filters for interaction augmentation comes with a high training cost. In contrast to the above methods of enhanced interactions, InteractE [18] and ArcE [19] employ the methods shown in Figure 2. InteractE enhances feature interactions by introducing multiple feature map permutations and by using large convolution kernels to extract latent information, and AcrE uses both atrous convolution and standard convolution to enhance interactions between different locations. The utilization of multiple feature maps combined with depth-wise circle convolution also causes an increased training cost, and the number of interactions can only be controlled by the scaling of the receptive fields.

In more detail, the internal 2D convolution needs to convert all relations and entities into convolution filters for interaction operations. Therefore, enhancing the interaction process of constructing internal filters is time-consuming and will cost much more time than the common externals, and simple 1D convolution finds it difficult to enhance interactions without introducing extra features and parameters. While InteractE employs checked feature reshaping to enhance interactions, reorganizing the elements of entity embeddings and relation embeddings into one or more new feature maps resembling a black-and-white checkerboard pattern, it only focuses on interactions which results in the lack of explicit knowledge extraction and the increase of training costs. The enhancement of the interactions by utilizing atrous convolution in AcrE is also limited.

In this paper, we propose an improved model IntME, which addresses the shortcomings of the above convolution-based models by using the approaches mentioned in Figure 3. Detailed operations will be discussed in the section that describes the model mechanisms. A reasonable number of interactions is the key to whether the latent knowledge extraction capability is strong in the model. Instead of finding a reasonable number of interactions starting from zero blindly, we introduce feature maps as the initial value of the interaction number and design a structure to fit the difference between the reasonable number of interactions and the initial value, which greatly reduces the difficulty of locating a reasonable number of interactions. In further detail, we introduce a checked feature reshaping feature map as the initial value of interaction number and use another path to expand the features linearly by matrix multiplication, so that each element of the enlarged feature maps contains information on both entities and relations. It is concluded from [18] that the fusion of the disordered elements tremendously facilitates the increase of the element interactions. Therefore, we disorder the feature map elements both before and after expansion by utilizing the element reordering to guarantee the quality of fused interactions. By channel scaling reshaping, the enlarged feature maps are reshaped to locate a reasonable number of elements in each channel before channel fusion, then shallowly fuse the channel features by pointwise filters to fit the difference. With the above operations, it is possible to train excellent potential knowledge extraction capability at a low cost. In addition, all the features are integrated in a linear way during fitting the difference, which reduces the loss of surface and explicit knowledge. Eventually, the features that contain the initial values of the interaction number and the features that contain the explicit knowledge information and the difference are fused by element-wise addition.

We evaluate our model on four representative benchmark datasets, and the implementation demonstrates that our model can achieve a state-of-the-art performance on medium and large datasets while having a decent performance on small datasets as well. In brief, our contributions are as follows:

• We increase the element interaction randomness by employing element reordering and preserve a reasonable interaction number by utilizing channel scaling reshaping before information fusion to further improve the knowledge extraction capability without extra parameters;
• We exploit matrix multiplication and pointwise filters to shallowly linearly fuse features in each channel, which can maintain interactions while retaining as much valuable explicit knowledge information as possible;
• Experiment evaluations demonstrate that our model outperforms powerful convolution-based models on four benchmark datasets in most cases, with robust generalization capability and a faster training process.

2. Related Work

In this section, we will introduce the related work on knowledge graph embedding models.

2.1. Translation-Based Models

TransE [8] and Rescal [12], as the typical representatives of the distance-based models and bilinear models, have opened the knowledge graph embedding research process. The vector geometric relation in low-dimensional linear space is the focus of the distance-based models to model entity vectors and relation vectors. TransE first uses simple vector distances as training targets for modeling positional relationships between vectors in low-dimensional space, but its training targets are not suitable for modeling complex relations. Therefore, TransH [10] thinks of solving this problem by utilizing the projection of entity vectors onto the hyperplane to solve the issue caused by the same distance of multiple relations. TransD [20], in contrast, takes into consideration not only the diversity of relations but also the diversity of entities, modeling both relations and entities under different kinds of entity spaces and relation spaces. Bilinear models are also called semantic matching models, and the training targets of all these models are to encode only entity vectors while treating the relations as linear change matrices of the head entities corresponding to the tail entities. Rescal [12] defines relations as full-rank matrices and reflects deep interactions through matrix computation between entities and relations, but it is uncomplicated to overfit in the training process, which makes it difficult to model large-scale knowledge graphs. DistMult [13] utilizes diagonal matrices instead of full-rank matrices for modeling relation matrices to prevent the overfitting problem, and the disadvantage it brings is that it cannot model asymmetric relations. ComplEx [14] extends the vector space to the complex space on the basis of DistMult and uses conjugate complexes to solve the problem of not being capable of modeling asymmetric relations in previous models. Rotation-based models are also a top research direction. RotatE [21] is inspired by the Euler decomposition, which considers that any complex numbers can be obtained by rotating them in the complex space, and the mapping of relational vectors to entity vectors can be modeled by rotating them in the complex space, and then many rotation-based models follow, such as QuatE [22] and DualE [23].

2.2. Neural Network Models

With the emergence of neural network models, knowledge graph embedding can also be addressed using neural network architectures. ConvE [15] connects relations and entities by matrix reshaping and processes them through convolution, then proposes the training targets and training strategies for convolution-based neural networks, but its modeling of the interactions between entities and relations is superficial. To enhance the interaction information, InteractE [18] enhances the interactions by checked feature reshaping, circle padding, and larger kernel size, then uses depth-wise convolution to process each feature map separately. But it relies too much on interactions, which leads to the lack of explicit knowledge modeling capability, and its performance is inferior on small datasets. AcrE [19] models the interactions among distant elements in a certain extent through the use of atrous filters, while the size of its convolution kernel is constant, so it has good modeling ability for explicit knowledge. ConvKB [24] performs convolution operations on all vectors and extracts complex interactions by the convolution combination. ConvR [16] splits the relation vectors into convolution kernels to enhance the interactions. JointE [17] not only utilizes simple 1D convolution to enhance the learning of explicit knowledge but also utilizes internal 2D convolution for intra-feature interactions and parameter saving. Current research figures that the deep nonlinear operations, which design filters and feature maps to increase the interactions as much as possible in convolution-based models, affect the extraction of surface and explicit knowledge. However, whether the increased interactions are reasonable has not been deeply investigated in previous research.

To summarize, translation-based models are more sensitive to linear surface features and can capture surface knowledge information. However, they are powerless to capture deep nonlinear features. Convolution-based models are devoted to increasing the interactions as much as possible to achieve the purpose of improving the potential knowledge extraction capability, but the improved interactions are not necessarily reasonable. Thus, it is necessary to design a new structure that can take into account both linear feature fusion and interaction number control, based upon which we propose the IntME.

3. Model

In this section, we will describe the composition of IntME, an improved model that combines the advantages of deep nonlinear and shallow linear fusion. We present a table of score functions for some classical models and several convolution-based models, and then we present the specific structural composition and calculation mechanisms of our model.

3.1. Notations

There are plenty of triplets in the knowledge graphs, and we use $\left|E\right|$ to denote the entity set and $\left|R\right|$ to denote the relation set. The triplets use $(h,r,t)$, where h and t refer to the head entity and tail entity in the entity set $\left|E\right|$. The knowledge graph embedding embeds the triplets $(h,r,t)$ into the n-dimensional space, so that they become $(e_h,e_r,e_t)$, where $e_h,e_r,e_t\in {\mathbb{R}}^n$. The score function will be used as the training target to ensure the consistency of predicted missing links and the true triplets. We collect the score functions of some classical models and convolution-based network models and summarize them in

Table 1. The score functions $f_r(e_h,e_t)$ embedding parameters of some knowledge graph embedding models. The $\langle e_h,e_r,e_t\rangle$ defines the inner product of three vectors which is calculated by $\sum h_i{r}_i{t}_i$. ⊙ indicates Hadamard Product. $\widehat{e_t}$ indicates conjugate vector for $e_t\in {\mathbb{C}}^n$, and $\overline{e_h}$ indicates 2D reshaping from $e_h$. The ∗ indicates convolution operation and ⋆ indicates depth-wise circle convolution. $vec(•)$ indicates vectorizin a tensor and $ve{c}^{-1}(•)$ means the inverse operation. $e_h^{\prime }$ indicates the matrixing of $e_h$. f is a nonlinear function, while g is a sigmoid function. $\varphi$ is the checked feature reshaping function and $\phi$ is a reordering function. ${\omega }_S$ indicates standard convolution filters and ${\omega }_A$ indicates atrous convolution filters. ${\omega }_h$ and ${\omega }_r$ indicate internal convolution filters whose elements are based on $e_h$ and $e_r$. ${\omega }_L$ indicates the large kernel convolution filters while ${\omega }_P$ means the point-wise convolution filters. $[e_h;e_r]$ means the concatenation of $e_h$ and $e_r$. ${[•]}_{chanel}$ indicates the channel scaling reshaing. The $H,W,W_1,W_2$ are both learnable weights.

Model	Score Function ${\mathit{f}}_{\mathit{r}}({\mathit{e}}_{\mathit{h}},{\mathit{e}}_{\mathit{t}})$	Embedding Parameters
TransE [8]	$-\left|\right|e_h+e_r-e_t\left|\right|$	$e_h,e_r,e_t\in {\mathbb{R}}^n$
DistMult [13]	$\langle e_h,e_r,e_t\rangle$	$e_h,e_r,e_t\in {\mathbb{R}}^n$
ComplEx [14]	$Re\langle e_h,e_r,\widehat{e_t}\rangle$	$e_h,e_r,e_t\in {\mathbb{C}}^n$
RotatE [21]	$-\left|\right|e_h\odot e_r-e_t{\left|\right|}^2$	$e_h,e_r,e_t\in {\mathbb{C}}^n,\left|e_{r_i}\right|=1$
ConvE [15]	$f\left(vec\left(f([\overline{e_h},\overline{e_r}]\ast \omega )\right)W\right)e_t$	$e_h,e_r,e_t\in {\mathbb{R}}^n$
HyerER [25]	$f\left(vec(e_h\ast ve{c}^{-1}\left(e_{rH}\right))W\right)e_t$	$e_h,e_r,e_t\in {\mathbb{R}}^n$
InteractE [18]	$g\left(vec\left(f(\varphi \left({\mathbf{P}}_{\mathbf{k}}\right)★\omega )\right)W\right)e_t$	${\mathbf{P}}_{\mathbf{k}}\in {\mathbb{R}}^{n_1\times n_2},e_t\in {\mathbb{R}}^n$
AcrE [19]	$f(vec\left(f([\overline{e_h};\overline{e_r}]+[\overline{e_h};\overline{e_r}]\ast {\omega }_S\ast {\omega }_A)W\right)e_t$	$e_h,e_r,e_t\in {\mathbb{R}}^n$
JointE [17]	$f([e_h;e_r]\ast {\omega }_{1D}^1\ast {\omega }_{1D}^2+vec\left([\overline{e_h}\ast {\omega }_r;\overline{e_r}\ast {\omega }_h]\right)W)e_t$	$e_h,e_r,e_t\in {\mathbb{R}}^n$
IntME	$f(vec\left(f(\varphi \left({\mathbf{P}}_{\mathbf{k}}\right)★{\omega }_L)\right)W_1+vec\left(f({\left[\phi (\phi \left(e_r^{\prime T}\right)\times \phi \left(e_h^{\prime }\right))\right]}_{channel}\ast {\omega }_P)\right)W_2)e_t$	${\mathbf{P}}_{\mathbf{k}}\in {\mathbb{R}}^{n_1\times n_2},e_h^{\prime },e_r^{\prime }\in {\mathbb{R}}^{1\times n},e_t\in {\mathbb{R}}^n$

.

3.2. IntME

The entire architecture of IntME is shown in Figure 4. The improved model has two forward paths: Path 1 and Path 2, respectively, corresponding to the processing process of interaction number’s initial value and the difference between the initial value and reasonable interaction number. Given a triplet, we obtain $e_h,e_r,e_t$ by embedding and processing it and feeding it into the two paths.

Path 1. We first transform $e_h$, $e_r$ to ${\mathbf{P}}_{\mathbf{k}}\in {\mathbb{R}}^{n_1\times n_2}$ using checked feature reshaping, then we use $\varphi \left({\mathbf{P}}_{\mathbf{k}}\right)$ to obtain output $M_{P_k}\in {\mathbb{R}}^{b\times w\times q}$, where $w\times q=n_1\times n_2$:

$$M_{P_k}=\varphi \left({\mathbf{P}}_{\mathbf{k}}\right);$$

(1)

after that, we perform depth-wise circle convolution by using ${\omega }_L$ to obtain the output $M_{circle}$:

$$M_{circle}=M_{P_k}★{\omega }_L,$$

(2)

where the large kernel convolution filter ${\omega }_L\in {\mathbb{R}}^{k\times s\times s}$, and $M_{circle}\in {\mathbb{R}}^{k\times w\times q}$. ⋆ indicates the depth-wise convolution operation. The initial value of the interaction number is obtained in this step. The details of depth-wise circle convolution and checked feature reshaping are presented in [18]. Then, we obtain the hidden-layer output of Path 1 $e_1$ using an activation function, concatenating and flattening:

$$e_1=vec\left(f\left(M_{circle}\right)\right)W_1,$$

(3)

where the $e_1\in {\mathbb{R}}^n$ and $W_1\in {\mathbb{R}}^{u\times n},u=k\times w\times q$.

Path 2. We first regard $e_h,e_r$ as matrix $e_h^{\prime },e_r^{\prime }\in {\mathbb{R}}^{1\times n}$. Then, we use element reordering function $\phi$, which means elements of the matrix rows and columns are rearranged separately in the given order set, to obtain the product results $M_{cross}$ of $\phi \left(e_r^{\prime T}\right)$ and $\phi \left(e_h^{\prime }\right)$:

$$M_{cross}=\phi \left(e_r^{\prime T}\right)\times \phi \left(e_h^{\prime }\right),$$

(4)

where the $M_{cross}\in {\mathbb{R}}^{n\times n}$. What can be seen is that the expanded feature maps have many times more new elements than before, but they are all calculated from the entity and relation elements, so the new elements can be equally used for increasing interactions. Then, we employ ${[•]}_{channel}$ channel scaling reshaping, which transforms the original feature maps to the channel mode and keeps the total number of elements unchanged, and utilize point-wise convolution to obtain the output $M_{point}$:

$$M_{point}={\left[\phi \left(M_{cross}\right)\right]}_{channel}\ast {\omega }_P,$$

(5)

where ${\left[\phi \left(M_{cross}\right)\right]}_{channel}\in {\mathbb{R}}^{o\times k_1\times k_2}$, $o\times k_1\times k_2=n\times n$. The ∗ donates the convolution operation and ${\omega }_P\in {\mathbb{R}}^{m\times 1\times 1}$ is the point-wise convolution filter. And $M_{point}\in {\mathbb{R}}^{m\times k_1\times k_2}$. ${[•]}_{channel}$ is the key step to finding a reasonable number of interactions based on the initial value of the interaction number of Path 1, changing o and thus affecting the number of interactions in the feature maps after linear fusion. The value of o can represent the number of interactions in Path 2. Then, we use $W_2$ to calculate the Path 2’s hidden-layer output $e_2$:

$$e_2=vec\left(f\left(M_{point}\right)\right)W_2,$$

(6)

where the $W_2\in {\mathbb{R}}^{v\times n}$, $v=m\times k_1\times k_2$ and $e_2\in {\mathbb{R}}^n$.

Finally, we obtain the final output e through element-wise addition:

$$e=e_1+e_2.$$

(7)

Then, we use prediction e and tail entity $e_t$ to obtain the score vector $e_{score}$:

$$e_{score}=e{e}_t.$$

(8)

The details of our score function are shown in Table 1. We adopt GELU [26] as Path 2’s feature-map activation function in Equation (6), while others use RELU [27].

3.3. Traning Strategy

Once we obtain the score vector $e_{score}$, we will multiply it by the transpose matrix of embedding $M_{embed}^T$ to transfer the result back to the previous dimension, at which point the result is related to the training strategy:

$$\mathbf{s}=e_{score}{M}_{embed}^T,$$

(9)

where the $M_{embed}^T\in {\mathbb{R}}^{n\times \left|E\right|}$; then, we will use sigmoid function g to set $\mathbf{s}$ in the range of 0 to 1 to obtain our final prediction y:

$$y=g\left(\mathbf{s}\right).$$

(10)

In this case, all the elements in y represent the probability that the corresponding ordinal number is authentic.

Considering the training speed and other aspects, we choose 1-N and 1-X among 1-1, 1-N, 1-X, N-1, and N-N as our training strategies. When we choose 1-N, the prediction $y\in {\mathbb{R}}^{\left|E\right|}$, and ith element represents the probability that the triplet $(h,t,e_i)$ is real. When the 1-X strategy is chosen, it will select one positive sample and X negative samples, and if the total number of prediction samples is $X+1$, then the prediction $y\in {\mathbb{R}}^{X+1}$.

Based on the previous work from [15,17,18], we still use binary cross-entropy loss as the loss function:

$$L=\frac{1}{N}\sum _i(\left(\widehat{y_i}log\left(y_i\right)\right)+(1-\widehat{y_i})(1-log\left(y_i\right))),$$

(11)

where the $\widehat{y}$ is the correct triplet’s value on prediction y. And we use layer normalization [28] in Path 2’s feature-map layer, while others use batch normalization [29]. As an aspect of parameter training, we prefer to adopt Adam optimizer and label smoothing [30], and we utilize dropout [31] to prevent the trouble of over-fitting.

4. Experiments

In this section, we first introduce our evaluation indicator selection and datasets, while we will give the structure-specific configuration of our model. After that, we will show the results of IntME on four benchmark datasets. Finally, we set up experiments to demonstrate the superiority of our approaches and the advantages of the structural choices.

4.1. Evaluation Indicator Selection

We still use the general metrics MR, MRR, and H@$rk$ for the evaluation of the model performance. MR is the mean rank, which is the average of the ranking of the probability or the score of the correct tail entity in the prediction result, and a lower MR means a better prediction result. H@$rk$ is Hits@$rk$, which indicates the proportion of the real tail entity ranking in the top $rk$ positions in the prediction results and usually takes the values of 10, 3, and 1 corresponding to H@10, H@3, and H@1 to evaluate the model, and their higher values indicate a better model performance. MRR is mean reciprocal rank, which is the average of the inverse of the ranking of the probability or score of the correct tail entity in the prediction results. It differs from MR in that the higher its value is, the better the model performs.

Giving the set $E_{test}=\{x_1,x_2,x_3,...,x_{|E_{test}|}\}$, then MRR, MR and Hits@$rk$(H@$rk$) are calculated by Equations (12)–(14).

$$MRR=\frac{1}{|E_{test}|}\sum _{i=1}^{|E_{test}|}\frac{1}{ran{k}_i}$$

(12)

$$MR=\frac{1}{|E_{test}|}\sum _{i=1}^{|E_{test}|}ran{k}_i$$

(13)

$$Hits@rk=\frac{1}{|E_{test}|}\sum _{i=1}^{|E_{test}|}f\left(ran{k}_i\right),f\left(ran{k}_i\right)=\left\{\begin{array}{c}1,x\le rk\hfill \\ 0,x>rk\hfill \end{array}\right..$$

(14)

Given a triplet in the test set, we will filter the triplet that existed in the validation and training sets by applying the approach from [8], then we evaluate the model’s performance using the prediction results of the tail entities with the above metrics.

4.2. Datasets

To validate the performance and robustness of our improved model on different types of datasets, we select representative datasets to validate our IntME based on the graph size and the number of relation categories. FB15K [8] and WN18 [8] are two classical datasets upon which many previous excellent works have been evaluated, and FB15k-237 [32] and WN18RR [1], which are subsets of the above datasets, respectively, are more capable of validating the performance of the model, so we select FB15k-237 and WN18RR as representative of the medium-sized datasets. YAGO3-10, as a subset of YAGO [2], has a large number of entities and contains a large number of triplets, so we select YAGO3-10 as a representative of large datasets. Alyawarra Kinship [33] can be considered a representative of a micro dataset compared to the above three datasets, so we select the above four datasets for our experiments. The detailed dataset information is shown in

Table 2. The detailed information of the datasets.

Dataset	$\left|\mathit{E}\right|$	$\left|\mathit{R}\right|$	Train	Valid	Test
FB15k-237	14,541	237	272,115	17,535	20,446
WN18RR	40,943	11	86,835	3034	3134
YAGO3-10	123,182	37	1,079,040	5000	5000
Kinship	104	25	8544	1068	1074

.

The whole Freebase has 1.9 billion triplets, and FB15k-237 is its subset with nearly 15,000 entities and 237 relation types; as a subgraph of Freebase, it has the corresponding information of a father knowledge graph. WN18RR is a subgraph of WordNet with 40,943 entities and 11 types of relations. The size of FB15k-237 and WN18RR entities and the number of types of relations can well verify the expression ability of the model for explicit and latent knowledge. YAGO3-10 has 123,182 entities and 37 relations, and its size is very suitable for verifying the performance of the model under a large number of knowledge systems compared with the previous two datasets. Kinship has only 104 entities and 25 relations, which can verify the learning ability of our model for surface knowledge.

4.3. Experiment Setup

We uniformly set the batch size to 256 and train 500 epochs in our experimental setup. We set the embedding dimension of both entities and relations to 200 and the label smoothing rate to 0.1. We define a three-layer dropout to prevent training overfitting and enhance the robustness of the network. We use different values and learning rates for different benchmark datasets, which are shown in

Table 3. The model’s detail settings. @ indicates the current path’s convolution filter number. $ke{r}_{size}$ indicates the large kernel size. $in{p}_{drop}$ indicates input dropout. $fea{t}_{drop}$ indicates feature-map dropout. $hi{d}_{drop}$ indicates hidden-layer dropout.

Dataset	$@{\mathit{Path}}_1$	$@{\mathit{Path}}_2$	${\mathit{ker}}_{\mathit{size}}$	lr	${\mathit{inp}}_{\mathit{drop}}$	${\mathit{feat}}_{\mathit{drop}}$	${\mathit{hid}}_{\mathit{drop}}$
FB15k-237	96	64	$9\times 9$	0.0005	0.2	0.3	0.7
WN18RR	96	64	$9\times 9$	0.003	0.1	0.2	0.8
YAGO3-10	64	64	$7\times 7$	0.003	0.1	0.3	0.3
Kinship	96	80	$9\times 9$	0.0007	0.1	0.8	0.6

.

In WN18RR, the reshaped feature map’s shape of our model on Path 2 is $64\times 25\times 25$, and the other three datasets are $100\times 20\times 20$. The checked feature reshaping time is only 1.

4.4. Experiment Results

We first compare the results of several benchmark models with our IntME after training on FB15k-237 and WN18RR. The benchmark models are chosen from distance-based models (i.e., TransE [8], KMAE [34]), bilinear-based models (i.e., DistMult [13], ComplEx [14]), and some convolution-based networks (i.e., ConvE [15], InteractE [18], JointE [17], HypER [25]). It is clear from

Table 4. The experimental results of FB15k-237 and WN18RR. The best results are shown in bold. The results of InteractE are re-run by its open-source code and others are taken from [17].

	FB15k-237					WN18RR
	MRR	MR	H@10	H@3	H@1	MRR	MR	H@10	H@3	H@1
TransE [8]	0.287	226	0.475	0.325	0.192	0.193	5230	0.445	0.370	0.003
DistMult [13]	0.178	409	0.352	0.204	0.092	0.332	4107	0.456	0.380	0.260
ComplEx [14]	0.234	472	0.407	0.265	0.146	0.394	5365	0.461	0.419	0.353
KMAE [34]	0.326	235	0.502	0.358	0.240	0.448	4441	0.524	0.465	0.415
ConvE [15]	0.316	246	0.491	0.350	0.239	0.460	5277	0.480	0.430	0.390
HypER [25]	0.341	250	0.520	0.376	0.252	0.465	5798	0.522	0.477	0.436
InteractE [18]	0.353	182	0.539	0.387	0.262	0.469	5039	0.531	0.482	0.438
JointE [17]	0.356	177	0.543	0.393	0.262	0.471	4655	0.537	0.483	0.438
IntME	0.360	172	0.543	0.395	0.267	0.475	4055	0.545	0.495	0.436

that IntME outperforms all the benchmark models on FB15k-237 and WN18RR.

The performance of IntME on FB15k-237 far exceeds that of ConvE and HypER, with a considerable improvement compared to InteractE. The improvement is about 2% on MRR, about 0.8% on H@10, about 2.1% on H@3, and about 1.9% on H@1. Compared with JointE, which is a state-of-the-art convolution-based model, IntME also demonstrates its excellent performance.

From the results of WN18RR, IntME also shows an overwhelming superiority. Different from JointE, which is not as good as DistMult in MR, IntME also performs well in MR. While outperforming ConvE and HypER, the addition of Path 2 makes IntME outperform InteractE comprehensively, improving MRR from 0.469 to 0.475, MR from 5039 to 4055, and H@10 & H@3 by about 2.6% and 2.7%, respectively.

We re-run the open source code of InteractE on FB15k-237 and WN18RR and compare it with the improved model IntME. The best MRR-based performance variation on the validation set is shown in Figure 5 and Figure 6. As can be clearly seen in Figure 5, IntME takes the overall lead over InteractE’s performance after about the 30th epoch, and the gap is very obvious. And in Figure 6, InteractE performs checked feature reshaping four times, while IntME only uses one time. Moreover, InteractE uses a convolution kernel size of 11 for WN18RR in the open source code, while we only use 9, so we train more slowly on the first 80 epochs but, after 80 epochs, both H@10 and H@3 surpass InteractE, and after 150 epochs MRR also surpasses it.

Next, we utilize YAGO3-10 to validate the performance of our model for link prediction on large datasets, and we select several bilinear models, DistMult [13], ComplEx [14], and several benchmark convolution-based models such as ConvE [15], HypER [25], JointE [17], and InteractE [18] as our baselines. Experimental results are shown in

Table 5. The experiment alresults of YAGO3-10. The best results are shown in bold. The results of InteractE on YAGO3-10 are re-run by its open-source code and others are taken from [17].

	YAGO3-10
	MRR	H@10	H@3	H@1
DistMult [13]	0.340	0.540	0.380	0.240
ComplEx [14]	0.360	0.550	0.400	0.260
ConvE [15]	0.440	0.620	0.490	0.350
HypER [25]	0.533	0.678	0.580	0.455
InteractE [18]	0.552	0.688	0.598	0.476
JointE [17]	0.556	0.695	0.605	0.481
IntME	0.556	0.690	0.600	0.483

.

In YAGO3-10, our model still gives a good performance; we perform optimally on two metrics, MRR and H@1, with a performance that is not weaker than JointE and is stronger than the other baselines. Compared to InteractE, IntME increases MRR by 0.7%, H@10 by 0.2%, H@3 by 0.3%, and H@1 by 1.4%. Our model is fainter than JointE in H@10 and H@3 metrics, and we think that our model is not quite strong enough for feasible interactions. If we continue to try channel scaling reshaping, we think we can reach the level of JointE or even surpass it. Furthermore, benefitting from the powerful nonlinear fitting capability, the neural network is much more powerful than the linear model for latent knowledge processing.

Kinship, the smallest of the four datasets, is appropriate for evaluating the model’s ability to extract explicit knowledge. We pick several models, including ComplEx [14], ConvE [15], RotatE [21], HAKE [35], and InteractE [18], as baselines. When applying Path 2, IntME exhibits highly extraordinary results in the test of explicit knowledge recognition capability, and the results are displayed in

Table 6. The experiment results of Kinship. The best results are shown in bold. The results are taken from [19].

	Kinship
	MRR	H@10	H@3	H@1
ComplEx [14]	0.823	0.971	0.899	0.823
ConvE [15]	0.833	0.981	0.917	0.738
RotatE [15]	0.843	0.978	0.919	0.760
HAKE [35]	0.852	0.980	0.928	0.769
InteractE [18]	0.777	0.959	0.870	0.664
IntME	0.872	0.984	0.939	0.799

. InteractE is the worst-performing model in the results. IntME outperforms all other baselines in MRR, H@10, and H@3. ComplEx scores the highest marks in H@1, while IntME follows. Our model improves on the InteractE by 12.2% on MRR, 2.6% on H@10, 7.9% on H@3, and 20.3% on H@1, respectively.

In summary, IntME shows an excellent performance on all four datasets. The analysis of datasets of various sizes supports our expectations for what IntME can reach in terms of link prediction. Meanwhile, it is demonstrated that IntME can be the state-of-the-art convolution-based model in medium and large knowledge graphs.

4.5. Ablation Study

We design an ablation study to examine whether the individual modules of our model actually work.

To verify whether channel scaling reshaping is able to locate the interaction difference between reasonable interactions and the output features of Path 1 through channel element adjustment, we set up two sets of experiments on FB15k-237 and WN18RR for verification. To simplify the factorization in 200-dimensional embedding, we only consider the form of the square feature maps, i.e., $k_1$ = $k_2$ in Equation (5). Thus, we set up 3 sets of square feature groups of different shapes—$100\times 20\times 20$, $64\times 25\times 25$, and $25\times 40\times 40$, respectively. We implement it on FB15k-237 and WN18RR with the optimal hyper-parameters, and the results are shown in

Table 7. The experimental results of different feature map shapes gained by channel scaling reshaping on FB15k-237 and WN18RR. The best results are shown in bold.

	FB15k-237				WN18RR
	MRR	H@10	H@3	H@1	MRR	H@10	H@3	H@1
$25\times 40\times 40$	0.356	0.542	0.394	0.263	0.473	0.541	0.491	0.435
$64\times 25\times 25$	0.358	0.541	0.394	0.265	0.475	0.545	0.495	0.436
$100\times 20\times 20$	0.360	0.543	0.395	0.267	0.473	0.547	0.491	0.433

.

The results of FB15k-237 indicate that the three shapes perform differently, with $100\times 20\times 20$ exhibiting the best performance, indicating the benefit of improved interactions for IntME performance. Overall, in contrast to the results of FB15k-237, the best results of WN18RR are $64\times 25\times 25$, and this suggests that increasing interactions should be carried out reasonably rather than arbitrarily.

As a result, channel scaling reshaping can produce plausible interaction improvement without the requirement for extra parameters and can quantify the difference through the adjustment of channel element number.

In Section 3, we employ the $\phi$ function, which is an element reordering function. In brief, it reorders both the row and column elements of the matrix according to different permutations. We anticipate that it can enhance the possibilities of interaction between different elements. Therefore, we perform controlled trials to verify our anticipation on FB15k-237 and WN18RR. The structures of the models in the trials are all optimally chosen and the results are shown in

Table 8. The results of controlled trials on FB15k-237 and WN18RR. The best results are shown in bold.

	FB15k-237				WN18RR
	MRR	H@10	H@3	H@1	MRR	H@10	H@3	H@1
IntME-$withou{t}_{\phi }$	0.354	0.536	0.388	0.263	0.471	0.543	0.489	0.432
IntME-$wit{h}_{\phi }$	0.360	0.543	0.395	0.267	0.475	0.545	0.495	0.436

.

According to the results, when $\phi$ is not applied, the performance of IntME at FB15k-237 is essentially identical to that of InteractE, with little improvement, and it is not significant on WN18RR, either. In contrast to the case of $\phi$ not applying, the IntME results are highly significantly improved on FB15k-237, and a slight improvement on WN18RR, especially improving FB15k-237’s results from 0.354 to 0.360 on MRR, from 0.536 to 0.543 on H@10, from 0.388 to 0.395 on H@3, and from 0.263 to 0.267 on H@1.

In summary, element reordering can effectively enhance the interaction quality through the disordered element interactions.

To verify the necessity of using Path 1 and Path 2 together in IntME, we implement an ablation study on FB15k-237 and WN18RR. The result is displayed in

Table 9. The ablation study results on FB15k-237 and WN18RR. The best results are shown in bold. All models are with optimal hyper-parameters.

	FB15k-237				WN18RR
	MRR	H@10	H@3	H@1	MRR	H@10	H@3	H@1
IntME${}_{path1}$	0.353	0.540	0.387	0.262	0.469	0.531	0.482	0.438
IntME${}_{path2}$	0.353	0.537	0.389	0.260	0.449	0.532	0.478	0.397
IntME	0.360	0.543	0.395	0.267	0.475	0.545	0.495	0.436

.

We can see from the results that it is necessary to employ Path 1 and Path 2 together. Without Path 1, Path 2 loses some capability to extract some potential knowledge. The decrease is less significant in FB15k-237 but, in WN18RR, MRR decreases from 0.475 to 0.449, H@10 from 0.545 to 0.532, H@3 from 0.495 to 0.478, and H@1 from 0.436 to 0.397. Without Path 2, IntME lacks much of its capability to extract explicit knowledge.

In summary, IntME is only complete if both Path 1, which represents the initial value of the interactions, and Path 2, with explicit knowledge information and the difference between more reasonable interactions and initial value, are used.

4.6. Training Cost

Our model has the advantages of not only robust generalization ability but also low training cost. InteractE, as a state-of-the-art convolution-based model using external common filters, and JointE, which utilizes internal alternate filters and is also a state-of-the-art model, are well suited as baselines for a training cost comparison. To confirm the consistency of comparison results, we set a 256 batch size and we choose the 1-N strategy in FB15k-237, WN18RR, and YAGO3-10. There is no open source code for JointE, so its code is rewritten in our environment according to [17].

As shown in Figure 7, our model has a minor improvement of training cost on FB15k-237 but brings a more powerful performance than InteractE, while it takes even less training cost than InteractE with optimal hyper-parameters on WN18RR and YAGO3-10. It benefits from the shallow linear operations in Path 2 and single feature map fusion in Path 1 and the complementary feature extraction capability of both Path 1 and Path 2. InteractE employs depth-wise circle convolution, which causes much cost in I/O operations instead of in calculation processes, and internal alternate convolution filters in JointE even cost several or tens of times the overhead of the above models in I/O, so that the time cost in an epoch would be much more.

MFAE [36] also has the advantage of low training cost, but our model can achieve a better performance than it with a lower training cost, and the comparison is displayed in

Table 10. The performance comparison between MFAE and IntME. The best results are shown in bold. The results of MFAE are taken from [36].

	FB15k-237				WN18RR
	MRR	H@10	H@3	H@1	MRR	H@10	H@3	H@1
MFAE	0.355	0.540	0.390	0.263	0.467	0.530	0.482	0.437
IntME	0.360	0.543	0.395	0.267	0.475	0.545	0.495	0.436

, all of which again prove the superiority of our model structure.

5. Conclusions

A concise and effective model for locating reasonable interactions, IntME is proposed in this paper. Current research does not delve into the necessity of reasonable interactions, and as the scale of knowledge systems enlarges, the roles of interactions in knowledge extraction capabilities become increasingly evident. Blindly increasing interactions does not better stimulate knowledge extraction; the more reasonable the increased interactions are, the stronger the knowledge extraction ability will be. IntME uses the idea of splitting reasonable interactions, using two paths to learn the initial value and difference separately, which greatly reduces the fitting difficulty. Also, using element reordering and feature reshaping, instead of internal filters, further accomplishes enhanced interactions, which maximizes the training cost reduction. Furthermore, IntME can achieve a state-of-the-art performance on four benchmark datasets compared to other convolution-based models, which represents the excellence of our model’s generalization ability and the superiority of our methods. Exploring the mechanisms of reasonable interactions may be a subsequent direction in which to improve the knowledge extraction capability.

In future work, we will focus our research on the combined model of vector embedding and graph embedding in low-dimensional space, using the information of proximity nodes to conform to the vector representation of the modeling triplets. Also, we will further research interaction effects on the knowledge representation capability to enhance the performance of the new model.