MM-Transformer: A Transformer-Based Knowledge Graph Link Prediction Model That Fuses Multimodal Features

Abstract

Multimodal knowledge graph completion necessitates the integration of information from multiple modalities (such as images and text) into the structural representation of entities to improve link prediction. However, most existing studies have overlooked the interaction between different modalities and the symmetry in the modal fusion process. To address this issue, this paper proposed a Transformer-based knowledge graph link prediction model (MM-Transformer) that fuses multimodal features. Different modal encoders are employed to extract structural, visual, and textual features, and symmetrical hybrid key-value calculations are performed on features from different modalities based on the Transformer architecture. The similarities of textual tags to structural tags and visual tags are calculated and aggregated, respectively, and multimodal entity representations are modeled and optimized to reduce the heterogeneity of the representations. The experimental results show that compared with the current multimodal SOTA method, MKGformer, MM-Transformer improves the Hits@1 and Hits@10 evaluation indicators by 1.17% and 1.39%, respectively, proving that the proposed method can effectively solve the problem of multimodal feature fusion in the knowledge graph link prediction task.

1. Introduction

In recent years, knowledge graphs have played a vital role in real-world tasks such as question answering systems [1,2,3], recommender systems [4,5], and information retrieval [6,7]. However, due to the issue of missing triples, knowledge graphs cannot encompass all knowledge. To address this problem, link prediction techniques have been introduced to predict possible missing relationship triplets [8] by using the symmetry between the complete knowledge graph and the real knowledge graph. Traditional link prediction methods, such as translation-based approaches [9] and neural network methods [10,11], often fall short in accurately capturing and processing the relationships between triples due to limitations in model design or the training process. Recently, some studies [12,13,14] have tackled this issue by enriching datasets and proposing new models to capture multimodal information. However, these methods typically project all modalities into a unified space, failing to effectively capture the complex interactions between modalities, thus limiting their performance. As a result, the multimodal link prediction task has been proposed to overcome these limitations.

Taking Figure 1 as an example, Stephen Curry and LeBron James were both born in Akron, Ohio. This similarity may lead to incorrect graph structure predictions, such as predicting that LeBron James is a player for the Golden State Warriors. Outdated predictions may also arise based on the text description “played for”. Visual features alone can only determine that LeBron James is a basketball player from his clothing but cannot determine which team he currently belongs to. However, by integrating multimodal knowledge such as geographic location and team information, it can accurately predict that LeBron James is a member of the Los Angeles Lakers. Inspired by multimodal representation learning [15], this paper fully leverages the graph structure, visual, and textual features of the knowledge graph and performs multimodal feature fusion based on the Transformer architecture to achieve link prediction on the knowledge graph. The contributions of this paper are summarized as follows:

(1)

This paper proposes a method for fusing multimodal features, which makes full use of structural, visual, and textual features, extracts these features through a specific modality encoder, and uses a Transformer for multimodal feature fusion. This method can effectively reduce the heterogeneity of multimodal entity representations.

(2)

By fusing feature information from different modalities at each layer, we can more comprehensively understand and represent entities and relationships in multimodal knowledge graphs and perform better in capturing the complex interactions of multimodal features.

(3)

Through case analysis, multimodal feature fusion effectively reduces the bias that may be caused by single modal features. By analyzing the contribution of each feature to the final result, the interpretability and credibility of the model are enhanced.

2. Related Work

To achieve multimodal link prediction, researchers have proposed a series of multimodal pre-training methods and pre-trained them on a corpus of image-text pairs, such as ViLBERT [16], VisualBERT [17], and UNITER [18], which effectively improve the processing ability of multimodal information. In addition, Xie et al. [14] proposed to integrate image features into a typical knowledge graph representation learning model for multimodal link prediction. On the other hand, Sergieh et al. [13] and Wang et al. [19] jointly encoded and fused visual and structural knowledge through simple concatenation and using an autoencoder for multimodal link prediction, respectively. Existing multimodal link prediction methods mainly focus on encoding image features into knowledge graph embeddings. Xie et al. [14] extended TransE to obtain visual representations corresponding to knowledge graph entities and knowledge graph structural information, respectively. Sergieh et al. [13], Wang et al. [19], and Zhao et al. [20] further proposed several fusion strategies to encode visual and structural features into a unified embedding space. Recently, Wang et al. [21] studied the noise introduced by irrelevant images corresponding to entities and designed a forget gate with the MRP metric to select valuable images for multimodal knowledge graph completion.

In order to improve the performance of multimodal feature fusion, researchers have proposed a series of innovative methods. Shankar et al. [22] proposed a progressive fusion method that establishes connections between different layers of the model so that the information of the deep fusion can be used by the shallow layer, avoiding information loss while maintaining the advantages of late fusion. Liang et al. [23] proposed an information theory-based method to define the upper and lower bounds of multimodal interaction and showed how these bounds accurately reflect the true degree of interaction. Finally, they explained how these theoretical results can help evaluate model performance, guide data collection, and select appropriate multimodal models. Jiang et al. [24] proposed a new method called HyperRep, which uses hypergraphs to capture complex high-order relationships between different data types and combines the information bottleneck principle to improve the fusion effect of multimodal data. Golovanevsky et al. [25] proposed a new one-to-many (OvO) attention mechanism, which achieves linear expansion with the increase in the number of modalities and reduces computational complexity. Zhang et al. [26] proposed a method called MLA, which reduces the interference between different modalities by alternately learning a single modality and captures the interaction between different modalities by sharing parts, thereby improving the effect of multimodal representation learning. The experimental results on different tasks demonstrate the effectiveness and versatility of these methods.

However, although previous research on knowledge graph link prediction based on multimodal features has made some progress, it still has obvious limitations such as architectural universality and modal contradictions. It is necessary to propose a unified model to more effectively expand the application of multimodal knowledge graph completion and solve the contradictions in modal fusion.

3. Methodology

Formally, the knowledge graph [27] is defined as $\mathcal{G}=<ℇ,\mathcal{R},\mathcal{T}>$, where $ℇ$ represents the set of entities, $\mathcal{R}$ represents the set of relations, and $\mathcal{T}=\left\{\right(h, r, t\left)\right|h, t\in ℇ, r\in \mathcal{R}\}$ represents the relation triple of the knowledge graph. There are many kinds of relationships in the world. For example, a “marriage relationship” is symmetrical, while a “parent-child relationship” is asymmetrical; “hypernym and hyponym” is a reversal relationship; and “father’s father is grandfather” is transitive. The key is to find ways to model and infer these patterns from observed facts, including symmetry, antisymmetry, and reversal and transitive relationships, so as to predict the missing links. In the knowledge graph based on multimodal features, each entity is represented not only by text features, but also by structural features and visual features. Therefore, this paper defines the modal feature set of the entity $\mathcal{M}=\{s, v, t\}$, where s, v, and t represent the structural modal features, visual modal features, and textual structural features of the entity, respectively.

Multimodal link prediction is one of the knowledge graph completion tasks. It is to predict the tail entity given the entity and relationship, expressed as ($e_h, r, ?)$. For a given entity $e_h$, according to the graph structure $S_h$ and image $I_h$ related to entity $e_h$, this paper models the distribution of the tail entity $e_t$, expressed as $p\left(e_t\right|\left(e_h, r, \left(S_h, I_h\right)\right))$. First, feature extraction is performed on the structure, vision, and text; then, the entity representation of structure–text–image fusion is modeled, and finally the missing entity is predicted based on the multimodal entity representation.

3.1. Overall Architecture

As shown in Figure 2, the model MM-Transformer in this paper uses the basic Transformer model to model the entity representation of structure–text–image fusion. A graph neural network model, GraphSAGE (Graph Sample and Aggregation), is used at the bottom layer to extract the structural features of the knowledge graph, and the pre-trained models ViT (Vision Transformer) and BERT (Bidirectional Encoder Representations from Transformers) are used to extract visual and text features. In the upper layer, an improved prefix-guided interaction module and a relevance-aware fusion module [28] are used to model the multimodal entity representation, and then link prediction is performed based on the multimodal entity representation.

3.2. Structural Feature Extraction

A knowledge graph is a graph-based data structure consisting of nodes and edges. Each node represents an “entity”, and each edge is a “relationship” between entities. A knowledge graph is essentially a semantic network. For example, if there is a relationship triple (entity 1, relationship, entity 2), then there will be a directed edge from entity 1 to entity 2 in the graph, indicating that there is a specified relationship between entity 1 and entity 2. Such a graph can capture the relationship and connection between entities well. Through the graph neural network model, we can learn the representation of nodes in the graph and calculate the importance of each node. For example, if a user has many friends, he may be more important in the social network. This can be achieved through the self-attention mechanism; that is, dynamically adjusting the weight according to the relationship between the user and his neighbor nodes, thereby reflecting the importance of the user in the network. In addition to the importance of nodes, we can also calculate the mutual relationship between nodes. For example, whether there is a direct friend relationship between two users, or whether there is an indirect association relationship between them (through common friends, etc.). This can be achieved by calculating the attention weights between nodes, so that the nodes are weighted and aggregated to reflect their mutual relationship in the network.

Assume that the nodes in the knowledge graph are represented as {$v_1,v_2,\cdots ,v_k$} and the edges are represented as {($v_i,v_j$)}. GraphSAGE [29] is used as the structural encoder to calculate the structural features of each node $v_i$. For example, for a node representing a person, its structural features can include its position information in the graph. GraphSAGE is a graph neural network model for learning node representation in graph structures. It adopts a sampling and aggregation strategy to update the node representation by aggregating the node’s neighbor information. Each node can be represented as a vector of fixed dimension, which contains the structural information of the node and its position relationship in the graph. Specifically, the feature representation obtained using GraphSAGE refers to the abstract representation of each node in the graph structure. These representation vectors capture the node’s neighbor information, which can help this paper understand the relationship between nodes, discover hidden patterns and rules, and capture the relationship and connection between nodes in the graph.

Let $x_i$ represent the hidden state of node $v_i$ in the graph neural network model. The following formula can be used to calculate the hidden state $x_i$ of node $v_i$:

$$x_i=\sigma \left(AGG\right(\left\{h_j^{(l-1)}|\forall j\in \mathcal{N}(v_i)\right\})·W^{\left(l\right)}),$$

(1)

where $\sigma$ is the activation function, AGG is the aggregation function, $h_j^{(l-1)}$ is the hidden state of node $v_j$ in the previous layer $l-1$, $\mathcal{N}\left(v_i\right)$ represents the set of neighbor nodes of node $v_i$, and $W^{\left(l\right)}$ is the weight matrix. GraphSAGE gradually updates the representation of nodes through multiple layers of aggregation operations to obtain the final structural feature output $X_S{\in \mathbb{R}}^{k\times d_S}$. In each layer of aggregation operation, the neighbor information of the node will be aggregated and weighted to finally obtain a new table for each node.

3.3. Visual Feature Extraction

This paper adopts ViT [30] as a visual encoder to extract image features. The image representation process involves dividing the image into fixed-size blocks and rearranging the pixel values of each block into a vector form, which is provided as input to the ViT model. These block vectors are regarded as part of the input sequence and represent the local area in the image. The ViT model then transforms each block vector into a low-dimensional vector representation to capture the visual features of the image. This transformation is implemented by a fully connected layer that maps the pixel values of the block vector into a smaller feature space.

Specifically, the image is divided into e blocks of a fixed size. The original pixel value of a given block is represented as $Z\in {\mathbb{R}}^{H\times W\times C}$, where $H$, $W$, and $C$ represent the height, width, and number of channels of the image, respectively. Then, the original pixel values are rearranged into a one-dimensional vector $z\in {\mathbb{R}}^{N\times C}$, where $N=H\times W$ represents the total number of pixels in the block. Next, a linear transformation is applied to map the block vector $z$ into a low-dimensional feature space to obtain the embedded block vector $x\in {\mathbb{R}}^{N\times d_V}$. This linear transformation is expressed as follows:

$$x=z\cdot W+b,$$

(2)

where $W$ is the weight matrix and $b$ is the bias vector. In this paper, the tiles of $a$ images are embedded and concatenated to obtain the visual feature output $X_V\in {\mathbb{R}}^{m\times d_V}$, where $m=(e\times a)$.

3.4. Text Feature Extraction

This paper uses BERT [31] as a text encoder to extract semantic features of text, including word-level and sentence-level semantic representations, as well as context-aware and multi-level semantic information. The text is fed into the BERT model for processing, and each input text is converted into a sequence of words (tokens), represented as $X=\{x_1, x_2, \cdots , x_n\}$, where $x_i$ represents the $i$-th word. For each input word $x_i$, an embedding operation is performed to convert it into a vector representation. A word embedding matrix $E$ is used, in which each row corresponds to the embedding vector of a word. The word embedding operation is expressed as follows:

$$Embedding\left(x_i\right)=E_{x_i},$$

(3)

where $E_{x_i}$ represents the embedding vector corresponding to word $x_i$. After the word embedding operation, the text feature output $X_T\in {\mathbb{R}}^{n\times d_T}$ is obtained.

3.5. Multimodal Feature Fusion

Existing multimodal feature fusion methods mainly focus on projecting the representations of different modalities into a unified space and exploiting the commonalities between modalities for prediction, but this approach may fail to preserve modality-specific knowledge. This paper alleviates this problem by fusing features from different modalities at multiple layers, thereby further leveraging their complementarity.

This paper uses the first $L_S$ layer of GraphSAGE as a structural encoder to obtain the basic graph structure information of the knowledge graph, that is, the structural feature output $X_S$. $X_S$ is used as the input of multi-head self-attention (MHA), and the structural representation is calculated as follows:

$$X_0^S=X_S+S_{pos},$$

$$\overline{X_l^S}=MHA\left(LN\left(X_0^S\right)\right)+X_{l-1}^S,l=1\cdots L_S,$$

(4)

$$X_l^S=FFN\left(LN\left(\overline{X_l^S}\right)\right)+\overline{X_l^S},l=1\cdots L_S,$$

where $S_{pos}{\in \mathbb{R}}^{k\times d_S}$ represents the corresponding position embedding and $X_l^S$ is the hidden state of the $l$-th layer of the structure encoder.

This paper uses the first $L_V$ layer of ViT as a visual encoder to obtain the basic feature information of the image, that is, the visual feature output $X_V$. $X_V$ is used as the input of MHA, and the visual representation is calculated as follows:

$$X_0^V=X_V+V_{pos},$$

$$\overline{X_l^V}=MHA\left(LN\left(X_0^V\right)\right)+X_{l-1}^V,l=1\cdots L_V,$$

(5)

$$X_l^V=FFN\left(LN\left(\overline{X_l^V}\right)\right)+\overline{X_l^V},l=1\cdots L_V,$$

where $V_{pos}{\in \mathbb{R}}^{m\times d_S}$ represents the corresponding position embedding and $X_l^V$ is the hidden state of the $l$-th layer of the structure encoder.

This paper uses the first $L_T$ layer of BERT as a text encoder to obtain the basic feature information of the text, that is, the text feature output $X_T$. $X_T$ is used as the input of MHA, and the text representation is calculated as follows:

$$X_0^T=X_T+T_{pos},$$

$$\overline{X_l^T}=MHA\left(LN\left(X_0^T\right)\right)+X_{l-1}^T,l=1\cdots L_T,$$

(6)

$$X_l^T=FFN\left(LN\left(\overline{X_l^T}\right)\right)+\overline{X_l^T},l=1\cdots L_T,$$

where $T_{pos}{\in \mathbb{R}}^{m\times d_S}$ represents the corresponding position embedding and $X_l^T$ is the hidden state of the $l$-th layer of the structure encoder.

In the MHA of the Transformer [32] model, the input consists of query (Q), key (K), and value (V) vectors, obtained by three linear transformations. Therefore, this paper performs multi-head attention calculations on mixed keys and values in each layer of the attention layer to control the degree of interaction between structural features and visual features and text features to reduce modal heterogeneity in advance. The structural head ${head}_i^S$, visual head ${head}_i^V$, and text head ${head}_i^T$ in the original ${head}_i=Attention(Q^i,K^i,V^i)$ are redefined, and the calculation process is as follows:

$$Q^i, K^i, V^i=x{W}_q^i, x{W}_k^i, x{W}_v^i,$$

$${head}_i^T=Attention(x^t{W}_q^t, x^t{W}_k^t, x^t{W}_v^t),$$

$${head}_i^S=Attention(x^s{W}_q^s, \left[x^s{W}_k^s, x^t{W}_k^t\right], \left[x^s{W}_v^s, x^t{W}_v^t]\right),$$

(7)

$${head}_i^V=Attention(x^v{W}_q^v, \left[x^v{W}_k^v, x^t{W}_k^t\right], \left[x^v{W}_v^v, x^t{W}_v^t]\right),$$

This paper also derives a variant formula of Equation (7):

$$\begin{array}{ll}{head}_i^m& =Attention(x^m{W}_q^m, \left[x^m{W}_k^m, x^t{W}_k^t\right], \left[x^m{W}_v^m, x^t{W}_v^t]\right)\\ & =softmax\left(Q_m{\left[K_m\right.;\left.K_t\right]}^T\right)\left[\begin{array}{c}{V}_m\\ V_t\end{array}\right]\\ & =\left(1-\lambda \left(x^m\right)\right)softmax\left(Q_m{K}_m^T\right)V_m+\lambda \left(x^m\right)softmax\left(Q_m{K}_t^T\right)V_t\\ & =\left(1-\lambda \left(x^m\right)\right)\underset{standard attention}{\underbrace{Attention\left(Q_m, K_m, V_m\right)}}+\lambda \left(x^m\right)\underset{cross-modal interaction}{\underbrace{Attention\left(Q_m, K_t, V_t\right)}}\end{array}$$

(8)

$$\lambda \left(x^m\right)={\displaystyle \frac{{\sum }_i{exp\left(Q_m{K}_t^T\right)}_i}{{\sum }_i{exp\left(Q_m{K}_t^T\right)}_i+{\sum }_j{exp\left(Q_m{K}_m^T\right)}_j}},$$

(9)

where $x^m$ represents the input of modality $m$, $m\in \{s, v\}$. $W_q^m$, $W_k^m$, and $W_v^m$ represent the weight matrices of the query, key, and value of modality m, respectively. $\lambda \left(x^m\right)$ represents the hybrid parameter of the normalized attention weight, which is obtained by dividing the sum of the cross-modal attention score by the sum of the standard attention score and the cross-modal attention score. It can be dynamically adjusted according to the input features to ensure the best balance between standard attention and cross-modal interaction. $\lambda \left(x^s\right)$ is used to control the degree of interaction between structural features and text features, and $\lambda \left(x^v\right)$ is used to control the degree of interaction between visual features and text features.

In order to mitigate the negative impact of irrelevant structures and images on text, this paper conducts cross-modal interaction based on tags to achieve alignment between structure, text, and image. This paper uses k, m, and n to represent the sequence lengths of the structural vector $x^s\in {\mathbb{R}}^{k\times d}$, the visual vector $x^v\in {\mathbb{R}}^{m\times d}$, and the text vector $x^t\in {\mathbb{R}}^{n\times d}$, which are the corresponding output features of multimodal features after MHA and fully connected feedforward network (FFN). For text tags, this paper calculates the similarity matrix with all structural tags and visual tags, respectively, and then performs softmax function processing, and performs average tag aggregation on the structural tags and visual tags. The specific process is as follows:

$$S=x^t{\left(x^s\right)}^T,S=x^t{\left(x^v\right)}^T,$$

(10)

$${Agg}_i\left(x^s\right)=softmax\left(S_i\right)x^s,(1\le i\le n),$$

$$Agg\left(x^s\right)=[{Agg}_1\left(x^s\right), \cdots ,{Agg}_n\left(x^s\right)],$$

(11)

$${Agg}_i\left(x^v\right)=softmax\left(S_i\right)x^v,(1\le i\le n),$$

$$Agg\left(x^v\right)=[{Agg}_1\left(x^v\right), \cdots ,{Agg}_n\left(x^v\right)],$$

(12)

where ${Agg}_i\left(x^s\right)$ represents the structural representation of the similarity-aware aggregation of the $i$-th text token, and ${Agg}_i\left(x^v\right)$ represents the visual representation of the similarity-aware aggregation of the $i$-th text token. The structural and visual hidden states of the similarity-aware aggregation are then merged into the text hidden state in the FFN layer, calculated as follows:

$$FFN\left(x^t\right)=ReLU\left(x^t{W}_1+b_1+Agg\left(x^s\right)W_3+Agg\left(x^v\right)W_4\right)W_2+b_2$$

(13)

where $W_1\in {\mathbb{R}}^{d\times d_n}$, $W_2\in {\mathbb{R}}^{d_n\times d}$, and $W_3{\in \mathbb{R}}^{d\times d_k}$ represent the newly added parameters of the structural hidden states for aggregation, and $W_4{\in \mathbb{R}}^{d\times d_m}$ represents the newly added parameters of the visual hidden states for aggregation.

4. Experiments

4.1. Experimental Setup

4.1.1. Datasets

This paper uses two publicly available multimodal link prediction datasets, namely, WN18-IMG: this dataset is an extended version of the knowledge graph WN18 [8] extracted from WordNet [33]. Different from the original WN18, each entity in WN18-IMG is accompanied by 10 images. This multimodal dataset provides richer information to support the link prediction task. FB15K-237-IMG: This dataset is a subset of Freebase [34], which is a large-scale knowledge graph that provides rich multimodal information to help the model understand the relationship between entities. Compared with the regular FB15K-237, each entity in FB15K-237-IMG [8] also has 10 images. Detailed statistics of these datasets can be found in

Table 1. Dataset statistics for multimodal link prediction.

Dataset	Ent	Rel	Train	Dev	Test
FB15K-237-IMG	14,541	237	272,115	17,535	20,466
WN18-IMG	40,943	18	141,442	5000	5000

.

During the evaluation process, we considered four effective entity metrics to measure the performance of the model which follow previous studies: (1) mean rank (MR) and (2) hit rate (Hits@1, Hits@3, and Hits@10).

4.1.2. Baselines

This paper compares several baseline models to fully demonstrate the superiority of this model. The baselines selected in this paper include the following:

• VisualBERT [17], a pre-trained vision–language model with a single-stream structure.
• ViLBERT [16], a pre-trained vision-language model with a two-stream structure.
• IKRL [14], which extends TransE to learn the visual representation of entities and the structural information of knowledge graphs respectively.
• TransAE [19], combines a multimodal autoencoder with TransE to encode visual and texture knowledge into a unified representation and uses the hidden layers of the autoencoder as the representation of entities in the TransE model.
• RSME [21], which designs a forget gate with MRP metric to select valuable images for multimodal knowledge graph embedding learning.
• MKGformer [28], a proposed hybrid Transformer model with multi-layer fusion to integrate visual and textual representations.

4.1.3. Experiment Details

This paper adopts a training method based on the Lightning framework. In order to ensure the reproducibility of the experimental results, this paper sets the random seed to 42. Since the task of this study is based on the knowledge graph, we chose the corresponding data class. For regularization, this paper chooses a smaller dropout, set to 0.1. During the training process, we used a batch size of 96, set the gradient accumulation step to 1, and set the number of working threads to 4, the learning rate to 4 × 10⁻⁵, the maximum sequence length to 64, and the validation batch size to 96. In order to improve the generalization ability of the model, this paper adopts the label smoothing technique and sets the label smoothing parameter to 0.3. We chose the pre-trained models ViT and BERT and used the binary cross entropy loss function (BCE) and the Adam optimizer. The computing equipment used in the experiment was configured as follows: the processor was Intel Core i5-12400F; the GPU was NVIDIA RTX 4070; the memory was 32 GB RAM; the operating system was Windows 10; and the deep learning framework was PyTorch 1.12.1. With this configuration, our model took about 41 h and 26 h and 30 min in the pre-training and training phases, respectively, for a total of about 67 h and 30 min.

5. Experimental Results

5.1. Overall Performance

The experimental results in

Table 2. Results of the link prediction on FB15K-237-IMG and WN18-IMG. The arrows (↑ and ↓) next to each metric in the table indicate whether performance increases or decreases when we move from one model to another or from one dataset to another. ↑ indicates an improvement or increase in performance (higher values are better), and ↓ indicates an improvement or decrease in average ranking (lower values are better).

Model	FB15K-237-IMG				WN18-IMG
	Hits@1 ↑	Hits@3 ↑	Hits@10 ↑	MR ↓	Hits@1 ↑	Hits@3 ↑	Hits@10 ↑	MR ↓
VisualBERT_base [17]	0.217	0.324	0.439	592	0.179	0.437	0.654	122
ViLBERT_base [16]	0.233	0.335	0.457	483	0.223	0.552	0.761	131
IKRL [14]	0.194	0.284	0.458	298	0.127	0.796	0.928	596
TransAE [19]	0.199	0.317	0.463	431	0.323	0.835	0.934	352
RSME [21]	0.242	0.344	0.467	417	0.943	0.951	0.957	223
MKGformer [28]	0.256	0.367	0.504	221	0.944	0.961	0.972	28
MM-Transformer	0.259	0.362	0.511	215	0.948	0.968	0.976	117

show that it is necessary to effectively integrate structural features, visual features, and text features to improve the link prediction performance of the knowledge graph. This shows that the model proposed in this paper is superior, especially in capturing multimodal information.

First, judging from the results on the FB15K-237-IMG dataset, MM-Transformer performs significantly better than other methods. Compared with the current multimodal SOTA method MKGformer, MM-Transformer improves the Hits@1 and Hits@10 evaluation indicators by 1.17% and 1.39%, respectively. In addition, MM-Transformer also performs better on the MR (Mean Rank) indicator, which is reduced to 215. Secondly, the performance on the WN18-IMG dataset is also outstanding. MM-Transformer’s scores on Hits@1, Hits@3, and Hits@10 are 0.948, 0.968, and 0.976, respectively, which are higher than RSME and other comparison methods. This further proves the universality and robustness of the model proposed in this paper on different datasets. These results show that MM-Transformer can more effectively integrate multimodal features, thereby improving the accuracy of link prediction.

Compared with multimodal methods that process different modal features separately, MM-Transformer combines multimodal features for joint learning, which helps to simultaneously consider the common characteristics and complementary relationships between different modal features. The experimental results clearly show that MM-Transformer has significant advantages in capturing and fusing multimodal features. This not only verifies the effectiveness of the model design, but also provides strong support for the use of multimodal fusion methods in knowledge graph link prediction tasks in the future.

5.2. Ablation Study

Table 3. Evaluation results with different modality feature combinations on FB15K-237-IMG. The arrows (↑ and ↓) next to each metric in the table indicate whether performance increases or decreases when we move from one model to another or from one dataset to another. ↑ indicates an improvement or increase in performance (higher values are better), and ↓ indicates an improvement or decrease in average ranking (lower values are better).

FB15K-237-IMG
	Hits@1 ↑	Hits@3 ↑	Hits@10 ↑	MR ↓
T	0.241	0.345	0.457	248
S + T	0.242	0.351	0.386	232
V + T	0.256	0.367	0.504	221
S + V + T	0.259	0.362	0.511	215

presents the evaluation results of fusing different modal features for the FB15K-237-IMG dataset. Here, this paper uses S to represent structural features, V to represent visual features of images, and T to represent text features of descriptions. It can be observed that when structural, visual, or text features are introduced, the performance of the model is significantly improved. This shows the effectiveness of multimodal feature fusion.

As can be seen from Table 3, when text features (T) are used alone, the performance of the model is relatively low. This shows that when only text features are used for link prediction, the accuracy and ranking performance of the model are limited. When structural features (S) are added, the model performance is significantly improved. This shows that structural features play an important role in providing basic information about entities and relationships, which helps to improve the link prediction performance of the model. Similarly, when visual features (V) are added, the model performance is also significantly improved. This shows that the advantage of visual features in capturing image detail information enables the model to utilize more useful information in link prediction, thereby improving performance. Most notably, when structural features, visual features, and text features (S + V + T) are fused simultaneously, the model achieves the best performance. This shows that by fusing multimodal features, the model can make full use of the complementary information between different modal features, thereby significantly improving the accuracy and effect of link prediction.

Overall, the experimental results in Table 3 fully demonstrate the effectiveness of multimodal feature fusion. Structural features provide basic information about entities and relationships, visual features capture details in images, and text features provide semantic understanding. By jointly learning these features, the model can more comprehensively capture the interaction between different modal features, reflecting both commonalities and complementarity, thereby significantly improving the link prediction performance of the knowledge graph.

5.3. Visual Analysis

To further verify the effectiveness of multimodal feature fusion, this paper selects a case to visualize the prediction scores of each modal feature, as well as the prediction scores, after multimodal feature fusion. As shown in Figure 3, by comparing the prediction results of different modal features, we can more intuitively understand the advantages of multimodal fusion. When using structural features for prediction, the system obtains the highest score for “LeBron James plays for the Golden State Warriors”. This is because structural features mainly rely on the existing entity and relationship information in the knowledge graph. If the structural information in the knowledge graph is influenced by outdated data or other factors, it may cause deviations in the prediction results. When using text features for prediction, the system obtains the highest score for “LeBron James plays for the Cleveland Cavaliers”. Text features mainly rely on descriptive text information and may be affected by the source and context of the text. If the text information is outdated or contains misleading descriptions, the prediction results are also prone to errors. When using visual features for prediction, the system obtains the highest score for “LeBron James plays for the Los Angeles Lakers”. Visual features rely on image information. In this case, by identifying relevant images of LeBron James wearing a Los Angeles Lakers jersey, the system can make more accurate predictions. However, if the image data are not comprehensive or are noisy, the prediction results may be affected.

After fusing the features of different modalities, the system’s prediction score changed, accurately predicting “LeBron James plays for the Los Angeles Lakers”. The effects of multimodal feature fusion include the following: (1) Reducing bias: Single-modal features are often limited by the information bias inherent to that modality. The fusion of multimodal features integrates information from each modality, reducing the bias impact of single-modal features. (2) Enhanced robustness: By combining structural, textual, and visual features, the complementarity of different modal features makes the model more robust when facing various types of data. For instance, when there is a conflict between structural and textual features, visual features can provide additional judgment basis, thereby improving prediction accuracy. (3) Improved accuracy: By comprehensively considering the information from multimodal features, the model can better understand the connections between entities and relationships, leading to more accurate predictions. This has been verified in this case. The prediction result “LeBron James plays for the Los Angeles Lakers” after multimodal fusion is consistent with the actual situation, demonstrating the effectiveness of the model in this paper for multimodal feature fusion.

Through the analysis of this case, the limitations of single-modal features and the advantages of multimodal feature fusion are clearly demonstrated. Figure 3 intuitively displays the prediction scores of each modal feature and the prediction scores after fusion, further verifying that multimodal fusion can significantly enhance the performance of knowledge graph link prediction.

6. Conclusions and Future Work

This paper studies the problem of link prediction for knowledge graphs. Inspired by multimodal representation learning, we propose an effective model called MM-Transformer for knowledge graph link prediction. The model comprehensively utilizes structural, visual, and textual features. Specific modality encoders are employed to extract these features, and Transformer is utilized for multimodal feature fusion to reduce the heterogeneity of multimodal entity representations. Furthermore, multi-head attention calculations are conducted on mixed keys and values at each layer within the attention mechanism to proactively reduce modality heterogeneity. In the feed-forward network (FFN) layer, token-level fine-grained fusion is implemented to mitigate the adverse impact of irrelevant structures and images on textual information. This research is highly significant for advancing the field of knowledge graph completion and offers valuable insights for addressing other knowledge-intensive tasks. Although MM-Transformer shows excellent performance in the multimodal link prediction task, it has certain limitations. (1) Computational complexity: the model requires a lot of computing resources and is therefore not very practical to deploy in resource-constrained environments. (2) Longer training time: due to its complex architecture, the training time is significantly longer compared to simpler models. (3) Data dependency: the performance of the model depends heavily on the quality and quantity of available multimodal data, which may not always be accessible.

In the future, we plan to do the following: (1) extend the model in this article to pre-training tasks for multimodal knowledge graph construction. By pre-training on large-scale multimodal data, the generalization and robustness of the model can be further improved, providing richer and more accurate representations for knowledge graph completion; (2) further optimize the model architecture and training strategy of MM-Transformer to improve its efficiency and performance on large-scale datasets. Specifically, we plan to explore more effective fusion methods and attention calculation methods in terms of multimodal feature fusion and attention mechanisms. We also plan to (3) cooperate with researchers in other fields to apply the idea of multimodal representation learning to more practical scenarios, such as medical diagnosis, autonomous driving, and intelligent recommendation systems. Through cross-domain cooperation, the scope of applications of the model can be broadened and the development of related fields can be promoted.