A Sentence-Matching Model Based on Multi-Granularity Contextual Key Semantic Interaction

Abstract

In the task of matching Chinese sentences, the key semantics within sentences and the deep interaction between them significantly affect the matching performance. However, previous studies mainly relied on shallow interactions based on a single semantic granularity, which left them vulnerable to interference from overlapping terms. It is particularly challenging to distinguish between positive and negative examples within datasets from the same thematic domain. This paper proposes a sentence-matching model that incorporates multi-granularity contextual key semantic interaction. The model combines multi-scale convolution and multi-level convolution to extract different levels of contextual semantic information at word, phrase, and sentence granularities. It employs multi-head self-attention and cross-attention mechanisms to align the key semantics between sentences. Furthermore, the model integrates the original, similarity, and dissimilarity information of sentences to establish deep semantic interaction. Experimental results on both open- and closed-domain datasets demonstrate that the proposed model outperforms existing baseline models in terms of matching performance. Additionally, the model achieves matching effectiveness comparable to large-scale pre-trained language models while utilizing a lightweight encoder.

1. Introduction

Sentence matching is essential in various natural language processing tasks and finds applications in question answering, information retrieval, recommendation systems, and natural language inference. For example, in question-answering systems, it is used for paragraph retrieval to match relevant paragraphs from text corpus with the given question. Additionally, it is employed in answer selection tasks to choose the most suitable answer from a set of candidates. Information retrieval and recommendation system tasks rely on sentence matching to establish relationships between queries and documents and rank candidate results accordingly. In natural language inference tasks, sentence matching is utilized to evaluate the semantic relationship between hypotheses and premises.

In recent years, pre-trained language models have shown great success in diverse natural language processing tasks. For instance, Garg et al. [1] employed a fully connected layer to combine the classification vector of BERT [2]. Reimers et al. [3] introduced SBERT, which encodes each sentence into a dense vector and measures the similarity between two vectors to assess the textual matching degree. These BERT-based models effectively capture overlapping terms in sentences. However, these methods have limitations in terms of utilizing single-granularity vectors, which are insufficient to fully represent textual information and the limited interaction between sentences.

Inspired by the matching aggregation framework [4], Tang et al. [5] enhanced the effectiveness of short text matching by considering multiple aspects of interaction that capture the similarities and differences between two sentences comprehensively. Wang et al. [6] utilized ChineseBERT [7] to incorporate the visual and phonetic information of Chinese characters into the pre-training model, which served as encoders for questions and answers. This approach enabled multi-granularity feature encoding. They introduced a multi-scale convolutional neural network [8] to extract local semantic information and developed a context-aware interactive module to model sentence interaction. Furthermore, they proposed a multi-perspective fusion mechanism to integrate local and interactive information, with the goal of capturing rich features from questions and answers. However, these methods have limitations in their focus on low-level local semantic interactions and their limited discriminative ability when dealing with candidate options that contain a high degree of overlapping terms.

Lin et al. [9] introduced BERT-SMAP (BERT with self-matching attention pooling), which incorporates a self-matching attention-pooling mechanism into sentence pairs. This mechanism allows the model to prioritize the key terms within the sentence pairs rather than the overlapping terms. Lu et al. [10] proposed MKPM (multi-keyword pair matching), a method for addressing the sentence-matching task. It utilizes attention mechanisms on sentence pairs to identify the most important keyword pairs from the two sentences, representing their semantic relationship and avoiding interference from redundancy and noise. However, these methods have limitations in terms of their performance on closed-domain datasets. In datasets with a high semantic similarity between texts in the same thematic domain, solely focusing on key terms may make it challenging for the model to distinguish between positive and negative examples.

Humans first consider the presence of matching keywords and then synthesize the overall meaning of sentences to assess text alignment. To address these issues, this paper proposes a sentence-matching model that incorporates multi-granularity contextual key semantic interaction. The model combines pre-trained language model embeddings with BiGRU contextual encoding to thoroughly consider sentence contextual information. It utilizes multi-scale CNN and multi-level CNN to extract different levels semantic information at word and phrase granularity. The model introduces multi-head self-attention and cross-attention to emphasize key semantics within sentences and align crucial information between sentence pairs. By integrating three interaction methods, it incorporates interactive information on similarity and dissimilarity, enabling focus on key semantics and local interaction information at different granularity levels. The sentence’s overall representation is obtained through max pooling, with the pre-trained language model’s CLS vector serving as the coarse-grained semantics of the entire sentence pair. By combining these components, the model comprehensively captures the semantic information of the entire sentence pair.

In Chinese, a language lacking natural word delimiters, semantic information is conveyed through lexical units. Additionally, the polysemy of Chinese words can introduce ambiguity in various contexts. Phrases formed by word combinations carry specific underlying semantics, and Chinese sentences with different word orders can exhibit entirely different meanings. Approaches that rely on single-granularity text semantic matching can only analyze semantics at a single level and may not accurately capture the semantic relationship between texts. Furthermore, they often overlook the fusion of features from different granularities. In contrast, MCKI explores key semantic information at the levels of words, phrases, and sentences, while fully considering the interaction and fusion of semantics across different granularities. This enables MCKI to achieve superior performance.

The main contributions of this paper can be summarized as follows:

• Introduction of the multi-granularity contextual key semantic interaction model (MCKI) for sentence matching, which combines multi-granularity feature extraction and key semantic interaction fusion. This enables the model to capture comprehensive semantic information in the text.
• Development of a semantic feature extraction mechanism that combines multi-scale CNN and multi-level CNN to extract contextual semantic information at different granularities and levels.
• Construction of a key semantic alignment mechanism by integrating multi-head self-attention and cross-attention, incorporating interaction information at different levels from low level to high level.
• The proposed model demonstrates superior matching performance compared to existing baseline models in the LCQMC and BQ datasets. Additionally, it achieves comparable results to large-scale pre-trained language models while using a lightweight encoder.

2. Related Work

With the advancement of deep learning techniques, attention mechanisms have been extensively explored by researchers to capture semantic interactions and determine the degree of matching. For example, Zhao et al. [11] presented an interactive attention network that employed a matching matrix to model the semantic exchange between source and target texts, improving the matching capability for low-frequency keywords. Fei et al. [12] introduced a deep neural communication model that utilized BiLSTM and tree encoders to capture semantic and syntactic features. Through attention mechanisms, they designed deep communication modules to capture local and global interactions between semantic and syntactic features, thereby enhancing semantic comprehension. However, these studies primarily focus on modeling English sentences and mainly address single-granularity text semantic matching. They primarily analyze semantics from a single level and do not take into account the characteristics of the Chinese language. As a result, accurately measuring semantic relationships between Chinese texts is hindered.

Several studies have investigated multi-granularity text semantic matching to comprehensively assess the semantic similarity and matching degree between texts. These studies have explored deep fusion techniques at the character and word levels. For example, Zhang et al. [13] utilized a soft alignment attention mechanism to enhance local information in sentences at different levels, capturing feature information and correlations from multiple perspectives. Yu et al. [14] employed interactive information to model diverse text pairs across various tasks and languages. Wang et al. [15] concurrently considered deep and shallow semantic similarity, as well as granularity at the lexical and character levels, enabling a deeper exploration of similarity information. Chang et al. [16] proposed semantic similarity analysis through the fusion of words and phrases.

Considering the characteristics of Chinese text, several research studies have employed character-level and word-level approaches to develop semantic matching models. For instance, Zhang et al. [17] proposed a deep feature fusion model that combines various deep learning structures to capture semantic information. They generate a similarity tensor using three matching strategies. Lai et al. [18] designed a lattice-based approach that constructs word grids by selecting character and word paths and utilized CNN to generate sentence representations. Chen et al. [19] introduced a neural graph matching network for Chinese text matching, constructing a graph that includes all possible characters and words in a sentence. They employed graph neural networks to generate graph representations for predicting the matching degree. Zhang et al. [20] presented a multi-granularity fusion model that combines LSTM encoding structures to extract features at different granularities. They employed an interactive matching layer to capture and fuse more interaction features. Zhao et al. [21] proposed a multi-granularity alignment model that utilizes attention mechanisms to capture interaction features between different granularities and merge them for obtaining matching representations. Lyu et al. [22] introduced a language knowledge-enhanced graph transformer to update character representations and semantic representations, obtaining the final sentence representation. Some studies attempted to model text representations using pinyin (phonetic transcription) and radicals. Tao et al. [23] proposed a radical-guided correlation model that extracts features from characters, radicals, and character–radical associations for Chinese text classification. Zhao et al. [24] proposed a multi-granularity interaction model based on pinyin and radicals for Chinese semantic matching. In addition to character and word information, they further incorporated multi-granularity features of Chinese pinyin and radicals. The interaction features between different granularities and sentences were aggregated to generate the final matching representation and predict the matching degree. Although these methods utilized multi-granularity information to comprehend text semantics from different perspectives, they faced limitations with the Word2Vec model for text embedding representation. Performance tended to decline when dealing with rare words or domain-specific terms.

In recent years, the widespread use of pre-trained language models has permeated various domains, benefiting tasks involving the matching of textual semantics. These models undergo extensive pre-training on large textual corpora, equipping them with the ability to understand grammar, semantics, and contextual intricacies in natural language. Wu et al. [25] merged BERT with tree models to investigate matching relationships in question-answering data. Zhang et al. [26] utilized BERT models and multi-feature convolutions to extract semantic features for textual semantic matching. Zou et al. [27] employed RoBERTa as the backbone model and employed a divide-and-conquer strategy by decomposing keywords and intents for textual semantic matching. Wang et al. [15] achieved multi-granularity feature encoding using ChineseBERT. They also developed a context-aware interaction module to model interactive information between sentences and utilized a multi-perspective fusion mechanism to integrate local and interaction information, capturing rich features of questions and answers. However, these approaches primarily focus on low-level local semantic interactions and overlook crucial semantic information within the text, making them susceptible to irrelevant terminologies. Lin et al. [9] introduced a self-matching attention-pooling mechanism in sentence pairs, prioritizing key terms rather than overlapping terms. Lu et al. [10] proposed a matching method based on multiple pairs of key terms, selecting the most significant word pairs from two sentences to avoid redundancy and noise interference. While emphasizing key terms helps eliminate irrelevant semantic information, its effectiveness is limited in domain-specific datasets where semantic similarity is higher within texts of the same theme. Solely focusing on key terms poses challenges for the model in distinguishing positive and negative examples.

3. Multi-Granularity Contextual Key Semantic Interaction Model

This section presents the detailed components of the proposed model, as illustrated in Figure 1. The multi-granularity contextual key semantic interaction model (MCKI) consists of several layers: the contextual representation layer, the multi-granularity feature extraction layer, the key semantic alignment layer, the multi-way interaction fusion layer, and the similarity prediction layer.

The contextual representation layer utilizes a pre-trained language model to obtain embedded representations of the sentence pairs. The BiGRU is then employed to capture contextual features of the sequences.

In the multi-granularity feature extraction layer, a combination of multi-scale CNN and multi-level CNN is used to extract semantic information at different granularities and levels from the contextual features.

The key semantic alignment layer first utilizes a multi-head self-attention mechanism to extract key semantic information within each sentence. Then, a cross-attention mechanism is applied to align the key semantic information between sentence pairs.

Next, the multi-way interaction fusion layer combines the interaction features of key semantics through multiple comparison operations.

Finally, the similarity prediction layer employs max pooling to obtain the final semantic vectors at different granularity levels. These vectors are concatenated with the CLS token vector to measure the semantic matching relationship.

3.1. Contextual Representation Layer

The contextual representation layer consists of two steps: embedding representation and context feature extraction. For two-sentence sequences, denoted as $P=\left\{p_1,p_2,\dots ,p_m\right\}$ and $Q=\left\{q_1,q_2,\dots ,q_n\right\}$, a pre-trained language model (such as BERT) is utilized as the encoder for the sentence pair. Special tokens, namely [CLS] and [SEP], are added to the sequences to distinguish them. The embedding representation of the sentence pair, denoted as ${\mathit{X}}_{emb}$, is obtained through the following encoding process:

$$X=\left[\left[CLS\right], P, \left[SEP\right], Q, \left[SEP\right]\right]$$

(1)

$${\mathit{X}}_{emb}=BERT\left(X\right)=\left[{\mathit{h}}_{cls},{\mathit{h}}_{p_1},\dots ,{\mathit{h}}_{p_m},{\mathit{h}}_{sep},{\mathit{h}}_{q_1},\dots ,{\mathit{h}}_{q_n},{\mathit{h}}_{sep}\right]$$

(2)

where $X$ represents the input sequence containing special markers and sentence pairs $P$ and $Q$. These sequences are encoded using a pre-trained language model, such as BERT, resulting in the embedded representation ${\mathit{X}}_{emb}\in {ℝ}^{\mathit{h}\times \left(m+n+3\right)}$ of the entire sequence. $\mathit{h}\in {ℝ}^h$ represents an embedding vector, with h denoting the dimensionality of the encoder’s hidden layer. Notably, ${\mathit{h}}_{cls}\in {ℝ}^h$ can serve as a coarse-grained feature, summarizing the semantic information of the entire sentence pair. Recurrent neural networks (RNNs) are commonly employed for modeling sequential data. However, RNN models are susceptible to problems like vanishing or exploding gradients. Hence, this study adopts an efficient model called a Bidirectional Gated Recurrent Unit (BiGRU) to further encode the embedded sequences and extract context features for each of the two sentences. Additionally, a linear layer is utilized to merge the embedding representations, facilitating the comprehensive utilization of both the original and contextual information of the sequences. The representation process is illustrated as follows:

$${\mathit{P}}_{context}=BiGRU\left(\left[{\mathit{h}}_{p_1},\dots ,{\mathit{h}}_{p_m}\right]\right)+Linear\left(\left[{\mathit{h}}_{p_1},\dots ,{\mathit{h}}_{p_m}\right]\right)$$

(3)

$${\mathit{Q}}_{context}=BiGRU\left(\left[{\mathit{h}}_{q_1},\dots ,{\mathit{h}}_{q_n}\right]\right)+Linear\left(\left[{\mathit{h}}_{q_1},\dots ,{\mathit{h}}_{q_n}\right]\right)$$

(4)

$$BiGRU\left(\mathit{H}\right)=Concat\left(\overrightarrow{GRU}\left(\mathit{H}\right),\overleftarrow{GRU}\left(\mathit{H}\right)\right)$$

(5)

where ${\mathit{P}}_{context}\in {ℝ}^{m\times 2d}$, ${\mathit{Q}}_{context}\in {ℝ}^{n\times 2d}$, and $m$ and $n$ represent the lengths of the sentences, while $d$ denotes the hidden dimension of the unidirectional GRU.

3.2. Multi-Granularity Feature Extraction Layer

In the field of natural language processing (NLP), convolutional neural networks (CNNs) have proven to be effective in extracting local features from sentences. They operate by applying convolutional operations to input sentences, which allows them to capture n-gram features and generate latent semantic representations that are rich in information for various downstream tasks. However, when dealing with Chinese text, words and phrases can vary in length and carry different meanings based on the number of characters they consist of. Using a fixed-size convolutional kernel might not be sufficient to capture the diverse range of phrase features, potentially resulting in the omission of crucial semantic information.

To address the limitations of a single fixed-size convolutional kernel in capturing diverse phrase features in Chinese text, this study proposes the use of multiple-scale convolutions to capture semantic features at different granularity levels, including individual words, phrases, and entire sentences. This allows for a comprehensive representation of sequence samples.

The multi-scale CNN performs convolutions on the input sentence using a series of differently sized kernels, each targeting specific n-gram semantic features. By employing multiple CNNs, the model can capture distinct features contained in different combinations of n-grams, overcoming the limitations of a single CNN. The network architecture is depicted in Figure 2.

Moreover, combinations of different phrases and expressions carry higher-level semantic information. By stacking different-scale CNNs, the multi-level CNN can capture advanced semantic features from low-level character sequences. This allows for the extraction of deeper semantic information. The network structure is illustrated in Figure 3.

The multi-scale CNN and multi-level CNN have the ability to capture different aspects of sentences. The multi-scale CNN captures a broader range of semantic information, while the multi-level CNN captures deeper semantic information. These two approaches complement each other. Similar to other CNN models used in NLP tasks, this study employs fixed-width convolutions to extract local features. The width of the convolutional kernel matches the hidden dimension of the input sequence.

For a given sequence of sentences, let us assume ${\mathit{c}}_i\in {ℝ}^d$ represents the contextual representation of the $i$-th word in the sentence, where $d$ denotes the dimension of the representation. The input sentence consists of $n$ words. To extract meaningful local features, we employ convolutional filters of different scales, denoted as $S=\left\{s_1,s_2,\dots ,s_t\right\}$, with a fixed kernel width of $l$. For example, generating feature $m_{ij}^{S_i}$ through window ${\mathit{c}}_{i:i+S_i-1}$:

$$m_{ij}^{S_i}=GELU\left({\mathit{W}}_j^{S_i}\left({\mathit{c}}_{i:i+S_i-1}\right)+b_j^{S_i}\right)$$

(6)

where ${\mathit{W}}_j^{S_i}\in {ℝ}^{S_i\times d}$ represents the weight of the $j$-th feature map at scale $s_i$, $GELU$ denotes the activation function, and $b_j^{S_i}$ represents the bias of the feature map.

The convolutional kernel, denoted as $s_i$, slides over the entire padded contextual representation matrix, denoted as $\mathit{C}$, to generate feature maps. This process is illustrated below:

$${\mathit{m}}_j^{S_i}=\left[m_{1j}^{S_i},m_{2j}^{S_i},\dots ,m_{nj}^{S_i}\right]$$

(7)

The local n-gram features of the sentences have been extracted using the multi-scale CNN. By concatenating the outputs of the convolutional kernel $s_i$, we obtain the input for the key semantic alignment layer, which is represented as follows:

$${\mathit{M}}^{S_i}=\left[{\mathit{m}}_1^{S_i},{\mathit{m}}_2^{S_i},\dots ,{\mathit{m}}_h^{S_i}\right]$$

(8)

where $h$ represents the number of feature maps, and ${\mathit{M}}^{S_i}\in {ℝ}^{n\times h}$.

For the multi-level CNN, it is necessary to repeat the above operation with the convolutional kernel $s_j$ on top of ${\mathit{M}}^{S_i}$ to realize the extraction of higher-level semantics.

After applying the multi-granularity feature extraction layer to the contextual features of the sentence, the resulting multi-granularity features of the sentence $\mathit{P}$ and $\mathsf{Q}$ are as follows:

$${\mathit{M}}_P=\left\{{\mathit{M}}_P^{S_0},{\mathit{M}}_P^{S_1},{\mathit{M}}_P^{S_2},\dots ,{\mathit{M}}_P^{S_t}\right\}$$

(9)

$${\mathit{M}}_Q=\left\{{\mathit{M}}_Q^{S_0},{\mathit{M}}_Q^{S_1},{\mathit{M}}_Q^{S_2},\dots ,{\mathit{M}}_Q^{S_t}\right\}$$

(10)

where ${\mathit{M}}_P^{S_i}\in {ℝ}^{m\times h}$, ${\mathit{M}}_Q^{S_i}\in {ℝ}^{n\times h}$, $t$ represents the number of convolutional kernels, ${\mathit{M}}^{S_0}$ represents the output features of the multi-level CNN, and ${\mathit{M}}^{S_i}$ represents the output features of different-scale convolutional kernels.

3.3. Key Semantic Alignment Layer

The key semantic alignment layer plays a crucial role in enabling the model to focus on important semantic features at different granularities within a sentence and align key semantic information across sentence pairs. The key semantic features for sentence pairs should meet the following conditions:

• They should possess rich semantic characteristics.
• They should be highly significant within the sentences and directly impact the overall meaning of the sentences.
• As key semantic information for sentence pairs, they should exhibit substantial correlations with each other.

These three features are determined by the contextual features at different granularities within the sentences, self-attention mechanisms within the sentences, and cross-attention mechanisms between sentence pairs. Specifically, the contextual features of the sentences are derived from the outputs of the multi-granularity feature extraction layer, represented by ${\mathit{M}}_P^{S_i}\in {ℝ}^{m\times h}$ and ${\mathit{M}}_Q^{S_i}\in {ℝ}^{n\times h}$, where $h$ denotes the number of feature maps. The overall structure of the key semantic alignment layer is illustrated in Figure 4.

3.3.1. Multi-Head Self-Attention

The transformer model utilizes the scaled dot-product attention (SDA) mechanism for self-attention. As transformer-based pre-trained language models have gained popularity, the use of the scaled dot-product function has become more prevalent. The process begins by calculating the dot product between $\mathit{Q}$ and $\mathit{K}$, which serves as a measure of similarity between them. A higher value indicates a stronger correlation. This result is then divided by the scaling factor $\sqrt{d}$, where $d$ represents the dimension of the vectors. The introduction of the scaling factor addresses the issue of gradient vanishing or exploding that may occur when dealing with large vector dimensions during the dot product computation. This ensures smoother and more manageable attention scores, promoting training stability. Subsequently, the softmax function is applied to ensure that the weighted coefficients sum up to one, achieving attention normalization and distribution. Finally, the attention expression is obtained by multiplying it with the matrix v. The SDA function is defined as follows:

$$SDA\left(\mathit{Q},\mathit{K},\mathit{V}\right)=Softmax\left(\frac{\mathit{Q}{\mathit{K}}^{\top }}{\sqrt{d}}\right)\mathit{V}$$

(11)

By introducing the multi-head mechanism, sequence features are mapped into different linear spaces, allowing for the extraction of relevant information from various representation spaces. This integration of information from different spaces enhances the expressive power of attention. The formula can be represented as follows:

$$MultiHead\left(\mathit{Q},\mathit{K},\mathit{V}\right)=Concat\left(\mathit{H}\mathit{e}\mathit{a}{\mathit{d}}_1,\dots ,\mathit{H}\mathit{e}\mathit{a}{\mathit{d}}_t\right)\mathit{W}$$

(12)

$$\mathit{H}\mathit{e}\mathit{a}{\mathit{d}}_i=SDA\left(\mathit{Q}{\mathit{W}}_i^Q,\mathit{K}{\mathit{W}}_i^K,\mathit{V}{\mathit{W}}_i^V\right)$$

(13)

where $\mathit{W}$, ${\mathit{W}}_i^Q$, ${\mathit{W}}_i^K$, and ${\mathit{W}}_i^V$ are learnable parameters, $t$ denoting the number of attention heads.

Using multi-head self-attention, the key semantic features ${\mathit{S}}_P^{S_i}$ for the sentence $\mathit{P}$ itself can be computed as follows:

$$S_P^{S_i}=Concat\left({\mathit{S}}_{P_1}^{S_i},\dots ,{\mathit{S}}_{P_t}^{S_i}\right)\mathit{W}$$

(14)

$${\mathit{S}}_{P_i}^{S_i}=SDA\left({\mathit{M}}_P^{S_i}{\mathit{W}}_i^Q,{\mathit{M}}_P^{S_i}{\mathit{W}}_i^K,{\mathit{M}}_P^{S_i}{\mathit{W}}_i^V\right)=Softmax\left(\frac{{\mathit{M}}_P^{S_i}{\mathit{W}}_i^Q{({\mathit{M}}_P^{S_i}{\mathit{W}}_i^K)}^{\top }}{\sqrt{h}}\right){\mathit{M}}_P^{S_i}{\mathit{W}}_i^V$$

(15)

In this equation, the dimension of ${\mathit{S}}_P^{S_i}$ matches the dimension of the input feature ${\mathit{M}}_P^{S_i}$, ${\mathit{S}}_P^{S_i}\in {ℝ}^{m\times h}$. Similarly, the crucial semantic features ${\mathit{S}}_Q^{S_i}\in {ℝ}^{n\times h}$ for the sentence $\mathit{Q}$ can be calculated.

3.3.2. Cross-Attention

Cross-attention calculates the attention between sentence pairs in both directions, allowing for the alignment of key semantic information between the two sentences. This attention is computed from the similarity matrix ${\mathit{F}}^{S_i}\in {ℝ}^{m\times n}$, which considers the lengths of sentence $\mathit{P}$ ($m$) and sentence $\mathit{Q}$ ($n$). The computation of the similarity matrix ${\mathit{F}}^{S_i}$ is as follows:

$${\mathit{F}}^{S_i}={\mathit{S}}_P^{S_i}{\left({\mathit{S}}_Q^{S_i}\right)}^{\top }$$

(16)

Afterwards, the softmax function is applied to normalize the similarity matrix ${\mathit{F}}^{S_i}$ in both directions. This generates attention weights ${\mathit{F}}_{PTQ}^{S_i}\in {ℝ}^{m\times n}$ row-wise for the attention from sentence $\mathit{P}$ to sentence $\mathit{Q}$ and ${\mathit{F}}_{QTP}^{S_i}\in {ℝ}^{n\times m}$ column-wise for the attention from sentence $\mathit{Q}$ to sentence $\mathit{P}$. The calculation is as follows:

$${\mathit{F}}_{PTQ}^{S_i}=Softmax\left({\mathit{F}}_{i:}^{S_i}\right)$$

(17)

$${\mathit{F}}_{QTP}^{S_i}=Softmax\left({\mathit{F}}_{:j}^{S_i}\right)$$

(18)

To obtain the attention-aware representation ${\mathit{R}}_P^{S_i}\in {ℝ}^{m\times h}$ for sentence $\mathit{P}$ with respect to sentence $\mathit{Q}$, the attention weights are multiplied with the representations. Similarly, the attention-aware representation ${\mathit{R}}_Q^{S_i}\in {ℝ}^{n\times h}$ for sentence $\mathit{Q}$ with respect to sentence $\mathit{P}$ can be obtained. The calculation is as follows:

$${\mathit{R}}_P^{S_i}={\mathit{F}}_{PTQ}^{S_i}{\mathit{S}}_Q^{S_i}$$

(19)

$${\mathit{R}}_Q^{S_i}={\mathit{F}}_{QTP}^{S_i}{\mathit{S}}_P^{S_i}$$

(20)

where ${\mathit{R}}_P^{S_i}$ represents the attention-aware representation of sentence $\mathit{P}$ with respect to sentence $\mathit{Q}$ after the $i$-th scale convolution and attention mechanism. Similarly, ${\mathit{R}}_Q^{S_i}$ represents the attention-aware representation of sentence $\mathit{Q}$ with respect to sentence $\mathit{P}$ after the $i$-th scale convolution and attention mechanism.

3.4. Multi-Way Interaction Fusion Layer

To capture the interaction information between sentences, the key semantic features ${\mathit{S}}^{S_i}$ and the attention-aware representation ${\mathit{R}}^{S_i}$ can be effectively integrated through three different comparison operations. These operations facilitate a deep interaction between the key semantic features of each sentence, enabling the integration of interaction information across different granularities, from low level to high level. The structure of the multi-way interaction fusion layer is illustrated in Figure 5.

For sentence P, the three comparison operations are defined as follows:

$${\mathit{G}}_{P_1}^{S_i}=FF{N}_1\left(\left[{\mathit{S}}_P^{S_i},{\mathit{R}}_P^{S_i}\right]\right)$$

(21)

$${\mathit{G}}_{P_2}^{S_i}=FF{N}_2\left(\left[{\mathit{S}}_P^{S_i}-{\mathit{R}}_P^{S_i}\right]\right)$$

(22)

$${\mathit{G}}_{P_3}^{S_i}=FF{N}_3\left(\left[{\mathit{S}}_P^{S_i}\odot {\mathit{R}}_P^{S_i}\right]\right)$$

(23)

where ${\mathit{G}}_{P_1}^{S_i}\in {ℝ}^{m\times 2h}$ (achieved through concatenation along the column direction), ${\mathit{G}}_{P_2}^{S_i}\in {ℝ}^{m\times h}$, and ${\mathit{G}}_{P_3}^{S_i}\in {ℝ}^{m\times h}$ represent the interaction features obtained after the three different operations:

• ${\mathit{G}}_{P_1}^{S_i}$ represents the concatenation of the key semantic features ${\mathit{S}}_P^{S_i}$ and the attention-aware representation ${\mathit{R}}_P^{S_i}$. In the interaction operation, concatenation is used to preserve all the information.
• ${\mathit{G}}_{P_2}^{S_i}$ performs element-wise subtraction, approximating the calculation of Euclidean distance and measuring the relevance between the two representations using difference information.
• ${\mathit{G}}_{P_3}^{S_i}$ performs element-wise multiplication, approximating the calculation of cosine similarity and measuring the similarity between the two representations.

These comparison operations enhance semantic information and capture semantic relationships. To optimize the model’s efficiency, a feed-forward neural network is employed to integrate the interaction information, resulting in the feature interaction matrix ${\mathit{G}}_P^{S_i}\in {ℝ}^{m\times l}$:

$${\mathit{G}}_P^{S_i}=FF{N}_4\left(\left[{\mathit{G}}_{P_1}^{S_i},{\mathit{G}}_{P_2}^{S_i},{\mathit{G}}_{P_3}^{S_i}\right]\right)$$

(24)

where $FF{N}_1$, $FF{N}_2$, $FF{N}_3$, and $FF{N}_4$ denote distinct feed-forward neural networks with non-shared parameters, conventionally represented as follows:

$$FFN\left(\mathit{Z}\right)=GELU\left(\mathit{Z}\mathit{W}+\mathit{b}\right)$$

(25)

and $\left[{\mathit{G}}_{P_1}^{S_i},{\mathit{G}}_{P_2}^{S_i},{\mathit{G}}_{P_3}^{S_i}\right]\in {ℝ}^{m\times 4h}$ is concatenated along the column dimension. The parameters ${\mathit{W}}_4\in {ℝ}^{4h\times l}$ and $\mathit{b}\in {ℝ}^l$ are adjustable, and $l$ represents the output dimension of the network. This operation enables the model to capture deeper levels of interaction information and reduces the complexity of the representation. Likewise, the feature interaction matrix ${\mathit{G}}_Q^{S_i}\in {ℝ}^{n\times l}$ can be obtained for sentence $\mathit{Q}$.

3.5. Similarity Prediction Layer

By applying the max-pooling operation, we map the feature interaction matrices ${\mathit{G}}_P^{S_i}$ and ${\mathit{G}}_Q^{S_i}$ to vector representations of the same dimension. This process allows for the extraction of valuable features at various granularities while reducing computational costs. The max-pooling operation for each granularity is defined as follows:

$${\mathit{p}}^{S_i}=MaxPooling\left({\mathit{G}}_P^{S_i}\right)$$

(26)

$${\mathit{q}}^{S_i}=MaxPooling\left({\mathit{G}}_Q^{S_i}\right)$$

(27)

where ${\mathit{p}}^{S_i}\in {ℝ}^l$ and ${\mathit{q}}^{S_i}\in {ℝ}^l$ represent the semantic feature vectors modeled by the model at scale $S_i$. By concatenating all semantic feature vectors from different scales with the CLS token vector, we obtain the final semantic interaction vector $\mathit{f}$:

$$\mathit{f}=Concat\left({\mathit{h}}_{cls},{\mathit{p}}^{S_0},{\mathit{p}}^{S_1},\dots ,{\mathit{p}}^{S_t},{\mathit{q}}^{S_0},{\mathit{q}}^{S_1},\dots ,{\mathit{q}}^{S_t}\right)$$

(28)

where $\mathit{f}\in {ℝ}^{h+2\left(t+1\right)l}$, and $h$ represents the dimensions of the CLS token vector, with a total of $t$ + 1 scales, and $l$ denotes the dimension of the feature vector at each scale. Finally, a fully connected layer is used to predict the matching relationship between the two sentences based on this vector.

The chosen loss function is the cross-entropy loss, represented by the following formula:

$$\mathcal{L}=-\frac{1}{N}{\displaystyle \sum _N}\left(ylog p+\left(1-y\right)\times log\left(1-p\right)\right)$$

(29)

In the equation, $y$ represents the true label, which is set to one when there is semantic matching between the sentence pairs and zero otherwise. $p$ denotes the predicted result, and $N$ represents the number of training samples.

4. Experimental Results and Analysis

4.1. Dataset

The experimental evaluation of the model in this study was conducted on the following two publicly available datasets, and the dataset statistics are presented in

Table 1. Dataset statistics.

Datasets	Total	Train	Validation	Test
LCQMC	260,068	238,766	8802	12,500
BQ	120,000	100,000	10,000	10,000

.

Large-scale Chinese Question Matching Corpus (LCQMC) [28]: LCQMC is a dataset designed for sentence pair matching in an open-domain context. It focuses on semantic intent matching, making it more generalizable than sentence rewriting datasets. Each sentence pair in LCQMC is labeled with zero or one to indicate whether the sentences match. The dataset consists of 149,226 positive sentence pairs and 110,842 negative sentence pairs.

Bank Question Dataset (BQ) [29]: The BQ dataset contains customer service logs from online banks spanning one year. It serves as a dataset for identifying sentence semantic equivalence, specifically for sentence pair semantic matching. Similar to LCQMC, each sentence pair in BQ is labeled with zero or one to indicate whether the sentences match. The positive and negative sentence pairs in BQ are balanced with a ratio of 1:1.

4.2. Implementation

The model architecture and system experiments in this study were implemented using the PaddleNLP framework. The framework’s pre-trained models and their initialized weights were utilized. The experiments were conducted using two NVIDIA GeForce RTX 3090 GPUs (Taiwan Semiconductor Manufacturing Company Limited, Hsinchu, Taiwan), each with a memory capacity of 24 GB.

In terms of the model architecture, for the input encoding stage, this study utilized 768-dimensional BERT-base, 312-dimensional ERNIE 3.0-nano, and 768-dimensional ERNIE 3.0-base for text embedding, respectively. For the feature extraction stage, the BiGRU unidirectional hidden dimension was set to 150. The multi-scale CNN adopts convolutional filters of size (3, 4, 5), while the multi-level CNN uses convolutional filters of size (3, 4). The feature map dimension for both was 300. The multi-head self-attention employs 20 heads. In the interaction fusion stage, the hidden dimension of the FNN was set to 150.

Regarding model training, the maximum sentence length was set to 128, and the batch size was 32. The optimization was performed using AdamW with ${\mathsf{\beta }}_1$ = 0.9 and ${\mathsf{\beta }}_2$ = 0.999. The learning rate was set to 5 × 10⁻⁵, and the dropout parameter was 0.1. The model was trained for 3 epochs. The evaluation metric used was accuracy (Acc), and the best-performing result was chosen as the experimental outcome.

In terms of model testing, the evaluation metric used was accuracy (Acc), and the optimal result was chosen as the experimental outcome. The following definitions were used: True Positives (TPs) represent the cases where the original sentences were matched and correctly classified, while True Negatives (TNs) indicate the cases where the original sentences were unmatched and correctly classified. False Positives (FPs) refer to the cases where the original sentences were unmatched but mistakenly classified, and False Negatives (FNs) denote the cases where the original sentences were matched but erroneously classified. The accuracy was calculated using the following formula:

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(30)

4.3. Baseline Models

In this study, representative methods from three categories, namely neural network-based methods, attention-based methods, and pre-trained language model-based methods, were selected as comparative models. The chosen models are as follows:

• Neural Network-based Methods:

Text-CNN [30]: a convolutional neural network model commonly used for sentence classification and matching tasks.

BiLSTM [31]: a variant of recurrent neural networks that processes sentences using LSTM units, capturing long-term and short-term dependencies from both forward and backward contexts.

BiMPM [32]: A sentence-matching model based on BiLSTM. It utilizes BiLSTM to represent contextual information and performs matching from two directions and four different perspectives. The matching results are then aggregated using BiLSTM.

• Attention-based Methods:

DRCN [33]: a densely connected co-attention recurrent neural network where each layer utilizes attention features and hidden features from all preceding recurrent layers.

MIPR: A multi-granularity interaction model based on pinyin and radicals for Chinese semantic matching. The interaction features between different granularities and sentences are aggregated to generate the final matching representation and predict the matching degree.

MKPM: A sentence-matching method based on multi-keyword pair matching. It employs attention mechanisms for sentence pairs and selects the most important keyword pairs from both sentences to represent their semantic relationships. This approach helps to avoid interference from redundancy and noise.

• Pre-trained Language Model-based Methods:

BERT: A renowned pre-trained language model. This study utilizes BERT-base as the encoder to generate representations for sentence pairs. The CLS vector output is fed into a linear layer to predict the matching relationship between the pairs.

SBERT: A semantic textual similarity method based on BERT. It employs a Siamese and triplet network structure to obtain semantically meaningful sentence representations.

MAGE: A method for multi-scale context-aware interaction in sentence pair matching, based on multi-granularity embeddings. It aligns semantic relationships using context-aware interaction modules and combines local semantic features with attention representations using a multi-perspective fusion approach. This study utilizes BERT-base as the encoder and predicts the matching relationship between sentence pairs using a linear layer.

MacBERT [34]: An enhanced Chinese pre-trained language model based on BERT. It replaces the MLM (Masked Language Model) task in BERT with the Mac (MLM as correction) task, which helps alleviate the discrepancy between pre-training and fine-tuning stages.

ERNIE 3.0 [35]: A large-scale knowledge-enhanced pre-trained language model that combines autoregressive and autoencoding networks. This study conducts experiments using both ERNIE 3.0-base and ERNIE 3.0-nano as encoders. Similar to BERT, the CLS vector output is fed into a linear layer to predict the matching relationship between sentence pairs.

4.4. Performance Comparison

Table 2. Accuracy of different models on LCQMC and BQ.

Methods	LCQMC	BQ
Text-CNN [28,29]	72.80	68.52
BiLSTM [28,29]	76.10	73.51
BiMPM [28,29]	83.40	81.85
DRCN [33]	85.90	83.30
MIPR [24]	85.29	84.45
MKPM [10]	86.71	84.11
BERT-base	86.75	84.67
SBERT	84.44	83.64
MAGE	87.58	84.84
MacBERT-base [34]	87.00	85.20
MacBERT-large [34]	87.60	85.60
Ernie 3.0-nano	86.12	83.02
Ernie 3.0-base	87.64	85.83
MCKI (BERT-base)	87.90	85.28
MCKI (Ernie 3.0-nano)	87.65	83.69
MCKI (Ernie 3.0-base)	89.08	86.08

presents the accuracy results of various models on the test sets of LCQMC and BQ datasets.

Overall, pre-trained language model-based methods outperformed the other two approaches. The performance improvement of these models was more pronounced on open-domain datasets compared to closed-domain datasets. This can be attributed to the emphasis on capturing essential semantic information, which allows for the elimination of irrelevant semantic details. As a result, the models demonstrated significant enhancements when handling sentence pairs with more noise. However, it is worth noting that the BQ dataset consisted of samples with a common theme, resulting in higher similarity between positive and negative instances. This increased similarity posed a challenge in distinguishing between them.

Among the neural network-based methods, BiLSTM demonstrated a substantial accuracy improvement compared to Text-CNN. This is attributed to the fact that text data can be viewed as sequential data, and the recurrent structure of BiLSTM is better suited for modeling sequence features. Unlike Text-CNN, which extracts diverse text features through convolutions of various sizes, BiLSTM takes into account the dependencies between contextual information present in the text. To further enhance accuracy, BiMPM, which builds upon BiLSTM, introduces bidirectional different-perspective matching to capture contextual features and aggregates matching information using BiLSTM.

Methods incorporating attention mechanisms, which enable the model to concentrate on crucial interaction information between sentences, also exhibit enhanced performance. DRCN introduced an attention mechanism in recurrent neural networks. MKPM employed bidirectional attention to select crucial words for keyword matching in sentence pairs, leading to significant improvements on the LCQMC dataset. This highlighted the importance of key semantics in sentence matching within an open-domain context. MIPR extracted semantic interaction features within and between sentences at four different granularities, resulting in notable improvements on the BQ dataset. This showcases the influence of multi-granularity semantics in sentence matching within a closed-domain setting.

Pre-trained language models demonstrate exceptional performance in language modeling by acquiring contextual representations from extensive data. BERT achieved an accuracy of 86.75% on the LCQMC dataset and 84.67% on the BQ dataset. In contrast, SBERT, which employs a dual-encoder architecture to model sentence representations but lacks explicit interaction information between sentences, exhibited lower effectiveness on both datasets when compared to the cross-encoder architecture of BERT. While attention mechanisms enable models to concentrate on important words or phrases in text, they can pose challenges in distinguishing between positive and negative examples in texts with similar themes. MAGE effectively handles this challenge through the use of a context-aware interaction module and a multi-perspective fusion module. Additionally, pre-trained language models such as MacBERT and Ernie 3.0, which have undergone extensive training on large-scale Chinese data, enhance performance by improving the training process.

The proposed method in this study demonstrated significant performance improvements compared to both neural network-based models and attention-based models, highlighting the effectiveness of building models upon pre-trained language models. Specifically, when compared to pre-trained language models, the method in this study achieved improvements of 1.15% and 0.61% on top of BERT, 1.53% and 0.67% on top of Ernie 3.0-nano, and 1.44% and 0.25% on top of Ernie 3.0-base, respectively, surpassing all baseline models. However, the improvement observed for the model based on Ernie 3.0-base in the BQ dataset is relatively smaller. This can be attributed to two factors. Firstly, as mentioned earlier, the BQ dataset poses greater difficulty in terms of matching compared to open-domain datasets. Secondly, Ernie 3.0, being a more advanced pre-trained language model, has undergone extensive knowledge-enhanced pre-training, equipping it with certain capabilities for modeling semantic matching relationships in sentence pairs. Consequently, the potential for improvement relative to the BERT-based models is relatively smaller.

Additionally, this study achieved performance that is comparable to BERT-base and even MacBERT-large by leveraging the capabilities of the lightweight Ernie 3.0-nano model on the LCQMC dataset. This observation suggests that, instead of relying on larger pre-trained language models, the approach presented in this study can attain similar performance levels by utilizing a more lightweight pre-trained language model. This strategy proves effective, particularly when resources are limited.

In comparison to the MAGE model, the proposed method achieved improvements of 0.32% and 0.44% on the two datasets. The approach presented in this study utilizes a fusion of multi-scale and multi-level CNNs to extract comprehensive contextual features, enabling the model to capture a wider and deeper range of information. By combining multi-head self-attention and cross-attention mechanisms, the model not only emphasizes key semantic information but also aligns significant details across different levels of semantics. As a result, significant improvements are observed compared to the MAGE model.

4.5. Ablation Study

To further evaluate the effectiveness of each module, this study included an ablation study on the model based on Ernie 3.0-nano using the BQ dataset. This study involved removing specific modules to assess their impact on performance, as displayed in

Table 3. Ablation study on BQ.

Index	Modules					Acc (%)
	Contextual Representation	Multi-Granularity Feature Extraction	Multi-Head Self-Attention	Cross-Attention	Multi-Way Interaction Fusion
1	√	√	√	√	√	83.69
2	-	√	√	√	√	83.25
3	√	-	√	√	√	83.32
4	√	√	-	√	√	83.35
5	√	√	√	-	√	83.38
6	√	√	√	-	-	83.21
7	√	√	√	√	-	83.26

. In the table, the symbol “√” denotes the inclusion of the corresponding module, while “-” denotes its removal.

To assess the effectiveness of the contextual representation module, the study conducted an ablation experiment by removing this module and directly extracting multi-granularity features from the original embeddings. The experimental results showed that the model’s accuracy decreased by 0.44%. Furthermore, this study investigated the impact of using different contextual representation modules, and the comparative results are presented in

Table 4. Accuracy of different contextual representation modules.

Modules	Acc (%)
LSTM	82.66
GRU	82.83
BiLSTM	83.44
BiGRU	83.57
BiGRU + embedding	83.69

. Compared to unidirectional LSTM and GRU structures, the bidirectional structure demonstrated an improvement of approximately 0.7% in accuracy. This improvement can be attributed to two factors. Firstly, the hidden dimension of the unidirectional structure is only half that of the bidirectional structure, leading to a decrease in overall model performance. Secondly, the bidirectional structure captures both forward and backward semantic dependencies in the text, effectively capturing contextual semantic information. Additionally, the GRU structure outperformed the LSTM structure, indicating that GRU is more effective in modeling sequential information in the text. Moreover, better results were obtained by combining the original embeddings, which allows for the comprehensive utilization of both the original information and contextual information from the text.

Additionally, to verify the effectiveness of the multi-granularity feature extraction module, its removal resulted in a decrease of 0.37% in model accuracy. This study also explored the replacement of this module with different convolutional methods, as depicted in

Table 5. Accuracy of different multi-granularity feature extraction modules.

Modules	Filters	Acc (%)
Single-scale CNN	(3)	83.35
Multi-scale CNN	(3, 4, 5)	83.41
Multi-level CNN	(3, 4)	83.37
Multi-scale CNN + Multi-level CNN	(3, 4, 5) + (3, 4)	83.69

. It was observed that a single-scale CNN can only capture features at a specific scale, limiting its ability to incorporate a broader range of semantic information. Conversely, a multi-scale CNN compensates for this limitation by capturing semantic information at multiple scales, while a multi-level CNN focuses on capturing higher-level semantic information. These two approaches possess distinct capabilities in capturing different semantic features. The multi-scale CNN extracts a wider range of semantic information, while the multi-level CNN delves into deeper semantic layers. By combining both approaches, the model achieves improved performance as they complement each other in capturing diverse semantic features.

This study also examined the impact of attention mechanisms on the model’s performance. By removing self-attention and directly fusing the multi-granularity contextual features with the aligned features from cross-attention, the model’s accuracy declined by 0.34%. This decrease can be attributed to the absence of important semantic information at various granularities. Similarly, removing cross-attention and fusing the multi-granularity contextual features with the output features from self-attention led to a decrease in model accuracy by 0.31%. This reduction occurred because the model solely focused on its own semantic information and lacked interaction between sentences. Furthermore, simultaneous removal of cross-attention and the multi-path interaction fusion module, relying solely on key semantic information at different granularities, resulted in a decrease in model accuracy by 0.48%. This finding indicates that aligning crucial semantic information between sentences and considering semantic interactions at various granularities significantly contribute to enhancing semantic matching capabilities.

Additionally, solely removing the multi-way interaction fusion module resulted in a decrease of 0.43% in accuracy. Furthermore, to further validate the effectiveness of different comparison methods, as presented in

Table 6. Accuracy of different feature fusion methods.

Methods	Acc (%)
(M, R)	83.35
(M-R)	82.91
(M⊙R)	83.05
(M-R, M⊙R)	83.24
(M, R, M-R)	83.30
(M, R, M⊙R)	83.51
(M, R, M-R, M⊙R)	83.69
(M; R; M-R; M⊙R)	83.19

, using only the concatenation of sentence semantic features achieved an accuracy of 83.35%. Using only semantic difference information led to an accuracy of 82.91%, and using only semantic similarity information resulted in an accuracy of 83.05%. These results indicate that combining all three types of comparison methods is an effective approach for capturing matching interactions. Moreover, the impact of feature fusion in different directions on performance was explored. In the table, “,” represents concatenation along the column direction of the tensor, expanding the hidden state dimension, while “;” represents concatenation along the row direction of the tensor, expanding the sentence length. The results showed that concatenation in the row direction did not effectively merge the compared information from multiple ways, which failed to reflect the interaction between sentences and resulted in decreased performance.

Lastly, a comparative experiment was conducted to validate the impact of the CLS token vector on model performance, as presented in

Table 7. Accuracy of different prediction modules.

Modules	Acc (%)
Max pooling	83.23
Avg pooling	83.18
(CLS, Max pooling)	83.69
(CLS, Avg pooling)	83.51
(CLS, Max pooling, Avg pooling)	83.09

. The experiment involved using different prediction modules. The results demonstrated that the best performance is achieved when the CLS token vector is concatenated with the output vector from the max-pooling operation. This finding suggests that in addition to fine-grained features, incorporating the CLS token as a coarse-grained representation of semantic information for the entire sentence pair effectively enhances semantic matching performance. Furthermore, the additional concatenation of the output vector from the average-pooling operation leads to a decrease in model performance. This decrease can be attributed to the introduction of redundant information, which affects the model’s ability to distinguish important features.

4.6. Parameter Sensitivity Analysis

In this section, a further evaluation was carried out to assess the impact of different combinations of multi-scale CNNs on model performance, as presented in

Table 8. Effect of different scale CNN combinations on model performance.

Multi-Scale CNN	Multi-Level CNN	Acc (%)
(3, 4, 5)	(3, 4)	83.69
(3, 4, 5)	(3, 4, 5)	83.47
(3, 4)	(3, 4)	83.54
(1, 2, 3)	(3, 4)	83.62
(2, 3, 4)		83.65
(3, 4, 5)	(1, 2)	83.50
	(2, 2)	83.49
	(2, 3)	83.57
	(3, 2)	83.44
	(3, 3)	83.64
	(4, 2)	83.40
	(4, 3)	83.44
	(4, 4)	83.46
	(4, 5)	83.53

. The results indicated that the best model performance was achieved when using a combination of multi-scale convolutional filters (3, 4, 5) and multi-level convolutional filters (3, 4). Increasing or decreasing the number of convolutional filters resulted in varying degrees of information redundancy or loss, which led to a decrease in performance.

In addition, the impact of the number of self-attention heads on model performance was investigated, as illustrated in Figure 6. It was observed that as the number of heads increased, the model performance improved, reaching its peak when the number of heads was 20. This improvement can be attributed to the average sentence length in the BQ dataset being 11.9 tokens, and with a model hidden dimension of 300, having 20 attention heads ensures that each attention unit has a dimension of 15, which is sufficient to capture most of the sentence patterns in the dataset. However, as the number of heads continues to increase beyond this point, the dimension per attention unit decreases, leading to a low-rank bottleneck problem [36]. This ultimately results in a decline in model performance.

5. Conclusions

This paper presents a sentence-matching model called the multi-granularity contextual key semantic interaction model (MCKI). The model incorporates pre-trained language model embeddings and BiGRU to capture contextual information effectively. It employs a semantic feature extraction mechanism that combines a multi-scale CNN and multi-level CNN to extract semantic contextual information at different granularities and levels. The introduction of a key semantic alignment mechanism by integrating multi-head self-attention and cross-attention mechanisms emphasizes key semantic information between sentence pairs. By combining three interaction methods, the model comprehensively considers both the similarity and difference between sentences, resulting in enhanced matching performance.

The effectiveness of the proposed model and its components was evaluated on public datasets. The findings demonstrated that in open-domain datasets, matching key semantic elements effectively eliminates irrelevant information, leading to significant improvements, especially when sentence pairs contain high levels of noise. Moreover, the proposed methodology in this study achieved comparable performance to large-scale pre-trained models by utilizing lightweight pre-training language models, which proves effective when resources are limited. However, in closed-domain datasets where sentence pairs share the same theme, considering semantic nuances at different levels and granularities aids the model in distinguishing overlapping terminologies within the subject. Nevertheless, the effectiveness of model enhancements remains limited in such scenarios. Additionally, marginal effects are observed when applying improvements to more advanced pre-trained models. The limitations and shortcomings of the current research should be acknowledged. This study primarily focused on combining existing models without thoroughly exploring improvements to these models. Future research will aim to enhance known models in order to achieve better performance. Furthermore, there is a consideration to integrate graph neural networks, which can leverage both the key semantic information and structural aspects of sentences to optimize matching performance.