A Short-Text Similarity Model Combining Semantic and Syntactic Information

Abstract

As one of the prominent research directions in the field of natural language processing (NLP), short-text similarity has been widely used in search recommendation and question-and-answer systems. Most of the existing short textual similarity models focus on considering semantic similarity while overlooking the importance of syntactic similarity. In this paper, we first propose an enhanced knowledge language representation model based on graph convolutional networks (KEBERT-GCN), which effectively uses fine-grained word relations in the knowledge base to assess semantic similarity and model the relationship between knowledge structure and text structure. To fully leverage the syntactic information of sentences, we also propose a computational model of constituency parse trees based on tree kernels (CPT-TK), which combines syntactic information, semantic features, and attentional weighting mechanisms to evaluate syntactic similarity. Finally, we propose a comprehensive model that integrates both semantic and syntactic information to comprehensively evaluate short-text similarity. The experimental results demonstrate that our proposed short-text similarity model outperforms the models proposed in recent years, achieving a Pearson correlation coefficient of 0.8805 on the STS-B dataset.

1. Introduction

Solving the similarities between two short texts is known as short-text similarity. It is a special form of text-matching task or textual entailment task that can be used in various downstream applications of natural language processing, for example, information retrieval [1], question answering [2], and document classification [3].

By learning linguistic expertise from massive amounts of textual data and expressing text (words and sentences) as continuously trainable embedding vectors, deep learning algorithms are better able to accomplish this [4]. Among others, BERT [5] has demonstrated excellent performance by unsupervised pre-training [6] on a huge corpus [7] and learning contextualized word representations through its multi-headed attention mechanism. However, while ordinary people can only understand words based on context when reading a domain-specific text, experts can reason with relevant domain knowledge. External knowledge plays a particularly crucial role for pre-trained language models to understand relevant domain information.

Pre-trained language models are increasingly being enhanced by external knowledge [8]. External knowledge is added to the original BERT model, thus supplementing the short text with more semantic information as well as fine-grained word relationship information. Examples include SemBERT [9] and KIM [10], which can consistently improve semantic text-matching performance. To address the inconsistency of the embedding vector space between words in short texts and entities in the knowledge graph (KG), Baidu ENRIE 2.0 [11] uses two knowledge strategies, phrase-level strategy, and entity-level strategy, to implicitly learn the prior knowledge of phrases and entities during the training process. Considering the possible knowledge noise problem caused by introducing external knowledge bases, K-BERT [12] loads pre-trained BERT models and introduces soft location and visible matrices to limit the influence of knowledge to overcome knowledge noise, and the better integration of external knowledge with the model. The above study demonstrates that introducing external knowledge allows the model to better understand the short text and learn additional external information.

However, even though their studies are aware of the importance of external knowledge for models to understand short texts in relevant domains, integrating external knowledge into pre-trained language models, they also ignore some issues: (1) How to effectively use the information in the knowledge base for short-text similarity computation; (2) How to model the relationship between knowledge structure and text structure. To overcome these challenges, in this paper, we propose an enhanced knowledge language representation model based on graph convolutional networks (KEBERT-GCN), where we directly form a new matrix by multiplying the external knowledge base construction matrix with the attention matrix Hadamard in the multi-headed attention mechanism of BERT and use it as the adjacency matrix of GCN [13] to effectively use the fine-grained word relationship information in the knowledge base to avoid knowledge noise. A graph is constructed with each token in the BERT containing semantic information as a node, and the relational features between each token are represented by the edges of the graph. Finally, to capture the relational features of knowledge structure and text structure, we extract the relational information between tokens using two layers of GCN to obtain the representation vector of sentences. In addition, we embed the relative position into the node representation to make the GCN position-aware [14], and experimental results on four popular short text similarity datasets show that, with the BERT model, the proposed KEBERT-GCN model can consistently improve the performance of short-text semantic similarity and performs significantly better than BERT.

Most of the current research focuses on short texts using language models to model the semantic similarity of short texts and does not fully utilize the sentences’ syntactic information. It has been previously shown that syntactic information is beneficial in improving short-text similarity performance [15,16]. Therefore, we also propose a calculation model for constituency parse trees based on tree kernels (CPT-TK), which combines syntactic information, semantic features, and attentional weighting mechanisms, as a way to obtain syntactic structural information and judge the syntactic similarity of short texts.

In summary, the main contributions of this paper are the following:

(1) An enhanced knowledge-based linguistic representation model based on graph convolutional networks, namely the KEBERT-GCN model, is proposed, which incorporates external knowledge in BERT, effectively uses fine-grained word relationship information in the knowledge base to avoid knowledge noise, and captures the relational features of knowledge structure and text structure to judge the semantic similarity of sentences. Experiments show that KEBERT-GCN significantly outperforms BERT on four publicly available short-text similarity datasets.

(2) A syntactic similarity modeling structure, CPT-TK, is proposed that combines syntactic information, semantic features, and attentional weighting mechanisms as a way to judge the syntactic similarity of sentences.

(3) Making full use of lexical, syntactic, and knowledge information, we propose a short-text similarity model that considers both semantic similarity and syntactic similarity, and our experimental results show that our model has better performance in computing short-text similarity on the STS-B dataset.

2. Related Work

2.1. Semantic Similarity

Semantic similarity is one of the important tasks in natural language processing, which aims to measure the semantic distance between given content blocks (words, sentences, or short texts). It plays an important role in various natural language processing tasks, such as text summarization [17], machine translation [18], keyword extraction [19], and question and answer [20].

The generation of pre-trained language models has led to the rapid development of natural language processing, which has achieved excellent results on semantic similarity tasks. A series of pre-trained language models such as ELMo [21], GPT [22], BERT [5], and XLNet [23] can be applied to semantic similarity tasks by fine-grained word embeddings, which have achieved excellent results. Among them, BERT is the most outstanding pre-trained language model, and AlBERT [24], RoBERTa [25], Sentence-BERT [26], and DeBERTa [27] are all variant models generated based on it, and they have achieved better semantic similarity performance by fine-tuning or modifying BERT.

Although fine-tuned language representation models have all been very successful, when the average person reads a domain-specific text they can only understand words in context and cannot reason using relevant domain knowledge. The integration of knowledge information is especially important for pre-trained models to understand short texts. To address the problem of “low information content of short text” in the semantic similarity calculation of short text, many researchers have expanded the information content of short text by introducing external knowledge to improve the semantic matching performance of short text. Tsinghua ERNIE [28] is a pioneer in this direction, based on a pre-trained model which incorporates entity information but ignores the relationships between entities. SemBERT [9] adds knowledge to the original BERT model by adding explicit contextual semantics to the pre-trained semantic role annotations. They introduce external knowledge but do not make good use of the fine-grained word relationship information in the knowledge base, WordNet (https://wordnet.princeton.edu/ accessed on 23 December 2022) [29] as a semantic concept knowledge base. It does not just arrange words in alphabetical order but also forms a ‘network of words’ based on their meaning. It is a semantic network covering a wide range of English words. The main relationship between words is that of synonyms; for example, car and automobile are synonyms. These synonyms are grouped into unordered sets of synonyms. Words with the same meaning are contained in the same concept node (synonym set), and the different synonym sets are connected by various semantic relations, forming a topological network structure. Synonymy relationships are the most fundamental semantic relationships in WordNet, which exist symmetrically between words, and synonym sets are made up of several synonyms linked with a pointer to synonymy relationships. In addition to synonymy, common semantic relations include subordination, antonymy, and part-whole relations, the most common of which is subordination, e.g., tree is the subordinate word for plant, which is the superordinate word for tree. In our work, we primarily use WordNet’s knowledge of synonymy and the topological network structure formed by the various semantic relations. We use them to construct a word similarity matrix, which allows the model to better capture the fine-grained word relations in the short text and to efficiently perform short-text semantic similarity calculations.

The problem of knowledge noise (some words are not important to the semantic understanding of the whole sentence, and too much knowledge admixture will make the whole sentence deviate from the meaning of the original text) may arise when retrieving the corresponding words in the short text from the knowledge base, which can interfere with the model judgment and reduce model performance. Our proposed KEBERT-GCN model, to avoid knowledge noise, extracts the fine-grained word relationship information of the short text in the knowledge base and constructs a word similarity matrix, which is combined with BERT’s attention matrix to enhance the model’s attention to semantically similar word pairs.

The KIM model [10] is based on the baseline model ESIM [30] and uses word relations to assist in judging the weight of attention, which improves the performance of ESIM on the semantic similarity task by influencing the weights so that the model imposes different emphasis on different words to introduce external knowledge. Although KIM is concerned with integrating external knowledge to capture word relationship information, it ignores the relational features between knowledge structure and text structure. The GCN [13] model can capture the relational features between nodes, and many studies have shown that it can effectively encode the contextual information of the input sentences [31,32]. GCN focuses on the word similarity matrix constructed with WordNet to extract word synonymy information in the short text, allowing the model to pay more attention to semantically similar word pairs. Whereas the graph is topological and cannot perceive the sequential information learned about the words, in many works, GCN relative position encoding (sequential encoding, for the positions of all nodes, in order, starting from 0) proved to have better results than BERT’s absolute position encoding (encoding that focuses on the relative positions between nodes, for each node, there is a relative position encoding concerning other nodes) [27,33,34] and is more conducive to models that capture contextual information about words.

2.2. Syntactic Similarity

Previously popular neural network approaches implemented sentence embeddings. Mikolov proposed Doc2vec [35] by referencing the word2vec [36] idea to learn the semantic representation of the whole sentence. Tien et al. [37] combined LSTM and CNN models to form sentence embeddings using pre-trained word embeddings. Tai et al. [38] proposed a Tree-LSTM model to evaluate sentence similarity. Most of them represent the short text as a vector by learning semantic information in the short text, without mining the syntactic information embedded in the short text. Also, most neural network-based sentence representation models have equal processing for each word in a sentence [35,39]. This is not in line with the way people normally read and understand sentences. Therefore, we take advantage of the attentional weighting mechanism [40] and consider using smooth inverse frequency (SIF) [41] to focus on the more important words in a sentence by assigning different weights to words. We also use pre-trained word embeddings, so that our model can avoid time-consuming learning and training.

We construct a selection parse tree of short text to obtain the syntactic structure information of short text, calculate the similarity between structure trees by calculating the number of common substructures between two trees $T_1$ and $T_2$ by the tree kernel method (a method to quantify the similarity between tree-structured data), and combine the attention weight mechanism with pre-trained word embeddings to calculate the syntactic similarity of two short texts. Existing tree kernel methods are classified into three types: subtree [42] (ST), subset tree [43] (SST), and partial tree [44] (PT). A subtree (ST) kernel captures the structure of all descendants including the target root node to the leaf nodes. In contrast, subset trees (SSTs) allow internal subtrees without any leaf nodes. The partial tree (PT) further relaxes the constraints of the SST, resulting in a more flexible substructure, which can have non-terminal leaf nodes. The tree kernel functions on $T_1$ and $T_2$ are generally defined as:

$$TK\left(T_1,T_2\right)=\sum _{nod{e}_1\in Se{t}_{T_1}}\sum _{nod{e}_2\in Se{t}_{T_2}}\Delta \left(nod{e}_1,nod{e}_2\right)$$

(1)

where $Se{t}_{T_1}$ and $Se{t}_{T_2}$ denote the set of nodes in the structure trees $T_1$ and $T_2$, respectively, and $nod{e}_1$ and $nod{e}_2$ denote the nodes in the two trees. Different $\Delta (·)$ functions represent different kernel spaces, and therefore different tree kernels can be generated.

Therefore, we also propose a syntactic similarity modeling structure, CPT-TK, which combines syntactic information, semantic features, and attentional weighting mechanisms to discriminate the syntactic similarity of sentences.

3. Methodology

3.1. Semantic Similarity Model Framework

In this section, we first construct the word similarity matrix, and then introduce the multi-headed attention in BERT and how to incorporate external knowledge into the BERT model, and finally, we model the relationship between the knowledge structure and text structure of the matrix obtained after BERT using two layers of GCN. As shown in Figure 1, our model (KEBERT-GCN) consists of three parts: the Encoder module, the Transformer module, and the GCN module. In the input phase, a (CLS) token is added to the first sentence, and a (SEP) token is added in the middle of two sentences and at the end of another sentence. In the Encoder module, we have the same model as BERT. The sum of the three parts of token, position, and segmental encoding is used. The word similarity matrix constructed by an external knowledge base is added to the Transformer module of BERT to obtain the score matrix and the token embedding carrying semantic information and inputting them into the two-layer GCN module to obtain a vector representation of the two sentences.

3.1.1. Construction of the Word Similarity Matrix

We build the word similarity matrix with the help of external knowledge bases. This paper first uses the knowledge base WordNet to build the word similarity matrix. Of course, our model is generic and compatible with other knowledge bases such as Wikipedia (https://en.wikipedia.org accessed on 22 December 2022) and Probase (Available online at https://www.microsoft.com/en-us/research/project/probase/ accessed on 23 December 2022) [45], and we can also use the word/concept similarity knowledge from these knowledge bases to construct word similarity matrices.

Given two short texts (i.e., sentences) $S_1$ = ($W_1$, $W_1$…$W_i$…$W_L\left(S_1\right)$) and $S_2$ = ($W_1$, $W_2$…$W_i$…$W_L\left(S_2\right)$), we construct the word similarity matrix S. We calculate the value of each element of S based on the semantic relationships in the WordNet knowledge base. For two words $W_a$ and $W_b$, if the pair of words are identical or synonymous in WordNet, the element value $S_{ab}$ is set to 1; if they are not synonyms, we calculate the elemental value $S_{ab}$ based on the method proposed by Wu-Palmer [46] for calculating word similarity based on topological distance in WordNet (at this point, $S_{ab}$ is a value from 0 to 1); if one or both of the words in the pair are not found in WordNet, we set the two words with similarity values set to 0.

We preprocessed all the datasets and calculated the word similarity matrix for each sentence pair in the dataset. This was used to enhance BERT’s attention to semantically similar word pairs by constructing the matrix S.

Figure 2 shows the heat map of the two-word similarity matrices. Their sentence pairs are derived from the STS-B dataset (each sentence pair has a rating from 0 to 5, with the higher ratings indicating that two sentence pairs are more similar). The first pair (“A man is cycling” and “A man is talking”) has a true similarity score of 0.6, and the second pair (“A deer jumps a fence” and “A deer is jumping over a fence”) has a true similarity score of 5.0.

3.1.2. External Knowledge Integration into BERT for Multi-Headed Attention

In this section, we implement a fine-tuning of the BERT-base model. The external knowledge matrix constructed above is added to the multi-headed note of BERT. In the encoding phase, our model is the same as BERT. The sum of the three components of tagging, location, and segmentation encoding is used. In the Transformer phase, our approach is also similar to BERT. Multi-head attention is a process of mapping the query vector, key vector, and value vector to the output vector. First, they undergo a linear transformation, and they are then fed into scaled dot-product attention (this is performed h times, which is called multi-head, each time counting one head, and the parameters W are different for each linear transformation of Q, K, and V), then the result of the h times scaled dot-product attention is stitched together, and the value obtained by another linear transformation is used as the multi-head. The result of the attention:

$$\begin{array}{c}hea{d}_i=Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)\\ MultiHead(Q,K,V)=Concat\left(hea{d}_1,\dots ,hea{d}_h\right)W^O\end{array}$$

(2)

where $W_i^Q$, $W_i^k$, and $W_i^V$ are the parameter matrices corresponding to the i-th attention head query, key, and value, respectively, and $W^O$ are the weight matrices when the h attention heads are spliced.

The BERT attention is computed using the scaled dot-product:

$$\begin{array}{c}Scores=Q{K}^T+MASK\\ Attention(Q,K,V)=softmax\left(\frac{\mathrm{Scores}}{\sqrt{d_k}}\right)V\end{array}$$

(3)

where MASK is the masked language model matrix and $d_k$ serves as a moderator to keep the inner product from being too large.

Unlike BERT, our model is injected with external knowledge and uses the word similarity matrix S to compute the Hadamard product so that the model focuses more on the word pairs with higher similarity in the sentence pairs, adjusted as follows:

$$\begin{array}{c}Scores=Q{K}^T\ast S+MASK\\ \mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Scores}{\sqrt{d_k}}\right)V\end{array}$$

(4)

3.1.3. Scores Matrix Integration into the GCN

The GCN is a widely used architecture for encoding graph information, where the information at each node in each GCN layer is communicated to the neighboring nodes through connections between them. The effectiveness of the GCN model for encoding contextual information on input sentence graphs was demonstrated by many previous studies. Using the GCN model, the relationships between words in a text can be modeled as a graph structure, enabling the better capture of both local and global information in the text.

Typically, graphs in the standard GCN model are constructed from word dependencies, represented by adjacency matrices $A_{i,j}$. Based on the adjacency matrices $A_{i,j}$, for a given node i, the l-th GCN layer collects the relevant information (semantic information, syntactic information, etc.) carried by its contextual words in $A_{i,j}$ and computes the output representation $h_i^{\left(l\right)}$ for i by:

$$h_i^{\left(l\right)}=Relu\left(\sum _{j=1}^n{A}_{i,j}\left(W^{\left(l\right)}·h_j^{(l-1)}+b^{\left(l\right)}\right)\right)$$

(5)

where $h_j^{(l-1)}$ denotes the output representation of node j in the $(l-1)$-th GCN layer, $W^{\left(l\right)}$ and $b^{\left(l\right)}$ are the trainable matrix and bias of the l-th GCN layer, respectively, and Relu is the activation function.

Before applying the GCN, for each sentence, we first go through the BERT to obtain the embedding $h_i$ of each token carrying semantic information, and next, we input $h_i$ into the GCN model. Unlike the standard GCN model, we use the score matrix scores as the adjacency matrix $A_{i,j}$, and each token as each node in the GCN, and we add relative position encoding [14] to the GCN so that it learns the relative position information of the token. Based on the fraction matrix S, for a particular node i, we specify the first GCN layer as follows:

$$h_i^1=Relu(\frac{1}{d_i}\sum _{j=1}^n{A}_{i,j}(h_j+h_P^j)W^{\left(1\right)}+b^{\left(1\right)})$$

(6)

where $d_i$ represents the degree of node i, here divided by $d_i$ for normalization. $A_{i,j}$ is the fraction matrix, $h_j$ is the Token embedding of node j, and $h_p^j$ represents the relative position representation of i and j.

Similar to the first GCN layer, we specify the second GCN layer as follows:

$$h_i^2=Relu(\frac{1}{d_i}\sum _{j=1}^n{A}_{i,j}({h_j}^1+h_P^j)W^{\left(2\right)}+b^{\left(2\right)})$$

(7)

where $h_j^1$ represents the input representation of node j at the first layer.

After two layers of GCN, the vector $h_i^2$ of each token is thus obtained, and then the vector of sentences is obtained by averaging pooling as:

$$h_s=\frac{1}{m}\sum _{i=1}^m{h}_i^2$$

(8)

where m is the total number of tokens, i.e., the length of the sequence of sentences.

We present the proposed method using “A man is cycling” and “A man is talking” as examples.

• The dataset is preprocessed and the word similarity matrix S of sentences $S_1$ and $S_2$ is constructed according to Section 3.1.1;
• In the BERT embedding stage, a (CLS) token is added to the first part of the first sentence $S_1$, and an (SEP) token is added to the middle of the two sentences and the end of the last sentence. The sum of the three parts. In the Transformer stage, external knowledge is incorporated into the BERT multi-headed attention, and the Hadamard product is calculated using $Q{K}^T$ and S to obtain the score matrix scores;
• The score matrix is incorporated into the GCN, using the score matrix scores to act as the adjacency matrix in the standard GCN model. Based on the score matrix, the token embeddings of sentences $S_1$ and $S_2$ obtained by BERT are used as input to the two-layer GCN, and the vector representation of each token is obtained after the two-layer GCN, and then the vectors of the two sentences are obtained separately by averaging pooling. Then, it is conventional to pick up a fully connected layer to get Logits, then a Softmax layer to obtain probabilities, after which the loss is calculated based on the true labels.

3.1.4. Setting of Loss Function

For the classification and regression tasks, we experimented with the following structures and objective functions, respectively.

Classification objective function: To solve the problem of sample imbalance in target detection, we use focal loss, which is a modification of the standard cross-entropy loss function and adjusts the category weights, easy-to-classify sample weights, and hard-to-classify sample weights in the loss function to improve the classification accuracy of the model. The cross-entropy loss function is shown in the following figure:

$$L=\left\{\begin{array}{cc}-ln{y}^{\prime },\hfill & y=1\\ -ln\left(1-y^{\prime }\right),\hfill & y=0\end{array}\right.$$

(9)

where: y denotes the label of the true sample and $y^{\prime }$ denotes the predicted value.

The focal loss function introduces a category weight factor $\alpha$ to adjust the weight size of samples of different categories, $\alpha$∈ (0, 1). To solve the problem of easy and hard-to-distinguish samples, focal loss also adds a factor $\gamma$$(\gamma >0)$ to the loss function to make the algorithm pay more attention to the hard-to-distinguish samples. The loss function after adding the adjustment factor is:

$$L_{\mathrm{FI}}=\left\{\begin{array}{cc}-\alpha {\left(1-y^{\prime }\right)}^{\gamma }ln{y}^{\prime },\hfill & y=1\\ -(1-\alpha )y^{\gamma }ln\left(1-y^{\prime }\right),\hfill & y=0\end{array}\right.$$

(10)

It further improves the accuracy of the classification model by adjusting the values of $\alpha$ and $\gamma$ to change the sample weights and make the model more focused on minority and hard-to-classify samples, improving the imbalance of sample categories at the algorithmic level.

The regression objective function. The similarity between the two sentence embeddings was calculated. We used the mean squared error loss as the objective function:

$$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n{\left(Y_i-{\widehat{Y}}_i\right)}^2$$

(11)

where n denotes the number of samples, and $Y_i$ and ${\widehat{Y}}_i$ predicted labels denote the true and predicted labels, respectively. In particular, we calculated the final loss function as:

$${\mathcal{L}}_{FI}={\mathcal{L}}_{bert-fl}+{\mathcal{L}}_{gcn-fl}$$

(12)

Due to the large number of parameters in the GCN and BERT models, BERT is also involved in the updating of the loss function through the (CLS) token to allow the model to learn better. Secondly, BERT is in the middle of the model, and the output of BERT is also supervised with labels to prevent overfitting and provide better node input to the GCN, which allows the model to learn semantic information better.

3.2. Syntactic Similarity Model Framework

The above methods compute the semantic similarity between short text and achieve good performance. However, the syntactic information of the sentence is ignored. In the case of “She’s very beautiful” and “Is she very beautiful”, for example, the two texts are similar according to the above calculation method. However, according to our human understanding, the two texts are not exactly similar. Therefore, we also propose a computational model of constituency parse trees based on tree kernels (CPT-PK) to judge the syntactic similarity of short texts.

3.2.1. Constructing the Constituency Parse Trees of the Short Text

First, we construct the CPT for short text. In this paper, we use previous work [47] to construct the CPT, which will be very simple to deploy, while featuring faster and easier parallelization to generate constituency short-text parse trees, obtaining new state-of-the-art results on Penn Treebank. Selected parse trees $T_1$ and $T_2$ for sentences $S_1$ and $S_2$ were constructed using this method, as shown in Figure 3.

3.2.2. Similarity between CPTs Is Calculated Using the Tree Kernel Method

Next, the syntactic similarity was calculated using the tree kernel method to calculate the similarity between CPTs as the syntactic similarity. Most tree kernels only consider syntactic information, and few consider both semantic information and attentional weighting. Our model cleverly combines syntactic information, semantic information, and attentional weighting mechanisms, absorbing the advantages of various techniques.

Our model uses PT kernels to compute syntactic similarity. Similarly to other tree kernel calculation methods, the PT tree kernel function PTK between trees $T_1$ and $T_2$ is defined as Equation (13):

$$PTK\left(T_1,T_2\right)=\sum _{nod{e}_1\in Se{t}_{T_1}}\sum _{nod{e}_2\in Se{t}_{T_2}}\Delta \left(nod{e}_1,nod{e}_2\right)$$

(13)

where $Se{t}_{T_1}$ and $Se{t}_{T_2}$ are the sets of nodes in trees $T_1$ and $T_2$, respectively, $nod{e}_1$ and $nod{e}_2$ are the sets of nodes in trees $T_1$ and $T_2$, respectively, and $\Delta$ ($nod{e}_1$,$nod{e}_2$) is the number of common segments in the tree kernel rooted at $nod{e}_1$ and $nod{e}_2$, respectively.

For the PT tree kernel, we define $\Delta$ ($nod{e}_1$,$nod{e}_2$) as in Equation (14):

$$\Delta \left(nod{e}_1,nod{e}_2\right)=\left\{\begin{array}{c}similarity\left(vecto{r}_1,vecto{r}_2\right)\times weigh{t}_1\times weigh{t}_2,nod{e}_1,nod{e}_2\in \mathrm{leafnodes}\hfill \\ \alpha \left({\beta }^2+{\displaystyle \sum _{l=1}^{l_{min}}}{\Delta }_l\left(c_{nod{e}_1},c_{nod{e}_2}\right)\right),nod{e}_1=nod{e}_2 \mathrm{and} nod{e}_1,nod{e}_2\in \mathrm{non}-\mathrm{leafnodes}\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(14)

In the above equation, $\alpha$ and $\beta$ are two decay factors: $\alpha$ is the height of the tree, $\beta$ is the length of the subsequence; $c_{nod{e}_1}$ and $c_{nod{e}_2}$ are the lists of child nodes of $nod{e}_1$ and $nod{e}_2$, respectively; $l_{min}$ is the length of the list of smallest child nodes between $c_{nod{e}_1}$ and $c_{nod{e}_2}$; ${\Delta }_l(·)$ denotes the number of common subsequences of length l in the list of child nodes; $vecto{r}_1$ and $vecto{r}_2$ are the word vectors of $nod{e}_1$ and $nod{e}_2$, respectively; $weigh{t}_1$ and $weigh{t}_2$ represent the weights of $nod{e}_1$ and $nod{e}_2$, respectively; $similarity(vecto{r}_1,vecto{r}_2)$ is the cosine similarity function of the two vectors; $leafnodes$ and $non-leafnodes$ denote the leaf nodes and non-leaf nodes in the tree. In this paper, the weights are calculated by referring to the calculation method of SIF, and the specific formula is shown in Equation (15):

$$SIF\left(\mathrm{node}\right)=\frac{\phi }{\phi +p\left(\mathrm{node}\right)}$$

(15)

where $\phi$ is the smoothing parameter in SIF and $p\left(node\right)$ is the word frequency of the words in the leaf nodes.

To better understand the above equation, we solve for ${\Delta }_l(·)$ by constructing a ’recursive’ function as follows. This is shown in Equation (16):

$${\Delta }_l\left(c_{nod{e}_1},c_{nod{e}_2}\right)=\Delta (p,q)\sum _{i=1}\left|n_1\right|\sum _{r=1}\left|n_2\right|{\beta }^{\left|n_1\right|-i+\left|n_2\right|-r}\times {\Delta }_{l-1}\left(n_1[1:i],n_2[1:r]\right)$$

(16)

In the above equation, p and q are the last child nodes, $\Delta (p,q)$ is calculated by Equation (13); and $n_1$ and $n_2$ are the child nodes of $c_{nod{e}_1}$ and $c_{nod{e}_2}$, respectively, and $|n_1|$ and $|n_2|$ denote the lengths of $n_1$ and $n_2$, respectively; $n_1[1:i]$ and $n_2[1:r]$ denote the subsequence from 1 to i in $n_1$ and from 1 to r in $n_2$, respectively; and ${\Delta }_{l-1}(·)$ is recursively calculated using Equation (13) and stops when the leaf node is reached.

In summary, the syntactic similarity of the two sentences is calculated. Firstly, the CPTs of the two sentences are constructed. Secondly, the similarity of each CPT is calculated according to Equation (13). Finally, all similarity values are summed and normalized to obtain the final syntactic similarity.

3.3. Short Textual Similarity Model Framework

In this paper, the similarity of short texts is calculated by fusing the semantic similarity model and syntactic similarity model, and the formula is shown in Equation (17).

$$\mathrm{Sentencesim}\left(S_1,S_2\right)=\theta \ast S\mathrm{emanticsim}+\left(1-\theta \right)\ast \mathrm{Syntacticsim}$$

(17)

where Semanticsim denotes the semantic similarity of the sentence pair, calculated by the model described in Section 3.1; Syntacticsim denotes the syntactic similarity of the sentence pair, calculated by the model described in Section 3.2; and $\theta$ denotes that the weights belong to $[0,1]$.

4. Experiments

In the experimental section, we first describe the dataset and experimental setup, and then describe the baseline model used in the experiment and some comparison methods, and then designed an ablation experiment and a comparison experiment to effectively evaluate our proposed model, and finally analyze the experimental results and discussed the effects of the two parameters.

4.1. Dataset and Experimental Setup

4.1.1. Dataset

In our experiments, we used four popular publicly available benchmark short-text similarity datasets: MRPC, STS-B, QQP, and SICK.

• MRPC (the dataset can be downloaded from https://www.microsoft.com/en-us/download/details.aspx?id=52398 accessed on 11 November 2022) [48]: Microsoft Research’s standard dataset for paraphrase recognition, which is extracted from news sources on the web and also carries a corpus of human-annotated sentence pairs, where each text pair is manually binary determined to be similar or not, explains whether the two sentences are paraphrased. It contains a total of 5801 text pairs, 4076 training pairs, and 1725 test pairs. In this experiment, we segmented 10% of the training data according to the GLUE [49] criterion as the validation set. The accuracy and F1 values are reported in the experiments.
• QQP (The dataset can be downloaded from https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs accessed on 11 November 2022) [50]: is a collection of question pairs from the community question-and-answer site Quora. The task is to determine whether a pair of questions is semantically equivalent. The QQP dataset consists of 404,350 question pairs, and each question pair is annotated with a binary value indicating whether the two questions are paraphrased or not. We used the partitioning approach in the article in [51] to randomly select 5000 annotated and 5000 non-annotated as the validation set, and externally 5000 annotated and 5000 non-annotated as the test set. We keep the remaining instances for training. As with MRPC, we report accuracy and F1 values.
• STS-B (the dataset can be downloaded from http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark accessed on 11 November 2022) [52]: text from image captions, news headlines, and user forums. This includes 5749 training data, 1500 development data, and 1379 test data. Each sentence pair was annotated by humans with a similarity score of 0–5 (a floating point number greater than or equal to 0 and less than or equal to 5. A score of 0 indicates that the two sentences are unrelated, while a score of 5 indicates that the two sentences are very related. Each score is the average of 10 scores marked by 10 different human annotators. We report the Pearson correlation coefficient (PCC) and Spearman correlation coefficient (SCC) as evaluation metrics.
• SICK (the dataset can be downloaded from http://clic.cimec.unitn.it/composes/sick.html accessed on 11 November 2022) [53]: contains 10,000 English sentence pairs from two pre-existing paraphrase datasets: an 8k imageFlickrbuilt dataset, and the SEMEVAL-2012 semantic text similarity video description dataset. It consists of 4439 training data, 495 development data, and 4906 test data. Each sentence pair was annotated by humans as 1–5 (a floating point number greater than or equal to 0 and less than or equal to 5). A score of 0 indicates that the two sentences are unrelated, while a score of 5 indicates that the two sentences are significantly related. Again, we report Pearson correlation coefficient (PCC) and Spearman correlation coefficient (PCC).

4.1.2. Baseline as well as Comparison Methods

To evaluate our proposed approach, we first designed an ablation experiment to evaluate the semantic similarity model, and we compared our proposed model with the state-of-the-art pre-trained model BERT-base. In contrast to the baseline model BERT-base, we used the same transformer model setup as BERT-base. The base model contains 12 layers, 12 self-attentive heads, and 768 dimensions of hidden dimensions. It was subjected to ablation experiments with our proposed two models without external knowledge (BERT-GCN) and with external knowledge (KEBERT-GCN).

We then designed a comparison experiment in which we compared our approach with the following methods.

• Neural network models include:
  • Skip-Thought [39]: a sentence-embedding model for predicting contextual sentences from central sentences.
  • Constituency Tree-LSTM [38]: a model that analyzes the syntactic information of sentences and calculates the similarity of sentences using constructed constituency parse trees.
  • SIF [41]: a sentence-embedding model considering word frequency and attention weighting mechanisms.
• Pre-trained neural network models include:
  • SemBERT [9]: a model for language representation using explicit semantic role annotation.
  • Tsinghua ENRIE [28]: a pre-trained language model based on BERT that introduces a priori knowledge of named entities in the knowledge graph.
  • Baidu ENRIE2.0 [11]: a sustainable learning framework with external knowledge and multi-task training.
  • CF-BERT [54]: a language model that introduces a weighting factor by emphasizing syntactic relations.
  • FEFS3C [55]: a joint method for computing frame and item-focused sentence similarity.
  • PromptBERT [56]: a cue-based sentence embedding method using denoising techniques for unsupervised training targets;

4.1.3. Other Settings and Implementation Details

We implemented the entire model on the PyTorch framework and trained the model on an NVIDIA GeForce RTX 3090 graphics card. We used the Adam optimizer and experimented with parameter settings on four datasets as shown in

Table 1. The main parameter settings on the four datasets.

Parameter	Learning Rate	Epochs	Max Length	Batch Size
MRPC	$2\times {10}^{-5}$	3	128	32
QQP	$3\times {10}^{-5}$	3	75	16
STS-B	$5\times {10}^{-5}$	3	128	64
SICK	$5\times {10}^{-5}$	3	80	64

.

In the semantic similarity model, we considered SemBERT, Tsinghua ENRIE, Baidu ENRIE2.0, CF-BERT, FEFS3C, and PromptBERT for comparison experiments with our model, which are all BERT-base-based variants in the experiments for a fair comparison.

In the syntactic similarity model, we used word vectors provided by paragm-sl999 (PSL) (https://www.cs.cmu.edu/~jwieting/ accessed on 10 January 2023). The wiki (https://dumps.wikimedia.org/enwiki/latest/ accessed on 10 January 2023) dataset was used to compute the SIF, and the two decay factors of Equation (14) did not have a significant impact on the performance of the model, and we set them to 0.1 by default.

4.2. Experimental Results and Analysis

To evaluate the performance of the model KEBERT-GCN, we performed ablation experiments on four publicly available datasets, and the results are shown in

Table 2. Ablation experiments on four datasets.

Model\Datasets	MRPC	SICK	STS-B	QQP
BERT	84.80/88.90	87.83/81.40	87.10/85.80	91.93/91.70
BERT-GCN (ours)	87.01/90.33	87.86/82.78	87.39/86.62	92.74/92.49
KEBERT-GCN (ours)	87.99/90.58	88.33/82.83	87.87/87.07	94.75/94.60

For public comparison, each performance value in the table is the average of five runs with different random seeds. The best average performance in each column is indicated in bold. The evaluation indicators of MRPC and QQP are accuracy: (accuracy)/F1 (macro F1 value); the evaluation indicators of SICK and STS-B are Pearson correlation coefficient (PCC)/Spearman correlation coefficient(SCC).

.

We conducted ablation experiments with our proposed two models without external knowledge (BERT-GCN) and with external knowledge (KEBERT-GCN). Where the fraction matrix injected with external knowledge is constructed as shown in Equation (4) in Section 3.1.2, while the matrix without external knowledge is the attention matrix of BERT, which is constructed as shown in Equation (3) in Section 3.1.2.

The experiments show that, compared to BERT, BERT-GCN outperforms BERT on all experimental datasets because the attention matrix itself contains semantic, syntactic, and inter-token contextual information. Compared with BERT-GCN, KEBERT-GCN constructs word similarity matrices through external knowledge to enhance the model’s performance on the GCN model with the addition of relative position information models the relationship between tokens and the synonymy information obtained through external knowledge as a graph structure, thus better capturing the local information, fine-grained word relationship information and word position information in the short text, so that it performs better on all four datasets, as well as demonstrates external knowledge. This also demonstrates that external knowledge contributes to the understanding of short textbooks by the pre-trained model.

In conclusion, our proposed models BERT-GCN and KEBERT-GCN show a better performance than the state-of-the-art BERT model on all four datasets.

We designed a comparison experiment to compare our proposed model with models that have performed relatively well in recent years. We experimented on the STS-B dataset and the experimental results are shown in

Table 3. Results from different models on the STS-B dataset.

Model	PCC	SCC
Skip-Thought	71.80	69.70
Constituency Tree-LSTM	71.90	-
SIF	72.00	-
BERT	87.10	85.80
$SemBER{T}_{BASE}$	87.30	-
$TsinghuaENRI{E}_{BASE}$	-	83.20
$BaiduENRI{E}_{BASE}$	87.60	86.50
$CF-BER{T}_{BASE}$	87.58	86.29
$FEFS3C_{BASE}$	-	86.78
$PromptBER{T}_{BASE}$	84.56	-
KEBERT-GCN (ours)	87.74	87.07
CPT-PK (ours)	76.42	75.31
KEBERT-GCN+CPT-PK (ours)	88.05	87.30

KEBERT-GCN and CPT-PK denote our proposed semantic similarity and syntactic similarity models, respectively, and KEBERT-GCN+CPT-PK is our proposed short textual similarity model. The results of each experiment in the table are the average of five runs using different random seeds. The best average performance in each column is indicated in bold. PCC: Pearson correlation coefficient; SCC: Spearman correlation coefficient.

.

The experiments showed that the pre-trained neural network model significantly outperformed the neural network models Skip-Thought, Constituency Tree-LSTM and SIF on the short text similarity task. The idea of the Skip-Thought model, which predicts central sentence contextual information with the help of the Skip-gram idea, achieved good results on STS-B. SIF takes into account the attentional weighting mechanism, but not the syntactic information of the sentence. Constituency Tree-LSTM captures the syntactic structural information of the sentence through a constituency parse tree, but ignores the importance of the attentional weighting mechanism. They are both deficient compared to our proposed syntactic similarity model (CPT-PK), which has better performance than them on the STS-B dataset, as the results show. However, compared to the pre-trained model, the performance does not appear to be outstanding, possibly because numerical items or special characters in the short text, for example, weaken the performance of the model.

SemBERT, Tsinghua ENRIE, and Baidu ENRIE both added external knowledge information on top of BERT. Compared with BERT, good results were achieved, and it was also demonstrated that adding external knowledge information could significantly enhance the original model, solve the problem of lack of semantic information in short texts, and improve the performance of text matching. While SemBERT, Tsinghua ENRIE, and Baidu ENRIE did not consider the possible noise problem brought by the introduction of external knowledge base. In order to avoid noise, KEBERT-GCN constructs the word similarity matrix, combined with BERT’s attention matrix, to enhance the model’s attention to semantically similar word pairs, and the experiments prove that, compared with them, KEBERT-GCN is more effective in using fine-grained word relationship information from the knowledge base for short-text matching, and is able to capture the relational features of the knowledge structure and text structure using the GCN model, showing better performance on the STS-B dataset. CF-BERT emphasizes syntactic relations to introduce weighting factors, taking into account the enhancement of complete sentences with key components in the sentences, but ignores the fine-grained word relationship information. FEFS3C considers ‘sentence meaning’ and ‘key sentence information’. PromptBERT uses Prompt to improve sentence representation and considers the sentence as a whole, but does not focus on the importance of external knowledge to the language model. It also does not focus on the relationships between words. Although they are fine-tuned or improved on BERT and achieve good semantic similarity performance, our focus is more comprehensive and has a higher PCC in the STS-B dataset than our model.

Meanwhile, experimental results show that the KEBERT-GCN+CPT-PK model combining semantic and syntactic information performs better than KEBERT-GCN in the dataset, demonstrating that the CPT-PK model compensates for the lack of syntactic structure information in the KEBERT-GCN model. Compared with more classical models in recent years, our model mines the semantic and syntactic information of short texts, allowing the model to more comprehensively judge short-text similarity, with a Pearson coefficient of 88.05%, showing better performance.

4.3. Influence of Parameters

We explored the effect of two parameters $\theta$ and $\phi$ involved in the model on the model performance. To obtain the parameter values that achieve the optimal model performance, we judge the optimal values of $\theta$ and $\phi$ by Pearson correlation coefficient (PCC) on the STS-B dataset.

As shown in Equation (15), we introduce the SIF attention mechanism, where $\phi$ is a smoothing parameter. The figure shows the results of the experiments on STS-B. As shown in the line graph, when the value of $\phi$ is small, it does not achieve the best performance on the dataset because the value of $\phi$ is smaller than the value of the p(node) word frequency. However, as the value of $\phi$ keeps increasing, PCC first increases to the maximum value and then decreases, which is exactly in line with the law that a lower frequency of words indicates more information. According to the graph, in this paper, we set $\phi$ = 0.0001.

As shown in Equation (17), we use a linear strategy to combine semantic similarity and syntactic similarity, where $\theta$ is a weighting parameter to weigh the values of these two components. Figure 4 shows the results of the experiments on STS-B. It can be seen that, as the value of $\theta$ increases, the PCC value also increases to its maximum value and then decreases. The optimal value is obtained between 0.8 and 0.9. Semantic similarity accounts for a large proportion of judging the similarity of short texts. This is because short texts usually do not strictly adhere to the syntactic structure of written language, and semantic similarity is dominant in the similarity of short texts. After a small span of fine-tuning, we set $\theta$ = 0.85 in this paper.

5. Conclusions and Future Work

Short-text similarity is one of the important applications of natural language processing. In previous studies, most of them have only focused on the semantic similarity of short texts, and little attention has been paid to the syntactic similarity of short texts. In this paper, we first propose the KEBERT-GCN semantic similarity model to improve the semantic similarity performance by constructing the relationship between knowledge structure and text structure and effectively using the fine-grained word relationship information in the knowledge base. Experiments show that it outperforms existing models that integrate external knowledge on the STS-B dataset. Then, we propose a tree kernel-based computation method for constituency parse trees (CPT-TK) that wisely combines syntactic information, semantic features, and attentional weighting mechanisms as a means to obtain syntactic structure information and judge the syntactic similarity of short texts. Finally, we combine these two models to propose a short-text similarity model that combines semantic and syntactic information. Our model achieves significant performance gains in the short-text similarity task, which shows that the combination of semantic and syntactic information is effective. These findings provide good hints and directions for further exploration and the improvement of techniques for short textual similarity.

In the future, it is worthwhile to focus on the following points: (1) When constructing word similarity matrices, WordNet does not include many proper nouns nor does it resolve word ambiguity. (2) KEBERT-GCN is a generic model based on the fine-tuning of the backbone framework of BERT, and we can replace BERT with RoBERTa [25], DeBERTa [27], and other BERT variants of BERT. It is worth further investigation in subsequent work. (3) CPT-TK is not very sensitive to numbers, such as telephone numbers and the prices of goods mentioned in short texts.