A Novel Hybrid Methodology of Measuring Sentence Similarity

Abstract

The problem of measuring sentence similarity is an essential issue in the natural language processing area. It is necessary to measure the similarity between sentences accurately. Sentence similarity measuring is the task of finding semantic symmetry between two sentences, regardless of word order and context of the words. There are many approaches to measuring sentence similarity. Deep learning methodology shows a state-of-the-art performance in many natural language processing fields and is used a lot in sentence similarity measurement methods. However, in the natural language processing field, considering the structure of the sentence or the word structure that makes up the sentence is also important. In this study, we propose a methodology combined with both deep learning methodology and a method considering lexical relationships. Our evaluation metric is the Pearson correlation coefficient and Spearman correlation coefficient. As a result, the proposed method outperforms the current approaches on a KorSTS standard benchmark Korean dataset. Moreover, it performs a maximum of a 65% increase than only using deep learning methodology. Experiments show that our proposed method generally results in better performance than those with only a deep learning model.

1. Introduction

With the advent of the big data era, many people are receiving services from it. For example, you can be provided with decision-making strategies for social and economic development, or you can automatically find the information you want on the web [1,2]. In addition, many new convenient services are emerging, such as product recommendations during shopping, human movement analysis, and health checks using data [3,4].

These days, natural language data such as social networks and news articles are pouring out; natural language processing have received tremendous attention [5]. Measuring the similarity between natural language sentences is important even within natural language processing [6]. Sentence similarity is a numerical measure of how symmetrical the meaning of two sentences is regardless of word order, sentence length, or symmetry in context. For example, many approaches such as a chatbot system, plagiarism checking system, and automatic classification system all depend on sentence similarity. Accurately measuring the similarity between two sentences is a crucial task

Research measuring sentence similarity has been conducted from various perspectives [6,7,8,9,10,11,12]. Including deep learning approaches and sentence structure approaches there are many ways to measure sentence similarity [10,11,12,13,14].

In a study on sentence similarity measurement using deep learning, Ref. [10] proposed a method by extracting features containing the entire information of a sentence via long short-term memory (LSTM). The authors of Ref. [13] proposed a method to measure sentence similarity using a combination of global features—the entire information of sentences extracted via bidirectional LSTM (Bi-LSTM)— and local features—the detailed information of sentences extracted through capsule networks.

In the natural language processing field, it is necessary to focus not only on the model using deep learning, but also on the structure of the sentence and the lexical relationship of the sentence. The author of Ref. [8] proposed a method measuring the similarity between two sentences based on the relationship between vocabulary within sentences, using WordNet composed from a knowledge base using vocabulary. In contrast, Ref. [9] proposed a method of decomposing and reorganizing vocabulary.

In this paper, our research aims to improve the performance of methods for measuring the similarity between the Korean language sentences by combining a deep learning methodology and a methodology that considers lexical relationships. We use several deep learning methodologies, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and bidirectional encoder representations from transformers (BERT), to measure sentence similarity. Additionally, we apply cosine the similarity to embedding vectors obtained from the language representation model. Finally, we calculate the final sentence similarity by combining the sentence similarity value calculated by the deep learning model and the value obtained from the cosine similarity. Experiments show that our proposed method performed better compared to those with only a deep learning model.

This paper is structured as follows: Section 2 mentions related work. Section 3 explains the proposed approach and data. Section 4 shows the result about our method. Section 5 discusses our research. Finally, Section 6 mentions the conclusion.

2. Related Work

Many approaches have been proposed to address the problem of measuring the similarity between sentences [15]. Research measuring the similarity between two sentences has been conducted for a long time from various perspectives. There are many approaches to calculate the similarity between two sentences, such as using sentence structure, considering a lexical relationship, and using deep learning.

The method of measuring the similarity between two sentences using the structure of sentences is a widely used method from the early days of natural language processing to the present era of deep learning. Since many researchers have studied it for a long time, many ideas have been proposed to measure the similarity between two sentences using sentence structure. The authors of Ref. [16] proposed a similarity measure method of two sentences based on sentence structure grammar. The authors of Ref. [17] proposed a similarity measure method of two sentences based on a part of speech tags. The authors of Ref. [6] proposed a similarity measurement method of two sentences based on the word order and sentence structure. The authors of Ref. [7] measured the similarity of two sentences by identifying statistics on sentence structure.

A method of measuring the similarity between two sentences considering lexical relationships is also one of the sentence similarity measures. The author of Ref. [8] proposed a method for measuring the similarity between two sentences using WordNet composed from a lexical-based knowledge base. The authors of Ref. [9] were presented with a method to calculate the similarity between two sentences using an approach that uses the word to calculate the similarity using repetitive and different parts of a sentence. The authors of Ref. [18] proposed a method to measure sentence similarity using word detection ambiguity and synonym extension.

Deep learning has recently developed significantly since hardware development and the opening of the big data era [19]. Sentence similarity studies using deep learning have shown good performance using various neural networks such as LSTM, gated recurrent units (GRU), CNN, and BERT [10,13,14,20,21].

The authors of Ref. [6] used LSTM, which has a good performance for sequential data processing. They evaluated the sentence similarity by applying the last hidden states extracted via LSTM to Manhattan distance. The authors of Ref. [20] combined CNN and LSTM. They extracted combined information from adjacent words through CNN and applied the last hidden states extracted via LSTM to Manhattan distances to assess the sentence similarity. The authors of Ref. [21] used Group CNN (G-CNN), which extracted representative local features, and bidirectional GRU (Bi-GRU), which has good performance in sequential data processing and applied the last hidden states extracted via Bi-GRU to Manhattan distances. The authors of Ref. [13] sequentially used Bi-LSTM, self-attention reflecting contextual information, capsule networks with a CNN structure. They, then, combined the last hidden states extracted via Bi-LSTM and local features extracted via capsule networks. The authors of Ref. [14] evaluated the similarity of two sentences using BERT, a language representation model that shows excellent performance in various natural language processing fields.

Unlike previous approaches, we proposed a novel method that combines deep learning and the method considering lexical relationships to measure the similarity between two sentences.

3. Methods and Data

Measuring accurately the similarity of sentence-to-sentence is an important task [6]. To measure similarity between sentences, we combined deep learning methodology and a method that considers lexical relationships. A description of the equations mentioned in Section 3 is given in Appendix A.

We used Python version 3.6.9 and TensorFlow version 2.4.1 from Seoul, Korea The computer specifications used in the experiment were 2 GeForce RTX 3090 and 32 Intel (R) Xeon (R) Silver 4215 CPUs @ 2.50 GHz.

3.1. Similarity Based on Deep Learning Model

The models used in this study were LSTM and GRU, CNN, G-CNN, capsule networks, and BERT. LSTM and GRU are a family of RNN, and G-CNN and capsule networks are neural networks using CNN. CNN and G-CNN were used as input values to the RNNs model. The sentence similarity using RNNs or capsule networks was calculated by applying the Manhattan distance, such as Figure 1a. Additionally, the sentence similarity using BERT was calculated through a special token, such as Figure 1b.

3.1.1. Sentence Similarity Using Word Embedding

The RNN is a neural network that shows good performance when processing sequential data [22]. When calculating the representation of each time step in text processing, it was determined through learning how much context information up to that point was reflected. However, in the case of RNN, if the sequence length was increased, gradient vanishing or gradient exploding problems could occur [23]. To solve this, LSTM and GRU were devised [23,24]. When calculating the similarity of two sentences, each sentence was input into the family model of RNNs to obtain the last hidden states ($h^l{}_L, h^r{}_L$) of each sentence [10,13,20,21]. The $h_L$ included the entire sentence information. Manhattan distance used in the family model of RNNs is as follows:

$$Si{m}_D= exp\left(-\Vert h^l{}_L-h^r{}_L{\Vert }_1\right)$$

(1)

The CNN showed good performance in image and text processing [13]. In text processing, CNN extracted local features, which were combined information, by grouping words appearing in sentences by kernel size ($k$), and increasing the amount of learning by using the number of filters ($N$). The equation of CNN is as follows:

$$Con{v}_i=f\left(W_k*w_{i-\frac{k-1}{2}:i-\frac{k+1}{2}}+b_k\right)$$

(2)

In Equation (2), $i$ refers to word index, $f(·)$ refers to the activation function. In addition, $W_k$ refers to a learning weight of a CNN having a size of ${ℝ}^{k\times E}$, $w_i$ refers to a word embedding vector having a size of ${ℝ}^E$, and $b_k$ refers to a bias vector. Through Equation (2), the feature map, which had a size of ${ℝ}^{L-k+1}$, was generated according to $L$, which refers to the number of words. Finally, the feature maps the size of ${ℝ}^{\left(L-k+1\right)\times N}$ were generated by the number of filters.

The G-CNN used three CNNs with different kernel sizes in parallel and obtained representative semantic information [21]. G-CNN integrated feature maps extracted from three CNNs into a feature of one and then created the most representative feature map by applying max pooling. The equation of G-CNN is as follows:

$$G-CN{N}_i=\max \left(CN{N}_i^k, CN{N}_i^{k+2}, CN{N}_i^{k+4}\right)$$

(3)

The capsule networks used two $Conv1$ and $Conv2$ which were used to extract subdivided information in the field of sentence similarity [13]. $Conv1$ had a typical CNN form, and $Conv2$ received a feature map of $Conv2$ as the input value. After that, to extract subdivided information, a kernel size corresponding to the overall size of the input value was used, and then a feature map the size of ${ℝ}^{C\times \frac{N}{C}}$ was generated by dividing the feature map into C-dimensions. The feature map $Conv2$ reshaped as many as C was defined as $R(Conv{2}_i)$ and, finally, Manhattan distance used in the capsule networks is as follows:

$$Si{m}_D=\frac{1}{C}{\displaystyle \sum }_i^{C}exp(-\Vert R\left(Conv{2}^l{}_i\right)-R{\left(Conv{2}^r{}_i\right)\Vert }_1)$$

(4)

3.1.2. Sentence Similarity Using BERT Embedding

BERT is a language representation model created by stacking several transformer encoder blocks [14]. The learning process of BERT is divided into a pre-training process and a fine-tuning process. In the pre-training process, after randomly masking word tokens in a sentence of a large corpus, the BERT model learned by predicting the masked word token. Fine-tuning is a process of learning a pre-trained BERT model with labeled data once more. We trained BERT on the similarity task between sentences in the fine-tuning process.

BERT is divided into BERT-base and BERT-large models according to the size of the model and, in this work, we used the BERT-base model. BERT-base model consists of 12 layers of transformer encoder blocks. There are several special tokens in the BERT model. The [CLS] token was placed at the beginning as a token indicating the beginning of the input. The [SEP] token was a token that distinguished between sentences. In the case of the sentence similarity task, the input was entered into the BERT model in the form of “[CLS] Sentence1 [SEP] Sentence2 [SEP]”. The similarity between sentences was calculated by inputting the output vector of the (CLS) token extracted via the BERT model into the dense layer, as shown in Equation (5).

$$Si{m}_D=\sigma \left(W{h}_{\left[CLS\right]}+b\right)$$

(5)

The $h_{\left[CLS\right]}$ is the vector representation of the (CLS) token extracted through the BERT model and has a matrix of ${ℝ}^{728}$. $W$ refers to weights that can be learned, with a matrix of ${ℝ}^{1\times 728}$, and $b$ refers to bias vector.

3.2. Similarity Based on Lexical Relationship

To measure structure similarity, the proposed approach calculated the word-to-word similarity between sentences. Using a language representation model in measuring similarity between words, improved similarity measuring between sentences [25].

In the word embedding model, to consider the lexical relationship, including semantic information, we used the embedding vector of words ($w_n$). Word embedding used in this study used Word2Vec, which learns by minimizing the dot product values of the target word vector and neighbor word vectors surrounding the target word vector [26,27].

In BERT, to consider the lexical relationship, including contextual information, we used the hidden states of the 12 transformer blocks, which were BERT-base model components. To calculate the word-to-word similarity, we excluded hidden states of the (CLS) token and (SEP) token. The embedding token of the word ($w_n)$ was calculated using the following equation:

$$w_n=\left({{\displaystyle \sum }}_{i=1}^T{h}_{i, n}\right)/T$$

(6)

In Equation (6), $n$ refers to the index of the word, $T$ refers to the number of transformer blocks.

The word-based similarity was calculated using the following equation:

$$si{m}_w\left(S_l,S_r\right)=\left({\displaystyle \sum }_{i=1}^{L}sim\left(w_i^l,S_r\right) \right)/L$$

(7)

In Equation (7), $S_l$ and $S_r$ refer to sentence 1 and sentence 2, respectively. $L$ is the number of words in $S_l$ and $sim$($w_i,S_r$) is the similarity of word $w_i$ in $S_l$ and $S_r$. To calculate the similarity, we first added all the words-to-sentences. In this process, we could obtain the sentence-to-sentence likeness of $S_l$ and $S_r$. However, the likeness became higher in proportion to the length of $S_l$, and this could cause a problem when the length of $S_l$ is too long. Thus, we divided the likeness into $L$, the length of sentence $S_l$. In this way, we could normalize the likeness and have similar values regardless of length. The similarity between word $w$ and $S$ was measured by selecting the max similarity between $w$ and every word in $S$ according to the following Equation (8).

$$sim\left(w,S\right)=\max \left(sim(w, w_j)\right)$$

(8)

In Equation (8), $sim(w, w_j)$ is the similarity between word $w$ and word $w_j$. $w_j$ refers to the $j$th word that appears in another sentence $S$. The similarity between words was measured using cosine similarity between word vectors. When measuring similarity between the target word $w$ and the sentence $S_r$, a comparison of several important words in a sentence was sufficiently meaningful. Thus, the maximum value of the cosine similarities was defined as the similarity between the target word $w$ and the sentence $S_r$.

Our method went through the process of calculating the similarity of words belonging to $S_r$ based on words belonging to $S_l$, as in the above formula, when comparing two sentences $S_l$ and $S_r$, and vice versa. In other words, we calculated the word in $S_l$ based on the word in $S_r$ and then calculated the arithmetic mean of the method, such as Equation (9), from two methods.

$$Si{m}_w=\frac{si{m}_w\left(S_l,S_r\right)+si{m}_w\left(S_r,S_l\right)}{2}$$

(9)

3.3. Novel Hybrid Sentence Similarity

In this study, we combined deep learning methodology and a method that considers lexical relationships using the equation below:

$$Similarity=\mathsf{\alpha }Si{m}_D+\left(1-\mathsf{\alpha }\right)Si{m}_w$$

(10)

In Equation (10), α is a weight that can adjust which information to focus on among deep learning methodology and a method that considers lexical relationships and is determined experimentally. Through Equation (10), the value of sentence similarity had a range of 0–1.

Table 1. Pseudo code about sentence similarity using BERT embedding.

Sentence Similarity Using BERT Embedding
$BER{T}_{output}\leftarrow BERT\left(Sentenc{e}_1, Sentenc{e}_2\right)$ $h_{\left[cls\right]}\leftarrow extract\_cls\left(BER{T}_{output}\right)$ $Si{m}_D\leftarrow W{h}_{\left[cls\right]}+b$

,

Table 2. Pseudo code about similarity based on lexical relationship.

Similarity Based on Lexical Relationship
$\mathit{i}\mathit{f} BER{T}_{output} \mathit{t}\mathit{h}\mathit{e}\mathit{n}$
$\mathit{f}\mathit{o}\mathit{r} n=1,N \mathit{d}\mathit{o}$
$\mathit{f}\mathit{o}\mathit{r} i=1,T \mathit{d}\mathit{o}$
$w_n\leftarrow w_n+h_{i,n}$
$\mathit{e}\mathit{n}\mathit{d}$
$w_n\leftarrow w_n/T$
$\mathit{e}\mathit{n}\mathit{d}$
$\mathit{e}\mathit{l}\mathit{s}\mathit{e}$
$w_n\leftarrow word2vec\left(wor{d}_n\right)$
$sim\left(w_i,S_r\right): \max (cosine similarity between w_i and each word in S_r$)
$\mathit{f}\mathit{o}\mathit{r} i=1,L \mathit{d}\mathit{o}$
$Si{m}_w\left(S_l, S_r\right)\leftarrow Si{m}_w\left(S_l, S_r\right)+sim\left(w_i,S_r\right)$
$\mathit{e}\mathit{n}\mathit{d}$
$Si{m}_w\left(S_l, S_r\right)\leftarrow Si{m}_w\left(S_l, S_r\right)/L$

and

Table 3. Pseudo code about novel hybrid sentence similarity.

Novel Hybrid Sentence Similarity
$Si{m}_w\leftarrow \frac{Si{m}_w\left(S_l, S_r\right)+Si{m}_w\left(S_r, S_l\right)}{2}$
$Similarity\leftarrow \alpha Si{m}_D+\left(1-\alpha \right)Si{m}_w$

shown below is the pseudo code to which our methodology was applied.

3.4. Data

BERT used in this study was the Bert-Kor-Base [28], which trains Korean texts. Additionally, Word2Vec is trained using the Korean raw corpus [29] applied Kkma [30], a Korean morpheme analyzer. Some studies have shown that splitting words into morphemes in Korean tends to perform well [31]. Word vector extracted through Word2Vec had an embedding size of ${ℝ}^{768}$, the same as BERT embedding size.

In this study, KorSTS [32], consisting of 8628 sentence pairs, was used as the experiment data. Training sets, development sets, and test sets consisted of 5749, 1500, and 1379 sentence pairs. The similarity score range of two sentences was composed from 0 to 5 points, as shown in

Table 4. KorSTS dataset example.

Language	Sentence 1	Sentence 2	Score
Korean	한 여성이 브로콜리를 자르고 있다.	한 여성이 칼로 브로콜리를 자르고 있다.	4.25
English	A woman is cutting broccoli.	A woman is cutting broccoli with a knife.
Korean	남자가 카누로 노를 저고 있다.	남자가 하프를 연주하고 있다.	0.667
English	The man is rowing a canoe.	The man is playing the harp.

.

In this study, the similarity score was normalized from 0 to 1 using the minimum–maximum scaling.

4. Result and Analysis

We compare the results of the model using only deep learning and the models using our method. We used the Pearson correlation coefficient and the Spearman correlation coefficient as the evaluation metrics.

The performance shown in

Table 5. Model performance comparison according to experiments.

	Method	Deep Learning		Hybrid
Model		Pearson	Spearman	Pearson	Spearman
LSTM [10]		0.362511	0.344804	0.557284	0.550898
CNN + LSTM [20]		0.35528	0.334058	0.556268	0.55126
G-CNN [21]		0.604134	0.577296	0.65184	0.640162
Capsule [13]		0.620881	0.599617	0.661728	0.65319
BERT [14]		0.838989	0.830135	0.842797	0.834181

was the average value of the results of the five experiments each. As shown in Table 5, we can see that the Pearson and Spearman correlation coefficients were both higher in consideration of both deep learning and lexical relationships than those using only deep learning.

5. Discussion

Our method resulted in a significant performance improvement, though the minor calculation cost increased compared to those using only deep learning. The proposed method had better performance than a single method using deep learning because it hybridizes methods that consider the semantic relationship between words. Since we used the pre-trained embedding model to calculate the relationships between words, we could distinguish different words with the same meaning. For example, ‘car’–‘car’ or ‘cookie’–‘biscuit’ had a high score of similarity. According to the experiment, the similarity calculation between sentences was improved by using the similarity information between words within a sentence. For example, consider the following sentences:

S1

= ‘미나는 저녁으로 치킨을 먹기 위해 자동차를 타고 식당에 갑니다.’

S1

= ‘Mina gets into the car and goes to a restaurant to eat chicken for dinner.’

S1

= ‘미나는 저녁식사를 치킨으로 해결하려고 승용차를 타고 식당에 갑니다.’

S1

= ‘Mina gets in her automobile and goes to a diner to make supper with chicken.’

These sentences have a similar meaning but are expressed in different words. Our model improved the similarity between the measured sentences by considering the semantic relationship between words in which the two sentences above are identical.

Given that we achieved performance improvements in all five models, we showed that our method generally increased the ability of the model to understand the semantic similarity of sentences. Especially on [18] model, our approach resulted in about a 65% performance improvement. Therefore, using our methods together gave a more accurate performance than using one deep learning method.

6. Conclusions

This study measured the similarity between the Korean language sentences by combining a deep learning methodology and a method that considers lexical relationships. In a deep learning methodology, we used five neural networks related to CNN, RNN, and BERT. Additionally, in a method that considers lexical relationships, we used a cosine similarity in embedding vectors extracted through the word representation model. Finally, we calculated the final sentence similarity by combining the output values of the two methods. As a result, our method combining two methods showed good performance compared to using only a deep learning model.

The method considering lexical relationships used in this study is one of several linguistic methods for measuring sentence similarity. As it can be seen from the experimental results, it could achieve good performance by combining the deep learning method and the linguistic method. Therefore, in future studies, we will further improve the performance of similarity between sentences by using various linguistic methods such as information on the order of words and parts of speech.