A Quantum Language-Inspired Tree Structural Text Representation for Semantic Analysis

Abstract

Text representation is an important topic in the field of natural language processing, which can effectively transfer knowledge to downstream tasks. To extract effective semantic information from text with unsupervised methods, this paper proposes a quantum language-inspired tree structural text representation model to study the correlations between words with variable distance for semantic analysis. Combining the different semantic contributions of associated words in different syntax trees, a syntax tree-based attention mechanism is established to highlight the semantic contributions of non-adjacent associated words and weaken the semantic weight of adjacent non-associated words. Moreover, the tree-based attention mechanism includes not only the overall information of entangled words in the dictionary but also the local grammatical structure of word combinations in different sentences. Experimental results on semantic textual similarity tasks show that the proposed method obtains significant performances over the state-of-the-art sentence embeddings.

1. Introduction

The parallelism of quantum computing has attracted more and more attention in different fields. Some scholars have combined quantum computing with natural language processing (NLP). In the quantum computing-based text representation model, the word vector was multiplied by its own transposed vector tensor to obtain a density matrix, and the weighted density matrix was summed to obtain a sentence tensor representation [1,2].

$$\rho ={\Sigma }_{i=1}^n{p}_i|w_i\rangle \langle w_i|,$$

(1)

$$Sim(S_1,S_2)=tr\left({\rho }_1{\rho }_2\right)=\sum _{i,j}{\lambda }_i{\lambda }_j{\langle w_i|w_j\rangle }^2,$$

(2)

where $tr$ denoted the trace operation. The probability that event $|u\rangle \langle u|$ belongs to a system was defined as the semantic measurement [3],

$${\mu }_{\rho }\left(\right|u\rangle \langle u\left|\right)=tr\left(\rho \right|u\rangle \langle u\left|\right)=\langle u\left|\rho \right|u\rangle$$

(3)

where ${\mu }_{\rho }\in [0,1]$.

The association between adjacent words with all of the entanglement coefficients set to 1 was discussed, ignoring the long-range modified relationship between non-adjacent words [4]. When the sentence structure is complex, the two words that have a direct modified relationship are not necessarily in close proximity. Especially for long sentences, the complex syntactic structure makes adjacent words not necessarily grammatically related, and the grammatically related words separated by several words. Take the following sentence for example with the constituency parser and the dependency parser (https://nlp.stanford.edu/software/lex-parser.shtml (accessed on 1 October 2020)) of $S_1$ shown in Figure 1.

According to the adjacent words to form a related phrase, the closest to the word $best$ is the nominal word $July$, but the two syntax trees in Figure 1 both show that there is no direct modification relationship between the two words. The dependent relation between $reading$ and $best$ is $nsubj$ (nominal subject), but the distance between them is 8 words, meaning that at least 9-gram can contain both of the words. If the 9-gram is considered, it will contain at least seven irrelevant words except $reading$ and $best$, thus introducing semantic errors. Furthermore, the distance difference between two words of different relation entities is different, which requires the size of the kernel function to change according to the change of the distance difference between the two ends of the relation entity. Obviously, this is difficult to achieve in reality. AS different weights of part-of-speech (PoS) combinations of entangled word have different influences on sentence semantics [5], PoS combination weight can be integrated with the attention mechanism to express the different modified relationship. Inspired by the density matrix and attention mechanism, a quantum language-inspired tree structural text representation model is established to reflect the association between variable distance words.

As different syntactic tree structures reflect different associations between words, different semantic association models between words according to the dependency parser and constituency parser of sentences are constructed. According to the association between relation entities in the dependency parser of the sentence, the text representation based on the dependency parser combines the word vector tensors of two words with relation entities to establish the semantics between long-distance dependent words with relation entities entanglement, so that distant words with a direct modified relationship can also be semantically related. According to the different degrees of modified relationship between words in the constituency parser, the text representation based on the constituency parser combines the semantic correlation coefficient with the distribution characteristics of adjacent words to establish the semantic association. The proposed model consists of two parts, as shown in Figure 2. The first part is composed of all of the tensor products of the two adjacent words in a sentence, combining the characteristics to establish the contribution of the short-range dependence between words to the semantics. The second part consists of the two words with long-range dependency of the direct modified relationship. Finally, the entanglement between adjacent words is integrated with the word entanglement of direct long-range modified relationship to form the sentence representation.

In brief, the contributions of this work are as follows.

(1)

A quantum language-inspired text representation model based on relation entity and constituency parser is established, including long-range and short-range semantic associations between words.

(2)

The combination of attention mechanism and entanglement coefficient reduces the semantic impact of indirect modified relationships between adjacent words and enhances the semantic contribution of direct modified relationships with long-range associations.

(3)

The attention mechanism contains not only the overall information of the related words in the dictionary, but also the local grammatical structure of different sentences.

(4)

The semantic association between words with variable distances is established by combining the dependency parser.

The rest of the paper is organized as follows. Section 2 summarizes some related literature on attention-based semantic analysis, the dependency tree and quantum based NLP. Section 3 explains the approach in detail. The experimental settings are presented in Section 4. Section 5 describes the experimental results and lists the detailed effects of different parameters. In Section 6, some conclusions are drawn.

2. Related Work

2.1. Attention-Based Semantic Analysis

The neural network-based methods use the attention mechanism to assign different semantic weights to words with good experimental results in many downstream tasks, such as LSTM [6], BiLSTM [7] and BERT [8]. Semantic analysis based on attention mechanism has been involved in many works [9,10,11] and can reflect the different weights of words in different texts. The attention mechanism is introduced to obtain different weight of words in order to extract enough key information. Semantic analysis can be applied to many problems such as image-text matching [12], question answering [13,14], knowledge extraction [15,16,17,18,19], and entailment reasoning [20]. To discover visual-textual interactions across different dimensions of concatenations, memory attention networks were adopted while marginalizing the effect of other dimensions [12]. A deep multimodal network with a top-k ranking loss mitigated the data ambiguity problem for image-sentence matching [21]. LSTM with a bilinear attention function was adopted to infer the image regions [22]. A mutual attention mechanism between the local semantic features and global long-term dependencies was introduced for mutual learning [23]. A scheme of the efficient semantic label extraction was developed to achieve an accurate image-text similarity measure [24].

Compared to the previous works mentioned above, the method proposed here is mainly based on the traditional attention mechanism, which mainly reflects the relation dependency between input and hidden without considering the relations of input words. The transformer mainly relies on attention mechanisms [25]. An improved self-attention module was proposed by introducing low-rank and locality linguistic constraints [26]. With the introduction of the self-attention mechanism, new models based on transformer obtained much success in semantic analysis on large datasets [27,28,29]. The new model BERT [30] and its variants [31,32] based on transformer divided the pretraining methods into feature-based methods and fine-tuning methods [33].

2.2. Dependency Tree

Relation extraction plays a very important role in extracting structured information from unstructured text resources. The dependency tree not merely expresses the semantics of sentences but also reflects the modified relationship relationship between words. Each note in dependency trees represents a word, and every word has at least one grammatically related word. The dependency tree is constituted by the head word, PoS of the head word, dependent word, PoS of the dependent word and the label of dependency. The works on the dependency tree are mainly divided into two categories: statistics-based models and deep-learning-based models [34]. Wang et al. structured a regional CNN-LSTM model based on a subtree to analyze sentiment predictions [35]. A reranking approach for the dependency tree was provided utilizing complex subtree representations [36]. A bidirectional dependency tree representation was provided to extract dependency features from the input sentences [37]. Zhang et al. [38] tried to upgrade the synchronous tree substitution grammar-based syntax translation model by utilizing the string-to-tree translation model. A graph-based dependency parsing model was presented by Chen et al. [39]. A bidirectional tree-structured LSTM was provided to extract structural features based on the dependency tree [40]. Fei et al. utilized a dependency-tree-based RNN to extract syntactic features and used the CRF layer to decode the sentence labels [41]. Global reasoning on a dependency tree parsed from the question was performed [42]. A phrase-based text embedding considering the integrity of semantic with a structured representation is reported [43].

2.3. Quantum Based NLP

In recent years, the application of quantum language models (QLM) in NLP has attracted more and more attention [44]. Aerts and Sozzo theoretically proved that under some conditions, the joint probability density of the two entities selected to define the uncertainty of selection to establish an entanglement between the concepts was reasonable [45]. Quantum theory is applied to neural networks to form quantum neural networks. To achieve comparable or better results, quantum network needed far fewer epochs and a much smaller network [46]. Quantum probability was first practically applied in information retrieval (IR) with significant improvements over a robust bag-of-words baseline [47,48,49,50,51,52]. At the same time, an unseparable semantic entity was used in IR, considering the pure high-order dependence among words or phrases [53]. On this basis, quantum entanglement (QE) was applied to terms dependency co-occurrences on quantum language models with theoretical proof of the connection between QE and statistically unconditional pure dependence [54].

Commonly, semantic representation generalized quantum or quantum-like use Hilbert spaces to model concepts, and the similarity is measured by scalar product and projection operators [55]. Density matrix representation can be used to many fields, such as document interaction [2], different modality correlations [56] and sentiment analysis [1,3,57]. The mathematical formalism of quantum theory could resist traditional model with significant effectiveness in cognitive phenomena [58]. A semiotic interpretation of the role played based on quantum entanglement was provided to find a finite number of the smallest semantic units to form every possible complex meaning [59].

3. Approaches

3.1. Read Text and Generate Syntax Tree

Each sentence is read, and the dependency tree and relation entity are generated. The word2vec embedding, PoS tagging and tree depth of each word in a sentence are stored, forming a quadruple $\mathcal{T}(\mathcal{A},\mathcal{P},\mathcal{H},\mathcal{R})$. The $\mathcal{A}$ array stores all of the words with the original sequences in the sentence.

$$\mathcal{A}=\{w_1,w_2,\cdots ,w_i,\cdots ,w_n\},$$

(4)

where $w_i$ is the ith word in the sentence and n is the total number of the words.

The $\mathcal{P}$ array stores the PoS of each word in sequence.

$$\mathcal{P}=\{P_1,P_2,\cdots ,P_i,\cdots ,P_n\},$$

(5)

where $P_i$ is the PoS of the ith word $w_i$.

The $\mathcal{H}$ array denotes the tree depth of the ith word $w_i$ in the dependency tree.

$$\mathcal{H}=\{h_1,h_2,\cdots ,h_i,\cdots ,h_n\},$$

(6)

where $h_i$ represents the tree depth of the ith word $w_i$.

The $\mathcal{R}$ array denotes the relation entity of the ith word $w_i$ in the dependency tree. We only consider whether an entity relationship exists between entangled words. If a relationship exists, the words, the associated vocabularies and the relationship between them are saved.

3.2. Entanglement between Words with Short-Range Modified Relationship

The model mainly describes the entanglement between two adjacent words. The attention mechanism is introduced to highlight word entanglement with a direct modified relationship while weakening the impact of the entanglement of indirect modified relationship on sentence semantics.

3.2.1. Normalize the Word Vector

Normalize the word2vec of an input word.

$$|w_i\rangle =\frac{\overrightarrow{s_i}}{|\overrightarrow{s_i}|},$$

(7)

where $\overrightarrow{s_i}$ is the d dimensional column vector of the ith word and $|\overrightarrow{s_i}|$ is the module of $\overrightarrow{s_i}$. Therefore, $|w_i\rangle$ is set as $|w_i\rangle ={\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_d\end{array}\right]}^{\mathrm{T}}$, with d dimensional column vector.

3.2.2. Embedding of Entangled Word

Two adjacent words are entangled together in order, forming the arrays

$$\mathcal{B}=\{\left(w_1{w}_2\right),\left(w_2{w}_3\right),\cdots ,\left(w_i{w}_{i+1}\right),\cdots ,\left(w_{n-1}{w}_n\right)\},$$

(8)

and

$$\mathcal{C}=\{\left(P_1{P}_2\right),\left(P_2{P}_3\right),\cdots ,\left(P_i{P}_{i+1}\right),\cdots ,\left(P_{n-1}{P}_n\right)\},$$

(9)

where $w_i{w}_{i+1}$ is the combination of the adjoint words $w_i$ and $w_{i+1}$, and $P_i{P}_{i+1}$ is the PoS combination of $w_i{w}_{i+1}$. The representation of $\left(w_i{w}_{i+1}\right)$ is defined by the tensor product between $|w_i\rangle$ and $|w_{i+1}\rangle$.

$$\begin{array}{cc}\hfill |w_i{w}_{i+1}\rangle & =|w_i\rangle |w_{i+1}\rangle \hfill \\ & ={\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_d\end{array}\right]}^{\mathrm{T}}{\left[\begin{array}{cccc}{b}_1& b_2& \cdots & b_d\end{array}\right]}^{\mathrm{T}}\hfill \end{array}$$

(10)

For simplicity and clarity, $|w_i{w}_{i+1}\rangle$ is rewritten into a square matrix form.

$$|w_i{w}_{i+1}\rangle =\left[\begin{array}{cccc}{a}_1{b}_1& a_1{b}_2& \cdots & a_1{b}_d\\ a_2{b}_1& a_2{b}_2& \cdots & a_2{b}_d\\ ⋮& ⋮& \ddots & ⋮\\ a_d{b}_1& a_d{b}_2& \cdots & a_d{b}_d\end{array}\right].$$

(11)

Similarly, $|w_{i+1}{w}_i\rangle$ is obtained,

$$|w_{i+1}{w}_i\rangle =\left[\begin{array}{cccc}{b}_1{a}_1& b_1{a}_2& \cdots & b_1{a}_d\\ b_2{a}_1& b_2{a}_2& \cdots & b_2{a}_d\\ ⋮& ⋮& \ddots & ⋮\\ b_d{a}_1& b_d{a}_2& \cdots & b_d{a}_d\end{array}\right].$$

(12)

Obviously, the inequality is obtained, $|w_i{w}_{i+1}\rangle \ne |w_{i+1}{w}_i\rangle$, which reflects the influence of the order of the entangled words.

3.2.3. Attention Mechanism

The attention mechanism consists of three components: the cosine similarity between the entangled words, the influence of PoS combination and the dependency tree depth difference of the two words.

The similarity between the adjacent words is defined as follows:

$$Sim(w_i,w_{i+1})=\frac{\langle w_i|w_{i+1}\rangle }{\left|\right|w_i\rangle |·\left|\right|w_{i+1}\rangle |},$$

(13)

where   

$$\langle w_i|w_{i+1}\rangle =\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_d\end{array}\right]{\left[\begin{array}{cccc}{b}_1& b_2& \cdots & b_d\end{array}\right]}^{\mathrm{T}}=\sum _{i=1}^d{a}_i{b}_i,$$

(14)

$$\left|\right|w_i\rangle |=\sqrt{\sum _{i=1}^d{a}_i^2},$$

(15)

$$\left|\right|w_{i+1}\rangle |=\sqrt{\sum _{i=1}^d{b}_i^2}.$$

(16)

The weight of PoS combination of the entangled words can reflect some common modified relationships between words [5]. The different combinations of the two adjacent words are set to different values to express the different contributions for grammatical structures. We use $t_{i,i+1}$ to represent the influence of the PoS combination of the ith word and the $(i+1)$th word, where $t_{i,i+1}\in (0,1)$, reflecting the global information of the corpus.

The last part is a parameter $d_{i,i+1}$, which is determined by the dependent relationship in the syntax tree and the tree depth difference. The tree depth difference is the absolute value of the difference between the tree depth of the ith word and that of the $(i+1)$th word.

$$\Delta h_{i,i+1}=|h_i-h_{i+1}|.$$

(17)

The weight of $\Delta h_{i,i+1}$ is set as follows:

$$d_{i,i+1}=\left\{\begin{array}{cc}{d}_1,\hfill & \hfill for\Delta h_{i,i+1}=0\\ d_2,\hfill & \hfill for\Delta h_{i,i+1}\le a,\\ d_3,\hfill & \hfill for\Delta h_{i,i+1}>a\end{array}\right.$$

(18)

where $d_1$, $d_2$ and $d_3$ satisfy the condition of $d_1>d_2>d_3$, $a=2$ and the value of a can be altered.

Hence, the attention mechanism is described as follows:

$$p_{i,i+1}=Sim(w_i,w_{i+1})\times t_{i,i+1}\times d_{i,i+1}.$$

(19)

3.2.4. Adjacent Words Entanglement-Based Sentence Representation

The adjacent words entanglement-based sentence representation is defined as follows.

$$|S_1\rangle =\sum _{i=1}^{n-1}{p}_{i,i+1}|w_i\rangle |w_{i+1}\rangle =\sum _{i=1}^{n-1}{p}_{i,i+1}|w_i{w}_{i+1}\rangle ,$$

(20)

where $p_{i,i+1}={\displaystyle \frac{p_{i,i+1}}{{\displaystyle \sum _{i=1}^{n-1}}{p}_{i,i+1}}}$.

3.2.5. Sentence Similarity

Since both the representations of $|S_1\rangle$ and $|w_i{w}_{i+1}\rangle$ are the second-order tensor with $d^2$ dimensions, the normalized dot product between the two sentence representations $|S_1\rangle$ and $|S_1^{{}^{\prime }}\rangle$ is defined as the sentence similarity of the sentence pair,

$$Sim\left(\right|S_1\rangle ,|S_1^{{}^{\prime }}\rangle )=\frac{\langle S_1|S_1^{{}^{\prime }}\rangle }{\left|\right|S_1\rangle |·\left|\right|S_1^{{}^{\prime }}\rangle |},$$

(21)

where $\langle S_1|S_1^{{}^{\prime }}\rangle$ denotes the inner product of $\langle S_1|$ and $|S_1^{{}^{\prime }}\rangle$, $\left|\right|S_1\rangle |$ and $\left|\right|S_1^{{}^{\prime }}\rangle |$ are the norms of $|S_1\rangle$ and $|S_1^{{}^{\prime }}\rangle$, respectively, and $\langle S_1|$ is the conjugate transpose of $|S_1\rangle$.

3.3. Optimize the Sentence Embedding

The ultimate goal of text representation is to enable computers to understand human language. For the calculation of the text semantic similarity in this paper, the calculated value $x_i$ is infinitely close to the artificial score $y_i$. Therefore, the Pearson correlation coefficient (Pcc) reaches the maximum value and the mean square error (MSE) reaches the minimum values. To maximize the Pcc and minimize the MSE, we optimize the sentence embedding using two approaches.

3.3.1. Entanglement between Words with Long-Range Modified Relationship

For long sentences or sentences with complex structures, two modified words are not necessarily adjacent. Aiming at the modified relationship of the long-distance association, a long-range dependent relationship $R_{i,j}$ is defined as the R array between words $w_i$ and $w_j$.

$$R=\left\{R_{i,j}\right\},(i,j=1,\dots ,n)$$

(22)

$$R_{i,j}=(r_{i,j},|w_i\rangle ,|w_j\rangle ),$$

(23)

where $r_{i,j}$ is a binary element, defined by whether there is a correlation entity between words $w_i$ and $w_j$. If there is a relation entity between words $w_i$ and $w_j$, $r_{i,j}=1$; if not, $r_{i,j}=0$. The entanglement between $w_i$ and $w_j$ is

$$p_{i,j}=r_{i,j}\times Sim(w_i,w_j).$$

(24)

Equation (24) indicates that only the word pair with relation entity is considered to expand the sentence semantics.

Therefore, the sentence embedding based on relation entity is altered as follows.

$$\begin{array}{cc}\hfill |S_2\rangle & =\sum _{i<j,i,j=1}^n{p}_{i,j}|w_i\rangle |w_j\rangle \hfill \\ \hfill & =\sum _{i<j,i,j=1}^n{p}_{i,j}|w_i{w}_j\rangle ,\hfill \end{array}$$

(25)

where ${\displaystyle \sum _{i<j,i,j=1}^n}$ defines that all of the long-range dependencies in the sentence are taken into account, and $p_{i,j}={\displaystyle \frac{p_{i,j}}{{\displaystyle \sum _{i,j=1}^n}{p}_{i,j}}}$.

3.3.2. Sentence Embedding Based on Constituency Parser and Relation Entity

The entanglement between the short-range modified words and the entanglement between the long-range modified relationship form a sentence representation. Hence, the optimized sentence representation is altered as

$$\begin{array}{cc}\hfill |T\rangle & =|S_1\rangle +|S_2\rangle \hfill \\ \hfill & =\sum _{i=1}^{n-1}{p}_{i,i+1}|w_i{w}_{i+1}\rangle +\sum _{i<j,i,j=1}^n{p}_{i,j}|w_i{w}_j\rangle .\hfill \end{array}$$

(26)

$|S_1\rangle$ uses the entanglement coefficient $p_{i,i+1}$ to highlight the semantic contribution of adjacent words with modified relationships, and to weaken the semantic contribution of the adjacent words without modified relationship. $|S_2\rangle$ only considers the semantic contribution of word pairs with a relation entity. Therefore, $|T\rangle$ includes not only the modified relationship between adjacent words, but also the long-range modified relationship between related words. In addition, the entanglement coefficient analyzes the relevant degree between words from multiple perspectives by the relation entity of the text, combining the local information of the words in the same text with the global information in a dictionary.

3.3.3. Reduce Sentence Embedding Dimensions

The semantics of some phrases cannot be expressed by any of their constituent words alone, nor can the semantics of these two words be simply added together, such as $lung$$cancer$. The semantics of $lung$$cancer$ is less than the semantics adding of $lung$ and $cancer$. To reduce the redundant information after word entanglement, we use the two methods of dimensionality reduction on the level of sentence embedding and of entangled word representation. For dimensionality reduction at the sentence level, we delete some smaller absolute values in the sentence embedding. At the entangled word level, we delete some smaller absolute values of the entangled word embedding to reduce the dimension of the sentence representation.

4. Experimental Settings

4.1. Parameters Definition

Some parameters and variables are defined in

Table 1. Definitions of the parameters and variables.

$\mathbf{Parameter}/\mathbf{Variable}$	$\mathbf{Definition}$
$|w_i\rangle$	word embedding of the ith word
$|w_i{w}_j\rangle$	entangled word embedding of the ith word and the jth word
$|T\rangle$	sentence embedding
$Sim(w_i,w_j)$	direction cosine between the word embedding of the ith word and the $(i+1)$th word
$t_{i,i+1}$	part of speech combination weight of the ith word and the $(i+1)$th word
$\Delta h$	depth difference between two words in the parser tree
$d_{i,j}$	weight of the depth difference between the ith word and the jth word
$p_{i,j}$	entanglement coefficient between the ith word and the jth word
y	annotated sentence similarity by humans
x	calculated sentence similarity by the proposed model
$\sigma$, $\gamma$	threshold value of annotated sentence similarity
$\lambda$	threshold value of relative error
$\Delta E$	relative error between the experimental result and annotated score of sentence similarity
D	dimensionality of the sentence representation
$D_1$	sentence dimension reduced at the level of sentence embedding
$D_2$	sentence dimension reduced at the level of entangled word embedding

.

4.2. Datasets

Datasets include the SemEval Semantic Textual Similarity (STS) Tasks (years of 2012 (STS’12), 2014 (STS’14), 2015 (STS’15)) and STS-benchmark (STSb). E. Agirre et al. selected and piloted annotations of 4300 sentence pairs to compose the STS tasks in SemEval, including machine translation, surprise datasets and lexical resources [60]. E. Agirre et al. added new genres to the previous corpora, including 5 aspects in STS’14 [61]. In 2015, sentence pairs on answer pairs and belief annotations were added [62]. STS-benchmark and STS-companion include English text from image captions, news headlines and user forums [63]. In the study, the public lib of word2vec with 300 dimensions is assigned (http://code.google.com/archive/p/word2vec (accessed on 1 October 2020)). The input vectors of fasttext are 300 dimensions (http://fasttext.cc/docs/en/english-vectors.html (accessed on 1 October 2020)). The total number of sentence pairs for each corpus is summarized in

Table 2. The number of sentence pairs in each corpus.

STS’12	STS’14	STS’15
MSRvid (750)	deft-forum (450)	answers-forums (375)	STSb (4225)
SMTeuroparl (459)	deft-news (300)	answers-students (750)	STS’12 (3108)
OnWN (750)	headlines (750)	belief (375)	STS’14 (3750)
MSRpar (750)	images (750)	images (750)	STS’15 (3000)
SMTnews (399)	tweet-news (750)	headlines (750)
	OnWN (750)

(http://groups.google.com/group/STS-semeval (accessed on 1 October 2020)). The corpora consist of English sentence pairs and the annotated similarities ranging from $0.0$ to $1.0$ (divided by 5). All the grammatical structures of sentences in the provided model are generated by Stanford Parser models package (https://nlp.stanford.edu/software/lex-parser.shtml (accessed on 1 October 2020)).

4.3. Experimental Settings

To make the calculated sentence similarity $x_i$ approximately equal to the annotated score $y_i$, we perform a simple classification of the calculated results based on the relative error $\Delta E$:

$$\Delta E=\frac{|x_i-y_i|}{y_i}.$$

(27)

When $\Delta E>\lambda$, we introduce the optimized models to recompute the sentence similarities. This is performed in order to reduce the difference between the calculated value and the labeled value:

$$min\left({\Sigma }_{\Delta E>\lambda }\right|x_i-y_i|+{\Sigma }_{\Delta E\le \lambda }|x_i-y_i\left|\right).$$

(28)

Then, we divide the calculated results into two parts. The first is computed by the short-range entanglement, and the other is modeled by the optimized models. When $\Delta E\le \lambda$, the calculated result is considered feasible and is stored as $x_{0i}$. When $\Delta E>\lambda$, the calculated result is too large and must be optimized. To exploit the advantages of the two dimensionality reduction models, we divide the sentence pairs with $\Delta E>\lambda$ into two parts according to the annotated scores $y_i$. When $\sigma <y_i\le 1$, the sentence similarity is recomputed by the sentence-level dimensionality reduction model with the result of $x_{1i}$. When $\gamma \le y_i<\sigma$, we utilize the entangled-word-level dimensionality reduction model to recompute the sentence similarity, with the calculated result of $x_{2i}$. The algorithm is listed in Algorithm 1.

Algorithm 1: Framework of sentence embedding based on constituency parser and relation entity for semantic similarity computation.

5. Experimental Results

5.1. Comparing with Some Unsupervised Methods

The Spearman’s rank correlation and Pearson correlation are used to compare the experimental results, as shown in

Table 3. Comparison of the Spearman’s rank correlation (Src) in each dataset.

Model	STS’12	STS’14	STS’15	STSb	Avg.
SimCSE-base [64]	0.702	0.732	0.814	0.802	0.763
IS-Bert-NLI [65]	0.568	0.630	0.752	0.692	0.661
Bert-flow [66]	0.652	0.694	0.749	0.723	0.705
DINO [67]	0.703	0.713	0.805	0.778	0.750
SBERT-base [68]	0.710	0.732	0.791	0.770	0.751
Provided model	0.641	0.774	0.844	0.830	0.772

and

Table 4. Comparison of the Pearson correlation coefficient (Pcc) in each dataset.

Dataset	ACVT [69]	$\mathbf{SCBOW}$-$\mathbf{att}$ [70]	$\mathbf{PP}$-$\mathbf{att}$ [70]	$\mathbf{No}$ $\mathbf{Tree}$-$\mathbf{Based}$ [4]	$\mathbf{Provided}$  $\mathbf{Model}$
12’MSRpar	0.58	0.58	0.50	0.59	0.71
12’MSRvid	0.83	0.83	0.85	0.90	0.92
12’SMTeuroparl	0.43	0.52	0.52	0.69	0.74
12’OnWN	0.70	0.73	0.73	0.84	0.82
12’SMTnews	0.54	0.66	0.67	0.78	0.81
STS’12	0.62	0.66	0.65	0.76	0.80
14’deft-forum	0.48	0.54	0.56	0.66	0.68
14’deft-news	0.74	0.74	0.76	0.78	0.75
14’headlines	0.72	0.72	0.72	0.81	0.82
14’images	0.81	0.81	0.83	0.87	0.87
14’OnWN	0.87	0.87	0.85	0.92	0.93
14’tweet-news	0.75	0.82	0.79	0.82	0.90
STS’14	0.73	0.75	0.75	0.81	0.82
15’answers-forums	0.69	0.69	0.69	0.86	0.88
15’answers-students	0.79	0.79	0.79	0.86	0.89
15’belief	0.70	0.78	0.78	0.87	0.88
15’images	0.82	0.84	0.85	0.85	0.86
15’headlines	0.79	0.79	0.77	0.89	0.86
STS’15	0.76	0.78	0.78	0.86	0.87

. Table 3 shows that the Srcs of STS’14, STS’15 and STSb are significantly higher than the comparison models, but the average Src of our model is higher than all the comparison models. The results in Table 4 show that, except for 4 corpora, the Pcc of all corpora has increased. The average Pccs of STS’12, STS’14 and STS’15 have all been significantly improved, especially the maximum improvement rate of the provided model compared to ACVT on STS’12 has reached $29\%$. The growth rates of the top three are $20.3\%$, $9.8\%$ and $7.2\%$ for STS’12.MSRpar, STS’14.tweet-news and STS’12.SMTeuroparl, respectively. Moreover, STS’12.SMTeuroparl achieves a growth rate of $42.3\%$. Comparing all the Pccs of the 16 corpora, The Pccs of 3 corpora are above $0.9$, of 3 corpora are below $0.8$, and only of 1 corpus is below $0.7$. Though the Pcc of STS’14.deft-forum is less than $0.7$, it is also increased slightly, with an increase of $0.02$.

To compare the semantic influence of the input word embedding, the entanglement coefficients of word2vec and fasttext for each corpus in

Table 5. The semantic influence of the input word embedding.

		${\mathbf{STS}}^{’}12$						${\mathbf{STS}}^{’}14$
		MSRpar	MSRvid	SMTeu	OnWN	SMTnews		deft-f	deft-n	headlines	images	OnWN	tweet-n
word2vec	Pcc	0.71	0.91	0.74	0.82	0.81		0.68	0.75	0.82	0.87	0.92	0.89
word2vec	MSE	0.022	0.017	0.047	0.020	0.022		0.053	0.033	0.029	0.022	0.022	0.015
fasttext	Pcc	0.51	0.84	0.42	0.74	0.60		0.51	0.63	0.76	0.76	0.79	0.86
fasttext	MSE	0.028	0.035	0.086	0.049	0.073		0.055	0.039	0.035	0.040	0.062	0.028
		${\mathbf{STS}}^{’}\mathbf{15}$						$\mathbf{STSb}$
		answ-for	answ-stu	belief	images	headlines
word2vec	Pcc	0.88	0.89	0.88	0.86	0.86		0.86
word2vec	MSE	0.016	0.017	0.023	0.033	0.030		0.028
fasttext	Pcc	0.78	0.76	0.78	0.81	0.81		0.78
fasttext	MSE	0.029	0.041	0.037	0.037	0.044		0.037

are set to the same value, that is, the influence of the syntax tree is mainly considered. For easy comparison, the entanglement coefficients $p_{i,j}$ of ‘word2vec’ and ‘fasttext’ are set to the same values for each corpus in the table; namely, the influence of the long-range dependency relation is mainly considered. Comparison of the experimental results in Table 5 shows that all of the MSEs of ‘word2vec’ are smaller than those of ‘fasttext’. There are 5 datasets with a difference of less than $0.01$ and 5 corpora with an MSE with a multiplied value greater than 2. The main reason for this result is that all of the MSEs are adjusted under the parameters of ‘word2vec’, which are not the optimal parameters of ‘fasttext’. Additionally, the maximum value of ‘word2vec’ is less than $0.055$, and that of ‘fasttext’ is less than $0.09$.

5.2. Influence of PoS Combination Weight $T_{I,J}$

We mainly discuss the influence of notional words on sentence semantics including noun, verb, adjective and adverb. The four PoS types of notional words are combined in pairs to form 16 different PoS combinations. The weights of other PoS combinations are set to $0.5$.

In

Table 6. Influence of parameters $t_{i,j}$ on sentence semantics of different corpora. The figures in bold-type refer to the maximum Pearson correlation coefficient of each corpus. The figures in red-type refer to the minimum mean squared error of each corpus.

Year	Dataset		$\mathbf{Pcc}$			$\mathbf{MSE}$
		${\mathit{t}}_{\mathit{i},\mathit{j}}=\mathbf{0.5}$	${\mathit{t}}_{\mathit{i},\mathit{j}}=\mathit{second}$	${\mathit{t}}_{\mathit{i},\mathit{j}}=\mathit{first}$	${\mathit{t}}_{\mathit{i},\mathit{j}}=\mathbf{0.5}$	${\mathit{t}}_{\mathit{i},\mathit{j}}=\mathit{second}$	${\mathit{t}}_{\mathit{i},\mathit{j}}=\mathit{first}$
	MSRpar	0.688	0.711	0.703	0.0248	0.0224	0.0235
	MSRvid	0.882	0.881	0.872	0.0248	0.0250	0.0270
2012	SMTeuroparl	0.728	0.732	0.732	0.0987	0.1013	0.1023
	OnWN	0.825	0.824	0.818	0.0179	0.0183	0.0196
	SMTnews	0.773	0.779	0.770	0.0195	0.0200	0.0198
2014	deft-forum	0.654	0.658	0.665	0.0515	0.0519	0.0432
	deft-news	0.750	0.747	0.752	0.0328	0.0337	0.0326
	headlines	0.805	0.806	0.794	0.0294	0.0294	0.0320
	images	0.872	0.876	0.873	0.0225	0.0215	0.0222
	OnWN	0.918	0.919	0.916	0.0257	0.0255	0.0262
	tweet-news	0.906	0.909	0.909	0.0161	0.0154	0.0151
	answers-forums	0.891	0.891	0.892	0.0214	0.0211	0.0209
	answers-students	0.842	0.843	0.843	0.0274	0.0275	0.0272
2015	belief	0.888	0.886	0.885	0.0233	0.0234	0.0232
	images	0.858	0.858	0.857	0.0326	0.0327	0.0328
	headlines	0.909	0.910	0.863	0.0200	0.0199	0.0301

, $t_{i,j}$ is set to different values to illustrate the influence of the parameters of the combination of PoS for different word entanglement. The word ‘first’ denotes the number combination of ‘$0.5$, $0.2$, $0.7$, $0.9$, $0.3$, $0.1$, $0.5$, $0.9$, $0.6$, $0.1$, $0.6$, $0.9$, $0.5$, $0.9$, $0.9$, $0.1$’, for which the numbers are scattered around 0.5. The word ‘second’ denotes the number combination of ‘$0.3$, $0.5$, $0.3$, $0.6$, $0.5$, $0.5$, $0.4$, $0.3$, $0.5$, $0.6$, $0.4$, $0.3$, $0.5$, $0.4$, $0.5$, $0.4$’, for which the numbers are concentrated around $0.5$. $t_{i,j}=0.5$ means that all sixteen weights of the PoS combinations are set to $0.5$. Comparing all of the Pccs, there are 10 out of 16 corpora achieving the maximum for $t_{i,j}=second$, with the greatest improvement by $0.037$ from STS’15.headlines. Additionally, the top three advances are derived from $t_{i,j}=second$, with an improvement of $0.037$ from STS’15.headlines, an increase of $0.023$ from STS’12.MSRpar and an increase of $0.012$ from STS’14.headlines. Comparing the three different MSEs of the same corpus in Table 6, the ratio of the minimums among the three PoS parameters combinations is 6:5:6, approximately evenly distributed. However, there are 8 out of 16 corpora reaching the maximum from $t_{i,j}=first$, and the other corpora average from $t_{i,j}=0.5$ and $t_{i,j}=second$. In brief, the Pcc and MSE are clearly affected by the distribution of the PoS combination weights. A more discrete parameter distribution corresponds to greater influence on the Pcc and MSE.

The detailed influence of $t_{i,j}$ on STS’12.OnWN is shown in

Table 7. Influence of parameters $t_{i,j}$ on sentence semantics of STS’12.surprise.OnWN.txt with the variables of $y_1=0.7$, $0.3<y_2<0.7$, $\nabla E_1=\nabla E_2=0.2$ and $D_1$ = 10,000.

${\mathit{t}}_{\mathit{i},\mathit{j}}$	${\mathit{D}}_2$	$\mathbf{Pcc}$	$\mathbf{MSE}$
$first$	$D_2$ = 75,000	0.80803	0.01465
$first$	$D_2$ = 80,000	0.81801	0.01962
0.5	$D_2$ = 75,000	0.81312	0.01404
0.5	$D_2$ = 80,000	0.82549	0.01792
$second$	$D_2$ = 75,000	0.81251	0.01413
$second$	$D_2$ = 80,000	0.82427	0.01833

. When $D_2$ changes, the trends of the Pcc and MSE with $t_{i,j}$ are still consistent. A higher concentration of $t_{i,j}$ corresponds to a larger Pcc and smaller MSE, as shown in Figure 3. Comparison of the Pccs in Figure 3a and the MSEs in Figure 3b shows that for $t_{i,j}=0.5$, the Pcc reaches the maximum and the MSE reaches the minimum. This result can be explained based on the entanglement coefficient $p_{i,j}=Sim\left(\right|w_i\rangle ,|w_j\rangle )\times t_{i,j}\times d_{i,j}$. When the entangled words $|w_i\rangle$ and $|w_j\rangle$ are determined, the cosine similarity of the two words is a fixed value. For the same input sentence, the dependency tree of the sentence is a certain structure; that is, the tree depth difference $\Delta h$ between the two words is a constant value so that the difference weight $d_{i,j}$ is a certain value. Hence, when the input sentence is known, $p_{i,j}$ is only changed by $t_{i,j}$. The value of $t_{i,j}$ has a strong effect. For example, when the two weights are $0.1$ and $0.9$, the weight of the PoS combination corresponding to $0.9$ is nine times that of the PoS combination corresponding to $0.1$. To compare the influence of other parameters, $t_{i,j}$ is set to $0.5$ for the experiments discussed below.

5.3. Influence of the Tree Depth Difference $\Delta H$

5.3.1. On STS-Benchmark

The influence of the tree depth difference $\Delta h$ on STS-benchmark is listed in

Table 8. Influence of parameters $p_h$ and $\Delta h$ on sentence semantics of STS-benchmark with the variables of $y_1=0.7$, $0.25<y_2<0.7$, $\nabla E_1=\nabla E_2=0.25$, $D_1$ = 10,000 and $D_2=$ 75,000.

${\mathit{d}}_{\mathit{i},\mathit{j}}$ Stays the Same	$\mathbf{\Delta }\mathit{h}$ Changes	$\mathbf{Pcc}$	$\mathbf{MSE}$
(1.5, 1.2, 0.8)	($\Delta h=0$, $0<\Delta h\le 2$, $\Delta h>2$)	0.85837	0.02869
	($\Delta h=0$, $0<\Delta h\le 3$, $\Delta h>3$)	0.85829	0.02868
	($\Delta h=1$, $1<\Delta h\le 3$, $\Delta h>3$)	0.85812	0.02868

. The list named ‘$\Delta h$ changes’ represents the condition, and the list labeled ‘$d_{i,j}$ stays the same’ describes the values under this condition. For example, the condition combination of ‘(1.5, 1.2, 0.8)’ and ‘($\Delta h=0$, $0<\Delta h<=2$, $\Delta h>2$)’ denotes that when $\Delta h=0$, $d_{i,j}=1.5$; when $0<\Delta h<=2$, $d_{i,j}=1.2$; and when $\Delta h>2$, $d_{i,j}=0.8$. The condition $\Delta h=0$ that denotes the tree depth difference between the two words is zero; that is, the two words have a direct modified relationship. When $\Delta h$ changes, the Pcc and MSE show very small changes. The main reason for this result is that the corpus consists mostly of short sentences, with sentence lengths of less than 10. A shorter sentence tends to have a simpler parser tree structure.

5.3.2. On STS’14.deft-News

The influences of the tree depth difference $\Delta h$ and the tree depth difference weight $d_{i,j}$ on STS’14.deft-news are illustrated in

Table 9. Influence of parameters $p_h$ and $\Delta h$ on sentence semantics of STS’14.deft-news.txt with the variables of $y_1=0.7$, $0.25<y_2<0.7$, $\nabla E_1=\nabla E_2=0.2$, $D_1$ = 10,000 and $D_2=$ 85,000.

${\mathit{d}}_{\mathit{i},\mathit{j}}$Stays the Same	$\mathbf{\Delta }\mathit{h}$Changes	$\mathbf{Pcc}$	$\mathbf{MSE}$
(1.5, 1.2, 0.8)	($\Delta h=0$,   $0<\Delta h\le 2$,   $\Delta h>2$)	0.75023	0.03280
	($\Delta h=0$,   $0<\Delta h\le 4$,   $\Delta h>4$)	0.74806	0.03296
	($\Delta h=0$,   $0<\Delta h\le 3$,   $\Delta h>3$)	0.74431	0.03340
	($\Delta h\le 1$,   $1<\Delta h\le 3$,    $\Delta h>3$)	0.74687	0.03313
	($\Delta h\le 1$,   $1<\Delta h\le 4$,   $\Delta h>4$)	0.74554	0.03312
	($\Delta h\le 1$,   $1<\Delta h\le 5$,   $\Delta h>5$)	0.74560	0.03310
	($\Delta h\le 1$,   $1<\Delta h\le 6$,   $\Delta h>6$)	0.74532	0.03313
	($\Delta h\le 1$,   $1<\Delta h\le 7$,   $\Delta h>7$)	0.74533	0.03313
	($\Delta h\le 1$,   $1<\Delta h\le 8$,   $\Delta h>8$)	0.74540	0.03312
	($\Delta h\le 1$,   $1<\Delta h\le 9$,   $\Delta h>9$)	0.74682	0.03302
$\Delta \mathit{h}$ Stays the Same	${\mathit{d}}_{\mathit{i},\mathit{j}}$ Changes	$\mathbf{Pcc}$	$\mathbf{MSE}$
($\Delta h=0$, $0<\Delta h\le 2$, $\Delta h>2$)	(1.5, 1.2, 0.6)	0.75083	0.03276
	(2.0, 1.2, 0.6)	0.75083	0.03276
($\Delta h=0$, $0<\Delta h\le 2$, $\Delta h>2$)	(2.0, 1.5, 1.0)	0.74690	0.03292
	(2.0, 1.5, 0.8)	0.75023	0.03280
($\Delta h=0$, $0<\Delta h\le 2$, $\Delta h>2$)	(1.5, 1.2, 0.6)	0.75083	0.03276
	(1.5, 1.5, 0.6)	0.75008	0.03285
($\Delta h=0$, $0<\Delta h\le 2$, $\Delta h>2$)	(2.0, 1.5, 1.0)	0.74690	0.03292
	(1.5, 1.2, 0.6)	0.75083	0.03276

. In the STS’14.deft-news corpus, there are some long-sentences with more than 20 words. The complexity of the sentence structure tends to increase with the sentence length.

We use the variable-controlling approach to discuss the impact of $\Delta h$ and $d_{i,j}$ on semantics from two aspects, as shown in the upper and lower parts of Table 9. The upper half shows the semantic changes when $\Delta h$ changes. The detailed influence of the tree depth difference $\Delta h$ between the entangled words with indirect modified relationship on sentence semantics is illustrated in Figure 4. As shown in Figure 4a, when $2\le \Delta h<4$, the Pcc decreases markedly, whereas for $\Delta h\ge 4$, the change curve of the Pcc is almost horizontal. Examination of Figure 4b shows that the MSE changes only slightly. The lower part of Table 9 shows the influence of the weight $d_{i,j}$ of the direct modified relationship and that of the indirect modified relationship on sentence semantics. The values in blue are the values of the variable changes, and the values in red indicate the maximum Pcc and the minimum MSE. When the weight of direct modified relationship is altered, the Pcc and MSE remain unchanged. However, comparing the second group and the third group, it is found that when $d_{i,j}$ increases, the Pcc decreases and MSE increases, implying that entangled words modified indirectly will produce semantic errors. Additionally, when all three weights increase, the Pcc decreases and the MSE increases. Hence, when two adjacent words are entangled together by the tensor, if their modified relationship is indirect, semantic errors will be introduced.

Comparison of the results presented in Table 8 and Table 9 shows that the influence on the sentence semantics of the long-range entanglement of words with modified relationship is more apparent than that of the short-range entanglement.

5.4. Influence of the Dimension Reduction

5.4.1. Influence on STS’15.Images

The influences of dimension $D_1$ of the sentence dimensionality reduction and dimension $D_2$ of the entangled word dimensionality reduction on STS’15.images are illustrated in Figure 5. As $D_2$ increases, the Pcc first decreases slowly and then fluctuates with smaller variables, as shown by the orange curve in Figure 5a. Excluding $D_1$ = 15,000, the overall change in the MSE is small; as $D_1$ increases, the MSE gradually increases, as explained by the orange curve in Figure 5b. As $D_2$ increases, the Pcc first increases and then decreases, and the MSE first decreases and then increases, as shown in Figure 5c and Figure 5d, respectively. When the Pcc reaches the maximum, the MSE is not the minimum, such that $D_2$ = 75,000. When $D_2$ = 80,000, the MSE reaches the minimum, but the Pcc is not at the maximum.

5.4.2. Influence on STS’15.Headlines

The effects of dimension $D_1$ of dimensionality reduction at the sentence level and dimension $D_2$ of the dimensionality reduction at the entangled word level on sentence semantics in STS’15.headlines are described in Figure 6. With the except of $D_1$ = 15,000, with increasing $D_1$, the Pcc first increases and then linearly decreases, while the MSE first decreases and then gradually increases, as shown by the orange curves in Figure 6a,b. However, considering all of the experimental results including $D_1$ = 15,000, the change curves of the Pcc and MSE both show large fluctuations, as presented by the blue curves in Figure 6a,b. Moreover, the Pcc of $D_1$ = 15,000 is higher than all of the Pcc values, and the MSE of $D_1$ = 15,000 achieves the minimum value. Comparisons of Figure 6c,d show that with the increasing of $D_2$, the Pcc first increases slowly and then decreases sharply, while the MSE decreases slightly and then increases rapidly.

5.4.3. Summary

Comparisons of the orange curves to the blue curves in Figure 5a,b and Figure 6a,b show that the change trend of the orange curves is smoother than that of the blue curves. The blue curves only consider one more point $D_1$ = 15,000 than the orange curve. Hence, the optimum value is related to the interval variables. A larger interval variable corresponds to greater fluctuations of the the curve, as observed by the comparison of the orange curves and the blue curves of Figure 5a,b and Figure 6a,b. A smaller interval variable makes it more likely that the optimal value will be obtained. This result shows that the optimal solution obtained when the interval variable is large may not even be the local optimal value.

6. Conclusions and Future Works

This paper proposes a quantum entangled word representation based on syntax trees to represent sentences. When the sentence structure is complex, the two words that have a direct modified relationship are not necessarily in close proximity. Introducing quantum entanglement between words that have long-range dependencies enables remote words to also directly establish modified relationships. Combining the attention mechanism based on the dependency tree with the quantum entanglement coefficient, the entanglement coefficient between words is related not only to the PoS combination of the two words and the distribution of the two words in the dictionary but also to the modified relationship between the words. Utilizing the dependency trees of sentences to establish long-distance connections between words only considering the entangled words reduces semantic errors. Moreover, the use of the dependency tree-based attention weight can reduce the influence of adjacent entangled words without directly modifying the sentence semantics, thereby more accurately expressing the sentence semantics. We also discuss the impact of PoS combination, tree depth difference, and dimensionality reduction of entangled words on the sentence semantics. As the maximum of Pcc and the minimum value of MSE obtained here are not the optimal solutions, and may not even be the local optimal solutions, in future works, we mainly consider how to obtain the optimal value easily and effectively by introducing the theory of convex optimization to generalize this model. In this model, the semantic expansion of associated words is obtained by the tensor product of word vectors of the two related words, which is not applicable to the short sentences consisting of only one content word.