RAG-TCGCN: Aspect Sentiment Analysis Based on Residual Attention Gating and Three-Channel Graph Convolutional Networks

Abstract

Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis task that mainly judges the polarity of a given aspect word in a review. Current methods mainly use graph networks to do aspect-level sentiment classification tasks, most of which use syntactic or semantic graphs, and utilize attention mechanisms to interact and correlate aspect terms and contexts to obtain more useful feature representations. However, these methods may ignore some insignificant syntactic structures and some implicit information in some sentences. The attention mechanism then easily loses the original information, which eventually leads to inaccurate sentiment analysis. In order to solve this problem, this paper proposes a model based on residual attention gating and three-channel graph convolutional network (RAG-TCGCN). Firstly, the model uses a three-channel network composed of syntactic information, semantic information, and public information to simultaneously optimize and fuse through the multi-head attention mechanism to solve the problem of sentences without significant syntactic structure and with implicit information. Through the residual attention gating mechanism the problem of loss of original information is solved. Experimental verification shows that the accuracy and F1 value of the model are improved on the three public datasets.

1. Introduction

Sentiment analysis is an important branch of natural language processing [1,2,3], which has attracted increasing attention in recent years. Sentiment analysis is considered to be the foundation for the realization of powerful artificial intelligence in the future and a prerequisite for machines to fully understand human language. Therefore, sentiment analysis is a key technique. Sentiment analysis can be divided into two types: fine-grained sentiment analysis and coarse-grained sentiment analysis [4]. Coarse-grained sentiment analysis can be divided into document-level and sentence-level sentiment analysis [5]. Early sentiment analysis efforts were mainly based on document-level and sentence-level, which were designed to detect the sentiment polarity (e.g.,: positive, neutral, or negative) of the entire document or sentence. However, this method leads to inaccurate sentiment analysis for some sentences with multiple polarities. For example: “The food at this restaurant is great, but the service is terrible!” Such a sentence is likely to be classified as neutral based on document-level and sentence-level sentiment classification models. However, for a person to know that it contains both positive and negative emotional polarities in terms of “food” and “service”. Based on this phenomenon, many researchers began to conduct ABSA research. ABSA is a fine-grained sentiment analysis method. Currently, ABSA research can be divided into two major trends: one is to improve the accuracy of sentiment analysis based on known model improvements [6,7,8], and the other is to study how to simultaneously detect aspect and sentiment [9,10].

Usually, aspect-based sentiment analysis needs to go through multiple steps, such as obtaining word embeddings, encoding syntactic information, and extracting semantic information, among which mining the most relevant opinion words is the key. With the successful application of the attention mechanism, some researchers combine the attention mechanism with a neural network to complete various tasks, and have achieved good results. Later, a large body of work used attention mechanism-based neural networks for sentiment analysis [11,12]. However, the attention mechanism introduces noise, which causes the model to learn context words that are not relevant to the current aspect [13,14]. For example, in the comment “The food is delicious but the service was mediocre”, when analyzing the sentiment polarity of “food”, we want the model to only focus on sentiment words related to “food” in “the food is delicious”, rather than sentiment words unrelated to “food” in the entirety of the review text. Recently, the rapid development of graph neural networks has attracted great interest, and a class of graph neural networks has been designed to extract syntactic information from dependency trees due to its enormous ability to learn structural representations [15,16,17,18,19]. Compared with attention-based models, although these models have made many improvements, their shortcomings cannot be ignored. First, sentences have different sensitivities to grammatical information and semantic information. Those sentences whose syntactic structure is not obvious have an especially low sensitivity to syntactic information, which means that in some cases syntactic information may not help the model to determine the sentence’s emotional polarity.

In response to the above problems, we propose the RAG-TCGCN model, which uses a three-channel network composed of syntactic information, semantic information, and public information to simultaneously optimize and fuse through a multi-head attention mechanism to solve the problem of sentences without significant syntactic structure and with some implicit information. The attention mechanism can capture words related to the emotional expression of the aspect term, and the essence of the attention mechanism is to increase the weight of some information and weaken others [20]. However, the part weakened by the attention mechanism may lead to the loss of some useful information, and the attention mechanism will bring noise. Therefore, we use the residual attention gating mechanism to solve this problem, which addresses noise through gating and alleviates information loss through a residual mechanism.

The main contributions of this paper are as follows:

1. For the aspect-level sentiment analysis task, a model based on residual attention gating and a three-channel graph convolutional network (RAG-TCGCN) was designed, in which the three-channel graph convolutional network can adaptively learn important information according to the characteristics of sentences.

2. We employed a Multi-Head Self-Attention (MHSA) mechanism to capture the key information output by the three-channel graph convolution to refine the aspect-wise sentence features.

3. We propose a gating mechanism for residual attention, which can capture words that are more important for the sentiment analysis of current aspect terms. At the same time, this mechanism can also weaken the problem of loss of original information in the attention mechanism, reduce the loss of model information, and make model information extraction more complete and useful.

4. Experiments show the importance of proper use of syntactic, semantic and syntactic, and semantic combination information, the necessity of an attention-gated residual mechanism, and prove that residual attention-gating and a three-channel graph convolutional network demonstrate effectiveness and robustness in sentiment classification tasks.

The rest of this paper is organized as follows. In Section 2, we briefly review related work on aspect-based sentiment analysis. In Section 3, the RAG-TCGCN model details are presented. Section 4 evaluates the model and analyzes the experimental results on three public datasets. Section 5 concludes the paper.

2. Related Work

Traditional sentiment analysis tasks are mainly for document-level and sentence-level classification tasks. In contrast, the aspect-level sentiment analysis task is an entity-oriented sentiment analysis task and is also a fine-grained sentiment analysis task. The goal is to extract, for several aspects of an entity, the sentiment polarity of each aspect. Commonly used methods include a sentiment dictionary, machine learning, and deep learning. We compare the advantages and disadvantages of these methods in

Table 1. Comparison of different sentiment analysis methods.

Method	Characteristic	Representative	Comparison Results
sentiment dictionary	According to the sentiment polarity of sentiment words provided by different sentiment dictionaries, sentiment polarity can be divided at different granularity.	SentiWordNet, NTUSD, How Net, et al.	(1) Performance: deep learning is more accurate than traditional machine learning for sentiment classification. (2) Time and hardware: traditional machine learning requires less time and hardware than deep learning training.
machine learning	Generally, sentiment polarity is divided through two stages: feature extraction and classification algorithm design.	SVM, NBM, LR, et al.
deep learning	Simple neural network, attention-based neural network, and graph convolution network are mainly used to divide the sentiment polarity.	CNN, LSTM, GCN, et al.

. The specific methods are detailed as follows:

Using a sentiment dictionary method to solve an aspect-level sentiment analysis task is closer to the solution idea of the traditional sentiment analysis method, and it is necessary to use the marked sentiment dictionary to determine the sentiment polarity. Common sentiment dictionaries are: SentiWordNet [21], NTUSD [22], and How Net [23]. Nguyen et al. [24] further optimized the dictionary-based approach, achieving better performance through the kernel recognition aspect of the tree connected to the opinion. Ding et al. [25] proposed a dictionary-based method that extracts the corresponding sentiment polarity for both explicit and implicit aspects.

Compared with sentiment dictionary methods, machine learning methods are used more in the field of aspect-level sentiment analysis, while traditional machine algorithms mainly include: Naive Bayes (NB), Maximum Entropy (ME), Logistic Regression (LR), and the Support Vector Machine (SVM), etc. Pang et al. [26] used a movie review text as a corpus to analyze the text and found that the support vector machine (SVM) and Bayesian algorithm (Bayes) have better classification results. Waila et al. [27] used Bayesian and SVM to analyze movie reviews with a semantic-based SO-PMI-IR algorithm. Zhao et al. [28] used a machine learning method to perform a sentiment analysis on the review text data in the catering field using the Bayesian, C4,5 method.

Deep learning methods are widely used in the field of aspect-level sentiment analysis, and its main research can be roughly divided into three categories: simple neural networks, attention-based neural networks, and graph convolutional networks. Simple neural networks mainly include: the Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) network, etc. Chen [29] proposed the TextCNN neural network model, and applied the CNN network model to the field of text classification for the first time, dividing sentences into word-level processing, so that the text features could be extracted by the model itself, but the sentence context information could not be learned. Zhang et al. [30] used a recurrent neural network (RNN) to train sentence vectors of fixed dimensions for sentences of different lengths, adding sequence features of words and enriching the representation of sentence vectors, but this model exhibits the phenomenon of exploding gradient and vanishing gradient. Nowak [31] and others proposed using an LSTM network for sentiment classification, which can effectively solve the defect of RNN’s loss of sentence information due to distance, but cannot receive parallel input of text. The emergence of attention-based neural networks has promoted further progress in sentiment analysis tasks. The main idea is to capture and establish the connection between aspects and opinion words through the attention mechanism. Strictly speaking, attention-based neural network models can also be used as a way to use sentence structure information, since the distance between aspects and opinion words is generally not too far. Wang et al. [32] proposed an attention-based approach to acquire meaningful information from specific aspects. Furthermore, given the advantages of the multi-head attention mechanism in modeling contextual semantic relations, Song et al. [33] proposed an attention encoder network to map the hidden states and semantic interactions between target words and context words. Chen et al. [34] addressed the long-distance dependencies between aspects and opinion words using a multi-layer attention network. Li et al. [35] introduced a multi-layer attention mechanism to capture aspects of distant opinion words. Since graph convolutional networks [36] were first proposed, and due to their excellent performance in processing graph structural information, they were quickly applied to various tasks in the field of natural language processing, and achieved great results in aspect-level sentiment analysis. Zhang et al. [15] transformed the syntactic dependency tree into a graph structure and used the syntax to extract information. Sun et al. [16] proposed using graph convolutional networks to learn node representations on dependency syntactic trees, and using the resulting node representations, along with other features, for sentiment classification. Wang et al. [37] defined a unified aspect-oriented dependency tree structure and proposed a relational graph attention network (GAT) to encode a new tree structure for sentiment prediction. Huang et al. [38] used a dependency graph to directly propagate sentiment features from the syntactic context of aspect targets. Zhao et al. [8] constructed a graph by considering the emotional similarity between aspects in a sentence, and then utilized a graph convolutional neural network combined with an attention mechanism to extract the emotional dependencies between different aspects in a sentence.

3. Methodology

In this section, the proposed RAG-TCGCN model will be introduced in detail. The overall framework of the RAG-TCGCN model is shown in Figure 1. Inspired by dualGCN [39], the key idea of this paper is to add a common information sharing channel based on the syntactic graph and semantic graph, and then use a residual attention gating mechanism to form a three-channel graph convolution network based on residual attention gating. This can solve the above-mentioned problems well. The RAG-TCGCN model framework in this paper mainly includes an input layer, a graph convolution module, an attention module, and an output layer. The graph convolution module consists of syntactic, semantic, and public information modules. The various components of the model are described in detail as follows.

3.1. Input Layer

The input layer is mainly composed of word embedding and sentence encoding.

(1) Word embedding: Given a sentence consisting of n words, $\mathrm{s}=\left\{{\mathrm{w}}_1{,\mathrm{w}}_2,\cdots {,\mathrm{w}}_{\mathrm{T}+1}\cdots {,\mathrm{w}}_{\mathrm{T}+\mathrm{m}},\cdots {,\mathrm{w}}_{\mathrm{n}-1}{,\mathrm{w}}_{\mathrm{n}}\right\}$ where $\left\{{\mathrm{w}}_{\mathrm{T}+1},\cdots {,\mathrm{w}}_{\mathrm{T}+\mathrm{m}}\right\}$. The representation has m aspects and each word in the sentence is looked up through the word embedding matrix $\mathrm{L}\in {\mathrm{R}}^{\left|\mathrm{V}\right|{\times \mathrm{d}}_{\mathrm{w}}}$ of pre-trained Glove [40], where |v| represents the size of the vocabulary and ${\mathrm{d}}_{\mathrm{w}}$ represents the dimension of word embeddings. Thus each word is embedded in a low-dimensional vector.

(2) Sentence encoding: In this paper, a bidirectional LSTM is used for encoding. In this way, the hidden representation of a given sentence, s, can be extracted. Given that two LSTM forward and reverse time series are connected to one output, this is expressed as follows:

$$\overrightarrow{\mathrm{H}}=\left\{\overrightarrow{{\mathrm{h}}_1},\overrightarrow{{\mathrm{h}}_2},\cdots ,\overrightarrow{{\mathrm{h}}_{\mathrm{T}+1}},\cdots \overrightarrow{{\mathrm{h}}_{\mathrm{T}+\mathrm{m}}},\overrightarrow{{\mathrm{h}}_{\mathrm{n}-1}},\overrightarrow{{\mathrm{h}}_{\mathrm{n}}}\right\}$$

(1)

$$\overleftarrow{\mathrm{H}}=\left\{\overleftarrow{{\mathrm{h}}_1},\overleftarrow{{\mathrm{h}}_2},\cdots ,\overleftarrow{h_{\mathrm{T}+1}},\cdots \overleftarrow{{\mathrm{h}}_{\mathrm{T}+\mathrm{m}}},\overleftarrow{{\mathrm{h}}_{\mathrm{n}-1}},\overleftarrow{{\mathrm{h}}_{\mathrm{n}}}\right\}$$

(2)

$$\mathrm{H}=[\overrightarrow{\mathrm{H}},\overleftarrow{\mathrm{H}}]$$

(3)

where $\overrightarrow{\mathrm{H}}$ represents forward, $\overleftarrow{\mathrm{H}}$ represents reverse, and H represents the final representation encoded by the bidirectional LSTM, which contains contextual information between aspect words and opinion words.

3.2. Syntactic Graph Convolution Moduler

Sun et al. [16] proposed encoding the dependency tree and combining the dependency paths between words to extend the GCN model. Therefore, this method is used to convert the dependency tree into a graph structure: ${\mathrm{G}}_{\mathrm{sy}}=\left({\mathrm{A}}_{\mathrm{sy}}{,\mathrm{H}}_{\mathrm{sy}}\right)$, where ${\mathrm{A}}_{\mathrm{sy}}$ is the adjacency matrix of the graph, and ${\mathrm{H}}_{\mathrm{sy}}$ the feature matrix. Then, this graph convolutional network is used to extract syntactic information, which is formulated as follows:

$${\mathrm{H}}_{\mathrm{i}}^{\mathrm{l}+1}=\mathrm{ReLU}\left({\displaystyle \sum }_{\mathrm{j}=1}^{\mathrm{n}}{\mathrm{A}}_{\mathrm{ij}}{\mathrm{W}}^{\mathrm{l}+1}{\mathrm{H}}_{\mathrm{j}}^{\mathrm{l}}{+\mathrm{b}}^{\mathrm{l}+1}\right)$$

(4)

$${\mathrm{H}}_1=\mathrm{H}$$

(5)

$${\mathrm{H}}^{\mathrm{sy}}=\left\{{\mathrm{H}}_1^{\mathrm{sy}}{,\mathrm{H}}_2^{\mathrm{sy}},\cdots {,\mathrm{H}}_{\mathrm{n}}^{\mathrm{sy}}\right\}$$

(6)

where ${\mathrm{A}}_{\mathrm{ij}}$ represents the adjacency matrix, ${\mathrm{W}}^{\mathrm{l}+1}$ represents the weight matrix, ${\mathrm{b}}^{\mathrm{l}+1}$ represents the bias value, ${\mathrm{H}}_1=\mathrm{H}$ represents the hidden state vector H from BiLSTM as the initial node representation in the syntactic graph, ${\mathrm{H}}_{\mathrm{i}}^{\mathrm{sy}}$ represents the hidden representation of the ith node, and ${\mathrm{H}}^{\mathrm{sy}}$ represents the grammar information representation.

3.3. Semantic Graph Convolution Module

Since some short sentences have ambiguous syntactic structures, if the model extracts syntactic information bluntly, it may lead to inaccurate prediction results. On the other hand, the syntactic structure is fixed, which may lead to aspects that are far from their opinion words or two words that should be connected on the dependency tree but are not, leading to missing important information. Inspired by Vaswani et al. [41], this paper designs a semantic graph module, which can directly capture the semantically related information of each word in the sentence through the attention mechanism, unlike the syntactic graph, which requires additional syntactic knowledge. This makes it more efficient than the syntactic graph structure as it is more flexible and can adapt to information that is not sensitive to syntactic information. The semantic information is formulated as follows:

$${\mathrm{A}}^{\mathrm{se}}=\mathrm{softmax}\left(\frac{{\mathrm{UW}}^{\mathrm{u}}\times {\left({\mathrm{PW}}^{\mathrm{p}}\right)}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}^{\mathrm{m}}}}\right)$$

(7)

$${\mathrm{H}}^{\mathrm{se}}=\mathrm{ReLU}\left({\displaystyle \sum }_{\mathrm{j}=1}^{\mathrm{n}}{\mathrm{A}}^{\mathrm{se}}{\mathrm{W}}^{\mathrm{l}+1}{\mathrm{H}}_{\mathrm{j}}^{\mathrm{l}}{+\mathrm{b}}^{\mathrm{l}+1}\right)$$

(8)

where ${\mathrm{A}}^{\mathrm{se}}$ is the score matrix of the attention mechanism, which is used as the adjacency matrix of the semantic module, while ${\mathrm{W}}^{\mathrm{u}}$ and ${\mathrm{W}}^{\mathrm{p}}$ represent the learning weight matrix, ${\mathrm{d}}^{\mathrm{m}}$ is the dimension of the input node feature, ${\mathrm{W}}^{\mathrm{l}+1}$ represents the weight matrix, ${\mathrm{b}}^{\mathrm{l}+1}$ represents the bias value, and ${\mathrm{H}}^{\mathrm{se}}$ represents the semantics Information representation.

3.4. Common Graph Convolution Module

According to Liina et al. [42], it can be seen that syntactic and semantic space are not completely separated, but are related to each other. As the syntactic structure of the sentence changes, the semantics will also change accordingly. Therefore, in order to better understand a sentence, it is necessary to establish the common information shared by syntactic and semantic spaces. Inspired by Wang et al. [18], a common graph convolution module with a parameter-sharing strategy was used to obtain information shared by both syntactic and semantic spaces. Public information is formulated as follows:

$${\mathrm{H}}_{\_\mathrm{csy}}{=\mathrm{H}}^{\mathrm{sy}}$$

(9)

$${\mathrm{H}}_{\_\mathrm{cse}}{=\mathrm{H}}^{\mathrm{se}}$$

(10)

$${\mathrm{H}}^{\mathrm{com}}=\frac{{\mathsf{\lambda }\mathrm{H}}_{\_\mathrm{csy}}{+\mathsf{\delta }\mathrm{H}}_{\_\mathrm{cse}}}{2}$$

(11)

where ${\mathrm{H}}_{\_\mathrm{csy}}$ stands for common syntax, ${\mathrm{H}}_{\_\mathrm{cse}}$ stands for common semantics, ${\mathrm{H}}^{\mathrm{com}}$ stands for common information representation, and $\mathsf{\lambda }$ and $\mathsf{\delta }$ stand for trainable parameters.

3.5. Attention Module

In this section, we divide attention into multi-head self-attention and residual-based gated attention. Multi-head self-attention is used to fuse the three-channel graph to refine features again, while residual-based gated attention, utilizing the residual mechanism directly, introduces the input information into the output result, reducing the loss of information and preventing the occurrence of network degradation. Therefore, we utilize an attention machine-based residual network to control the information flow, which makes information extraction more complete and useful.

3.5.1. Multi-Head Self-Attention

Multi-head self-attention (MHSA) uses multiple attention heads to capture information in parallel. Each attention head focuses on different aspects, and finally the information of each attention head is combined to obtain more refined characteristic information. Taking three identical ${\mathrm{H}}^{\mathrm{three}}=\left[{\mathrm{H}}^{\mathrm{sy}};{\mathrm{H}}^{\mathrm{com}};{\mathrm{H}}^{\mathrm{se}}\right]$ as input, MHSA is defined as:

$${\mathrm{Attention}}_{\mathrm{i}}\left({\mathrm{H}}^{\mathrm{three}},{\mathrm{H}}^{\mathrm{three}},{\mathrm{H}}^{\mathrm{three}}\right)=\mathrm{Soft}\max \left(\frac{{\mathrm{H}}^{\mathrm{three}}{\mathrm{H}}^{\mathrm{three}}{}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}\right){\mathrm{H}}^{\mathrm{three}}$$

(12)

$${\mathrm{head}}_{\mathrm{i}}={\mathrm{Attention}}_{\mathrm{i}}\left({\mathrm{H}}^{\mathrm{three}},{\mathrm{H}}^{\mathrm{three}},{\mathrm{H}}^{\mathrm{three}}\right)$$

(13)

$$\mathrm{MHSA}\left({\mathrm{H}}^{\mathrm{three}},{\mathrm{H}}^{\mathrm{three}},{\mathrm{H}}^{\mathrm{three}}\right)=\left({\mathrm{head}}_1\oplus {\mathrm{head}}_2\oplus \cdots \oplus {\mathrm{head}}_{\mathrm{h}}\right)·{\mathrm{W}}^{\mathrm{O}}$$

(14)

$${\mathrm{H}}^{\mathrm{left}}=\mathrm{MHSA}\left({\mathrm{H}}^{\mathrm{three}}{,\mathrm{H}}^{\mathrm{three}}{,\mathrm{H}}^{\mathrm{three}}\right)$$

(15)

where h represents the number of attention heads in multi-head attention, $\oplus$ represents vector connections, ${\mathrm{W}}^{\mathrm{O}}$ represents parameter matrix, ${\mathrm{head}}_{\mathrm{i}}$ represents the output of the ith attention head, and ${\mathrm{H}}^{\mathrm{left}}$ represents the refined feature representation after fusion.

3.5.2. Residual Attention Gating

In a text, the importance of each word to the emotional expression of the aspect word is different, and generally only a few words play a decisive role in the emotional expression. The residual mechanism directly introduces the input information into the output result, reducing the loss of information and preventing the occurrence of network degradation. Therefore, we utilize an attention-machine-based gated residual network to control the information flow, which makes information extraction more complete and useful. Attention gating is defined as follows:

$${\mathrm{O}}_{\mathrm{att}}=\mathrm{Soft}\max \left(\frac{{\mathrm{qk}}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}\mathrm{v}\right)$$

(16)

$$\mathrm{Gat}=\mathrm{sigmoid}\left({\mathrm{O}}_{\mathrm{att}}\right)$$

(17)

$${\mathrm{H}}^{\mathrm{right}}=\mathrm{mul}\left({\mathrm{O}}_{\mathrm{att}},\mathrm{Gat}\right)$$

(18)

where q, k, and v represent the output of the LSTM after linear transformation, and ${\mathrm{H}}^{\mathrm{right}}$ represents the processed representation of the original text information.

3.6. Output Layer

In this part, the information of the syntactic graph, semantic graph, and public graph needs to be refined through multi-head self-attention, fused with the original text information that has been specially processed, and finally the result of sentiment classification is output. However, before feature fusion, the corresponding aspect vectors ${\mathrm{h}}_{\mathrm{a}}^{\mathrm{left}}$ and ${\mathrm{h}}^{\mathrm{right}}$ need to be obtained from ${\mathrm{H}}^{\mathrm{left}}$ and ${\mathrm{H}}^{\mathrm{right}}$ using the average pooling operation. The formula is as follows:

$${\mathrm{h}}_{\mathrm{a}}^{\mathrm{left}}=\mathrm{f}\left({\mathrm{H}}^{\mathrm{left}}\right)$$

(19)

$${\mathrm{h}}^{\mathrm{right}}=\mathrm{f}\left({\mathrm{H}}^{\mathrm{right}}\right)$$

(20)

$${\mathrm{h}}_{\mathrm{a}}{=\mathrm{h}}_{\mathrm{a}}^{\mathrm{left}}{+\mathrm{h}}^{\mathrm{right}}$$

(21)

where f () represents the average pooling function, and Equation (21) represents the residual connection calculation, which combines the three-channel network feature representation from multi-head self-attention with the special processing information of the input original text to better generate a useful and complete feature representation, ${\mathrm{h}}_{\mathrm{a}}$. Next, through the softmax function, the probability, P, of emotional polarity is obtained, and the formula is as follows:

$$\mathrm{P}=\mathrm{soft}\max \left({\mathrm{h}}_{\mathrm{a}}\mathrm{W}+\mathrm{b}\right)$$

(22)

where W represents the learning weight and b represents the bias value. Finally, through the continuous optimization model of loss function, L is obtained:

$$\mathrm{L}=-{\displaystyle \sum }_{\mathrm{i}}^{\mathrm{c}}{\displaystyle \sum }_{\mathrm{j}}^{\mathrm{z}}\mathrm{P}\log \widehat{\mathrm{P}}$$

(23)

4. Experiment

4.1. Datasets

This paper selected three public datasets, Laptop, Restaurant, and Twitter comments [43], for experiments. Laptop and Restaurant were selected from the SemEval-2014 Task [44]. The three datasets contain three different sentiment polarities, negative, neutral, and positive, and each sentence in these datasets is labeled with an aspect and its corresponding polarity. The specific data distribution is shown in

Table 2. Statistics of datasets.

Dataset	Division	Positive	Negative	Neutral
Rest14	Training	2164	807	637
	Testing	728	196	196
Lap14	Training	994	851	455
	Testing	341	128	167
Twitter	Training	1507	1528	3016
	Testing	173	169	336

.

4.2. Dataset Implementation and Parameter Settings

The experimental environment of this paper is shown in

Table 3. Experimental environment.

System	Windows 10
CPU	Intel(R) Core(TM) i7-10510u
GPU	NVIDIA GeForce MX250
Language	Python 3.8
Tool	Pycharm 2021

. For all experiments, the pre-trained 300-dimensional Glove word vector was used to initialize the word embedding, the 1-layer BiLSTM, 2-layer GCN, and Adam optimizer were used for optimization, and the remaining detailed experiments were carried out. The parameters are shown in

Table 4. Experimental parameters.

Parameter Settings	Value
embed_dim	300
batch_size	16
rnn_hidden	50
input_dropout	0.7
gcn_dropout	0.1
num_epoch	50
learning_rate	0.002

.

4.3. Baseline Methods

In this paper, the following mainstream models were selected for comparison and run in the same experimental environment. The results are shown in

Table 5. Experimental results comparison on three publicly available datasets.

Models	Restaurant		Laptop		Twitter
	Accuracy	Macro-F1	Accuracy	Macro-F1	Accuracy	Macro-F1
IAN	78.60	-	72.10	-	-	-
AOA	80.53	69.84	72.88	67.48	72.25	69.96
RAM	80.23	70.80	74.49	71.35	69.36	67.30
MGAN	81.25	71.94	75.39	72.47	72.54	70.81
ASGCN-DG	80.77	72.02	75.55	71.05	72.15	70.40
ASGCN-DT	80.86	72.19	74.14	69.24	71.53	69.68
CDT	82.30	74.02	77.19	72.99	74.66	73.66
BiGCN	81.97	73.48	74.59	71.84	74.16	73.35
R-GAT	83.30	76.08	77.42	73.76	75.57	73.82
DGEDT	83.90	75.10	76.80	72.30	74.80	73.40
DualGCN	84.27	78.08	78.48	74.74	75.92	74.29
RAG-TCGCN	84.09	77.02	78.80	75.04	76.66	75.41

The symbol ‘-’ indicates this result is not available in their work. The best are in bold.

.

(1)

IAN [12]: Simultaneous modeling of aspect words and context information to make aspect words and context information interactively fuse with attention.

(2)

AOA [45]: Simultaneous modeling of aspects and text using long short-term memory neural networks to focus on what is important in sentences.

(3)

RAM [34]: Mem-Net is enhanced with deep Bi-LSTM and position weighting. The gated recurrent unit network is nonlinearly combined with multiple attention results using a recurrent network to obtain the final feature representation.

(4)

MGAN [46]: Fine-grained attention is proposed to solve the loss caused by coarse-grained attention, and then combined with coarse-grained attention to predict the sentiment polarity of sentences.

(5)

ASGCN [15]: GCNs on syntactic dependency trees are built and generate aspect-oriented sentence representations by applying masking and an attention mechanism. Finally, two variants of ASGCN are proposed, namely, ASGCN-DG based on an undirected dependency graph and ASGCN-DT based on a directed dependency tree.

(6)

CDT [16]: BiLSTM is used to obtain the feature representation of the sentence and further enhance the embedding by a direct convolution operation on the dependency tree.

(7)

BiGCN [47]: A conceptual hierarchy is built on the syntactic and lexical graphs to distinguish various types of dependencies or lexical word pairs, and a two-layer interactive graph convolutional network is designed to take full advantage of these two graphs.

(8)

R-GAT [37]: An aspect correlation tree rooted in aspect terms by reshaping the dependency parse tree is constructed, which uses relational graph attention network coding.

(9)

DGEDT [6]: A dual-transformer structure is designed to enable interaction enhancement between planar representations learned from the transformer and graph-based representations.

(10)

DualGCN [39]: The dual-graph convolutional network considers both the complementarity of the syntactic structure and semantic relevance.

The comparison models selected in this paper can be divided into two categories: attention mechanism and graph neural network. IAN, AOA, RAM, and MGAN mainly use an attention mechanism to complete classification tasks. ASGCN, CDT, BiGCN, R-GAT, DGEDT, and DualGCN all use dependency trees to obtain syntactic information and utilize graph neural networks for classification tasks.

4.4. Comparative Results and Analysis

The results of comparison with all baseline models are shown in Table 5. We drew the following conclusions:

(1)

On the three datasets of Restaurant, Laptop and Twitter, our model performance is better than the attention-based and syntax-based models, which shows that the RAG-TCGCN model performs better in encoding syntactic and semantic information through the adaptive fusion of syntactic and semantic information.

(2)

On the three datasets, the performance of the syntax-based model and the attention-based model is very different. The syntax-based model (ASGCN, CDT, BiGCN) is superior to the attention-based model (IAN, AOA, RAM). The main reason is that syntactic structure can more effectively capture the relationship between aspects and corresponding sentiment words, and extract more useful information.

(3)

Compared with attention-based models (MGAN, IAN), our model demonstrates significant improvement. The IAN model mainly obtains the initial feature representation through an LSTM pre-training model, and then obtains relatively rich semantic feature information through an attention mechanism. The effect of the model highly depends on whether the attention mechanism can accurately establish the connection between aspects and context. However, due to the complexity of a sentence and the inadequacy of the attention mechanism in capturing long-distance dependent information, this introduces some irrelevant information and generates noise, resulting in a poor performance of the model. However, our model constructs the relationship between aspects and opinion words by using syntactic information. Therefore, the noise introduced by the attention mechanism is avoided.

(4)

Compared with syntax-based models (R-GAT, DGEDT), our model's performance demonstrates significant improvement. Syntax-based models mainly obtain local features through syntactic structures and establish word-word relationships, but they ignore the global information and the semantic information between words. However, when there are sentences with obscure grammatical structure or complex sentences, extracting features only by syntactic knowledge leads to poor results.

(5)

Compared with the DualGCN model, our RAG-TCGCN model shows an improvement of 0.32% and 0.74% on Laptop and Twitter accuracy, respectively, and 0.3% and 1.12% on F1. Although a DualGCN model extracts information features syntactically and semantically, due to the complexity of sentences, each sentence has a different sensitivity to syntax and semantics. Therefore, important information cannot be obtained by self-learning according to the characteristics of sentences. On the basis of syntax and semantics, our model adds public information channels to form a three-channel network, which can learn adaptively and fuse according to the characteristics of sentences. This allows it to obtain good results.

4.5. Ablation Study

For this section, further ablation studies were carried out to verify the effectiveness of each module in RAG-TCGCN, and the results are shown in

Table 6. Experimental results of ablation study.

Models	Restaurant		Laptop		Twitter
	Accuracy	Macro-F1	Accuracy	Macro-F1	Accuracy	Macro-F1
SynGCN	82.57	75.06	76.90	72.60	74.59	73.13
SemGCN	82.48	73.12	76.42	72.19	75.18	73.87
Comom	81.50	71.99	77.22	73.14	74.45	73.39
TCGCN	82.84	74.70	77.85	73.97	75.78	74.59
RAG-TCGCN	84.09	77.02	78.80	75.04	76.66	75.41

. The SynGCN (syntactic graph convolutional network) model represents the syntactic feature channel network; the semGCN (semantic graph convolutional network) model represents the semantic feature channel network; the common model represents the public information channel network; the TCGCN (three-channel graph convolutional network) represents the three-channel graph convolutional network.

It can be seen from Table 6 that, firstly, on the Restaurant and Laptop datasets, the syntax-based SynGCN model performed better than the semantics-based SemGCN model, suggesting that a rich knowledge of grammar can reduce dependent parsing errors. SemGCN models performed better than SynGCN models on Twitter data, mainly because there are more comments with obscure syntax structure in the Twitter dataset compared with the Restaurant and Laptop datasets. Secondly, on the three datasets, the performance of a single SynGCN, SynGCN, or Comom model is lower than that of a three-channel graph convolution network (TCGCN). The main reason is that, due to the complexity of sentences, each sentence has a different sensitivity to semantics and syntax. Therefore, it is difficult for a single-channel network to adaptively extract important information according to the characteristics of sentences. However, compared with our model, the performance of the three-channel graph convolution neural network (TCGCN) is still insufficient, mainly because the TCGCN model over-refines features, leading to the loss of some important original information, resulting in poor performance. Our model adds a residual attention gating mechanism to reduce the loss of important original information. Overall, our model achieved the best performance.

4.6. Case Study

To better understand our model, we compared different models in several test examples. The experimental results are shown in

Table 7. Visual analysis cases of IAN, SynGCN, SemGCN, Comom, and RAG-TCGCN.

Moedel	Aspect	Attention Visualization	Prediction	Label
IAN	food	Great food but terrible service	Neutral	Positive
	dinner	My wife and I recently visited the bistro for dinner and have a wonderfull experience	Positive	Neutral
	Windows11	Did not enjoy the new Windows11 and touchscreen functions	Neutral	Negative
SynGCN	food	Great food but terrible service	Positive	Positive
	dinner	My wife and I recently visited the bistro for dinner and have a wonderfull experience	Positive	Neutral
	Windows11	Did not enjoy the new Windows11 and touchscreen functions	Positive	Negative
SemGCN	food	Great food but terrible service	Positive	Positive
	dinner	My wife and I recently visited the bistro for dinner and have a wonderfull experience	Positive	Neutral
	Windows11	Did not enjoy the new Windows11 and touchscreen functions	Negative	Negative
Comom	food	Great food but terrible service	Positive	Positive
	dinner	My wife and I recently visited the bistro for dinner and have a wonderfull experience	Positive	Neutral
	Windows11	Did not enjoy the new Windows11 and touchscreen functions	Negative	Negative
RAG-TCGCN	food	Great food but terrible service	Positive	Positive
	dinner	My wife and I recently visited the bistro for dinner and have a wonderfull experience	Neutral	Neutral
	Windows11	Did not enjoy the new Windows11 and touchscreen functions	Negative	Negative

. The Attention Visualization column in the table shows the different attention scores for each model, using darker to lighter colors depending on how high or low they scored. In the first example, “Great food but terrible service”, the sentence contains two aspects, “food” and “service”, as well as two sentiment words, “great” and “terrible”. The attention-based IAN model does not capture the connection between the two well, resulting in inaccurate predictions. In the second and third examples, IAN models still fail to correctly establish the relationship between aspect words and sentiment words due to the complexity of the sentences. Although it is possible to directly establish a connection between an aspect word and sentiment words through syntactic structure, this connection does not exist in a sentence with no obvious syntactic structure. In the third example, the SynGCN model does not capture the feature representation of the keyword “not”, resulting in incorrect prediction results, while the SemGCN, and Comom models can focus on the semantic correlation between words and capture this “not” feature representation, thereby obtaining correct prediction results. In the second example, when using the syntax or semantics, public information convolution modules incorrectly allocate the highest attention to “wonderful”. Treating “wonderful” as a sentiment word for “dinner” eventually leads to errors in sentence judgment. However, our proposed RAG-TCGCN model adaptively reduces the attention to the irrelevant word “wonderful” and adaptively increases the score of “dinner”.

5. Conclusions

In this article, we proposed a RAG-TGCN model for ABSA tasks. It solves the shortcomings of attention-based and graph convolution operations in the dependency tree. First, in order to alleviate the problems of traditional dependency methods and lack of semantic information, we used a multi-head attention mechanism to assist and enhance syntax and semantics to obtain more important information. At the same time, in order to alleviate the noise caused by the attention mechanism and the challenge of losing some of the original important information due to the over-refined features, we added a residual attention gating mechanism to alleviate the loss of original important information. Experimental results on three benchmark datasets show that our model outperforms other baseline methods. In addition, ablation studies and case studies verify the role of each component of our proposed model. In future, we want to integrate domain knowledge so that the model can perform other classification tasks.