1. Introduction
Sentiment analysis has a wide range of applications across various industries and fields—for example, consumer feedback analysis [
1] and public opinion analysis [
2,
3]. As a subtask of sentiment analysis, aspect-level sentiment classification (ASC) aims to automatically distinguish the sentiment polarity of aspects in text [
4], which is one of the research hotspots in the field of natural language processing. Based on the sentence shown in
Figure 1, given the aspects ‘food’, ‘service’, and ‘environment’, aspect-level sentiment classification models should predict the corresponding polarities as positive, negative, and negative, respectively. An aspect typically refers to an entity or an attribute of an entity mentioned in text, which can be represented by a word or a text segment. Aspect-level sentiment classification offers finer granularity of sentiment information compared to sentence-level and document-level classifications. It has garnered increasing attention from academia and industry.
In recent years, aspect-level sentiment classification models based on deep learning have outperformed traditional models based on artificial features by automatically learning contextual information related to a given aspect in a sentence [
5,
6,
7,
8,
9]. Among these models, the attention mechanism-based ones are noted for their simplicity and effectiveness, receiving considerable attention [
10,
11]. This type of model can more accurately capture contextual information related to a specific aspect in a sentence through attention mechanisms, improving sentiment polarity judgment. While attention-based models have historically achieved excellent performance, they can often be susceptible to noise. For example, assuming that the given aspect is ‘service’ in
Figure 1, attention-based models may consider both ‘good’ and ‘horrific’ as its relevant contexts, making it challenging to accurately predict sentiment polarity. To address the above issue, researchers have endeavored to integrate dependency syntax information into aspect-level sentiment classification models. This integration aims to enhance the models’ ability to capture context information related to specific aspects, thereby improving performance. For example, Zhang et al. [
12], Sun et al. [
13], and Wang et al. [
14] leveraged the dependency syntax information of sentences through graph convolutional networks [
15] and achieved notable performance improvements. From the example depicted in
Figure 1, we can clearly observe the critical role of dependency syntax information. The aspect ‘service’ in the sentence depends on the word ‘horrific’, which conveys its negative polarity.
Promising performance has been achieved by existing aspect-level sentiment classification models that incorporate dependency syntax information. However, these methods suffer from two shortcomings: (1) They only leverage the dependency relations between words in the dependency syntax tree, disregarding the categories of these relations. As shown in
Figure 1, the words dependent on the aspect ‘food’ include ‘The’ and ‘good’. Without considering the categories of dependency relations, classification models still struggle to determine which word is crucial for ‘food’. When the related words are uncommon, knowing the categories of dependency relations becomes crucial for classification models to distinguish the importance of words. Therefore, a robust aspect-level sentiment classification model should simultaneously consider both the dependency relations between words and their categories. (2) Existing models typically predict the sentiment polarity of each aspect independently, overlooking the sentiment relationships between aspects within a sentence. Specifically, these models first decompose multiple aspects within a sentence. They then take the sentence and one aspect at a time as the input to predict the sentiment polarity for each aspect. Although the above strategy simplifies the design of ASC models, it fails to capture the sentiment relationships between multiple aspects within a sentence. For example, for the two aspects ‘food’ and ‘service’ shown in
Figure 1, it is easy to infer that their sentiment polarities are opposite because there is a conjunction ‘but’ between them. When the sentiment polarity of an aspect is expressed implicitly—for example, ‘food and service are horrific’—it becomes crucial for ASC models to understand the sentiment relationships between multiple aspects in a sentence in order to infer the sentiment polarities of these aspects.
To address the two aforementioned issues, we propose an aspect-level sentiment classification model based on hybrid graph neural networks. Specifically, we begin by constructing a hybrid graph based on both the dependency syntax tree and the adjacency relations between aspects. The constructed hybrid graph includes two types of nodes: words and aspects. It incorporates two types of edges: the dependency relations between words or between words and aspects (represented above the sentence in
Figure 1) and the adjacency relations between aspects (represented below the sentence). Then, building upon the context coding layer (
Section 3.1), we introduce multiple layers of hybrid graph neural networks to facilitate information transfer among words, between words and aspects, and among aspects (
Section 3.2). In the process of information transmission, we consider not only the dependency relations between words and their categories but also the sentiment relationships between aspects. Finally, the classification layer takes the semantic vector representations of aspects learned by the hybrid graph neural networks as the input and jointly predicts the sentiment polarities of all aspects in a sentence (
Section 3.3).
Existing models [
5,
16] use LSTM or graph neural networks to capture the sentiment relationships between multiple aspects within a sentence. However, a limitation of their models is its tendency to predict multiple aspects in a sentence as having the same sentiment polarity. By modeling both dependency types between words and adjacency relations between aspects, our model can place greater emphasis on connectives like
and and
but. This enables our model to effectively capture whether multiple aspects have the same or opposite sentiment polarities. Our experimental results based on three commonly used datasets show that our proposed model achieves performance comparable to or better than recent benchmark methods.
3. Method
A sentence may contain m aspects, where each aspect can be a word or a text segment within the sentence, and n represents the total number of words in the sentence. Most previous studies take the sentence S and one aspect as the input at a time to predict the sentiment polarity of that aspect, overlooking the sentiment relationships between multiple aspects with the sentence. Differing from these methods, our proposed model is based on a hybrid graph neural network, which takes both the sentence and all aspects contained within it as the input to jointly infer the sentiment polarities of these aspects. The main contribution of our work is the development of a hybrid graph neural network that fully leverages both the dependency syntax information and the relationships between aspects.
As shown in
Figure 2, the proposed aspect-level sentiment classification model comprises the input layer, the encoding layer, the hybrid graph neural network layers, and the classification layer. The input layer transforms words in a sentence into low-dimensional semantic vectors. The encoding layer then utilizes a BiLSTM network to further learn the contextualized representations of both words and aspects. The hybrid graph neural network layers first organize the aspects and words in a sentence into a hybrid graph. Subsequently, they capture relationships between words, between words and aspects, and among aspects. This approach enables the model to jointly learn improved semantic vector representations of multiple aspects within a sentence. Finally, the classification layer is responsible for generating the sentiment classification results. In the following subsections, we will elaborate on the input layer and the encoding layer (
Section 3.1), the hybrid graph neural network layers (
Section 3.2), and the classification layer, along with the loss function (
Section 3.3).
3.1. Input Layer and Encoding Layer
Most natural language processing models based on deep learning typically represent words and other inputs as semantic vectors. Specifically, the input layer first converts the
i-th word
in a sentence into the corresponding static word vector
, according to the pretrained GloVe [
35] embeddings. Then, based on the pretrained BERT [
36], the
i-th word
in the sentence is represented as the dynamic contextualized word embedding
. Lastly, BIO tags, which are commonly used in sequence tagging, are introduced to mark the aspect words in a sentence. The corresponding tag vector of the
i-th word
is denoted as
. The tags B, I, and O represent the first word of an aspect, the subsequent words of an aspect, and words not in an aspect, respectively. Concatenating the above three types of information yields the complete semantic representation of a word, which serves as the input to the encoding layer:
The encoding layer further learns the representation of each word conditioned on its sentence-level context as follows:
where BiLSTM refers to a bidirectional long short-term memory network.
An aspect in a sentence can be a single word or a text segment containing multiple words. For an aspect consisting of a single word, we directly use the word representation in Equation (
2) as its semantic representation. For an aspect containing multiple words, we compute its semantic representation by averaging the word representations. For convenience, we denote the semantic representations of
m aspects in a sentence as
, respectively.
3.2. Hybrid Graph Neural Network Layers
We combine multiple aspects in a sentence, along with words not belonging to these aspects, into a hybrid graph to jointly learn better semantic representations of these aspects. By introducing the hybrid graph neural network layers, we consider not only the dependency relations of words and their categories but also the sentiment relationships between multiple aspects.
3.2.1. Construction of Hybrid Graph
For a sentence
S, we construct a hybrid graph
, where
G is an undirected graph,
V is the set of nodes, and
E is the set of edges in the graph. There are two types of nodes: aspect nodes and word nodes. Specifically, each aspect in the sentence corresponds to an aspect node
in the graph, and each word not in any aspect corresponds to a word node
, where
m is the number of aspects in the sentence and
c is the number of words not in any aspect. As shown in
Figure 3, the boxes denote aspect nodes, and the circles represent word nodes.
A hybrid graph contains two kinds of edges: dependency edges and adjacency edges. Dependency edges exist between an aspect node and a word node, or between two word nodes. These dependency edges are derived from the dependency syntax tree of the sentence. Adjacency edges exist between aspect nodes and are based on the adjacency relationship between two aspects in a sentence.
The definition of a dependency edge is as follows: (1) A dependency edge exists between two word nodes if and only if there is a dependency relation between the two corresponding words in the dependency tree. (2) When there is a dependency relation between a word in an aspect and a word not in any aspect, a dependency edge exists between the corresponding aspect node and the word node. (3) The type of the dependency relation in the syntax tree is directly used as the category of the corresponding edge in the graph. There are two points that need to be noted. First, a hybrid graph does not contain the dependency edges between words within the same aspect because each aspect is treated as a cohesive unit. Second, there may be multiple dependency edges between an aspect node (which contains multiple words) and a word node. For convenience in processing, only the dependency edge with the smallest span (the shortest distance between two words involved) is retained.
The definition of an adjacency edge is as follows: (1) There is an adjacency edge between any two aspect nodes in the graph. (2) If two aspects are adjacent in the sentence, the category of the adjacency edge is ‘adjacent’ (AS.adj in
Figure 1); otherwise, the category of the edge is ‘interval’ (AS.inter in
Figure 1).
3.2.2. Information Transfer Mechanism
The hybrid graph neural network layers can effectively model the semantic relationships between nodes by transferring information among neighbor nodes, thereby enhancing the learning of semantic representations of nodes. To model the semantic relationships between nodes that are not directly connected, we stack
L hybrid graph neural network layers. Specifically, we initialize the corresponding aspect nodes and word nodes in a graph with outputs from the encoding layer. The aspect nodes are initialized as the obtained semantic vectors
, and the words’ nodes are initialized as the learned contextualized vectors
as defined in Equation (
2). For convenience, the initial representations of all nodes are uniformly referred to as
. Each edge category is randomly initialized as a low-dimensional vector, which serves as the parameters of our model. Then, we calculate the semantic representations of nodes at the
l-th (
) hybrid graph neural network layer based on the representations at the
-th layer. Formally, we obtain the semantic representation of the node
at the
l-th layer as follows:
where
is the set of indices of nodes adjacent to the node
,
represents the weight obtained via the attention mechanism for the corresponding node,
is the semantic vector of the edge between nodes
and
, FFN
l is a multi-layer feed-forward neural network used to calculate the edge category-enhanced node representation
, and Gate
l is a gating mechanism used to fuse representations of the node itself and its neighbor nodes. Lastly, the node representations
at the
L-th graph layer are used as the final semantic representations of
m aspects in the sentence
S, respectively.
In our hybrid graph neural network, the information transfer mechanism comprises three steps. First, edge category information is integrated with related nodes through a multi-layer feed-forward neural network (FFNl), resulting in edge category-enhanced node representations. Next, we employ an attention mechanism to selectively aggregate information from neighboring nodes, ensuring that more significant information receives greater weight. Finally, a gated mechanism is utilized to combine a node’s own information with that of its neighbors. This helps in filtering out noise and irrelevant information, leading to more accurate sentiment predictions. To some extent, the attention mechanism functions as a local information filter, while the gated mechanism serves as a global information filter, together improving the model’s contextual understanding.
3.3. Classification Layer and Loss Function
Based on outputs of the last graph neural network layer, the classification layer calculates the final classification results as follows:
where
is the predicted result for the
p-th aspect, and
and
are the parameters of the classification layer.
The commonly used cross-entropy loss function is adopted to guide the model training, defined as follows:
where
is the ground-truth label (one-hot vector) of the
p-th aspect and
represents the expected value of the predicted result about the ground-truth label.
4. Experiments
4.1. Datasets and Settings
To verify the effectiveness of our proposed method, we compared it with benchmark methods based on three datasets: Laptop, Restaurant, and MAMS. The Laptop and Restaurant datasets are derived from review data provided by the SemEval 2014 competition [
37]. The MAMS dataset [
38] contains at least two aspects with different sentiment polarities in each sentence, making the dataset more challenging. The statistics of the three datasets are shown in
Table 1. The officially provided Laptop and Restaurant datasets were split into only training and test sets, without validation sets. To determine the optimal hyper-parameters, we randomly selected 10% instances from the official training set to create our validation set. The remaining 90% instances were used as our training data. For MAMS, we directly used the officially provided validation set.
In the Laptop dataset, more than 70% of sentences contain multiple aspects, and about 60% of sentences contain more than one aspect in the Restaurant dataset. For MAMS, each sentence contains multiple aspects. From the above statistical results, it is evident that modeling the sentiment relationships between multiple aspects in a sentence is necessary. The experimental results in the following subsections further verify this necessity.
In our experiments, we used pretrained 300-dimensional GloVe word embeddings as static word vectors, which were not fine-tuned during model training. Additionally, we used the pretrained BERT-base model with 12 hidden layers to obtain dynamical word embeddings, which have 768 dimensions. The semantic vectors of BIO tags and the edge categories were randomly initialized from a uniform distribution between [−0.01, 0.01]. Other parameters in the model were set to their default initial values in PyTorch 1.8. To mitigate overfitting, we applied the dropout strategy [
39] to the input layer. We used the Biaffine dependency parser [
40] to obtain the dependency syntax tree of each sentence. The values of the hyper-parameters were determined based on achieving the best performance on the validation sets as depicted in
Table 2.
As with most previous work for this task, we used accuracy (denoted as
) and
as the performance metrics.
is the ratio of correctly predicted instances (
T) to the total number of instances (
N). It is calculated as follows:
The
score is a harmonic mean of precision (
P) and recall (
R). For each class, it is calculated as follows:
where
,
and
are the true positives, false positives, and false negatives, respectively. In a multi-class setting, we first calculate the
score for each class separately, treating that class as the positive class and all others as negatives. Then, the
is defined as:
where
C is the number of classes and
is the
score for the
i-th class.
We further utilized
(Receiver Operating Characteristic) and
(Area Under the Curve) metrics to evaluate the performance of our model [
41]. Specifically, we calculated the macro-averaged
curve with the one vs. rest approach.
4.2. Results
We verified the effectiveness of our proposed hybrid graph neural network model based on the ASC task in two scenarios: HGNN (without the BERT word embeddings) and HGNN-BERT (with the BERT word embeddings). Specifically, we compared our method with the following two types of benchmark methods.
The first type of method only uses static word embeddings, such as GloVe, and includes the models that utilize the dependency syntax information as follows:
IAN [
11]: Two LSTM networks are employed to learn the semantic representations of an aspect and the corresponding sentence, respectively. The interaction between these representations is also taken into account.
RAM [
42]: The multi-layer attention mechanism and memory network are combined to learn the representation of context related to a specified aspect, which is then used for classification.
MIAD [
16]: Multiple aspects in a sentence are considered as a sequence, and an LSTM network is adopted to capture the sentiment relationships between them.
ASGCN [
12]: A graph convolution neural network based on a dependency syntax tree is proposed to better capture the long-distance dependencies between a specified aspect and other words in a sentence.
The second type of method uses dynamical word embeddings from pretrained language models, such as BERT, and includes the models that utilize the dependency syntax information and external knowledge as follows:
The experimental results based on the three datasets are shown in
Table 3. It should be noted that the results of IAN and RAM based on the MAMS dataset were extracted from the previous work [
46], while the results of T-GCN, dotGCN, and APARN were from [
25]. We executed the authors’ provided code to obtain the results of ASGCN on the MAMS dataset. We enhanced the publicly available ASGCN code with BERT word embeddings to obtain the results of ASGCN-BERT. The results for other benchmark methods were extracted from the corresponding literature.
From the experimental results in
Table 3, we can conclude the following: (1) Integrating the dependency syntax information benefits the aspect-level sentiment classification (ASGCN-BERT, KDGN and IDGNN vs. AEN-BERT). (2) Modeling sentiment relationships between multiple aspects in a sentence is necessary and improves ASC performance (MIAD vs. RAM; SDGCN-BERT vs. AEN-BERT). (3) By leveraging both the dependency syntax information and the sentiment relationships between multiple aspects, our proposed model achieves a performance that is comparable to or better than baseline methods (HGNN-BERT vs. methods in Part 2; HGNN vs. methods in Part 1). (4) While our method may not always outperform KDGN, IDGNN, and APARN, this is because they leverage additional information such as constituent syntax trees in KDGN and semantic information in IDGNN and APARN.
Based on the above analyses, our proposed hybrid graph neural network ASC model is proven to be effective.
4.3. Ablation Study
To verify the effectiveness of key components of our proposed model, we conducted ablation experiments on the relatively larger Restaurant and MAMS datasets, and present the results in
Table 4. Specifically, we compared the proposed model (HGNN-BERT) with the following three variants:
W/o dependency edges: The hybrid graph does not include edges based on the dependency syntax tree, meaning our model does not utilize information from the dependency syntax tree.
W/o categories of edges: The category of edges is not considered in the hybrid graph neural network layer. In other words, we replace
in Equation (
3) with
.
W/o adjacency edges: The hybrid graph does not consider the adjacency relationships between multiple aspects. That is to say, our model does not leverage the sentiment relationships between multiple aspects.
W/o AS.adj edges: Only the edges between adjacent aspects are excluded.
W/o AS.inter edges: Only the interval edges are excluded.
W/o an attention mechanism: An average operator is used in place of the attention mechanism.
W/o a gated mechanism: An addition operator is used as a replacement.
From the results in
Table 4, several observation can be made: (1) All three variants of HGNN-BERT show varying degrees of performance degradation. These results clearly demonstrate the necessity of introducing dependency edges and adjacency edges in the hybrid neural network layer, as well as capturing the category information of these edges. (2) When the category of edges is not considered, the variant model’s performance substantially declines on both datasets, despite many previous models ignoring this type of information (Line 3 vs. Line 1). (3) Omitting the consideration of sentiment relationships between multiple aspects results in performance degradation, particularly evident for the MAMS dataset (Line 4, 5 and 6 vs. Line 1). As expected, the edges between adjacent aspects have more of an impact on performance than the edges between non-adjacent aspects (Line 5 vs. Line 6). (4) When the attention mechanism is not used, the performance based on both datasets degrades significantly (Line 7 vs. Line 1). Excluding the gated mechanism leads to a small but consistently decreased performance based on both datasets (Line 8 vs. Line 1).
On one hand, these findings highlight the advantage of jointly predicting sentiment polarities of multiple aspects in a sentence, compared to independently predicting them as in previous approaches. On the other hand, these results confirm that the attention and gated mechanisms are both effective and necessary for information transfer.
4.4. Performance on and
To further validate the effectiveness of our model, we conducted a comparative analysis against baseline models using the
and
metrics. Due to the lack of open-source code for KDGN and APARN, our evaluation was limited to comparing our model (HGNN-BERT) with the IDGNN model. The performance of these two models across all three datasets is depicted in
Figure 4.
Our model consistently achieved superior values across all three datasets, highlighting its enhanced ability to discriminate between different classes across various threshold settings. Specifically, our model attained scores of 0.93, 0.95, and 0.95, showcasing its high sensitivity and specificity in the ASC task. In comparison, the IDGNN model yielded scores 0.92, 0.94, and 0.94, falling short of matching our model’s performance. This disparity indicates that our model is more adept at handling the complexities of the classification problem, particularly in situations where class distinctions are subtle or when the data are imbalanced.
4.5. Weights of Dependency Types
To further analyze which types of dependencies are important for our model, we averaged the weights assigned to each dependency type in the last graph neural network layer. The top 10 dependency types with their average weights based on the Restaurant dataset are listed in
Figure 5. We can observe the following: (1) Our model assigns higher weights to
nsubj and
acomp, which are typically employed to express a subjective attitude toward a particular aspect (e.g.,
the macbook looks very beautiful). (2) The
amod and
advmod types also receive significant attention, as they frequently occur in phrases like
great battery and
extremely useful. (3) More importantly, our model assigns considerable weights to
conj and
cc. In the sentence
I have been impressed with the battery life and the performance for such a small amount of memory, the aspects
battery life and
performance are connected by a
conj edge, while
performance and
and are connected by a
cc edge. With these higher weights, our model can easily deduce that the two aspects share the same sentiment polarity. Overall, these observations suggest that our proposed model is capable of allocating suitable weights to various dependency types, thereby better learning the context information related to the aspects.
4.6. Effects of Hyper-Parameters
To explore the influence of different values of hyper-parameters on the performance of our HGNN-BERT model, we varied these values and report the corresponding accuracies in
Figure 6. Specifically, we varied the number of hybrid graph neural network layers
, the dimension of the category vectors of edges
, the size of BiLSTM hidden units
, and the dropout rate
, respectively. Please note that we changed one hyper-parameter at a time and kept the others fixed.
From the results shown in
Figure 6, we can observe that our model yielded relatively stable results when
,
,
and
. This indicates that our proposed model is not highly sensitive to these hyper-parameters.
4.7. Time Analysis
Time consumption is crucial for practical applications, as it directly impacts the usability and scalability of models. We compared the proposed model with baseline models in terms of time efficiency. Specifically, we ran SDGNCN-BERT, ASGCN-BERT, IDGNN, and our HGNN-BERT using an RTX 2080Ti GPU (MSI, Shanghai, China) with a batch size of 32. We did not include the time required for dependency parsing, as this can be performed in advance during training. During testing, the dependency parsing time was considered. From
Table 5, we can observe the following: (1) Compared to traditional graph-based models (SDGCN-BERT and ASGCN-BERT), our model consumes more training and test time. The main reason for this is that the hybrid graph we used is more complex than the aspect graph or dependency graph used in these model. (2) IDGNN designs a dependency graph and a global-word graph, based on which it transfers two types of information via GAT and GCN, respectively. Our model requires less time than IDGNN while achieving better performance.
There is no open source code available for the baseline models KDGN and APARN, and it is difficult to reproduce these two models. Therefore, we do not list their training and test time. Theoretically, our model outperforms KDGN and APARN in terms of time efficiency. The reasons are for this as follows: (1) KDGN requires more external tools: a dependency parser, a constituency parser, and a knowledge retriever. The constructed knowledge-aware dependency graph is also more complex than our hybrid graph. (2) As described in the limitations of APARN [
25], the path aggregation layer is more time-consuming than the existing methods. Overall, our proposed model performs well in terms of time consumption.
4.8. Case Study
To further explore the effectiveness of our proposed HGNN-BERT, we present several examples in
Table 6, along with the predictions of three models: SDGCN-BERT, IDGNN, and HGNN-BERT. In the first example, SDGCN-BERT makes a mistake on both aspects because it focuses more on
like. With the help of the dependency information (
not, version) and (
version, feature), both IDGNN and HGNN-BERT make correct predictions. In the second example, both SDGCN-BERT and HGNN-BERT effectively capture the sentiment relationships between adjacent aspects, resulting in accurate predictions. For example 3, the sentiment polarity for the aspect
vegetables is implicitly expressed. Our model assigns higher weights to the relationships (
but,
needed,
cc) and (
vegetables,
needed,
nsubj), while also considering the relationship between the aspects
meat and
vegetables, thereby accurately predicting the different polarities of them. The above-mentioned examples clearly demonstrate that it is essential to use both types of dependency relation and the sentiment relationships between aspects.
All three models failed to classify examples 4 and 5 correctly. Understanding the meanings of text like just put in my pocket and around is very challenging for current ASC models. We leave the study of these issues for future work.
5. Conclusions
In this paper, we propose a novel aspect-level sentiment classification model based on a hybrid graph neural network, achieving a performance comparable to or better than that of recent baseline models based on three commonly used datasets. Specifically, we design several hybrid graph neural network layers on the encoding layer to leverage both dependency syntax information (relations and their types) and sentiment relationships between multiple aspects. The primary distinction between our model and existing methods lies in our comprehensive consideration of relationships: among words, between words and aspects, and among aspects. In contrast, existing methods usually focus on only a subset of these relationships. Comprehensive experimental results confirmed both the effectiveness and efficiency of our method. The main drawback of our method is its reliance on the results of dependency parsing. When the dependency parsing results are suboptimal or in scenarios where a suitable dependency parser is unavailable, our method is not applicable.
In our future work, we plan to explore the application of hybrid graph neural networks in other natural language processing tasks. For instance, in multi-modal tasks, we can utilize a hybrid graph neural network to model interactions between objects from different modalities.