An Attention-Based Multi-Representational Fusion Method for Social-Media-Based Text Classification

Abstract

There exist various text-classification tasks using user-generated contents (UGC) on social media in the big data era. In view of advantages and disadvantages of feature-engineering-based machine-learning models and deep-learning models, we argue that fusing handcrafted-text representation via feature engineering and data-driven deep-text representations extracted by performing deep-learning methods is conducive to enhancing text-classification capability. Given the characteristics of different deep neural networks, their complementary effect needs to be investigated. Moreover, contributions of these representations need to be adaptively learned when it comes to addressing different tasks or predicting different samples. Therefore, in this paper, we propose a novel fused deep-neural-network architecture with a hierarchical attention mechanism for text classification with social media data. Specifically, in the context that handcraft features are available, we employ the attention mechanism to adaptively fuse totally data-driven-text representation and handcrafted representation. For the generation of the data-driven-text representation, we propose a data-driven encoder that fuses text representations derived from three deep-learning methods with the attention mechanism, to adaptively select discriminative representation and explore their complementary effect. To verify the effectiveness of our approach, we performed two text-classification tasks, i.e., identifying adverse drug reaction (ADR)-relevant tweets from social media and identifying comparative-relevant reviews from an E-commerce platform. Experimental results demonstrate that our approach outperforms other baselines.

1. Introduction

With the rapid development of Web 2.0 technology, people tend to express their ideas and share their experiences on social media. User-generated content (UGC) on social media has been widely used in various applications, such as exploring factors affecting usage intention [1], investigating public sentiment and attention of recycled water [2], stock market forecasting [3], and user interest mining [4]. Social media has enabled a plethora of text-classification tasks, such as recognizing comparative opinion-related reviews [5], identifying fake news and other malicious social media contents [6], classifying opinion spam [7], recognizing adverse drug-reaction (ADR)-related text from Twitter [8] or health forums [9], predicting depression degrees [10], detecting rumors [11], and collecting event-related tweets [12].

To train a text classifier using traditional machine-learning methods, some handcrafted features are extracted and then fed to a classifier algorithm. The performance of the model relies heavily on the quality of feature engineering. For a specific task, there may exist various external domain-specific knowledge resources, which can be naturally encoded into a conventional machine-learning model via feature engineering. However, UGC on social media generally presents unique characteristics. For example, there exist a variety of colloquial terms, creative phrases, inevitable misspellings, and other irregular expressions. Moreover, people frequently express their opinions using implicit language, which necessitates global semantic understandings. Therefore, a traditional feature-engineering-based model faces huge challenges to deal with these characteristics, since it can only explore superficial language information, failing to capture semantic meanings.

In recent years, deep-learning methods have gained increasing attention for addressing various natural-language-processing (NLP) tasks, including text classification [13,14]. The adoption of deep-learning methods, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Bidirectional Encoder Representation from Transformers (BERT) [15], can capture discriminative information by automatically performing feature learning. However, generic deep-learning models do not necessarily have advantages over classic machine-learning models since they ignore external domain-specific knowledge bases and experts’ involvement. For example, Zhou and Li [16] pointed out that traditional machine-learning models exploring various handcrafted features still outperform deep-learning models for paper-section identification.

In view of the abovementioned advantages and disadvantages of feature-engineering-based machine-learning models and deep-learning models, we argue that fusing handcrafted features via feature engineering (referred to as “handcrafted-text representation”) and data-driven features extracted by performing deep-learning methods (referred to as “deep-text representations”) is conducive to enhancing text-classification capability. In terms of fusion strategies, although prior studies have endeavored to incorporate certain knowledge into classic deep neural networks [17], this fusion paradigm needs careful design. Moreover, it is difficult to incorporate various types of knowledge. Explicitly fusing them with attention is a strategy that shows more initiative.

Moreover, when faced with different tasks or samples, different deep neural networks have varied performances. RNNs are well-suited to NLP applications requiring comprehension of sequence dependency. CNN is a competitive method where recognizing local and position-invariant information is important [14]. It presents outstanding capability of capturing local features due to the use of convolutional filters. For example, the key phrase “I like” plays an important role to indicate a positive sentiment polarity in the sentiment-classification task. In addition to the strong capability of capturing long-range dependency via self-attention, BERT is also expert at addressing text requiring global semantic understandings [15].

Given these characteristics of different deep neural networks, we argue that on one hand, there is no deep-learning model that can outperform other models. To deliver a high-performing text classifier, multiple representations derived by different deep-learning methods generally need to be simultaneously explored, since they can capture multi-facets of information, thus complementing each other. On the other hand, contributions of these representations might vary significantly when it comes to addressing different tasks or predicting different samples.

In this paper, we propose a novel fused deep-neural-network architecture with a hierarchical attention mechanism for social-media-based text-classification. Specifically, in the context that handcraft features are available, we employ the attention mechanism to adaptively fuse totally data-driven deep-text representation and handcrafted-text representation. For the generation of deep-text representation, we propose a data-driven encoder that fuses text representations derived from BERT, CNN, and Long Short-Term Memory (LSTM) with the attention mechanism, in order to adaptively select discriminative representation and explore their complementary effect. To assess the generalization capability of our proposed approach, we conducted extensive experiments with two case studies and three real-world datasets. The empirical results demonstrate the effectiveness of our proposed approach. This study is an extension of a preliminary version of the proposed approach [5] from the following aspects. On one hand, we have extended the prior study [5] by proposing the concept of hierarchical attention and simultaneously exploring handcrafted-text representation and deep-text representations. Moreover, apart from comparative-relevant text identification on the phone dataset, we have applied the approach to another task (i.e., adverse drug-reaction-relevant text identification) and added another two datasets to further verify the generalization of the approach.

The remainder of this paper is organized as follows: We review related work in Section 2. Our proposed approach is described in detail in Section 3. We present experimental datasets and settings in Section 4. Experimental results are reported and analyzed in Section 5. We provide conclusions in Section 6.

2. Related Work

2.1. Social-Media-Based Text Classification

In terms of conducting text classification using UGC on social media, apart from the supervised learning, prior studies also investigated the effectiveness of semisupervised learning [7] and active learning [12]. Earlier studies mainly relied on feature engineering, i.e., extracting shallow linguistic features and domain-specific features. For examples, Sarker and Gonzalez [8] explored various features including n-grams with term frequency–inverse document frequency (tf-idf), medical semantic features, ADR-lexicon match-based features, negation features, synonym-expansion features, change-phrase features, sentiword-score features, and topic-based features. Zheng and Sun [12] leveraged n-grams and hash values, and computed the cosine similarity based on bag-of-words features with tf-idf. Yang et al. [9] mainly focused on topics using the latent dirichlet allocation (LDA). Regarding the used classification algorithm, a support vector machine (SVM) was widely used [8,12]. Recent studies have turned to utilizing deep-learning methods; for example, Bing et al. [11] utilized dual coattention to fuse hidden embeddings encoded from user profiles, source tweets, and comments. They explored various deep neural networks, including BERT, CNN, and Bidirectional Gated Recurrent Units (Bi-GRU).

2.2. Deep Learning for Text Classification

In recent years, we have witnessed great success of applying deep learning for text classification. Among deep neural networks, RNN is considered to be an effective architecture for addressing sequential text data since it is capable of capturing long-range dependency. Given the gradient-vanishing problem of conventional RNN, researchers pay attention to LSTM [18], a popular variant of RNN. By incorporating three gates (input gate, forget gate, and output gate) to manage the flow of information into and out of a memory cell, LSTM mitigates the gradient-vanishing issue that conventional RNNs suffer from, and presents strong capability of capturing and maintaining long-term dependencies. Furthermore, researchers also investigated the effectiveness of Bidirectional LSTM [19] due to its capability of considering information conveyed by both previous and next words. Although CNN was originally proposed to process images, its effectiveness has been verified for various NLP tasks. For example, Kim et al. [20] proposed the TextCNN with one convolution layer for sentence classification. Zhang et al. [21] used character-level features. Prior studies have found that RNN and CNN performed differently when addressing different types of tasks [22]. Vu et al. [22] found that CNN and RNN can provide complementary information. Yin et al. [23] compared scenarios in which CNN and LSTM had respective advantages, and pointed out that CNN dominated in tasks that needed key-phrase matching, while RNN was suitable to encode long-range context dependency and order information. Pretrained language models, such as Embeddings from Language Models (ELMo) [24], OpenAI Generative Pre-trained Transformer (GPT) [25], BERT [15], and Enhanced Representation through Knowledge Integration (ERNIE) [26,27] can capture global semantics effectively, and have boosted various NLP tasks. Representations derived by ELMo are a function of output embeddings of different layers of bidirectional LSTM. ELMo can effectively address the issue of modeling polysemy. GPT and BERT are two landmark Transformer-based pretrained models. GPT uses autoregressive language modeling as the pretraining objective that maximizes all words’ conditional probabilities given their previous words. Compared with GPT, BERT treats bidirectional words as contexts and uses two self-supervised tasks to pretrain a model. Previous work also tailored domain-adaptive and task-aware pretraining models [28]. Beyond leveraging a single method, some researchers also proposed hybrid networks by combining different deep-learning architectures. For example, Wang et al. [29], Liu and Guo [30] utilized CNN to extract local features, and then leveraged LSTM to capture long-dependency features.

2.3. Attention in the NLP Domain

Attention, originally proposed for machine translation and inspired by human biological systems, has gained increasing attention in neural-network literature. The attention model is capable of adaptively concerning the input’s certain parts [31]. In addition to being utilized to fuse multimodal information [32], the attention mechanism has been gaining traction in the NLP domain, such as machine translation [33], question answering [34], and text classification [19,35]. One of the typical ways of implementing the attention mechanism in deep neural networks is by focusing on relevant parts of sequences [19,36]. Considering the hierarchical structure of a document (i.e., a document is composed of sentences; a sentence is composed of words), Yang et al. [35] proposed a hierarchical attention network (HAN) at two abstraction levels, i.e., at sentence level and word level, for document classification. HAN is capable to pay attention to important words in a sentence and important sentences in a document. Another type of attention mechanism is called multirepresentational attention, which involves multiple feature representations to capture several facets of the input. Learned attentions describe important weights of different representations. For example, for the same input sentence, Kiela et al. [37] learned importance weights over various word embeddings to improve sentence representation. Moreover, self-attention, an attention mechanism relating to a single sequence’s different positions, is the key to the Transformer’s success [38].

2.4. Research Gap

We identified several research gaps. First, although prior studies have proposed hybrid deep neural networks, they generally combine CNN and RNN in sequence, e.g., using the output of CNN as the input of RNN [29,30]; the complementary nature of various text representations is worth exploring in line with the multirepresentational attention research. Second, the complementary effect of deep-text representation and handcrafted-text representation is overlooked for finding relevant text from social media.

3. The Attention-Based Multirepresentational Fusion Method

Our proposed attention-based text-representation fusion approach is described in Figure 1. Word embedding aims to transform a word into a vector based on the word’s context. In prior studies, Word2Vec [39] and Glove [40] are popular word-embedding methods. However, they are incompetent at distinguishing the same word with different contexts. In recent years, BERT [15] can solve this problem. Therefore, as the first step, we adopted BERT to generate word embeddings, which served as inputs of CNN and LSTM. In addition, the [CLS] output was regarded as the BERT-generated text representation, waiting for fusion in the data-driven encoder. Subsequently, three deep-text representations, respectively derived by BERT, CNN, and LSTM, were fused to form the data-driven representation with a bottom-level attention mechanism. Meanwhile, different handcraft-feature subsets were transformed to respective hidden embeddings. Another bottom-level attention mechanism was adopted to fuse these hidden embeddings to obtain the handcrafted-text representation. Afterwards, we employed a top-level attention layer to obtain the fused text representation, followed by a fully connected layer to predict the probability for each class.

3.1. Deep Neural Networks Explored in Our Method

3.1.1. BERT

BERT aims to pretrain deep bidirectional text representations using large amounts of unlabeled text by jointly taking both left and right contexts into account [15]. The Transformer applies self-attention to calculate in parallel the relationship between each word pair in the text, thus modeling the influence each token has on another [38]. As shown in Figure 1, in BERT architecture, the input $E_i$ (i = 1, 2, …, t) corresponding to i-th word in the sentence is the sum of the word’s token embedding, position encoding, and segment embedding. Two self-supervised tasks, i.e., the masked-language model (MLM) and next-sentence prediction (NSP), were performed to guide the language modeling in BERT. The MLM task masks a certain percentage of tokens at random and attempts to predict those masked tokens. The NSP task is aimed at predicting whether two sentences are adjacent, i.e., one sentence following the other. Based on the pretrained model, BERT can be applied to downstream tasks with little architecture modification by jointly fine-tuning all parameters, which is relatively inexpensive compared to pretraining. In this study, we used a fine-tuned BERT model.

3.1.2. CNN

The typical CNN framework originally proposed by Kim et al. [20] was used in this study (as shown in Figure 2). Let ${\mathit{x}}_i\in R^m$ denote the $m$-dimensional word embedding of the $i$-th word in the sentence. Therefore, $X_{1:n}={\mathit{x}}_1$⊕${\mathit{x}}_2$⊕…⊕${\mathit{x}}_n$ represents a sentence of length $n$ (with the padding operation if necessary), where ⊕ stands for the concatenation operator. In each convolution operation, a filter $W\in {\mathrm{R}}^{h\times m}$ was performed on $X_{i:i+h-1}$ to generate a feature using the equation $c_i=f$($W·X_{i:i+h-1}+b$), where $f$ stands for a nonlinear activation function and $b$ is a bias term. After being applied to all sliding windows of $h$ words in the sentence, the filter produced a feature map $\mathit{c}=\left[c_1, c_2, \dots , c_{n-h+1}\right]$. The feature map $\mathit{c}$ was then fed into a max-pooling layer, i.e., keeping the maximum value $\widehat{\mathit{c}}$ = $\max \left\{c\right\}$ as the final feature obtained by the filter. The max-pooling layer enabled the filter to capture most important information. Suppose $k$ filters are used, the obtained text representation via CNN is $\widehat{c}$ = $\left[{\widehat{c}}_1, {\widehat{c}}_2,\dots ,{\widehat{c}}_k\right]$. Three types of filters with $h=3, 4, 5$ were implemented in this study.

3.1.3. LSTM

As shown in Figure 3, there are three input values of each LSTM cell, i.e., the input value ${\mathit{x}}_t$ of the network at the current cell, the output hidden value of the previous cell ${\mathit{h}}_{t-1}$, and the output cell state of the previous cell ${\mathit{c}}_{t-1}$. The condition of the unit and the flow of information can be protected and controlled via three gates (i.e., forget gate, input gate, and output gate). The three gates of each LSTM cell are described in detail below.

Suppose $\sigma$ and ⊙ represent the sigmoid activation function and dot product, respectively. $W$ and $U$ stand for learnable weight matrices and $\mathit{b}$ denotes learnable bias vector. The forget gate controls which part of the information needs to be discarded (forgotten) based on ${\mathit{h}}_{t-1}$ and ${\mathit{x}}_t$:

$${\mathit{f}}_t=\sigma (W_f{\mathit{x}}_t+U_f{\mathit{h}}_{t-1}+{\mathit{b}}_f)$$

(1)

The input gate determines which part of the information should be stored by calculating ${\mathit{i}}_t$ and ${\tilde{\mathit{c}}}_t$:

$${\mathit{i}}_t=\sigma (W_i{\mathit{x}}_t+U_i{\mathit{h}}_{t-1}+{\mathit{b}}_i)$$

(2)

$${\tilde{\mathbf{c}}}_t =tanh(W_c{x}_t+U_c{\mathit{h}}_{t-1}+{\mathit{b}}_c)$$

(3)

Based on the above calculation, we can update the current cell state as:

$${\mathit{c}}_t={\mathit{f}}_t\odot {\mathit{c}}_{t-1}+{\mathit{i}}_t\odot {\tilde{\mathit{c}}}_t$$

(4)

According to the following equations, the output gate determines which parts of the cell state ${\mathit{c}}_t$ should be outputted.

$${\mathit{o}}_t=\sigma (W_o{\mathit{x}}_t+U_o{\mathit{h}}_{t-1}+{\mathit{b}}_o)$$

(5)

$${\mathit{h}}_t={\mathit{o}}_t\odot \tanh \left({\mathit{c}}_t\right)$$

(6)

3.1.4. Data-Driven Encoder with Attention Mechanism

Not all deep-text representations contribute equally to predicting the label of a text. Different deep-learning methods excel at handling different forms of text. We need to adaptively assign varying importance weights to deep-text representations for different samples. Therefore, we introduced the attention mechanism to recognize important and suitable deep-text representations, and aggregated all deep-text representations to form a fused text representation which is totally data-driven. Specifically, we first fed each $e_{\left\{1,2,3\right\}}$ through a one-layer MLP (multilayer perceptron) to obtain its corresponding hidden representation ${\mathit{d}}_{\left\{1,2,3\right\}}$, where ${\mathit{e}}_1$, ${\mathit{e}}_2$,${\mathit{e}}_3$ are derived by BERT, CNN, and LSTM, respectively. We then measured the importance of each $\mathit{d}$ as the similarity of $\mathit{d}$ with an introduced context-query vector ${\mathit{u}}_d$, and computed the attention weight ${\alpha }_i$ through a Softmax function. The data-driven text vector ${\mathit{h}}_1$ was computed as a weighted sum of ${\mathit{d}}_1$, ${\mathit{d}}_2$ and ${\mathit{d}}_3$.We randomly initialized the context-query vector ${\mathit{u}}_d$ and jointly learned it during the training process.

$${\mathit{d}}_i=\tanh \left(W_{1i}{\mathit{e}}_i+{\mathit{b}}_{1i}\right)$$

(7)

$${\alpha }_i=\frac{\exp \left({\mathit{d}}_i^T{\mathit{u}}_d\right)}{{{\displaystyle \sum }}_{i=1}^3\exp \left({\mathit{d}}_i^T{\mathit{u}}_d\right)}$$

(8)

$${\mathit{h}}_1={\displaystyle \sum }_{i=1}^3{\alpha }_i{\mathit{d}}_i$$

(9)

3.2. Feature Engineering

Feature engineering plays an important role for pursuing a satisfactory performance in the conventional machine-learning community. In general, extracted features vary according to tasks, since experts typically resort to external knowledge bases to extract discriminative features. If extracted features can be divided into different subsets, we employ the attention mechanism to combine them. Given the different possible dimensions of different feature subsets, we first fed each feature subset ${\mathit{r}}_{\left\{1,2,\dots , n\right\}}$ through a one-layer MLP to obtain its corresponding hidden representation ${\mathit{f}}_{\left\{1,2,\dots , n\right\}}$ where $n$ represents the number of feature subsets. Subsequently, similar to the attention mechanism in the data-driven encoder, the attention weight of each ${\mathit{f}}_{\left\{1,2,\dots , n\right\}}$ was assessed as the similarity of $\mathit{f}$ with a learnable context-query vector ${\mathit{u}}_f$, and normalized with the Softmax function. The weighted sum of all ${\mathit{f}}_{\left\{1,2,\dots , n\right\}}$ yielded the handcrafted-text representation.

$${\mathit{f}}_i=\tanh \left(W_{2i}{\mathit{r}}_i+{\mathit{b}}_{2i}\right)$$

(10)

$${\beta }_i=\frac{\exp \left({\mathit{f}}_i^T{\mathit{u}}_f\right)}{{{\displaystyle \sum }}_{i=1}^n\exp \left({\mathit{f}}_i^T{\mathit{u}}_f\right)}$$

(11)

$${\mathit{h}}_2={\displaystyle \sum }_{i=1}^n{\beta }_i{\mathit{f}}_i$$

(12)

3.3. Top-Level Representation Fusion with Attention Mechanism

When designing MLPs, we ensured that the dimension of ${\mathit{d}}_{\left\{1,2,3\right\}}$ equals that of ${\mathit{f}}_{\left\{1,2,\dots , n\right\}}$. We introduced another learnable context vector ${\mathit{u}}_h$, with which importance weights of the data-driven text representation and handcrafted-text representation (i.e.,${\mathit{h}}_1$ and ${\mathit{h}}_2$) could be measured. Then weights were normalized with the Softmax function. The final text representation $\mathit{v}$ was obtained as the weighted sum of ${\mathit{h}}_1$ and ${\mathit{h}}_2$.

$${\delta }_i=\frac{\exp \left({\mathit{h}}_i^T{\mathit{u}}_h\right)}{{{\displaystyle \sum }}_{i=1}^2\exp \left({\mathit{h}}_i^T{\mathit{u}}_h\right)}$$

(13)

$$\mathit{v}={\displaystyle \sum }_{i=1}^2{\delta }_i{\mathit{h}}_i$$

(14)

3.4. Loss

The learning objective function of the model is to minimize the popular cross-entropy-loss function. The cross-entropy loss is calculated as:

$$L=-{\displaystyle \sum }_{i=1}^N{y}^{\left(i\right)}log{\widehat{y}}^{\left(i\right)}$$

(15)

where ${\widehat{y}}^{\left(i\right)}$ denotes the possible probability under each class category, $y^{\left(i\right)}$ represents the true label, and $N$ denotes the number of samples.

4. Experimental Datasets and Settings

4.1. Experimental Datasets

To verify the effectiveness of our proposed method, we applied it to addressing two text-classification tasks. The first case study was aimed at identifying ADR-relevant tweets from social media (ADR). The second case study was regarding the identification of comparative-relevant reviews from an E-commerce platform (CR). It is noteworthy that for both case studies, in order to alleviate the imbalance issue, oversampling was performed on the training and validation set, while the test set kept its original distribution.

4.1.1. Case Study 1: Identifying ADR-Relevant Tweets from Social Media (ADR)

We collected an open-source dataset (http://diego.asu.edu/Publications/ADRClassify.html (accessed on 1 March 2020)) from Twitter based on the user id and tweet id [8]. Due to the removal of some tweets, we obtained a total of 7060 samples, which included 6304 ADR-irrelevant samples and 756 ADR-relevant samples. To preprocess these tweets, we conducted tokenization, lemmatization, text conversion to lowercase, and removal of hypertext mark-up language (HTML) tags, et al.

4.1.2. Case Study 2: Identifying Comparative-Relevant Reviews from an E-Commerce Platform (CR)

We constructed two real-world datasets that were composed of product reviews from JD (https://www.jd.com (accessed on 1 January 2021)), a famous E-commerce platform in China. We selected two electronic product categories (i.e., phone and camera, referred to as “CR_Phone” and “CR_Camera”, respectively) since they are updated frequently, and customers generally compare different brands and models before making a purchase. In the CR_Phone dataset, we collected 132,696 reviews, 12 product brands, 106 products, and 1011 product subcategories. In the CR_Camera dataset, we obtained 47,364 reviews, 3 explored brands, 47 products, and 121 product subcategories. The time spans of the two collected datasets were 5 February 2017–10 October 2019 and 20 June 2014–22 October 2021, respectively. We randomly labeled 20,000 reviews including 3284 comparative-relevant reviews and 16,716 comparative-irrelevant reviews on the CR_Phone dataset, and annotated 871 comparative-relevant samples and 10,939 comparative-irrelevant samples on the CR_Camera dataset. We conducted preprocessing on the two datasets. Specifically, traditional Chinese characters were converted to simplified Chinese characters, half-angle symbols were converted to full-angle symbols, and meaningless symbols and emoticons were removed.

4.2. Evaluation Metrics

We employed precision, recall, F1-score, and accuracy (Acc), which are popular metrics for evaluating classifiers’ performances. These metrics were computed as follows, based on the confusion matrix shown in

Table 1. Confusion matrix.

		Actual
		Positive	Negative
Prediction	Positive	True positive (TP)	False positive (FP)
	Negative	False negative (FN)	True negative (TN)

:

$$\mathrm{precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(16)

$$\mathrm{recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(17)

$$\mathrm{F}1-\mathrm{score}=\frac{2\ast \mathrm{precision}\ast \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}$$

(18)

$$\mathrm{Acc}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}$$

(19)

4.3. Experimental Procedure

For the ADR task where various external knowledge bases are available, we first conducted handcraft-feature engineering, following [8]. Shallow linguistic features, domain-specific knowledge-based features, and other discriminative features constituted three feature subsets, as shown in

Table 2. Explored features for identifying ADR-related text from social media.

Feature Type	Feature	# Features 1
Shallow linguistic features	N-grams with tf-idf	10,000
Domain knowledge-based features	Medical semantic features	1150
	The ADR lexicon match-based features	2
	The negation features	405
Other discriminative features	Synonym-expansion features	3003
	Change-phrase features	4
	Sentiword-score features	5
	Topic-based features	588
	Total	15,157

¹ # denotes the number of features.

. For more detailed information regarding these features, please refer to work [8]. We compared our approach with several baselines, including using individual deep-learning methods (i.e., BERT, CNN, and LSTM), performing a data-driven method that fuses three different deep-text representations using the attention mechanism, and applying a feature-engineering method that fuses various feature subsets using the attention mechanism.

For the CR task, we ignored the utilization of handcraft-feature engineering and only explored the data-driven encoder in our approach, verifying the flexibility of our approach. We performed the following baselines:

1.

The traditional method refers to the existing state-of-the-art approach, which extracts rules with CSR (class sequence rule) as features and adopts SVM as the classification algorithm (i.e., CSR+SVM).

2.

Individual deep-text representation by using BERT, CNN, and LSTM, respectively.

3.

Different fusion strategies include model-level fusion, which combines different classifiers with majority voting, and feature-level fusion, which fuses multiple representations by simply concatenating all features.

For all of the experiments in this study, we employed 10-fold cross-validation. We conducted our experiments on a Linux server with GPU and implemented deep-learning methods with PyTorch. We resorted to an open source code (http://diego.asu.edu/Publications/ADRClassify.html (accessed on 15 March 2020)) to perform the feature engineering for the ADR case study. For BERT, we adopted the BERTbase model. After parameter tuning, the learning rate, batch size, and pad size were 0.001, 32, and 40 on the ADR dataset. These parameters were 1 × 10^–5, 32, and 120 on the CR_Phone dataset, and 1 × 10^–5, 32, and 50 on the CR_Camera dataset. In terms of the number of filters in CNN, we used 256 filters for each size; therefore, 256 × 3 filters were implemented on each of the three datasets.

5. Results and Discussion

As shown in

Table 3. Experimental results for the ADR task.

	Model	Precision (%)	Recall (%)	F1-Score	Acc (%)
Individual deep-learning-based representation	BERT	35.42	78.89	0.4879	83.06
	CNN	35.93	78.34	0.4931	83.05
	LSTM	38.80	79.72	0.5191	84.79
The bottom-layer fusion	Data-driven	45.47	67.22	0.5374	88.07
	Feature engineering	33.43	63.47	0.4357	83.09
The top-layer fusion	Our method	45.71	69.31	0.5476	88.20

, our method yielded the highest F1-score (0.5476) and Acc (88.20%) for the task of identifying ADR-relevant tweets from social media. The data-driven method with bottom-layer fusion outperformed all of the individual deep-learning methods. These improvements resulted from the effectiveness of combining different deep-learning models with attention. Moreover, our method with hierarchical attention can deliver improved performance as compared to both the data-driven method and the feature-engineering method. The reason lies in the fact that our method benefits from the complementary effect of data-driven deep-text representation and the handcrafted representation obtained by feature engineering.

The experimental results for identifying comparative-relevant reviews are shown in

Table 4. Experimental results for the CR task.

	Model	CR_Phone				CR_Camera
		Precision (%)	Recall (%)	F1-Score	Acc (%)	Precision (%)	Recall (%)	F1-Score	Acc (%)
	CSR+SVM	50.18	83.35	0.6262	86.47	40.91	77.89	0.5088	87.22
Individual deep-learning-based representation	BERT	67.55	92.08	0.7807	91.68	41.59	82.82	0.5534	85.38
	CNN	68.06	92.05	0.7820	91.58	42.16	82.47	0.5561	85.36
	LSTM	66.43	92.93	0.7738	91.30	42.12	83.41	0.5575	85.01
Different fusion strategies	model-level	68.76	92.60	0.7888	91.86	43.07	83.18	0.5660	86.00
	feature-level	72.73	88.09	0.7923	92.36	44.68	79.65	0.5648	86.10
	Our method	87.02	77.31	0.8176	93.63	50.28	77.64	0.6085	88.95

. Our proposed method obtains F1-score improvements of 0.1914 and 9.97% on CR_Phone and CR_Camera, respectively, as compared to CSR+SVM. Adopting individual deep-text representation outperforms CSR+SVM in terms of F1-score due to the automatic feature learning via deep neural networks. Multirepresentational fusions present superiority over using individual deep-learning-based representation, regardless of the adopted fusion strategy. This finding shows that simultaneously combining different representations can contribute to enhancing capability of identifying comparative-relevant text, since different representations can capture varying facets of information and hence complement each other. In comparison to other fusion strategies, our attention-based fusion method achieves higher performance values on the two datasets. The result demonstrates the effectiveness of the attention mechanism in fusing multiple representations, since the attention mechanism can adaptively learn the importance weights of different representations.

Figure 4a,b show average attention-weight distributions of bottom-level representations (i.e., various feature subsets and multiple deep-text representations, respectively) for the ADR task. Figure 4c presents attention weights assigned to three representations obtained by applying BERT, CNN, and LSTM, respectively, on the CR_Phone dataset. As shown in Figure 4a, we find that all of the attention weights assigned to the three different feature subsets were around 33.33%, presenting an even distribution for both ADR-relevant and ADR-irrelevant tweet identification. These results demonstrate the effectiveness of each feature subset.

As shown in Figure 4b,c, attention-weight distributions for different tasks present a significant difference. For identifying ADR-relevant tweets, BERT plays the dominant role with an attention weight of 43.28%, while CNN contributes the least. On the contrary, for identifying comparative-relevant reviews, CNN dominates with an attention weight of 49.97%, while BERT only contributes 17.73%. Moreover, even for the same task, when identifying relevant and irrelevant text, contributions of different deep-text representations may also present a huge difference. For example, as shown in Figure 4c, the attention distribution of BERT and LSTM varies hugely for comparative-relevant and comparative-irrelevant text recognition.

We explain the reasons for the abovementioned results as follows: In the case of ADR, people prefer colloquial and implicit expressions to describe their ADR experiences, rather than using professional terms. For example, the first example for the ADR task in

Table 5. Attention weights of relevant text for the two tasks.

Content	BERT	CNN	LSTM
Case study 1: Identifying ADR-relevant tweets
eqinnastra all the time take trazodone last night and it really help but it be difficult to wake up.	0.5171	0.1699	0.3130
just wonder why doctor prescribe dangerous addictive medication without any warning to patient paxil for instance.	0.2445	0.2804	0.4751
really bad rls from Seroquel.	0.2427%	0.3600	0.3973
Case study 2: Identifying comparative-relevant reviews
给我老爸用，感觉性价比很不错，相比之下845处理器比710和675还是s高些，总体满意.	0.4468	0.4351	0.1181
It’s a present for my father. It is, in my opinion, quite cost-effective. The 845 processor outperforms the 710 and 675 processors. In overall, I am satisfied.
很赞，比现在的安卓系统快多了，不后悔，比我的vivo-x20强太多了，唯一比不上vivo的就是照相了，vivo的手机现在拍照超清.	0.0947	0.4785	0.4268
It’s fantastic. It’s noticeably quicker than the Android system. I have no regrets about purchasing. It is far superior to my vivo-x20. The only thing that cannot compare with vivo is photography. Vivo’s phones take crystal-clear photographs.
非常流畅，在系统上一些细节感比安卓机要好很多.	0.1769	0.5990	0.2241
It runs without glitches. Some of the system’s details are far superior than those of Android.

implies “drowsiness” without using the word. To predict this sample correctly, capturing local information is incompetent; we need to capture global semantic and sequence information. Therefore, BERT and LSTM play their important roles while CNN contributes the least for the example. On the contrary, for comparative-relevant text identification, comparative opinion is generally encoded in obvious sequences and keywords, such as “比…高” in Chinese, “superior”, “outperform”; therefore, CNN’s outstanding ability of capturing local information is particularly important. These results demonstrate the advantage of the attention mechanism, since it can adaptively assign importance weights according to expression characteristics of different tasks.

Figure 5a shows average attention weights of feature-engineering-based and data-driven representations for the ADR task. As shown in Figure 5a, for ADR-relevant tweets, the average attention-weight distribution of deep learning vs. feature engineering is approximately 6:4. On one hand, deep-learning methods contribute more as compared to feature engineering due to the implicit, colloquial, and diverse expressions. On the other hand, the results verify their complementary effect. We provide some examples with attention weights of deep learning and feature engineering in

Table 6. Top-level attention-weight distribution of several ADR-relevant tweets.

Tweet	DL 1	FE 2
do trazodone actually make people feel like high omfg it make me sleepy	0.5441	0.4559
I think that Vyvanse is making me grind my teeth...	0.5521	0.4479
rocephin ciprofloxacin me barely able to stay awake	0.5077	0.4923
why do I not remember going to bed or waking up (the answer: probably seroquel)	0.6457	0.3543
I run on Vyvanse and RedBull.” So done with that life. Vyvance cooked my brain like a stove top	0.6623	0.3377

¹ DL refers to the fused data-driven text representation. ² FE refers to the fused feature-engineering-based text representation.

. Moreover, we manually divide tweets into two categories, i.e., implicit tweets and explicit tweets. Implicit tweets refer to using questions, rhetorical questions, or metaphors to express the occurrence of adverse reactions. Explicit tweets directly indicate adverse reactions caused by taking drugs, such as “make me sleepy”. As shown in Figure 5b, deep-learning methods are adept at identifying implicit tweets, while handcrafted-feature engineering is effective in recognizing explicit tweets.

6. Conclusions

In this paper, we propose a hierarchical attention-based neural-network architecture that fuses multiple text representations for social-media-based text-classification. First, we employed an attention mechanism to decide contributions of deep-text representations derived by performing BERT, CNN, and LSTM. If handcrafted features were available, we also leveraged the attention mechanism to fuse different feature subsets, and designed a top-level attention layer to fuse these handcrafted-text representations and data-driven deep-text representations. Experimental results demonstrate the advantages of our proposed approach over other baselines on three real-world datasets for two relevant text-identification tasks. Our approach verifies the complementary effect of handcrafted and data-driven text representations, as well as that of different learning methods. The incorporation of the attention mechanism can adaptively assign weights to different representations. Moreover, contributions of different deep-text representations vary significantly according to different tasks.

This study has implications for researchers. Our study advances multisource information fusion by treating distinct deep-learning method-derived text representation as separate sources, and investigates the complementary effect of handcrafted features and totally data-driven representations with the attention mechanism. Our approach is scalable by adding any other text representations.

In the future, we would like to apply our approach to other social media-based text-classification tasks. Furthermore, it would be interesting to incorporate strategies that avoid the introduction of redundant information among various representations.