Polarity and Subjectivity Detection with Multitask Learning and BERT Embedding

Abstract

In recent years, deep learning-based sentiment analysis has received attention mainly because of the rise of social media and e-commerce. In this paper, we showcase the fact that the polarity detection and subjectivity detection subtasks of sentiment analysis are inter-related. To this end, we propose a knowledge-sharing-based multitask learning framework. To ensure high-quality knowledge sharing between the tasks, we use the Neural Tensor Network, which consists of a bilinear tensor layer that links the two entity vectors. We show that BERT-based embedding with our MTL framework outperforms the baselines and achieves a new state-of-the-art status in multitask learning. Our framework shows that the information across datasets for related tasks can be helpful for understanding task-specific features.

1. Introduction

In order to accurately extract and manipulate text meaning, a Natural language processing (NLP) system must have access to a notable amount of knowledge about the world and the domain of discourse. With the data, the NLP system is reliant on the extraction of meaning from text that has resulted in an exponential interest in NLP tasks such as sentiment analysis [1], microtext normalization [2], and others. The rise of e-commerce has given rise to reliance on sentiment analysis for market research. The tasks in NLP are interrelated and could benefit from sharing each other’s learning. Sentiment analysis is a subtask under NLP that predicts different affect states. Subjectivity detection and polarity detection are subtasks under sentiment analysis. Subjectivity detection aims to remove ’factual’ or ’neutral’ content, i.e., objective text that does not contain any opinion. Polarity detection aims to differentiate the opinion into ’positive’ and ‘negative’.

Multitask learning (MTL) [3] has displayed remarkable success in the field of image recognition. This success can be primarily attributed to learning-shared representations from multiple supervisory tasks. MTL acts as a regularizer by introducing an inductive bias and it reduces the risk of overfitting of the model. Extending MTL’s success to NLP, we propose an MTL model to extract both sentiment (i.e., positive or negative) and subjectivity (i.e., subjective or objective) of a sentence. In multitask framework, we aim to leverage the inter-dependence of these two tasks to increase the confidence of individual tasks in prediction; e.g., information about sentiment can help in the prediction of subjectivity and vice-versa. For sentence-level classification, the neutral class cannot be ignored because an opinion document can contain many sentences that express no opinion or sentiment. A sentence is opinionated if it expresses or implies a positive or negative sentiment. A sentence is non-opinionated if it expresses or implies a neutral sentiment.

Recently, Bidirectional Encoder Representations from Transformers (BERT) [4] has caused a stir in the NLP community by presenting state-of-the-art results in a wide variety of NLP tasks. BERT’s critical technical innovation is applying the bidirectional training of Transformer, a popular attention model, to language modeling. In our proposed framework, we use BERT as an embedding for our input sentences. The results show that our MTL framework surpasses baselines in both tasks. We evaluate our proposed approach on subjective and polarity datasets [1,5,6]. Our famework can be used to extract insights from text or review to provide insights for market research.

The rest of the paper is structured as follows: Section 2 introduces related works about sentiment analysis and multitask learning; Section 3 presents the proposed model, and Section 4 conducts experiments on multitask learning; Section 5 concludes the paper.

2. Related Work

This section introduces related work in multitask learning, sentiment analysis, and subjectivity detection.

Sentiment analysis, in particular, has seen growth in multitasking usage [7,8]. Ref. [9] uses MTL for adversarial text classification, mitigating the shared and private latent feature spaces from interfering with each other. Ref. [10] proposes an MTL approach that allows joint training of the primary and auxiliary tasks, improving the performance of rumour verification.

Subjectivity detection is an NLP task that consists of differentiating objective data (“factual” or “neutral” content) from subjective data (opinions). Mishra et al. (2018) [11] propose a multitask deep neural framework for document-level sentiment analysis that learns to predict the overall sentiment expressed in the given input document as the primary task and simultaneously learning to predict human gaze behavior and auxiliary linguistic tasks such as part-of-speech and syntactic properties of words in the document as the secondary task. Extracting subjective text segments poses a tremendous challenge that only a few works have attempted. Ref. [6] applied a graph-min-cut based technique to separate the subjective portion of the text from the irrelevant objective portions. Ref. [12] introduced a novel architecture for filtering out neutral content in a time- and resource-effective manner.

Glove [13] is a word vector technique that leverages both global and local statistics of a corpus in order to come up with a principled loss function that uses both these. We finetuned Glove with the dataset and created an embedding of L X 300 dimension where L is the maximum length of the input text.

The authors in [12] proposed a Bayesian network-based extreme learning machine (BNELM) model. BNELM augments the standard recurrent neural network structure to generate a predictor that can take advantage of the beneficial properties of extreme learning machines and Bayesian networks. Emotion recognition is a task very close to sentiment classification. Rashkin et al. (2019) [14] used BERT to detect emotions and applied it to a dialogue system.

The authors in [15] propose an evaluation of MTL on semantic sequence prediction on data-dependent conditions. They derive attribute of datasets that make them favorable for MTL by comparing performance with information-theoretical metrics of the label frequency distribution. Generic MTL [3,16] has a rich history in machine learning. It has widespread applications in other fields, such as genomics [17], NLP [18,19,20,21], and computer vision [22,23,24]. The MTL paradigm provides an effective platform for achieving generalization. The inspiration is that if the tasks are related, the model can learn jointly, taking into account the shared information, which is expected to increase its generalization ability. Various tasks can exploit the inter-relatedness to improve individual performance through a shared representation. Overall, it provides three principal benefits over the single-task learning paradigm:

• It helps in achieving generalization for multiple tasks;
• Each task improves its performance in association with the other participating tasks;
• Offers reduced complexity because a single system can handle multiple problems or tasks simultaneously.

The performance improvement from MTL [3] is due to the extra information in the training signals of related tasks:

• Implicit data augmentation: Learning only one task carries the risk of overfitting that task while learning jointly enables the model to obtain a better representation by averaging noise patterns. MTL effectively increases the sample size we are using to train our model by sharing the learnt features.
• Attention focusing: If the data are insufficient and high-dimensional, it can be challenging for a model to distinguish between relevant and irrelevant features.
• Eavesdropping: We can allow the model to eavesdrop through MTL; i.e., tasks challenging to learn for one model are learnt by the other model.
• Representation bias: MTL biases the model to prefer representations that other tasks also prefer, which helps the model to generalize new tasks in the future.

Subjectivity classification models classifies sentences into two classes: subjective and objective [25]. An objective sentence states some factual information, whereas a subjective sentence expresses personal feelings, views, judgments, or beliefs. We explore this relation in our MTL-based framework. Polarity classification models classifies sentences into two classes: positive and negative. If a sentence is classified as subjective or opinionated, we determine whether it expresses a positive or negative opinion, making sentiments and subjectivity closely related. We solve two tasks with a single network. Given a sentence sentence $s_i$, we assign it both a sentiment tag (positive/negative) and a subjective tag (subjective/objective), see

Table 1. Statistics of the datasets used in this paper after preprocessing and output of keras tokenizer.

Dataset	Train	Dev	Test	Max Length	Avg. Length	Vocabulary
POL	7.2K	800	2K	40	15	16.5k
SUBJ	7.2K	800	2K	85	17	18.5k

.

3. Proposed Multitask Learning (MTL) Based Framework

MTL is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. We use MTL, where a single framework performs two classification tasks simultaneously, i.e., subjectivity detection and polarity classification simultaneously. We assign a sentiment tag (pos or neg) and a subjectivity tag (yes or no) to each text, see Algorithm 1.

The methods of MTL have often been subdivided into two groups with a familiar dichotomy: hard parameter sharing vs. soft parameter sharing [26]. In hard parameter sharing, the model weights are shared between multiple tasks whereas in soft parameter sharing, all tasks have task specific models with separate weights. We have used hard parameter sharing, which is the most commonly used approach to MTL. It is generally applied by sharing the hidden layers between all tasks while keeping several task-specific output layers. In our experiments, we have used two different datasets and shared their information via Neural Tensor Network (NTN) [27]. NTN infuses knowledge sharing capabilities between individual tasks. Our model (Figure 1) shows that the information across datasets for related tasks can be helpful for understanding task-specific features.

Algorithm 1: Multitask BERT based Sentiment and Subjectivity

Result: Class

1. $E_{BER{T}_{T_i}}$ = BERT(S)

2. $L_{T_i}$ = BILSTM($E_{BER{T}_{T_i}}$)

3. $F_{T_i}$ = TDFC($L_{T_i}$)

4. $F_{T_i}$ = Drop($F_{T_i}$)

5. $S{A}_{T_i}$ = Attention($F_{T_i}$)

6. $D_{T_i}$ = FC($S{A}_{T_i}$)

7. $D_{T_i}$ = Drop($D_{T_i}$)

8. $F{n}_{T_i}$ = Flatten($D_{T_i}$)

9. $X_{T_i}$ = FC($F{n}_{T_i}$)

10. N = NTN([$F{n}_{T_1}⨁F{n}_{T_2}$])

11. $C_{T_i}$ = $X_{T_i}⨁N$

12. $Y_{T_i}$ = FC($C_{T_i}$)

Result: BERT Embedding

initialization

1. Token = BERTTokenizer(S)

2. id = Map(Token, ID)

3. S-new = Pad(S, maxlen)

4. embedding = transformer(S-new)

3.1. Embedding

We used two different feature extraction for our experimental setup, namely, BERT- and Global Vectors (Glove)-based embeddings. Glove embedding acts as a baseline for our proposed MTL setting.

BERT Embedding

We use pre-trained BERT [28] and computed sentence-level BERT embeddings. We used padding to normalize the variable-length input sentences to a fixed-length. The dimension of each embedding is L x 768, where L is the maximum length of the input text. In our experiments, we use the BERT base model with 12 encoder layers (i.e., transformer blocks) and feed-forward networks with 768 hidden units and 12 attention heads to capture transformer-based contextual word representations from both directions. The “Attention Mask” in BERT is simply an array of 1s and 0s indicating the presence or absence of padding tokens. This mask tells the “Self-Attention” mechanism in BERT not to include these PAD tokens in its interpretation of the sentence. Our proposed model uses the output of the final layer of BERT.

3.2. Bidirectional LSTM Layer

To obtain the input’s context-rich representation, we fed the embeddings to the Bidirectional Long Short-Term Memory (biLSTM) Layer of size 128. The output of the LSTM layer is used for both subjectivity and sentiment analyses.

In the next layer, we fed the LSTM output to obtain two matrices using two fully connected layers, which are input into two different tasks.

$$i_t=\sigma \left(W_{x^i}{x}_t\right)+W_{hi}{h}_{t-1}+W_{ci}{c}_{t-1}+B_i$$

$$f_t=\sigma (W_{xf}{x}_t+W_{hf}{h}_{t-1}+W_{cf}{c}_{t-1}+b_f)$$

$$c_t=f_t{c}_{t-1}+i_{t}tanh(W_{xc}{x}_t+W_{hc}{h}_{t-1}+b_c)$$

$$o_t=\sigma (W_{xo}{x}_t+W_{ho}{h}_{t-1}+W_{co}{c}_t+b_o)$$

$$h_t=o_{t}tanh\left(c_t\right)$$

3.3. Self Attention Network

Self-attention is an attention mechanism [29] associating the various positions of a single sequence to compute its representation. We use the self-attention mechanism in the next layer as it prioritizes the words necessary for classification:

$$P=tanh\left(H_{*}{W}^{ATT}\right),$$

(1)

$$\alpha =softmax\left(P^T{W}^{\alpha }\right),$$

(2)

$$s_{*}=\alpha H_{*}^T,$$

(3)

where $W^{ATT}\in {\mathbb{R}}^{D_t\times 1}$, $W^{\alpha }\in {\mathbb{R}}^{L\times L}$, $P\in {\mathbb{R}}^{L\times 1}$, and $s_{*}\in {\mathbb{R}}^{D_t}$. In Equation (2), $\alpha \in {[0,1]}^L$ provides the relevance of words for the task multiplied by Equation (3) by the context-aware word representations in $H_{*}$.

The output of the attention layer is flattened before feeding to the next layer, which is the Neural Tensor Network (NTN).

3.4. Neural Tensor Network (NTN)

We use the Neural Tensor Network [27] model to combine both tasks. NTN consists of a bilinear tensor layer that links the two entity vectors. In standard neural networks, the entity vectors are simply concatenated, whereas NTN can relate these vectors multiplicatively:

$$\begin{array}{c}\hfill s_{NTN}=tanh(s_{subj}{T}^{[1:D_{ntn}]}{s}_{pol}^T+(s_{subj}\oplus s_{pol})W+b),\end{array}$$

where $T\in {\mathbb{R}}^{D_{ntn}\times D_t\times D_t}$, $W\in$${\mathbb{R}}^{2D_t\times D_{ntn}}$, $b,s_{+}\in {\mathbb{R}}^{D_{ntn}}$, and ⊕ stands for concatenation. The vector $s_{+}$ contains shared information of both sentiment and subjectivity.

3.5. Classification

For the two tasks, we use two different softmax layers (task specific) for classifications detailed below.

3.5.1. Sentiment Classification

We use the output of NTN and concatenate it with the output of self-attention to determine the sentiment of the input text using the softmax layer.

$$\begin{array}{cc}\hfill {\mathcal{P}}_{pol}& =\mathrm{softmax}(s_{pol}{W}_{pol}^{softmax}+b_{pol}^{softmax}),\hfill \\ \hfill {\widehat{y}}_{pol}& =\underset{j}{\mathrm{argmax}}\left({\mathcal{P}}_{pol}\left[j\right]\right),\hfill \end{array}$$

where $W_{pol}^{softmax}\in {\mathbb{R}}^{D_t\times C}$, $b_{pol}^{softmax}\in {\mathbb{R}}^C$, ${\mathcal{P}}_{pol}\in {\mathbb{R}}^C,j$ is the class value (0 for negative and 1 for positive), and ${\widehat{y}}_{sen}$ is the estimated class value.

3.5.2. Subjectivity Classification

We use only the output of the attention layer as sentence representation for subjectivity classification since subjectivity detection is a subtask of sentiment analysis. We use the softmax layer to obtain the final output:

$$\begin{array}{cc}\hfill {\mathcal{P}}_{subj}& =\mathrm{softmax}((s_{subj}\oplus s_{+})W_{subj}^{softmax}+b_{subj}^{softmax}),\hfill \\ \hfill {\widehat{y}}_{subj}& =\underset{j}{\mathrm{argmax}}\left({\mathcal{P}}_{subj}\left[j\right]\right),\hfill \end{array}$$

where $W_{subj}^{softmax}\in {\mathbb{R}}^{(D_t+D_{ntn})\times C}$, $b_{subj}^{softmax}\in {\mathbb{R}}^C$, ${\mathcal{P}}_{subj}\in {\mathbb{R}}^C$, j is the class value (0 for objective and 1 for subjective), and ${\widehat{y}}_{subj}$ is the estimated class value.

4. Experiments

This section introduces the experiments performed and compares the experiments with the baselines on both single task and multitask settings available for polarity and subjective detection tasks.

4.1. Dataset

We have used the same number of sentences for both the models. Both the datasets [1,5,6] are balanced with an equal number of classes as well:

• POL: The dataset contains 5331 positive and 5331 negative processed sentences. We selected 5000 sentences from each class randomly, i.e., 5000 positive and 5000 negative sentences.
• SUBJ: The dataset contains 5000 subjectively and 5000 objectively processed sentences.

Both the datasets can be downloaded from here (https://www.cs.cornell.edu/people/pabo/movie-review-data/ accessed on 5 May 2022).

4.2. Baselines and Model Variants

We implemented three different models and compare our results with six state-of-the art models, as described in

Table 2. Accuracy comparision of different models.

	Framework	Subjective	Polarity
Baselines	SenticNet 6 [30]	-	92.8%
	Subjectivity detector [6]	92%	-
	AdaSent [31]	95.5%	83.1%
	CNN+MCFA [32]	95.2%	83.2%
	Multitask uniform layer [33]	93.4%	87.1%
	Multitask shared-layer [33]	94.1%	87.9%
	$MT{L}_{sharedNTNGlove}$	92.3%	92.1%
BERT Embedding	$BILST{M}_{pol}$	-	77.5%
	$BILST{M}_{subj}$	93.5%	-
	$MT{L}_{sharedNTN}$	95.1%	94.6%

.

4.3. Hyperparameters and Training

• Trainable parameters for the MTL model: 14,942,052.
• Trainable parameters for the individual models: 1,923,746.

ADAM [34] is an optimization algorithm that can be applied instead of the classical stochastic gradient descent algorithm to update network weights based on training data iteratively. The code is run for 20 epochs with 64 GB RAM and 32 GB Nvidia v100 Tesla.

$$\begin{array}{cc}\hfill \theta =& \{U^{[z,r,h]},W^{[z,r,h]},W_{*},b_{*},W^{ATT},\hfill \\ & W^{\alpha },T,W,b,W_{*}^{softmax},b_{*}^{softmax}\}.\hfill \end{array}$$

We use categorical crossentropy ($J_{*}$; * is $subj$ or $pol$) as the loss function:

$$\begin{array}{c}\hfill J_{*}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{j=0}^{C-1}{y}_{ij}^{*}log{\mathcal{P}}_{*i}\left[j\right],\end{array}$$

where N is the number of samples which is 10K, i is the index of a sample, j is the class value, and the following is the case.

$$\begin{array}{c}\hfill y_{ij}^{*}=\left\{\begin{array}{c}1,\mathrm{if}\mathrm{expected}\mathrm{class}\mathrm{value}\mathrm{of}\mathrm{sample}i\mathrm{is}j,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

4.4. Results and Discussions

We divided the dataset into train, validation, and test set for our experiments. The dataset was divided into train and test as 80:20 with random shuffling. The training dataset was further divided into train and validation as 90:10. We used the ADAM algorithm as an optimizer with categorical cross-entropy to calculate the loss.

We implemented single task frameworks and multitask frameworks to compare the performance. The models are trained on two different datasets, so the output shows if the tasks are related; they can share essential knowledge, which can benefit both task-specific outputs. We observed that, with the addition of NTN layer, which fuses the representations of polarity and subjectivity, our MTL model performed better. In the polarity detection case, NTN improved the accuracy by 15% with a single task framework and 10% with the MTL framework, whereas in the subjectivity detection case, NTN improved accuracy by 2–3% with a single task framework and no improvements were found for MTL frameworks. The comparisions with baseline also suggests an improvement in performance across both tasks. Table 2 depicts the comparision of our results to baselines, a single task and multitask frameworks. The loss and accuracy graphs of our proposed BERT embedding based MTL network is shown in Figure 2. We have also released our code and datasets on Github (https://github.com/shwetapardeshi1/Polarity-and-sentiment-detection accessed on 5 May 2022).

5. Conclusions

Multitask learning often aids in improving the performance of similar tasks. Related tasks often have interdependence on each other and function better when solved in a joint framework. In this paper, we present a BERT-based MTL framework that combines sentiment and subjective detection. In the current paper, we proposed a multilayer multitask LSTM for the main task of polarity detection and subjectivity detection. We used a Neural Tensor Network to combine the task to improve the functionality of individual tasks. The polarity task in MTL improved by at least 15% when compared to single-task performance. In comparison, the subjectivity detection task has improved only by 2–4% in a similar setting. As shown in the experiments, BERT embedding for individual tasks are not as good as the MTL setting. As the tasks are related, so are the features that the model learnt to classify. However, the key takeaway from our experiment is that linguistic tasks such as polarity and subjectivity detection are related tasks and, when trained on different datasets, they still surpass the baselines.