Multi-Task Aspect-Based Sentiment: A Hybrid Sampling and Stance Detection Approach

Abstract

This paper discusses the challenges associated with a class imbalance in medical data and the limitations of current approaches, such as machine multi-task learning (MMTL), in addressing these challenges. The proposed solution involves a novel hybrid data sampling method that combines SMOTE, a meta-weigher with a meta-based self-training method (MMS), and one-sided selection (OSS) to balance the distribution of classes. The method also utilizes condensed nearest neighbors (CNN) to remove noisy majority examples and redundant examples. The proposed technique is twofold, involving the creation of artificial instances using SMOTE-OSS-CNN to oversample the under-represented class distribution and the use of MMS to train an instructor model that produces in-field knowledge for pseudo-labeled examples. The student model uses these pseudo-labels for supervised learning, and the student model and MMS meta-weigher are jointly trained to give each example subtask-specific weights to balance class labels and mitigate the noise effects caused by self-training. The proposed technique is evaluated on a discharge summary dataset against six state-of-the-art approaches, and the results demonstrate that it outperforms these approaches with complete labeled data and achieves results equivalent to state-of-the-art methods that require all labeled data using aspect-based sentiment analysis (ABSA).

1. Introduction

Discharge summaries are virtual worlds where medical staff share their attitudes and behavior towards patients, diseases, and treatments by writing comments in a text summary format. Likewise, online and offline medical applications enable discharge summaries to allow other medical staff to express their opinions about a specific medical decision [1]. Sentiments and opinions that appear in the medical staff’s behavior and attitudes have an important influence on their decisions. Hence, profoundly understanding and analyzing these summaries can help medical staff and the medical health center make decisions [2].

Sentiment analysis, also known as opinion mining, is the domain that analyzes text to infer sentiments, emotions, or evaluations. In the literature, this domain is given many terms like sentiment analysis and opinion mining as well as many names for various tasks, for example, subjectivity analysis, opinion extraction, emotion analysis, etc. [3]. Sentiment analysis studies infer the expressed opinions across various levels, including document, sentence, and aspect levels. Therefore, various methods have been suggested to address tasks in three categories: deep neural network models, traditional machine learning, and rule-based methods [4].

Multi-model aspect-based sentiment analysis (MABSA) is of great value to sentiment analysis tasks. There are three subtasks, including sentiment extraction (SE), for example, good, bad, like, etc.; aspect-level sentiment classification (ASC), for example, positive, negative, and natural; and aspect term extraction (ATE), related to the diagnosis of the disease, which aims at associating each aspect with its respective polarity separately [5]. MABSA functionally operates at the intersection of information retrieval, natural language processing, and artificial intelligence. The aspect is the focus of the opinionated polarity, which may be explicitly mentioned or implicitly indicated through a set of indicators in the text. The explicit aspect, also known as a target, appears explicitly in the sentence, while the implicit aspect, also known as a category, is mentioned in the text implicitly through its indicators. In the literature, the former is known as aspect-target sentiment analysis (ATSA), and the latter is called aspect-category sentiment analysis (ACSA). Given a sentence review, ABSA first extracts a set of aspects and then identifies their respective polarities. MABSA is mainly performed through three steps: detection, classification, and aggregation. The detection step requires extracting the evaluated aspects in the text; the classification step classifies the sentiment into a predefined set of polarities, e.g., positive, negative, and neutral, in response to the extracted aspects; and the aggregation step produces a concise summary [6].

However, ATE and ASC components are considered two-step pipelined tasks, and SE is utilized separately. These three smaller components created for multi-task learning (MTL) have been combined to form an end-to-end model. Utilizing MTL can concurrently attain aspects of the opinions and sentiment polarities with a shared encoder. Learning about each subtask can save a certain amount of computing resources and enhance speed [7].

However, compared to single-task learning systems, MTL-based MABSA confronts more difficulties. The first issue is that different annotated subtasks may source an imbalanced label classification. In an MTL scenario, it is challenging to create enough examples with an equivalent number of data labels for each subtask. This issue is made more explicit in a sequence marking task when the class “others” makes up the majority of the tag sets. A subtask label occurrence might not be enough to train a robust neural network. Taking an applied discharge summary corpus in this research as an instance, the number of examples with the most commonly labeled data is compared against the unlabeled data. With a small dataset, very unbalanced data may mean that some classes do not provide enough support for an MTL neural network to perform as intended [8].

To overcome the problem, as mentioned earlier, self-training groups are looking for a robust way to develop aspect knowledge from the domain, such as choosing input based on confidence and creating automated pseudo-labels to reduce the effects of incomplete and unbalanced data [9]. By training a model with additional data that the model automatically labels, a model’s prediction power can be improved. However, utilizing self-training always creates noisy pseudo-labels [10], which could lead to the issue of progressive drift. Re-weighting is a straightforward and common approach to solving this issue. Some self-training techniques used utility thresholds and uncertainty to assess the data quality of data that had been pseudo-labeled before giving low-quality pseudo-labels lower weights [11].

Assigning weights is more difficult in the following ways when using a self-learning method in an MTL and sequence-labeling MABSA. First, smaller weights are required for the generated pseudo-labeled data to reduce the effects of noise. Second, minority-labeled classes are assigned higher weights to assist the model in learning from insufficient data [12]. Finally, distinct subtasks may require different weights to be coordinated thoroughly because they have varying convergence rates and degrees of priority (primary tasks vs. auxiliary tasks). Typical re-weighting studies probably concentrated on one of the aforementioned circumstances. They created a predefined function to assign weights to the data, which fixed the denoising or unbalanced distribution issues. It is challenging to manually create a predefined weighting function that satisfies all three objectives while also adapting well to changing conditions [13,14].

To address the abovementioned issues, such as imbalanced and insufficient data and MABSA sub-problem weight reassigning, this study presents a Mix-weightier with the Meta-based Self-training method (MMS). MMS utilizes self-training to collect additional data and calculate different weights for several jobs in various situations. The proposed MMS model contains three components: the Student Model, the Instructor Model, and the extra meta-weightier. The student model has a similar construction as the IM. For the student model to receive thorough and objective supervision during training, the meta-weights are utilized to reassign weights to both the gold and false labels. In contrast, the IM only uses gold-labeled data to train it, avoiding the influence of false labels. The meta-weightier offers subtask-specific weights of hybrid labeled data and data with no labels for the student model. Moreover, this paper utilized a 3-step update technique based on meta-learning to train the student model of MMS and the meta-weightier simultaneously to ensure the meta-weightier can provide accurate weights. The meta-weightier can use the recent response from the student model to build weights for current inputs by retaining two sets of parameters. Since it has already been updated using a similar input, the meta-weightier provides the student model with a better convergence. In the end, the student model makes final predictions using the IM and meta-weights guidance.

In summary, the primary contributions of this paper are as follows:

• The proposed hybrid techniques SMOTE-OSS-CNN deal with solving issues when utilizing an imbalanced dataset.
• This paper developed MMS, which combines a conventional instructor-student structure with a novel meta-weightier to achieve self-training. To reduce the effects of noise, coordinate sub-tasks, and balance class labels in MABSA, the meta-weightier can produce weights specific to each subtask.
• To jointly update the meta-weightier and student models, this paper develops a three-step meta-training technique. By using the suggested technique, MMS will be able to use the student model’s most recent feedback to guide MMS in the direction of a temperature gradient.
• In MABSA challenges, this paper outperforms state-of-the-art methods with complete labeled data while using only 40% of the training data to obtain better performance. The experimental findings show how effective the suggested MMS is.

The rest of the research is ordered as follows: Literature Review is introduced in Section 2; the mechanism of our proposed model is explained in Section 3; results of the experiments are discussed in Section 4; lastly, concluding observations are provided in Section 5.

2. Related Work

Recent MABSA research shows that multi-task knowledge is a common technique [14,15]. Compared to pipeline techniques, MTL-based methods address SE, ASC, and ATE concurrently instead of extracting aspect keywords first and then finding sentiment polarity. The most recent studies concentrate on ways to improve interactions between sub-tasks. The paper by [16] suggested a unique Gated Bridging Mechanism for filtering out unnecessary information and exchanging important information between distinct sub-tasks. The work by [17] provided a message-sharing mechanism across several tasks using a common set of latent variables that jointly learned numerous correlated tasks at both the word token and the document levels. In the research by [18], an MTL technique has been proposed to encode cooperation signals between distinct sub-problems in a stacked-layer network.

However, in MTL-based sequence labeling data, for instance, MABSA, special data features and needs should also be considered in addition to improving information interactions. Self-training is a potentially useful method to address the issues of potentially inadequate and unbalanced data features. Making up fictitious labels is a helpful technique for self-learning. Translating, rotating, and flipping are popular processes used in computer vision to produce fictional label examples [19]. Due to the uncertainty of language, creating pseudo-labeled data in the Natural language processing (NLP) discipline is more complex [20]. For token-level sequence labeling tasks, randomly removing, adding, and replacing specific tokens in a phrase may result in semantic incoherence and unintended consequences [21]. So, in NLP activities, self-labeling is a more successful technique. In work by [22], Using a prompt model with self-training for single-shot tasks was recommended. To provide two viewpoints on an instance through weak and robust augmentations, the structure from linear motion (SFLM) established an automated label on the weak enhanced version and fine-tuned it with the highly enhanced version. In the research by [23], sentiment augment was developed to create task-specific data from a bank of billions of unlabeled phrases for a given task. These data were categorized using a supervised instructor model. The paper by [24] is comparable to ours, concentrating on using self-training to overcome the scarceness of the label difficulty of sequence labeling tasks. However, they only investigated a single-task setting, whereas this paper is attempting to handle tasks of multiple sequence labeling.

Because pseudo labels are noisy, it might be problematic for the neural model to converge effectively, which presents a problem for self-training. The produced pseudo labels need to be carefully chosen to lessen noise’s effects. The estimated softmax score was used in some early research [25]. Additionally, a portion of more recent active learning research relied on model output scores to choose samples [26]. However, a forecast with a score of high softmax cannot be completely reliable. This is because the quantity of training data and the generalization capacity of these models, trained for producing pseudo labels, are constrained. As a result, they may confidently assign incorrect labels [27]. Because of this, some research [27,28] used curricular learning to choose data examples ranging from simple to complicated. Some studies on self-learning used data uncertainty as a selection factor for the data [26,29,30]. In this study, this paper picks pseudo-labeled data using a model that accounts for utility and data uncertainty. The paper by [18] proposes a Relation-Aware Collaborative Learning (RACL) framework that enables subtasks to work collaboratively through multi-task learning and relation propagation mechanisms within a stacked multi-layer network. Extensive experiments on three real-world datasets demonstrate that RACL significantly outperforms state-of-the-art methods for the complete ABSA task.

Recently, deep neural network (DNN)-based models have shown power in feature representation for many NLP tasks, where the features with the same semantic context are mapped to close points in the latent space [31]. For instance, the words improve and are well-prepared, usually represented by close points in the embedding space due to their semantic relation in the medical corpus. BERT-SPC [32] introduced simply concatenating the sentence and the aspect term as a pseudo-sentence to model the representation in response to the aspect. More recently, BERT-Pair [33] investigated BERT for ABSA through various forms of auxiliary sentences, including pseudo-sentence, question answering, and natural language inference, to address aspect category detection and polarity identification.

3. Proposed Method

The MABSA task is developed in this part. The idea is adopted from [34] with different datasets and proposed with a different technique, with an overview of MMS and a detailed explanation of the meta-weigher training procedure. Figure 1 shows that the proposed MMS contains three components: the student model, the instructor model, and the meta-weigher. The IM uses gold data to create pseudo labels, learn the task, and determine the level of uncertainty for unlabeled data. The student model and the meta-weigher are collaboratively trained to assign gold and fake data subtask-specific weights. Under the guidance of the IM and the meta-weigher, the student model is trained with pseudo and gold labels before conducting the final interference.

MMS uses two distinct instructor and student models, allowing the instructor to only be trained with gold data to minimize the effects of noise and provide high-quality pseudo labels. The student model can reduce insufficient and unbalanced data impacts by adding more training data without requiring further manual annotations. It is possible to take the appropriate precautions to stop the student model from gradually drifting since automatically labeled data are noisy. To achieve this, this paper creates the meta-weigher to offer subtask-specific weights. In contrast to human experts who employ fixed weight functions, the meta-weigher takes input from the student model into account and may be dynamically changed while being trained. This is similar to curriculum learning regarding how it directs the student model to be educated using a learnable input sequence and distinguishes gold data and pseudo.

3.1. Task Description

ABSA is a common MTL problem containing SE, ASC, and ATE sub-tasks. According to the baseline research, each sub-task is designed as a task that involves labeling sequences [35]. Comparison of many existing related works, based on ABSA refers to Aspect-Based Sentiment Analysis [36], TASD refers to Target Aspect Sentiment Detection [37], ACSA refers to Aspect-Category Sentiment Analysis [38], and AOPE refers to Aspect-Opinion Pair Extraction Sentiment is also extracted by this paper proposed ABSA task [39]. The symbol (s) mentions sentiment, (o) mentions opinion, (c) mentions category, and (a) mentions aspect. Explicitly, given the input sentence with $L$ tokens, MMS purposes for predicting a label sequence for each sub-task. The inside, beginning, and outside (IBO) tagging text schema is utilized for ATE and SE sub-tasks, where “B” refers to the beginning, “I” refers to the inner, and “O” refers to the other. For the ASC sub-task, MMS utilizes the set of labels $\left(very pos, pos, neu, neg, and vert neg\right)$, very positive, positive, neutral, negative, and very negative, respectively. Some label in the subtasks refers to these labels as not having an aspect word and not having a polarity of sentiment, which, in this step, is ignored.

3.2. Self-Train Learning

The self-training process of MMS will be thoroughly explained in this part. There are five steps in the MMS self-training: setting up the instructor model, creating pseudo labels, setting up the student model, and this model self-training using the meta-weigher.

The first phase, creating artificial instances, oversamples the under-represented class distribution using the SMOTE-OSS-CNN. On the other hand, the learning model’s decision-making is directly influenced by the development of synthetic examples closer to the decision boundary.

The second phase is setting up the instructor model. The IM is a supervised technique with a labeled dataset $D_{lab}$ for setting up. To reduce the effect of noise from unlabeled data on MMS, unlabeled data is not used in this training phase.

$$H_s=BER{T}_{Tea}\left(w_1,\dots , w_L\right)$$

(1)

$${\widehat{Y}}_s^t=Argmax (Softmax(FN{N}_{Tea}^t(H_{\mathrm{S}})))$$

(2)

$\left(w_1,\dots , w_L\right)$ refer to the input of the $L$ token and $S$ sentence task $t \in \left(SE, ASC, and ATE\right)$. $H_s$ is the hidden representation of $S\in D_{Lab}$. Feed-forward neural network (FNN) with two layers of activation of the $ReLU$. $BER{T}_{Tea}$ is the encoder of the instructor model. The instructor and the student model utilize BERT-large as encoders to allow for a fair comparison with research [40,41]. $Los{s}_{Tea}$ is utilized to update the instructor model, which is calculated as the average of the three tasks’ cross entropy CE.

$$Los{s}_{Tea}=\frac{1}{\Vert Tasks\Vert } {{\displaystyle \sum }}_{t \in Tasks}{{\displaystyle \sum }}_{w_i\in s}CE \left({\widehat{Y}}_{w_i}^t, Y_{w_i}^t\right)$$

(3)

where ${\widehat{Y}}_{w_i}^t\in {\widehat{Y}}_s^t,Y_s^t$ is the gold label sequence, whereas $Y_{w_i}^t\in Y_s^t$. ${\widehat{Y}}_s^t$ is a predicted label sequence.

The third phase is creating pseudo labels. MMS employs the initialized instructor model to produce data pseudo labels of unlabeled $D_{unl}$, utilizing modeling tools to choose these produced labels and estimate data uncertainty. $D_{unl}$ is extracted from the initial dataset without label annotation.

$S_{unl}$, specifically, for each sentence without a label, considering dropout, given $N_{aug}$ forward passes through the instructor model (dropout can introduce stochasticity). With each run using the same model parameters, data augmentation is the term used by MMS to describe the averaged representations of this unlabeled data. Then, for each token within a sub-task $t$, an averaged representation is translated into a soft pseudo label.

$$Logtit{s}_{s,pseu}^t=\frac{1}{N_{aug}}{{\displaystyle \sum }}_{i \in N_{aug}}FN{N}_{Tea}^t \left(H_{s,i}\right)$$

(4)

$${\widehat{Y}}_{S,pseu}^t=Softmax \left(Logit{s}_{S, pseu}^t\right),$$

(5)

where $H_{s,i}$ signifies the BERT hidden statuses of a sentence $S$ from the $ith$ forward pass in the instructor model. MMS applies the dropout proposed by [42] to calculate data ambiguity $unc$ in the following Equation, based on labels of soft pseudo ${\widehat{Y}}_{S,pseu}^t.$

$$un{c}_{pseu}^t=\frac{{\widehat{Y}}_{S,pseu}^t\times Log {\widehat{Y}}_{S,pseu}^t}{{{\displaystyle \sum }}_{S^{,}\in D_{unl}}{\widehat{Y}}_{S^{,},pseu}^t\times Log {\widehat{Y}}_{S^{,},pseu}^t}$$

(6)

The above Equation is based on the idea that if this paper can apply the same model to forecast the same example more than once and the results are various, this paper can compute the entropy from these additional trials to estimate the uncertainty for this example.

The last data ambiguity $un{c}_{pseu}$, for example, is averaged over various tasks $t$. The occurrences are complex samples, as evidenced by higher data uncertainty. MMS may not provide helpful information if it provides the self-training student model with too many simple instances. However, a high level of uncertainty can imply that these data examples are noisy. For this reason, MMS only employs pseudo-labeled data with ambiguity ranging from $L\%$ to $U\%$, where L and U are hyperparameters. $D_{pseu}^{unc}$ stands for the pseudo-label data that has been filtered by uncertainty.

Moreover, the prediction utility of the instructor model for unlabeled data is measured by $uti{l}_{pseu}$, which this paper defines. The variance between $uti{l}_{pseu}$ and $un{c}_{pseu}$ is that $uti{l}_{pseu}$ is the level of the token and $un{c}_{pseu}$ is a measure of instance level. $uti{l}_{pseu}$ is to sample the sentences whose label sequences have more than no other tags, while $un{c}_{pseu}$ purposes for selecting more confident and reliable sentences. The following Equation of $uti{l}_{pseu}$ is calculated by

$$uti{l}_{pseu}=\frac{1}{\Vert ta{g}^{\prime }\Vert } {{\displaystyle \sum }}_{tag \in ta{g}^{\prime }}Softmax \left(logit{s}_{s, pseu}^{tag}\right),$$

(7)

where $tag{s}^{\prime }$ is the all-no other tags set from SE, ASC, and ATE. ABSA considering the unbalanced label distributions, MMS chooses examples of which $util>{\beta }^{tag}$ to get more no other tags, creating a pseudo-labeled dataset $D{ }_{pseu}^{util}$. ${\beta }^{tag}$ is a threshold hyperparameter, the data samples uncertainty applied upper bound ($U\%$) 80%, 75%, and 65%, lower bound ($L\%$) 20%, 30%, and 35%, and utility threshold (${\beta }^{tag}$) 0.25, 0.3, and 0.4, respectively. The ultimately chosen batch of pseudo-labeled data is by the following Equation.

$$D_{pseu }=D_{pseu }^{util }\cap D_{pseu }^{unc}$$

(8)

To optimize the information gain for the student model without adding excessive noise, MMS can selectively choose pseudo-labeled data with the right level of uncertainty and utility.

The fourth phase is setting up the student model. The student model is set up with $D_{lab}$, which is similar to the instructor model. Prepared student model aids avoid gradual drift when the student model demeanors self-training with data of pseudo-labeled.

The fifth phase is using a meta-weigher to self-train the student model. The selected pseudo-labeled data set is MMS.$D_{lab}$ and $D_{pseu}$ to form $D_{mix}$ is combined first using MMS. The student model uses $D_{mix}$ to accomplish self-training. The student model applied the BERT encoder to input the sentence $S\prime \in D_{mix}$ to obtain the representation $H_{S\prime }$ by using the first Equation and next applying the second Equation to get prediction labels ${\widehat{Y}}_{S\prime }^{t }$ Then, the following Equation is used to calculate $Loss{ }_{Mix}^t$.

$$Loss { }_{Mix}^t=CE\left({\widehat{Y}}_{{S^{\prime }}_{mix}}^t, Y_{{S^{\prime }}_{mix}}^t\right),$$

(9)

For predicting a label sequence use ${\widehat{Y}}_{S\prime { }_{mix}}^t$. The pseudo or gold label sequence can be presented by $Y_{S\prime { }_{mix}}^t$. The input length present by $L$ and a loss sequence present by $Loss{ }_{Mix}^t$ is with this length. MMS retains the right to use this loss sequence as the meta-input weigher for calculating the weights particular to each subtask. Then, $Loss{ }_{Mix}^t$ is observed as input parts of meta-weigher to computing the particular to each subtask and weights $W^t$ of time-changing.

$$W^t=\mathrm{meta}-\mathrm{weigher}\left(\left[Loss { }_{Mix}^t;H_{tab};H_{epo};H_{S^{\prime }}\right]\right),$$

(10)

where (;) illustrations chain and a two-layer forward neural network present meta-weigher. The epo and tab are consistent embeddings of epoch and label, inspired by fully connected layers, the model encoded by two separate embedding layers. The student model learning progress is indicated by $H_{epo}$, which is normalized between 1 and 100 as an integer. A one-hot vector is assigned to each label, and the one-hot vector is entered into the relevant embedding layer to produce $H_{tab}$, It identifies the relevant class. Similar to the previous example, this number likewise maps into a one-hot vector and is fed into the appropriate embedding layer to produce $H_{epo}$. This paper leaves out $H_{S^{\prime }}$, $H_{tab}$, and $H_{epo}$ from the inputs of meta-weigher for clarity. Next, by using $W^t$ weighted for each sub-task $Loss{ }_{Mix}^t$ to gain the final $Los{s}_{Stu}$.

$$Los{s}_{Stu}=\frac{1}{L} {{\displaystyle \sum }}_L{w}^t. Loss { }_{Mix}^t$$

(11)

The following section provides specifics on how to update the student model using $Los{s}_{Stu}$. Algorithm 1 illustrates the whole MMS training.

Algorithm 1 MMS Self-learning

Input: The instructor model, the student model, the meta-weigher; Epoch’s maximum training iterations; the unlabeled data set $D_{unl}$; the labeled data set $D_{lab}$; Output: A validation set’s predictions. 1: for i = 0 toward Epoch do 2: Setting the instructor model within $D_{lab}$. 3: Producing pseudo labels for $D_{unl}$ within the prepared instructor model. 4: mingling the $D_{lab}$ and pseudo-labeled $D_{unl}$ toward form $D_{mix}$. 5: Prepared the student model within $D_{lab}$. 6: By training the student model and meta-weigher within $D_{mix}$ by using the three-meta-update approach 7: A conclusion based on the set of validation. 8: end for

3.3. Training Model

This part describes a 3-step meta-updating technique for training the student model and the meta-weigher. Figure 2 shows the workflow. Using this technique, MMS can weight-matching data instances using the most recent feedback ($Los{s}_{Stu}$) from the student model. The calculated weights may coordinate three sub-tasks in ABSA and reduce the effects of noise in $D_{mix}$.

According to $Los{s}_{Stu}$ loss sequence, Once updated, the student model from $ts=0$ to $ts=1$ is based on the following Equation.

$${\theta }_{stu}^1={\theta }_{stu}^0-{\lambda }_{stu} {\nabla }_{{\theta }_{stu}} {\propto }_m Los{s}_{Stu}z$$

(12)

$TimeStep$ presents by $ts$. ${\propto }_m$ is a hyperparameter. The learning rate is shown by ${\lambda }_{stu}$ for the SM. However, the meta-weigher is not updated at this stage. Then, the student model applies data with gold labels to compute $Los{s}_{Lab}$. Its assistance with the meta-weigher leads the student model to the suitable gradient way. Utilizing the revised student model parameter ${\theta }_{stu}^1$ and gold-labeled data, $Los{s}_{Lab}$ is computed. Next, the meta-weigher is updated by the following Equation.

$$\left({\theta }_{mix}^1={\theta }_{mix}^0-{\lambda }_{mix} {\nabla }_{{\theta }_{mix}} ({\propto }_m Los{s}_{Stu}|{\theta }_{stu}^1\right)$$

(13)

The learning rate for the meta-weigher is presented by ${\lambda }_{mix}$. $Los{s}_{Lab}$ is only applied for training meta-weigher at this stage, without modifying other methods.

Then, MMS gains an updated meta-weigher $ts=1$. Next, the student model’s principle was previously changed using the data from the current batch, and the meta-weigher may direct the gradient more appropriately. The meta-weigher takes the feedback of the existing batch from the student model $Los{s}_{Mix}$ as input, the model of the original student $ts=0$, in the following Equation, officially updated with the supervision technique of the upgraded meta-weigher.

$$\left({\theta }_{stu}^1={\theta }_{stu}^0-{\lambda }_{stu} {\nabla }_{{\theta }_{stu}} ({\propto }_m Los{s}_{Stu}|{\theta }_{mix}^1\right)$$

(14)

where $Loss{ }_{stu}^2$ is correspondingly calculated with $Los{s}_{Lab}$ that relies on ${\theta }_{stu}^0$ and ${\theta }_{mix}^1$. Algorithm 2 shows the three-step meta-updating complete process. Algorithm 2 is used to compute the time costs without/with, and the results show that conducting a three-step meta-update will cost an additional 0.40 times the run cost.

Algorithm 2 The method of 3-step meta-updating for the student model and meta-weigher

Input: A prepared the student model, the meta-weigher; the mixed label; the pseudo-labeled dataset $D_{mix}$; and the labeled dataset $D_{lab}$; Output: Updated parameters in the second time step for the student model and the first time step for the meta-weigher. 1: calculating weighted $D_{mix}$ with $Los{s}_{Stu}$. 2: Between time step 0 and the first-time step, updating the student model within $Los{s}_{Stu}$. 3: Between time step 0 and the first-time step, update the meta-weigher within $D_{lab}$, which relies on the student model in the first-time step. 4: Between time step 0 and time step 2, updating the student model within $D_{mix}$, which relies on the meta-weigher in the first-time step.

4. Experiments and Results

This section presents the settings used in this paper’s empirical experiments, including the hyperparameter initialization, applying the proposed to polarity detection, the benchmark datasets, and the comparative methods.

4.1. Data Collection

The DS datasets as the objective corpus are examined in this research wherein 1237 de-identified DS, obesity illness, and 15 comorbidities were considered as an illustration in

Table 1. The i2b2 obesity corpus statistics.

Diseases	Absent	Present	Unmentioned	Questionable	Total
Asthma	1	75	529	1	606
CHF	7	239	344	0	589
CAD	16	331	240	4	591
Obesity	3	245	354	4	606
Diabetes	12	396	181	6	595
Depression	0	90	519	0	609
GERD	1	98	500	3	602
Gallstones	3	93	513	0	609
Hypercholesterolemia	9	246	343	1	599
Gout	0	73	534	2	609
Hypertriglyceridemia	0	15	594	0	609
Hypertension	10	441	149	0	600
OSA	0	88	510	7	604
OA	0	89	513	0	602
Venous Insufficiency	0	14	592	0	606
PVD	0	83	525	0	608
Sum	62	2616	6940	28	9644

. This dataset was downloaded from the website www.i2b2.org/NLP/Obesity/ (15 May 2023), which was used to determine a correlation among various medical terms. The experiment was performed to evaluate the proposed MMS, the treatment quality, and health care based on the ABSA using the DS of the patients. This paper empirically estimated the performance of the proposed solutions on a real benchmark.

Referring to the parameters in Table 1, the words “Absent” and “Present” correspondingly indicated that every DS provided the data on certain illnesses only and specific illnesses together with other associated diseases. The words “Questionable” and “Unmentioned” implied that each DS may relate to other diseases and does not refer to data about other related illnesses, respectively.

4.2. Active Learning Method

The purpose of the active learning (AL) method is to reduce the problems and costs related to the manual annotation step in supervised and semi-supervised machine learning approaches [43,44]. Reduction in the manual annotation burden becomes exceptionally critical in the medical domain because of qualified experts’ high costs for annotating medical documents. This technique is applied for different biomedical tasks, for example, text classification, medical named entity recognition, and de-identifying medical records. The common AL technique is to choose the samples randomly. AL methods iteratively use ML approaches, and a human annotator can drastically decrease by being involved in the learning process.

Figure 3 shows the AL general cycle to extract information from the document. As an iterative cycle, the query strategy was used to select the informative samples from unstructured medical text documents. A human annotator does the labels, and these samples are used for extracting data and building an ML-based model at every recurrent cycle. This technique has not been fully explored for biomedical information extraction [30]. The key idea of AL is to test the effectiveness of the suggested model by decreasing the number of samples that need manual labeling. The major problem is identifying the practical examples available to train a model, producing better effectiveness and performance [45].

To accomplish the abovementioned objective, the authors collected the opinions and views (through questionnaires) from various teachers, lecturers, linguistics (English language) doctoral students, and annotators capable of teaching, understanding, and reading English. The dataset was supplied based on AL and human annotators, and they were asked to label each sentence using one of five terms: very positive, positive, neutral, negative, and very negative. The final label or polarity of the sentences was decided based on the annotators’ majority vote, wherein the dataset was labeled sentence-wise. Two illnesses were chosen randomly from a list of sixteen diseases: Asthma with 606 and Obesity with 606 DSs. The annotator aimed to create a gold standard to train data labeled at the sentence level containing 10.439 sentences. The sentiment tag for each discharge summary based on the sentences was assigned with the polarity of +1, 0.5, 1, 0, −0.5, and −1, corresponding to each positive, positive, neutral, negative, and very negative, as shown in

Table 2. The gold standard corpus statistics.

Diseases	Very Positive	Positive	Neutral	Negative	Very Negative	Total
Obesity	141	215	180	41	29	606
Asthma	213	150	211	21	11	606

. These datasets were utilized to assess the final results. To show the unbalanced label distributions in ABSA, the ratios have been a list of various tags with three different tasks. ASC sub-task denotes the ratio of very positive, positive, neutral, negative, and very negative. In this research, MMS adopted the idea from the study [1], which applied 10-fold cross-validation to randomly select the training data as validation sets for adjusting hyper-parameters. The other data are utilized for MMS training.

Table 3. The percentage of tags ratio refers to the various tags for each sub-task. For the SE and ATE sub-tasks, it shows the tags O, I, and B for all tag percentages. For the ASC sub-task, it shows the tags ratio of very positive, positive, neutral, negative, and very negative.

Dataset	Statistics	Training	Validation	Testing
Obesity	No. Sentence	2155	510	990
	No. Token	31,120	7240	10,560
	Aspect labels (%)	10.93	9.99	13.20
	Opinion labels (%)	9.02	7.29	8.90
	The ratio of tags (SE) (%)	7/2/77	7/3/79	8/3/77
	The ratio of tags (ATE) (%)	9/2/89	9/2/11	8/2/88
	The ratio of tags (ASC) (%)	8/3/3/4/6	8/3/3/4/4	9/3/3/4/5
Asthma	No. Sentence	1160	220	870
	No. Token	3090	8210	9620
	Aspect labels (%)	9.89	10.01	12.41
	Opinion labels (%)	9.15	7.40	8.97
	The ratio of tags (SE) (%)	9/3/80	9/4/90	9/4/79
	The ratio of tags (ATE) (%)	8/3/90	9/3/12	9/2/80
	The ratio of tags (ASC) (%)	8/3/3/4/6	8/3/3/4/7	9/3/3/4/6

shows the statistical details of the used datasets.

4.3. Word Embedding Model

BERT, one of the most accepted pre-trained models, can deal with succession task problems such as textual classifiers, question answering, relation extractions, and SA. This model can be trained using vast textual corpora; thus, it does not require dealing with the parameters tuning. Therefore, fine-tuning was performed for the hyper-parameters. Two models were considered inclosing, BASE and LARGE, wherein they are different in terms of the number of hidden layers, attention heads, the size of the feed-forward networks hidden, and highest sequence length parameters (the sizes of the accepted input vectors) (12 or 24), (12 or 16), (768 or 1024), and (512 or 1024), respectively [46,47]. The applied BASE model hyper-parameters are enlisted in

Table 4. The values of the fine-tuned Hyperparameters.

Hyperparameter	Value
Maximum sequence length	128
Parameters	110 M
Hidden size	768
Learning rate	0.00003
Epochs	8
Gradient accumulation steps	16
Attention heads	12
Hidden layers	12

. Compared to the LARGE model, the BASE model is better in terms of less complexity, small dataset, and low sequence length. Conversely, the LARGE model is comprised of a large number of parameters, leading to an over-fitting and small dataset that is inappropriate for training the LARGE model. In addition, the sequence length of LARGE is also a restriction because a small length is unsuitable for the LARGE model as the model has limitations in fine-tuning the parameters, leading to poor training.

These parameters were selected after performing the experiments with various feasible values. The choice of large values of epoch and learning rate could not lower the loss values. The BERT was used with two distinct tokens [SEP] and [CLS], wherein the former is used for the separations of segments, and the latter performs the classifications. The classifier from the first input token represented the whole sentence sequence, and the hidden layer of size (H) had the same size as the output vector [48]. Thus, the output of the transformed one was the closing hidden layer state used as an input for the first token. The vector was $\mathrm{C}\in {\mathrm{R}}^{\mathrm{H}}$, wherein the output is utilized as input for the full-connected classification layer. The layer matrix parameters for the classifier were $\mathrm{W}\in {\mathrm{R}}^{\mathrm{KxH}}$, where K signifies the category number. The likelihood of every category was calculated using the softmax function, which can be calculated by the probability of each category and presented by the following equations [49].

$$p=sof \left(C{W}^T\right)$$

(15)

where p is the probability of each category, and sof is the softmax function.

However, the transformer is the BERT base. The word sequence is taken from two different sentences, presented by y and x. [SEP], After x and y, the token is located, while [CLS], before x, the token is located. The embedding function presented by E and the normalization layer shown by NL, the embedding function is given below;

$${\widehat{h}}_i^0=E\left(x_i\right)+E\left(i\right)+E\left(1_x\right)$$

(16)

$${\widehat{h}}_{j+\left|x\right| }^0=E\left(y_j\right)+E \left(j+\left|x\right|\right)+E\left(1_y\right)$$

(17)

$${\widehat{h}}_{.}^0=Dro \left(NL\left({\widehat{h}}_{.}^0\right)\right)$$

(18)

where is the dropout layer [50].

The embedding techniques are passed through blocks of transformer M. Applying the activation function of Element-Wise Gaussian Error Linear Units (GELU), the function of Multi-Heads Self-Attention (MHSA), and Feed-Forward Layer (FFL) [51], by each block of transformer it is calculated as follows:

$${\widehat{h}}_{.}^{i+1}=Skip (FFL, Skip \left(MHSA, h_{}^i\right)$$

(19)

$$Skip\left(f,h\right)=NL \left(h+Dro\left(f\left(h\right)\right)\right)$$

(20)

$$FF\left(h\right)=GELU \left(h W_1^{⊺}+b_1\right) W_2^{⊺}+b_2$$

(21)

where $h^i\in {ℝ}^{\left(\left|y\right|+\left|x\right|\right)*d_h}$, $W_1\in {ℝ}^{4d_h*d_h}$, $W_2\in {ℝ}^{4d_h*d_h}$, $b_1\in {ℝ}^{4d_h}$, $b_2\in {ℝ}^{4d_h}$ and each new ${\widehat{h}}_i$ the position is equivalent to the following:

$$\left[\dots ,{\widehat{h}}_i,\dots \right]=MHSA \left(\left[h_{1,}\dots ,h_{\left|y\right|+\left|x\right|}\right]\right)=W_{0 }Concat\left(h_i^1,\dots ,h_i^N\right)+b_0$$

(22)

Instead, it is true in each head of attention that:

$$h_i^j={{\displaystyle \sum }}_{k=1}^{\left|y\right|+\left|x\right|}Dro\left({\propto }_k^{\left(i,j\right)}\right) W_{v^{h_k}}^j$$

(23)

$$a_k^{\left(i,j\right)}=\frac{exp\frac{{\left(W_{Q^{h_i}}^J\right)}^{⊺}{W}_{k^{h_k}}^J}{\sqrt{d_h}/N}}{{{\displaystyle \sum }}_{k^1=1}^{\left|y\right|+\left|x\right|}exp\frac{{\left(W_{Q^{h_i}}^J\right)}^{⊺}{W}_{k^{h_k}}^J}{\sqrt{d_h}/N}}$$

(24)

where $h_i^j\in {ℝ}^{\left(d_h/N\right)}$, $W_0\in {ℝ}^{\left(d_h*d_h\right)}$, $b_0\in {ℝ}^{d_h}$ and $W_Q^j,W_K^j,W_V^j\in {ℝ}^{d_h/N*d_h)}$, with N equal to the attention head’s number [52].

Figure 4 displays the word clouds of frequently occurring sentiment terms in medical documents. The world or text cloud is an illustration of the textual data. The text mining approaches enable highlighting of the texts of high-frequency terms as sentences, paragraphs, or documents, making more visual engagement than the ones represented manually. Four health documents’ sentiment (terms) word cloud (Figure 4) clearly shows the text cloud results. It was observed that most of the arrangements of sentiment terms in the medical documents are connected to the status, improve, stable, failure, etc. In general, these terminologies play a fundamental role in evaluating the treatment quality.

4.4. Lexicon Generation

This section presents one of the novelties of this work. The lexicon limitation is one of the most significant challenges in the medical domain. The novel mechanisms to build and integrate the lexicon-based sentimental scores have been introduced into the learning process of beep learning through an attention mechanism to address SA tasks. This section was separated into two sections, as follows:

(i)

This section compares the proposed BOW approach with SentWordNet [53] and UMLS [54], VADER [55], and TextBlob lexicon [56], relying on the semantic SA method that suffers from the issue of neglecting a neutral score. This problem is solved by applying the POS (PENN) tagging techniques like (JJ.* |NN.* |RB.* |VB.*) retrieved from www.cs.nyu.edu/grishman/jet/guide/PennPOS.html) (15 May 2023). Next, two lists of the terms were generated, wherein BOW is the first, and four lexicons are fused as the second list that relied on the hypernym’s procedure.

(ii)

In the second section, the sentiment-specific word embedding models are proposed to learn the sentimental orientation of features in the existing language models, like GloVe, Word2Vec, FastText, BERT, and TF-IDF, from the global context in a specific domain. Inspired by this intuition, this paper introduces a weak supervised solution to build a domain-specific sentiment lexicon. Specifically, this paper proposes leveraging a small seed of sentiment words with the feature distribution in the embedding space of a specific domain to associate each word with a domain-specific sentiment score. The key idea is to learn a set of cluster embeddings used to build the lexicon by looking at their neighbors in the latent space. To achieve this, this paper introduces an unsupervised neural network trained to minimize the error reconstruction, i.e., analogous to autoencoder, of a given input as a linear combination from the cluster’s matrix. The model does not require labeled data for training purposes, so constructing a sentiment lexicon in the low-resource language is possible. Finally, the obtained results were applied to assign the training dataset relying on the medical documents. The input to the model is a list of sentence indexes in the vocabulary, which is modeled by simply averaging its corresponding features’ vectors. The modeled input dimension is reduced to $k$ clusters to compute the relatedness probability to each cluster. The model is trained to approximate the modeled input as a linear combination of cluster embeddings from $C$. An example of the proposed model is shown in Figure 5. The sentiment polarity was estimated via the:

$$\mathrm{Polarity}\mathrm{of}\mathrm{text}\mathrm{sentiment}=\left(\mathrm{neg}-\mathrm{pos}\right)/\left(\mathrm{pos}+\mathrm{neg}+\mathrm{neu}\right)$$

(25)

$$m_t=\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{l}_w^i,$$

(26)

where $l_w^i$ denotes the feature vector of the word $w_i$. Hence, $m_t$ denotes the sentence representation that captures the input sentence’s global sentiment representation.

$$r_t=C^T. v_{t, }$$

(27)

where $r_t$ denotes the reconstructed vector, and $v_t$ is a weighted vector over $k$ clusters. Particularly, $v_t$ can be read as the probability that the input belongs to the clusters. It is computed by reducing the sentence representation $m_t$ from d dimension to $k$ dimension and then applying a $softmax$ nonlinearity to yield nonnegative weights:

$$v_{t}softmax\left(W. m_{t}b\right),$$

(28)

where $W$ is the projection parameter, and $b$ is the bias, which is learned during the training process.

$$C\in R^{k\times d}.$$

(29)

where $C$ denotes the cluster matrix.

In the next stage, the numerical representation for each sentence was obtained. The processed dataset was submitted words-wise to an embedding pre-trained model BERT. After that, SentiWordNet, TextBlob, VADER, UMLS, and statistical techniques were used to develop a specialized vocabulary (medical domain) that determined the polarity of each sentence. Various sizes of the lexicon (number of terms) were also examined to test this method’s reliability [57]. The lexicon’s capacities were also evaluated with different sizes of lexicons to investigate whether the largest or smallest lexicon of sentiment can produce better results [58]. The largest lexicon produced the highest performance, and the smallest lexicon resulted poorly. Figure 6 compares the obtained results relying on the number of words in each lexicon, indicating the values of lexicons 1,2, 3, 4, 5, 6, and 7 are 10.000, 20.000, 30.000, 40.000, 50.000, 60.000, and 70.000, respectively.

After creating the lexicons, the ratio of the sentiments in assigning the polarity was non-equivalent, indicating the model over-fit on the unbalanced dataset. To solve these issues, SMOTE-OSS-CNN (One-Side-Selection) and (Condensed Nearest Neighbors) were proposed for the imbalanced datasets following the balance of the number of samples for all classes in the dataset. Balancing was accomplished by generating the synthetic minority samples so that the number of samples for the minority class was almost equal to that of the majority class. Consequently, the models achieved a high accuracy using the SMOTE-OSS-CNN approach, producing synthetic samples of the minority class [59]. In brief, the proposed models achieved a high level of accuracy when the SMOTE-OSS-CNN approach was employed to produce the artificial samples of the minority class. The binary classification was performed as the first experiment of the proposed method, which was achieved for four cases of illness: asthma with neutral and positive labels, asthma with neutral and negative labels, obesity with neutral and positive labels, and obesity with neutral and negative labels.

5. Baselines

To validate the performance of the proposed solution, this paper compares it with state-of-the-art techniques; all the basslines have been experimented on based on the datasets that were applied in this paper.

• IMN-GloVe is a collaborating MTL method for SE, ASC, and ATE, which can be applied at both the token word and document levels. IMN proposed a message transfer system that may change across tasks via shared latent variables. MMS compares their findings utilizing BERT-large for a pre-trained linguistic model for a fair comparison.
• BERT-single, BERT is a primary component of our BERT-based proposed solution. To illustrate the effectiveness of the auxiliary sentence, this paper also compared it with BERT-single, where ACSA is regarded as a text classification problem.
• AEN-BERT is an attentional encoder network for the ATSA task, which models context and target through an attention-based encoder. AEN-BERT is the pre-trained model BERT fine-tuning to tackle ATSA tasks. In our experiment, this paper adopts it to tackle the ACSA task.
• BERT-SPC, a BERT-based approach, introduced concatenating the aspect and sentence in one long text, then fine-tuned BERT to address the ATSA task. In our experiment, this paper adopts it to tackle the ACSA task.
• LCF-BERT proposed a local context focus mechanism that used the context features mask and context dynamic weight layer to capture the local aspect context. An addition layer based on BERT is applied to exploit the connection between the local and global context.
• BERT-Pair, a BERT-based fine-tuning model that considered the aspect category manually annotated as an auxiliary sentence. It introduced different variants to convert ACSA to sentence-pair classification tasks, such as question answering and natural language inference.

6. Evaluation Metrics

The suggested model in this work has been evaluated using recall, precision, accuracy, and F-measure (F1). The performance and efficacy of a given approach model are assessed using a variety of measures. Because not all measures are appropriate for a particular problem, a model is crucial in this last stage of development. Sometimes, a new assessment metric can be offered to assess the final approach. The measurements chosen may impact how a model’s effectiveness and performance are compared and evaluated. A simple confusion matrix is a two-by-two that lists all incorrect and correct patterns predicted by a classifier model [60]. For instance, True Negative (TN) shows the number of negative samples that the classifier correctly predicted as negative, False Positive (FP) shows the number of negative samples that the classifier incorrectly predicted as positive, and so on. False Negative (FN) shows how many positive samples the classifier incorrectly identified as negative, and True Positive (TP) shows how many positive samples the classifier correctly predicted as positive [61].

MMS applies BERT-large with 1024 hidden dimensions. The maximum number of sentences that may be entered is 100. To update MMS, Adam Optimizer is utilized. The batch size is 64. The learning rate ${\lambda }_{Stu}$ of the linear layers and BERT-large on the maximum of BERT-large is equal to 8 × 10⁻⁵ and 4 × 10⁻⁵, respectively. The 8 × 10⁻⁵ is referred to ${\lambda }_{Mix}$. The method applied a threshold ${\beta }^{tag}$, lower (L%) and upper (U%), and limits of doubts for the objects brought to the student model by MMS are presented in Table 3. In Algorithm 1, the total training epoch is 500, and $N_{aug}$ is set to 2.

7. Results

The key findings of the suggested MMS are shown in

Table 5. The comparative results of the aspect category polarity on Asthma and Obesity datasets were measured by the F1 metric with multi-sub tasks. Some results of the state-of-the-art methods are retrieved from the original papers, while the others are our implementation. The best scores are highlighted in bold.

Model	Asthma				Obesity
	SE-F1	ASC-F1	ATE-F1	ABSA-F1	SE-F1	ASC-F1	ATE-F1	ABSA-F1
IMN-GloVe	82.0%	84.0%	87.3%	88.67%	86.9%	88.56%	85.5%	93.3%
BERT-Single	85.14%	87.25%	90.14%	90.07%	86.87%	87.98%	86,68%	92.06%
AEN-BERT	88.3%	89.9%	93.83%	80.74%	88.08%	90.0%	92.28%	93.93%
BERT-SPC	89.9%	91.6%	93.3%	90.11%	88.83%	88.55%	92.51%	93.89%
LCF-BERT	88.7%	91.1%	92.8%	91.72%	89.3%	89.86%	93.6%	94.4%
BERT-Pair	91.37%	91.1%	92.67%	93.60%	88.7%	86.82%	92.1%	92.4%
MMS (our)	87.85%	89.05%	95.94%	96.4%	94.45%	86.64%	89.35%	93.0%
MMS (10%)	87.92%	84.61%	85.26%	92.74%	91.04%	86.24%	88.58%	86.4%
MMS (20%)	91.82%	89.25%	89.87%	93.76%	92.78%	89.07%	90.89%	89.05%
MMS (40%)	92.07%	90.86%	91.25%	93.39%	92.78%	90.24%	91.54%	89.35%
MMS (70%)	93.78%	91.37%	91.77%	90.07%	93.04%	89.95%	91.47%	93.60%
MMS (100%)	91.37%	91.77%	86.82%	96.90%	93.72%	93.78%	93.43%	94.76%

. Since MMS is a self-training technique, this paper investigates how well it performs with various quantities of manually annotated data. It is important to note that while this paper reported MMS performance using the datasets in 10%, 20%, and 40% with benchmarking baselines, all labeled data from the corresponding dataset are shown in Table 5. Additionally, MMS has been evaluated using additional unlabeled dataset augmentation with 70% and 100% of the data. Between 70% and 100% of labeled data, respectively, have been used in both settings. The introduction of the unlabeled dataset augmentation is intended to provide extra pseudo-labeled data to aid in learning the student technique and the meta-weigher. More labeled data for MMS leads to increases in all MMS F1 metrics of various sub-tasks, as shown in Table 5. In terms of ABSA-F1, which has just 30% labeled data, this paper suggested MMS performs better than the state-of-the-art techniques. In addition, MMS sightly best performs the state-of-the-art method BERT-Pair in the Asthma and Obesity datasets, with only 40% labeled data. The 40% labeled data of Asthma has been adopted, and ABSA-F1 of MMS is lower than the results of full-data supervised AEN-BERT. This conclusion results from the magnitude of the asthma data being less than half that of the obesity data. The produced pseudo data may be excessively noisy from an unreliable instructor model, making it insufficient to train a robust instructor model to direct a student model.

Next, MMS uses additional unlabeled data from another dataset for self-training together with 70% and 100% of the annotated data from the datasets for Obesity and Asthma as labeled data. The Dataset section describes the extra unlabeled data that was used. The reductions in Asthma are visible with more labeled data, whereas the improvements in Obesity begin to be restricted. In Asthma, for example, ABSA-F1 measurements rise by 1.05% (40% vs. 70%) and 1.71% (70% vs. 100%). In Obesity, the equivalent increases are 0.75% (40% vs. 70%) and 0.77% (70% vs. 100%). These findings suggest that MMS can, to some extent, decrease the need for labeled data.

To summarize, MMS may obtain equivalent outcomes with only 40% labeled data compared to the strongest baselines in each dataset. MMS uses more stringent data choices to prevent noise effects when using 70% and 100% labeled data. In such cases, where there is a lack of data, MMS can obtain equivalent outcomes with only 50% labeled data. MMS uses more stringent data choices to prevent noise effects when using 70% and 100% labeled data. Because the issue of inadequate data is not evident in this situation, MMS creates fewer pseudo-labels and focuses on re-weighting data instances in $D_{mix}$. Then, the final improvement of MMS is 1.41% compared with the BERT-Pair method in Obesity and 0.84% with LCF-BERT in Asthma.

More labeled data for MMS, all MMS F1 scores of distinct sub-tasks are generally growing. With only 20% labeled data from ABSA-F1, this paper suggested MMS beats IMN-GloVe in all sub-tasks. In the same settings, MMS is comparable to the performance of LCF-BERT in Asthma and Obesity datasets, achieving improved results in Asthma. By applying the same BERT encoder and 40% labeled data, MMS can attain the enhancements of around 6% ABSA-F1 for Asthma and approximately 1% ABSA-F1 for Obesity, compared with the state-of-the-art methods. Following all baselines, this paper considers the first sentiment label predicted for an aspect phrase to be an ASC result, and this paper excludes conflicting sentiment labels from the ASC subtask. This exclusion may cause the ratio sum in ASC to be less than 100%. Additionally, as mentioned earlier, this paper assigns an enhanced dataset for each dataset to further investigate MMS’s data usage.

On the other hand, the effects of the quantity of data with and without labels. This paragraph investigates the effects of various amounts of annotated data on MMS. The impacts of incorporating and removing unlabeled data on model performance are first contrasted (see Table 5). All models often demonstrate the growing F1 measures by using additional labeled data. Across all of the used labeled data ratios, MMS (including unlabeled data) may consistently enhance the ABSA-F1 measurements by 10% to 40%. The largest discrepancy between MMS labeled data and unlabeled data emerges when employing 10% labeled data in Asthma, accounting for 4.10%. The greatest noticeable gains for Obesity are provided by 1.10% with 40% labeled data. MSM with 10% tagged data shows a modest improvement for the obesity and asthma datasets. The improvement is visible when utilizing more than 20% tagged data. This is due to the difficulty in managing a workable instructor model with only 10% labeled data. Effective initialization of the instructor model is required for MMS to prevent progressive drift.

This paper illustrates the produced weights and contrasts them with different common weighting techniques since the MMS meta-weigher seeks to automatically learn various weighting strategies under various scenarios (weighers). Weighers’ inputs are virtual losses in the range of 0 to 10. The virtual losses are uniformly distributed, automatically created numbers. These losses replicate all potential inputs to our weighers within a reasonable range, which helps visualize the learned distribution of weighers. Afterward, several weighers are fed the losses. For more information, see Figure 7, including four different cases (a) to (d). The explicit mathematical functions are manually used in the construction of the weighers. The weigher is this paper’s suggested meta-weigher, which is trained using various ABSA sub-tasks and datasets. We take advantage of visualization to reveal certain details about the suggested model. To illustrate the effectiveness of our MMS model, this paper first shows the confusion matrix as a heat map. Figure 8 displays the MMS heat maps on the DS dataset. Overall, the sentiment classes and the performance of our model are balanced. In terms of convergence capability, the quantity of trainable parameters, and training time, our suggested model outperforms IMN-GloVe. Our models require a great deal less training time as compared to BERT-Pair. As a result, our model learns more rapidly than its equivalent, as seen in Figure 9a, where training time is recorded on a single NVIDIA Quadro M5000. Our MMS model is more parameter-efficient and takes less time to train, whether the context is global or local. The training curve is displayed with training and testing losses in Figure 9b. For all the compared models, this paper uses the same loss function. While LCF-BERT is prone to overfitting, in this paper, MMS converges as quickly as its competitors.

8. Conclusions

In conclusion, this research proposes a multi-task approach for aspect-based sentiment analysis, utilizing deep neural network models for sentiment detection. To address the issue of insufficient and unbalanced data in the MABSA task, the SMOTE-OSS-CNN, and meta-based self-training approach MMS are suggested. MMS contains the meta-weightier, the student model, and the instructor model, and a 3-step meta-updating technique is suggested to avoid noise in the automatically labeled data. Overall, this study has implications for discharge summary sentiment analysis.

Future work will involve dealing with numerous sequence labeling jobs under more challenging low-data conditions. Moreover, this paper intends to explore data augmentation to address imbalanced data to increase performance stance detection. Additionally, feature extraction is required for greater contextual information, for example, information on the aforementioned discharge summary, discharge in the same thread, and disease connection networks to enhance sentiment stance detection on discharge summary.