Hate Speech Detection by Using Rationales for Judging Sarcasm

Abstract

The growing number of social media users has impacted the rise in hate comments and posts. While extensive research in hate speech detection attempts to combat this phenomenon by developing new datasets and detection models, reconciling classification accuracy with broader decision-making metrics like plausibility and faithfulness remains challenging. As restrictions on social media tighten to stop the spread of hate and offensive content, users have adapted by finding new approaches, often camouflaged in the form of sarcasm. Therefore, dealing with new trends such as the increased use of emoticons (negative emoticons in positive sentences) and sarcastic comments is necessary. This paper introduces sarcasm-based rationale (emoticons or portions of text that indicate sarcasm) combined with hate/offensive rationale for better detection of hidden hate comments/posts. A dataset was created by labeling texts and selecting rationale based on sarcasm from the existing benchmark hate dataset, HateXplain. The newly formed dataset was then applied in the existing state-of-the-art model. The model’s F1-score increased by 0.01 when using sarcasm rationale with hate/offensive rationale in a newly formed attention proposed in the data’s preprocessing step. Also, with the new data, a significant improvement was observed in explainability metrics such as plausibility and faithfulness.

1. Introduction

Although half of the world’s population uses social media for networking, many have encountered hostility in some form or another [1], such as those categorized under crimes against minorities [2]. Victims of online hate suffer from psychological and emotional distress like depression, anxiety, and in severe cases, physical harm, which potentially leads to suicide [3,4,5,6]. Over the years, continuous research has been conducted to develop hate speech detection methods that could effectively reduce online hate. In order to achieve such methods, understanding online hate and predicting hate crimes in their various manifestations are very important [7]. Therefore, to improve hate detection, hate speech datasets [8,9,10,11,12,13] have been developed over the years, which mostly focus on hate speech attributes closely related to hate expressions (e.g., abusive comment, hate speech, offensive, and disrespectful content), the target of hate (e.g., origin, gender, religion, and sex orientation), and retweets (now termed posts) or the like are made available for the purpose of progressive research and development.

A distinction is recognized between hate speech and offensive comments in this study. Hate speech is described as statements that incite violence or prejudice against specific groups based on protected attributes such as race, religion, or sexual orientation. These expressions typically aim to harm and marginalize, reflecting deep-seated prejudice.

Offensive comments, which are potentially hurtful or rude, are generally not directed to incite direct harm nor are they always targeted at specific groups. These may include profanity or derogatory remarks made in a general context without the intention to incite violence or discrimination against a group.

In this paper, a clear distinction between these two types of content is maintained to tailor the model’s response appropriately. Not only is the presence of hate or offense identified, but also the context and the target, which are considered crucial for effective moderation on social platforms. This approach allows for more nuanced detection and response strategies, which are essential for maintaining the integrity and inclusivity of online communication spaces.

Some studies [14,15,16,17] have proposed models for improving automatic hate speech detection. However, these models often lack inductive reasoning due to an overestimation of state-of-the-art approaches for hate speech detection [18,19]. These advanced models typically rely on triggered vocabularies for predictions, leading to substantial evidence of racial bias in major published hate speech datasets [20,21]. Moreover, as hate speech detection models become more complex, it becomes increasingly challenging to clarify their decision-making process [22].

To address these concerns, a benchmark dataset named “HateXplain” [13] was introduced. This dataset is unique in that each data point is annotated with labels indicating whether the content is hate, offensive, or normal, along with details such as the mentioned target communities and text snippets known as rationales [23]. These rationales, which are identified by human annotators, provide context that is instrumental in teaching the model not only to recognize hate speech but also to understand the context that characterizes such language, enhancing both the robustness and interpretability of the model.

During the training phase, these rationales inform a specialized attention mechanism within a neural network architecture, focusing the model’s attention more on words and phrases within these rationales. This targeted learning approach helps the model to prioritize the most critical attributes that indicate offensive/hate speech, leading to improved accuracy and reliability in classifications.

Furthermore, the use of rationales extends into the operational phase of the model. Here, the trained model applies the patterns it has learned to new, unseen texts. When it identifies words or phrases that mirror the rationales from the training dataset, the model scrutinizes these pieces of text with heightened attention. This method not only improves the accuracy of classification but also ensures that each decision made by the model can be traced back to clear, understandable linguistic evidence, thereby enhancing the transparency and accountability of automated hate speech detection systems.

Thus, incorporating rationales into the training and operational phases of model bridges the gap between traditional text analysis and a deep, contextual understanding of language used in hate speech. This not only enhances the model’s performance but also ensures that each decision made by the model is transparent and explainable, which is crucial for applications in sensitive areas such as online content moderation.

While recent studies have highlighted flaws in previous hate speech detection methods, including issues with model performance and datasets, the evolving nature of language and the constant emergence of new ways social media users spread hate speech pose ongoing challenges. Platforms like Twitter or Facebook often inspire users to employ figurative language in conveying their opinions. This includes the concealment of hate comments under the guise of sarcasm [24], a common form of passive-aggressive speech.

Therefore, sarcasm represents a crucial aspect for enhancing hate speech detection. However, as far as the knowledge of this study extends, there has not been any previous research in this sector specifically addressing hate concealed under the pretext of sarcasm. As a solution to this problem, this research proposes a new dataset that also focuses on sarcasm alongside hate. For the study, 3199 data points were selected from the benchmark HateXplain dataset [13] and annotators were hired to label them based on sarcasm as well as choose the accompanying rationale. LIME [25] was utilized for generating important tokens as rationale to assist in the classification task. The BERT model, as documented in [26], gives significance to the HateXplain data rationales provided by human annotators, but it tends to misclassify the label (offensive speech—OF). Notably, for this study’s data, BERT assigned the label (hate speech—HS).

This study aimed to enhance hate speech detection by incorporating a sarcasm-based rationale in addition to the conventional hate/offensive rationale. The main objectives were to develop a novel methodology for integrating these rationales into the existing model frameworks and to assess the impact of this integration on model performance, specifically in terms of classification accuracy and explainability.

The growing prevalence of online hate speech necessitates robust detection mechanisms that can effectively handle the nuances of language, including sarcasm, which is often used to disguise hateful intents. By addressing these challenges, the research seeks to contribute to safer online environments and more effective moderation tools.

The contributions of this research are as follows:

1.

A new dataset for hate speech detection is presented, comprising two kinds of labeled categories: (i) sarcastic/non-sarcastic; and (ii) hate speech/offensive/normal. Rationale selection is based on whether it fits into the categories of sarcastic and hate/offensive;

2.

The sarcasm rationale is incorporated in cases where the data are both sarcastic and hate speech/offensive. This involves preprocessing steps before utilizing it in the model’s attention calculation;

3.

The inclusion of sarcasm-based rationale alongside hate speech/offensive-based rationale improved both the classification and explainability metrics of the BERT model.

2. Related Works

The proliferation of online platforms has significantly increased incidents of hate speech, impacting vulnerable individuals and communities profoundly [3,5,6,27]. Comprehensive studies [4,28,29,30] have documented these effects, emphasizing the urgent need for effective detection mechanisms. Current research primarily focuses on developing computational models to detect explicit and implicit hate speech [11,12,13]. However, traditional methods often overlook nuanced expressions such as sarcasm and indirect speech that can mask hateful intentions [31,32,33], a critical gap highlighted in the literature. This study addresses this by enhancing model sensitivity to not only detect overt hate speech but also subtler, sarcastic expressions often overlooked by current methodologies [14,15,16,17,34,35].

Dataset: Previous datasets designed to detect hate speech primarily focus on expressions related to hate, encompassing abusive comments, hate speech, offensive language, and disrespectful content. A common issue in much of the preceding research is the tendency to conflate hate speech with abusive or offensive language [34]. Furthermore, some studies merged offensive and hate language into a single category, while only a limited number [34,36] made efforts to distinguish offensive from hate speech. In the published HateXplain dataset, subjectivity is recognized as a pivotal aspect, acknowledging that numerous messages may be offensive without qualifying as hate speech. For example, in the United States, the term “nigga” is commonly used in online language by the Black community in a manner that is not necessarily intended to be malevolent toward another individual [35]. Building on the categorization framework that separates hate from offensive language [34], the HateXplain dataset adopts three classes: hate, offensive, and normal. Moreover, HateXplain incorporates the concept of using rationales [23]. Since human annotators are required to highlight a span of text supporting their labeling decision, this enriches the rationale annotation process. As a result, HateXplain provides the first benchmark hate speech dataset with human-level explanations, enhancing interpretability and transparency.

Despite the progress made by HateXplain in addressing major issues related to hate speech, a contemporary challenge persists—the use of sarcasm as a disguise for spreading hate speech [24]. Given the non-physical and implicit nature of online violence, it often appears subtle. Social media platforms frequently host stereotypical, racial, and gender jokes, sometimes disguised as sarcasm. While seemingly harmless, the repetition of such jokes can lead to negative psychological effects, resembling a form of cyberbullying [37,38]. Despite existing research on sarcasm detection [39,40] and multi-modal detection [41], there is a noticeable gap—no research has explored the connection between hate and sarcasm for automatic hate speech detection. In our work, assuming hate is hidden under the guise of sarcasm, and rationale can provide human-level explanations, we collected data from HateXplain, which is already labeled with rationale based on hate/offensive/normal for new dataset creation. This new dataset is different from existing datasets in that it introduces two categories, and based on each category (sarcasm and hate/offensive), we provide rationale for human-level explanations.

Model: Despite numerous models claiming to achieve state-of-the-art performance on specific datasets, their ability to generalize is often questionable [18,19]. Challenges emerge when these models misclassify comments related to identities frequently under attack (e.g., gay, black, and/or Muslim) as toxic even when lacking any malicious intent [42]. The prevalence of biased predictions, stemming from an overemphasis on specific trigger vocabulary, can result in further discrimination against groups already targeted by online abuse [20,21]. Another pressing issue in current methodologies is the lack of decision-making transparency. As hate speech detection models grow in complexity, elucidating their decisions becomes increasingly challenging [22]. Therefore, it is important to focus on interpretable rather than performance-based models. HateXplain tackles the challenge of model explainability by concurrently learning target classification and the rationale behind human decisions. This dual learning approach aims to enhance both aspects. In this study, existing state-of-the-art models such as CNN-GRU [14], BERT [26] and BiRNN [43] were leveraged. Drawing insights from HateXplain, where BERT demonstrates slightly superior results in classification, bias, and a key explainability metric (faithfulness), as well as being the only model that can provide a vector representing the attention for each token, the study opted to employ BERT for a more comprehensive comparison with HateXplain.

3. Methods

This section explains the preprocessing steps and model employed in evaluating the dataset. The preprocessing steps encompass HateXplain dataset management, the data from the newly proposed dataset, and a novel preprocessing method designed for data exhibiting both sarcasm and hate/offensive characteristics. In the model details, the utilization of ground truth attention for training is clarified along with the model’s requirement to produce a vector representing the attention for each token.

3.1. Dataset Selection

Among the various available hate speech datasets, HateXplain was selected for this research due to its unique features. HateXplain provides annotated rationales for each labeled instance, which is critical for developing models that not only predict hate speech but also offer insights into the reasons behind these classifications. This transparency is vital for the interpretability of models in sensitive applications like content moderation. Additionally, HateXplain’s focus on both hate speech and offensive language, annotated in a real-world context, aligns well with the objectives of this study to handle nuanced expressions like sarcasm effectively.

3.2. Preprocessing

Conversion of text data: Essential preprocessing tasks are efficiently executed to ensure data uniformity and usability. Text normalization and cleaning involve converting all text to lowercase, removing HTML tags, and substituting URLs and other non-textual elements with placeholders to standardize data. The Ekphrasis tokenizer, designed for social media, segments and annotates text with emotional and stylistic cues critical for detecting sarcasm and hate speech. Advanced tokenization for the model input employs BertTokenizer to refine text into tokens suitable for BERT models, appending special tokens like ‘[CLS]’ and ‘[SEP]’ to facilitate proper sequence formatting, while also preparing sequence input IDs and attention masks. Integration with SpaCy enhances the grammatical and semantic handling of the text, supporting accurate model training and effectiveness. These comprehensive preprocessing steps meticulously prepare the dataset, addressing language complexities and nuances vital for the effective detection of sarcasm and hate speech, thus enhancing model accuracy and interpretability.

Conversion of rationale to ground truth attention: Ground truth attention is established for posts identified as toxic (hatespeech/offensive) using a baseline method for scenario (i). Furthermore, for scenario (ii), where posts are classified as hatespeech/offensive in the first category and sarcasm in the second category, ground truth attention is derived from the annotator’s rationales depicted in Figure 1. In this process, each rationale is transformed into an attention vector with values of 0 and 1. Tokens within the rationale are assigned a value of 1 in the attention vector, while the rest receive a 0. The average of these attention vectors is then calculated to obtain the ground truth attention for each post. Subsequently, a softmax function is applied to normalize the attention.

An inherent challenge in the ground truth attention vector arises from the potential proximity of values between rationale and non-rationale tokens. To address this, we utilize the temperature parameter ($\tau$) within the softmax function. This strategic application of the temperature parameter facilitates an adjustment in the probability distribution, amplifying the emphasis on the rationales. The fine-tuning of this parameter is a meticulous process conducted through adjustments in the validation set.

In the conversion process of toxic (hatespeech/offensive) rationale, if at least two out of three annotators label the data as hatespeech/offensive, the rationale is considered. However, for sarcasm data, even if just one annotator labels the data as sarcasm, the rationale is converted. This distinction is made due to the subjective nature of understanding sarcasm in text data, which can vary significantly from person to person.

Proposed method for the conversion of rationale to ground truth attention: In this study, an innovative algorithm tailored for data exhibiting both hateful/offensive and sarcastic characteristics is introduced, see Algorithm 1. In this process, similar to the previous approach, each rationale is transformed into an attention vector with values of 0 and 1. Tokens within the rationale are assigned a value of 1 in the attention vector, while the rest receive a 0. Here, the attention vector for toxic (hate/offensive) after conversion is denoted as $H=(h_1,h_2,h_3,\dots ,h_n)$ in the 1st category, and the attention vector for sarcasm, denoted as $S=(s_1,s_2,s_3,\dots ,s_n)$, corresponds to the 2nd category. This procedure is applied to a sentence represented by tokens $t_1,t_2,t_3,\dots ,t_n$ and its corresponding rationale after vector conversion $r_1,r_2,r_3,\dots ,r_n$.

Algorithm 1 Calculate ground truth attention vector

1: procedure Calculate Ground Truth Attention  $\mathrm{Sentence}\mathrm{tokens}{t}_1,t_2,t_3,\dots ,t_n$; $\mathrm{Rationale}\mathrm{vector}\mathrm{after}\mathrm{conversion}{r}_1,r_2,r_3,\dots ,r_n$;  Hate/Offensive Attention Vector, $\mathbf{H}=(h_1,h_2,h_3,\dots ,h_n)$; $\mathrm{Sarcasm}\mathrm{Attention}\mathrm{Vector},\mathbf{S}=(s_1,s_2,s_3,\dots ,s_n)$ 2:     for $i\leftarrow 1$ to n do 3:         Calculate combined attention score for token $t_i$: 4:         if $s_i>h_i$ then 5:            $r_i\leftarrow {\displaystyle \frac{s_i+h_i}{2}}$ 6:         end if 7:         if $s_i<h_i$ then 8:            $r_i\leftarrow h_i$ 9:         end if 10:         if $s_i=h_i$ then 11:            $r_i\leftarrow h_i+{\displaystyle \frac{s_i+h_i}{2}}$ 12:         end if 13:     end for 14:     return Rationale Attention Vector r 15: end procedure

Drawing on the concept of an attention vector, a new attention vector is established for tokens within the rationale that simultaneously convey hateful/offensive and sarcastic elements. This new attention vector is utilized for the calculation of the ground truth attention. Specifically, if, in sarcasm rationale, a vector element $s_i$ surpasses a hate rationale vector element $h_i$, the final newly assigned vector element is $r_i=0.5$. When a sarcasm rationale vector element $s_i$ is less than the hate rationale vector element $h_i$, the final new assigned vector element $r_i=1$.

And lastly, when the sarcasm rationale vector element $s_i$ is equal to the hate rationale vector element $h_i$ and if both of them are 1, then the final new assigned vector element $r_i=1.5$, as shown in Figure 2.

This approach refines the interpretation of attention dynamics in the context of sentences exhibiting both hatespeech/offensive and sarcasm. Following this, the average of the attention vectors is calculated to obtain the ground truth attention for each post, as in the previous process shown in Figure 1. Once the updated ground truth attention is obtained, a softmax function is applied to distribute the concentration of rationale, mirroring the final ground truth attention calculation as previously demonstrated.

3.3. Model

In this section, the details of the BERT model are provided. The model was trained on both HateXplain and the newly proposed dataset. Here, how the cross-entropy loss is calculated is explained for ground truth attention.

• Bidirectional Encoder Representations from Transformers (BERT)

In the BERT model, there are approximately 12 layers, and each layer consists of 12 attention heads, resulting in a total of 144 attention heads. Typically, a subset of these heads, denoted as “supervised heads”, is utilized in the last layer of BERT for attention supervision, Figure 3a. HateXplain [13] employs this term to describe the selected attention heads. For each supervised head, the attention weights corresponding to [CLS] are utilized, and the cross-entropy loss is calculated against the ground truth attention vector. This process ensures that the final weighted output corresponding to [CLS] focuses on words aligned with the ground truth attention vector. The same procedure is repeated for all supervised heads. The resultant loss from attention supervision is the average of the cross-entropy losses from each supervised head, further multiplied by the regularization parameter, denoted as $\lambda$. The utilization of attention weights associated with [CLS] in conjunction with the cross-entropy loss helps align the model’s attention with the ground truth attention vector.

Following the computation of ground truth attention, the BERT model undergoes training using the preprocessed data and ground truth attention. Subsequently, a loss from attention supervision is computed, employing cross-entropy between attention values and the ground truth attention as the loss function for attention, Figure 3b. The total loss, denoted as $L_{total}$, is determined by the sum of the prediction loss ($L_{prediction}$) and the attention loss ($\lambda ·L_{attention}$). In this equation, $\lambda$ acts as a weighting factor, regulating the influence of the attention loss on the overall loss.

4. Experimental Conditions

4.1. Dataset

This section is divided into two categories: Benchmark Dataset HateXplain [13] and New Dataset Creation.

Benchmark Dataset—HateXplain: This dataset was obtained from platforms like Twitter and Gab, which were previously explored in hate speech studies. The corpus, comprising tweets and Gab posts, was annotated with labels (hatespeech/offensive/normal), target communities, and text snippets marked by annotators supporting the labels. Each data point was evaluated by three independent annotators, and if there was a consensus among two or more annotators on a specific category, the data were officially categorized accordingly.

New Dataset Creation: From the benchmark dataset, HateXplain (which is publicly available in online), 3199 data points were randomly selected and each underwent manual scrutiny to determine its sarcastic nature, illustrated in Figure 4a. Three independent annotators meticulously labeled each data offering a nuanced perspective on the presence or absence of sarcasm, complemented by a detailed rationale for their labeling decisions, illustrated in Figure 4b.

About 12 anonymous annotators, along with recruited volunteers, were engaged in the labeling task for this project. Exclusively provided with textual data (posts) from the selected entries in the HateXplain dataset, these annotators focused on identifying sarcasm. Drawing inspiration from the use of rationales in detecting hate/offensive comments, as observed in the benchmark dataset, our approach involved creating rationales [23] for sarcasm in comments. When an annotator labeled a data point as sarcasm, they highlighted a specific text span they believed conveyed the sarcastic nature of the comment/post. Given the subjective nature of sarcasm perception, even if just one annotator categorized a data point as sarcastic, it was officially placed into the sarcastic category.

4.2. Data Distribution

All 3199 data points were categorized into two groups: (i) toxic; and (ii) sarcasm. Within the first category, which comprises toxic (hatespeech/offensive) data, there were 1637 instances labeled as toxic, 1552 labeled as normal, and 10 labeled as undecided. The distribution of toxic data was visualized using a heatmap.

Table 1. Splitting of data into different sets.

Data Categories	Total Amount	Training	Validation	Testing
All category	3199	2599	300	300
Toxic (Hate + Offensive)	1637	1320	164	153
Sarcasm	1487	1198	145	144
Toxic under Sarcasm	755	608	79	68

Here, the table shows that data in both categories were not balanced and how they were distributed in different sets. Toxic under sarcasm shows which sarcasm data were used in the new method.

illustrates the distribution of toxic data, presenting two perspectives: (a) non-sarcasm and sarcasm data; and (b) other data and toxic data under sarcasm. In perspective (b), “other” encompasses sarcasm data not classified as toxic along with all non-sarcasm data.

Observing Figure 5, it is apparent that, while the total number of sarcasm data points was 1487, only 755 data points could be utilized under the proposed method as they fall under toxic data. Here, the last three data points from the undecided section could not be utilized as they do not conform to toxic data criteria. The heatmap depicted in the following figure illustrates the distribution of data from both perspectives.

4.3. Dataset Splitting

The set of 3199 data points was divided into three subsets for training, validation, and testing. As the dataset was imbalanced, a stratified split was performed on the dataset to ensure a minimum class balance was maintained across all categories and the best performance was achieved. Table 1 shows the total amount of toxic data (hatespeech/offensive), sarcasm data, and data that are both sarcastic and toxic (toxic under sarcasm). Furthermore, the splitting of these data in different sets are shown here.

Through both criteria—(i) non-sarcasm and sarcasm data; and (ii) other data and toxic data under sarcasm—the data distribution can be observed for all subsets.

Figure 6 above illustrates the distribution of data across all subsets: training, validation, and testing. Each set exhibits two distinct criteria. Specifically, only criterion (ii) data were utilized in accordance with the proposed method. Furthermore, the graphs reveal that sarcasm comprises nearly half of the data for all labels within the first category (hatespeech, offensive, normal).

4.4. Hyperparameter Settings

In the hyperparameter tuning phase, our approach to training the BERT model involved the incorporation of specific settings. A dropout rate of 0.1 was applied after the fully connected layer, utilizing a batch size of 4 and adopting a learning rate of 2 × 10⁻⁵. For the calculation of the ground truth attention, we opted for a configuration with six supervised heads, accompanied by an attention lambda ($\lambda$) value of 0.001. Notably, in the selection of rationale tokens, we adhered to the practice of choosing the top five tokens, aligning with the average length of annotation spans within the dataset. This careful calibration of hyperparameters contributes to the robustness and effectiveness of our model during training and attention calculation.

4.5. Metrics for Evaluation

In the quest to uncover hate speech masked within sarcasm, a range of metrics covering diverse dimensions of hate expression was considered. Taking cues from the HateXplain paper’s approach to hate speech classification, emphasis was placed on two primary types of metrics: performance-based and explainability-based.

• Performance-based metrics: Priority was given to these metrics, with accuracy, macro F1-score, and AUROC score reported for both the baseline and proposed method in accordance with established standards. These metrics serve to assess the classifier’s performance in distinguishing among the three classes: hate speech, offensive speech, and normal speech;
• Explainability-based metrics: We adhered to the framework outlined in the ERASER benchmark by to gauge the explainability aspect of the model. This assessment involved measuring the plausibility and faithfulness. Plausibility denotes the persuasiveness of the interpretation to humans, while faithfulness pertains to the accuracy with which it reflects the genuine reasoning process of the model [44,45].

  1.

  Plausibility: To assess plausibility, metrics for both discrete and soft selection were employed. Specifically, the Intersection-Over-Union (IOU) F1-score and token F1-score for the discrete case were reported, along with the Area Under the Precision-Recall curve (AUPRC) score for soft token selection [45]. The IOU metric was utilized for credit assignment in cases of partial matches. This metric, defined at the token-level, measures the overlap between two spans divided by the size of their union [45]. A prediction was considered a match if the overlap with any ground truth rationale exceeded 0.5. These partial matches were used to compute the IOU F1-score. Token-level precision and recall were also calculated, and these were used to derive token-level F1 scores (token F1). Additionally, to assess the plausibility of soft token scoring, the AUPRC was reported by sweeping a threshold over the token scores;

  2.

  Faithfulness: To assess faithfulness, two metrics were reported: comprehensiveness and sufficiency [45].

  -

  Comprehensiveness: This metric evaluates the extent to which the model’s predictions are affected by the removal of predicted rationales. For each post $x_i$, a contrast example ${\tilde{x}}_i$ was created by removing the predicted rationales $r_i$ from $x_i$. Let $m{\left(x_i\right)}_j$ be the original prediction probability provided by a model m for the predicted class j. Then, $m{(x_i\mid r_i)}_j$ represents the predicted probability of ${\tilde{x}}_i$ (i.e., $x_i$ without $r_i$) by the model m for class j. Comprehensiveness was computed as $m{\left(x_i\right)}_j-m{(x_i\mid r_i)}_j$. A higher value of comprehensiveness suggested that the rationales significantly influenced the prediction;

  -

  Sufficiency: This metric measures the adequacy of the extracted rationales for the model to make a prediction. Sufficiency was calculated as $m{\left(x_i\right)}_j-m{\left(r_i\right)}_j$, where $m{\left(r_i\right)}_j$ represents the predicted probability of the rationales $r_i$ by the model m for class j.

5. Results

In this section, the performance and explainability score of the BERT model for both the baseline and new method is shown. The tokens for the explainability calculation were chosen using the LIME method. Rationales were comprised of the top five tokens, consistent with the baseline methodology, reflecting the average length of annotation spans in the dataset.

Table 2. Model’s performance and explanability result.

Method	Performance			Explanability
				Plausibility			Faithfulness
	Acc. ↑	Macro F1 ↑	AUROC ↑	IOU F1 ↑	Token F1 ↑	AUPRC ↑	Comp. ↑	Suff. ↓
Baseline	0.7425	0.6883	0.8264	0.0577	0.2799	0.3884	0.1789	0.1363
Proposed Method	0.7492	0.6983	0.8450	0.0429	0.3232	0.5667	−0.0655	0.0546

The symbol ↑ denotes improvement in metric values when there is an increase, whereas the symbol ↓ signifies enhancement when there is a decrease in metric values.

presents the outcomes derived from both the the baseline and proposed method. The proposed method was employed on the newly constructed proposed dataset, while the baseline approach was implemented on the HateXplain dataset [13] section of the proposed dataset. The performance metrics show an increase of 0.01 in the macro F1 score, 0.186 in the AUROC, and a slight increase in accuracy. In the explainability metrics, a significant increase was observed in two metrics (token F1 and AUPRC), from plausibility and on sufficiency from faithfulness. There is an increase of 0.04 in the token F1 and 0.17 in AUPRC. For sufficiency, it is understood that a lower value signifies better sufficiency, and a decrease of 0.08 was observed.

6. Discussion

The main results are reported in Table 2.

Performance: The results indicate a gradual enhancement in the performance metrics, with the macro F1 score exhibiting a marginal increment of 0.01 alongside improvements in other metrics when utilizing the proposed method in combination with sarcasm rationale. Figure 7 provides a detailed breakdown of the performance results via a confusion matrix.

Performance metrics from the confusion matrix are illustrated in Figure 7a,b for both the baseline and the proposed method using True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN) for each class.

• Baseline Performance Metrics Calculation

  Precision:

$$\begin{array}{cc}\hfill & \mathrm{For}\mathrm{hatespeech}:P=0.796\hfill \\ \hfill & \mathrm{For}\mathrm{offensive}:P=0.533\hfill \\ \hfill & \mathrm{For}\mathrm{normal}:P=0.811\hfill \end{array}$$

  Recall:

$$\begin{array}{cc}\hfill & \mathrm{For}\mathrm{hatespeech}:R=0.796\hfill \\ \hfill & \mathrm{For}\mathrm{offensive}:R=0.480\hfill \\ \hfill & \mathrm{For}\mathrm{normal}:R=0.795\hfill \end{array}$$

  F1-Score:

$$\begin{array}{cc}\hfill & \mathrm{For}\mathrm{hatespeech}:F1=0.796\hfill \\ \hfill & \mathrm{For}\mathrm{offensive}:F1=0.506\hfill \\ \hfill & \mathrm{For}\mathrm{normal}F1=0.803\hfill \end{array}$$

• Proposed Method Performance Metrics Calculation

  Precision:

$$\begin{array}{cc}\hfill & \mathrm{For}\mathrm{hatespeech}:P=0.815\hfill \\ \hfill & \mathrm{For}\mathrm{offensive}:P=0.520\hfill \\ \hfill & \mathrm{For}\mathrm{normal}:P=0.814\hfill \end{array}$$

  Recall:

$$\begin{array}{cc}\hfill & \mathrm{For}\mathrm{hatespeech}:R=0.778\hfill \\ \hfill & \mathrm{For}\mathrm{offensive}:R=0.500\hfill \\ \hfill & \mathrm{For}\mathrm{normal}:R=0.781\hfill \end{array}$$

  F1-Score:

$$\begin{array}{cc}\hfill & \mathrm{For}\mathrm{hatespeech}:F1=0.796\hfill \\ \hfill & \mathrm{For}\mathrm{offensive}:F1=0.510\hfill \\ \hfill & \mathrm{For}\mathrm{normal}:F1=0.797\hfill \end{array}$$

The performance metrics from both the baseline and proposed method revealed a comparison in their effectiveness. The proposed method demonstrated improvements in precision for the hate speech and normal speech categories compared to the baseline. However, there was a slight decrease in precision for offensive language classification. The recall for all categories remained relatively similar between the baseline and proposed methods. Overall, the F1-scores for the hate speech and normal speech categories remained consistent, while there was a slight increase in the F1-score for offensive language classification with the proposed method.

Explainability: A significant improvement was evident in three out of five explainability metrics, encompassing both plausibility and faithfulness. Within the realm of explainability metrics, notable advancements were observed in two metrics (token F1 and AUPRC) under plausibility and sufficiency under faithfulness. The HateXplain [13] paper demonstrates that using rationale compared to not using it, alone, resulted in an increase in only one out of five explainability metrics for the BERT model, specifically in comprehensiveness within faithfulness. In contrast, our dataset and proposed method enhanced three metrics. Overall, it is evident that relying solely on a model’s performance metric is inadequate. Models with slightly lower performance metrics but significantly higher scores for plausibility and faithfulness may be preferable depending on the specific task. Therefore, the presence of hate speech concealed within sarcasm could augment a model’s ability to provide more interpretable outcomes.

LIME: LIME [25], or Local Interpretable Model-agnostic Explanations, enhances the interpretability of complex machine learning models, particularly for text data. It operates by generating new samples around the instance being explained through perturbation—slightly modifying the original data point to create a dataset of “synthetic” samples that are similar to the original instance. LIME then monitors how the predictions change with these modifications. Subsequently, LIME constructs a simpler, interpretable model that approximates the original model’s behavior within a local region around the perturbed instance. This model identifies and highlights the words or phrases that significantly influence the model’s decisions. By demonstrating how predictions shift when key text elements—termed rationales—are included or excluded, LIME helps in assessing the model’s robustness and its reliance on specific features for decision-making. This methodology is crucial for applications requiring high interpretability, such as content moderation or hate speech detection, where understanding the basis of the model’s decisions is essential for user trust and the validation of the model’s effectiveness.

Figure 8 shows how LIME selected tokens for both baseline and proposed method along with the prediction probabilities for the toxic category, as evidenced by an example of hate speech data. For both the baseline and proposed method, the token selection was diverse, which led to distinct label predictions for each method. In Figure 8, the classification approaches of the baseline and proposed methods are compared using the BERT model, as illustrated by LIME. Figure 8a shows the baseline method utilizing BERT. Here, BERT marked tokens via LIME for a text example, demonstrating how each token influenced the classification, with the offensive category predicted with the highest probability. This highlights the baseline’s sensitivity to specific trigger words, classifying them typically as offensive without deeper contextual analysis.

Conversely, Figure 8b depicts the proposed method, also applying BERT with LIME to mark tokens. This method, however, assigned the highest prediction probability to the hate speech category, aligning more closely with the contextual use of the terms within the dialogue, hence providing a more accurate representation of the original label, which was hate speech. This contrast between the two figures illustrates the proposed method’s enhanced ability to contextualize and accurately classify complex expressions of hate speech, leveraging the nuanced understanding capabilities of the BERT model.

The details of Figure 8 is explained in

Table 3. Example of the rationales predicted by baseline and proposed method compared to human annotators.

Model	Text	Label
HateXplain Annotator	user he is infected with jihadi virus he will spread it to others	HS
Our Annotator	user he is infected with jihadi virus he will spread it to others	SARC
Baseline	user he is infected with jihadi virus he will spread it to others	OF
Proposed Method	user he is infected with jihadi virus he will spread it to others	HS

The orange highlight marks what the HateXplain [13] annotators found important for the prediction. The green highlight marks what our annotators found important for the prediction. The underlined words/tokens were identified by the model as important using LIME [25], and the lime highlight marks the highest probability tokens from this set for both the baseline and proposed method. (OF—offensive speech; HS—hatespeech; SARC—sarcasm).

. The first two rows depict the “rationale” chosen by the annotators of the HateXplain paper [13] and our annotators, which they deemed essential for the classification task. The last two rows showcase significant tokens using LIME for the BERT model, encompassing both the baseline and proposed method. From Table 3, the details can be observed for the baseline ground truth attention calculation; therein, even though the BERT model marked the same “rationale” as the HateXplain human annotators, it assigned the wrong label (offensive speech—OF). But, for the proposed ground truth attention calculation, the BERT model marked almost the same “rationale” as that of our annotators and assigned the right label (hate speech—HS).

7. Limitations

Several limitations are inherent in our work. Firstly, the absence of external context poses a significant challenge. Relying solely on the published dataset resulted in a lack of contextual information for each post, thereby complicating the annotation process for annotators. Providing additional context could facilitate a more accurate assignment of posts as sarcastic or non-sarcastic. Additionally, our work is limited to the English language, neglecting potential insights from other languages.

8. Conclusions and Future Work

This study introduces a novel dataset derived from the established HateXplain dataset, comprising 3199 posts extracted from Gab and Twitter. In the original HateXplain dataset, each post is annotated with labels such as normal, hate, or offensive, along with target communities (e.g., black, Muslim, LGBTQ+) and rationales marked by annotators. The data from HateXplain were re-labeled based on sarcasm by our annotators, with rationales selected based on their identification of sarcastic content. Additionally, a new methodology is proposed for calculating ground truth attention when the data exhibit both toxic (hate/offensive) content and sarcasm, aiming to enhance the decision-making process of models. This approach was tested on the BERT model, which incorporates sarcasm along with hate/offensive data to improve hate speech detection, particularly for toxic comments disguised within sarcasm. The results demonstrate that utilizing sarcasm to identify concealed toxic content led to a slight improvement in the performance metrics, while also providing plausible and faithful rationales that enhanced the model’s decision-making capabilities.

Looking ahead, the plan is to explore the application of other Large Language Models (LLMs) to our methodology. Given the complexity and resource requirements of LLMs, this extension was not feasible within the current study’s time frame. Future work will focus on comparing the efficacy and generalizability of our approach using a range of LLMs. This step is essential for verifying our model’s robustness across various architectures and settings, providing deeper insights into its adaptability and performance. We also aim to incorporate context information into data annotation and utilize it in our model training process.