MaLang: A Decentralized Deep Learning Approach for Detecting Abusive Textual Content

Abstract

Cyberbullying is a growing and significant problem in today’s workplace. Existing automated cyberbullying detection solutions rely on machine learning and deep learning techniques. It is proven that the deep learning-based approaches produce better accuracy for text-based classification than other existing approaches. A novel decentralized deep learning approach called MaLang is developed to detect abusive textual content. MaLang is deployed at two levels in a network: (1) the System Level and (2) the Cloud Level, to tackle the usage of toxic or abusive content on any messaging application within a company’s networks. The system-level module consists of a simple deep learning model called CASE that reads the user’s messaging data and classifies them into abusive and non-abusive categories, without sending any raw or readable data to the cloud. Identified abusive messages are sent to the cloud module with a unique identifier to keep user profiles hidden. The cloud module, called KIPP, utilizes deep learning to determine the probability of a message containing different categories of toxic content, such as: ‘Toxic’, ‘Insult’, ‘Threat’, or ‘Hate Speech’. MaLang achieves a 98.2% classification accuracy that outperforms other current cyberbullying detection systems.

1. Introduction

While cyberbullying is a widely recognized problem for adolescents [1], racism, misogyny, homophobia, and other causes of cyberbullying do not just disappear with the granting of a diploma or degree. These attitudes and the large percentage of type-A personalities present in STEM-related businesses [2] lay a fertile ground for cyberbullying of adults. The ever-increasing prevalence and importance of electronic communication mechanisms in organizations today including social networks, along with the ever-expanding global cyber-marketplace, are the foundations necessary to facilitate cyberbullying in the workplace [3]. Cyberbullying in the workplace is typically conducted using social networks and social media, email, and text (SMS) messages, which allow for cyberbullies to often remain anonymous [4]. Additionally, the recent migration of employees to online workplace environments due to the COVID-19 pandemic and consequent reliance on social media networks and the Internet for conducting everyday work may further exacerbate the problem [5,6].

Although the subjects of child and adolescent cyberbullying are well-documented in academic research [7,8,9,10,11,12], workplace cyberbullying research is still in its infancy and there is a need for further research on the topic of workplace cyberbullying [3]. There is a need for automatic intelligent detection of cyberbullying in organizational contexts [13].

Cyberbullying in the workplace is an ongoing and significant problem. Studies conducted in Finland found from 12.6% to 17.4% of employees felt they had experienced cyberbullying on a monthly basis [4]. From 14% to 21% of higher education faculty have been documented to suffer from cyberbullying [14]. Cyberbullying can last for an extended period of time [15]. Various studies have shown how cyberbullying increases employee stress and job dissatisfaction [16,17]. These effects of workplace cyberbullying in turn may cause a decrease in worker productivity and if not resolved after being reported to human resources (HR), may result in litigation.

In this paper, a decentralized deep learning approach called Malicious Language Detector (MaLang) is proposed. While cyberbullying may include text, images, photographs, video, and audio content [18], the MaLang system only analyses textual content in user devices. The decentralized methodology facilitates keeping an employee’s data private and secure. MaLang is simulated in a workplace environment to demonstrate the applicability of intelligent automated detection of cyberbullying messages for improving workplace safety and well-being. Previous machine learning and deep learning techniques are re-combined in a novel way to improve the overall accuracy of hostile content in an organization’s electronic messaging systems. The two-part MaLang method helps to minimize deployed size, improve employee privacy, and increase hostile message detection.

2. Related Research Work

Around the world, more than one billion people use messaging platforms, such as WhatsApp or Facebook Messenger, and more than 30 billion messages are exchanged daily [19]. However, not all of these messages are beneficial to individuals or organizations as a lot of toxic textual content is shared in public, private, and workplace spaces. Abusive textual content includes words or phrases that either insult or show hate speech toward a person or community, or that pose a threat to a person, or project obscene opinions. Various applications have been deployed to control and minimize the sharing of such content [20].

Earlier information retrieval (IR) and natural language processing (NLP) methods are applied for resolving the text-related classification problems such as sentiment analysis [21]. IR and NLP have been previously applied for the detection of cyberbullying text-oriented messages [11]. Various other cyberbullying detection techniques have been previously developed, with most being dependent on textual and user features [7]. A simple program was developed by making use of a dictionary of keywords to perform cyberbullying detection over online chats [22], and performance was improved by collecting a much larger sample of labeled data [23].

Sensitive topics in the offensive and cyberbullying-oriented comments of YouTube such as racism or sexual aggression are considered in building the detection model [24]. The roles of bullies and other participants are studied and segregated into the groups of accuser, bully, and victim [21]. Feature engineering is a commonly used method for detecting cyberbullying from text-related representations. A twitter-based model extracts features from tweets and classifies them into the different groups mentioned in [21]. In addition, to generate additional features, the domain knowledge of data is also used for improving the performance of the classifiers. The performance of Support Vector Machines (SVM) is improved for the detection of cyberbullying as a result of adding features such as the occurrence and frequency of bad language, number of pronouns, length of comments, use of capital letters, and emoticons [20]. Similarly, pronoun counts are also considered a feature with the representation of n-grams and skip-grams [25]. The usage and application of soft computing for detecting cyberbullying are reviewed by [26].

A combination of textual and social network features is utilized for cyberbullying detection in [27]. Social network analysis (SNA) features are represented as a graph with the combination of nodes and edges used for discriminating the activities of cyberbullying. The usage of textual features along with the term frequency–inverse document frequency method improved the detection of cyberbullying. SNA has been further extended for the labeled features such as racism, sexual, and intelligence [28,29]. The classification of more indirect and hidden labels related to bullying can improve the accuracy of cyberbullying detection.

Zhao et al. [30] presented an insult-related word embedding method for extracting the features of bullies. They also used the additional features that were extracted from a dictionary of general words used in social networks [31]. However, their method works well only on a balanced dataset and suffers in real time with imbalanced datasets [32].

The issue of real time deployment and huge data processing is addressed by various researchers who used neural network-based models [33]. A simple convolutional neural network (CNN) model is utilized over pre-trained word vectors to perform classification based on the number of tokens presents in the target text [34,35]. The CNN model was used on 50 million tweets for detecting cyberbullying [35,36]. A pronunciation-based CNN is proposed to tackle the challenges of cyberbullying such as misspellings in the messages and posts in social media [37]. Due to their ability to identify spoken language, CNNs are frequently used to build models for detecting bullying tweets [8,38,39,40,41,42]. Abusive language detection was performed by training Bidirectional Encoder Representations from Transformers (BERT) on a Reddit Abusive Language Dataset. Hate speech detection in different languages was achieved using various hybrid deep learning models such as Bidirectional Encoder Representations from Transformers (BERT) [41], Graph Neural Network Model [43], BERT-CNN [44,45], Gated Recurrent Units (GRU) [46], CNN-GRU and mBERT, CNN-LSTM (Long Short-Term Memory) [10], and Heterogeneous Neural Interaction Networks [47].

Machine learning ensemble models for cyberbullying detection have also been proposed [12,48], though more infrequently. The deep learning architectures used for experimentation include the specialization in regularization, convolution, and bi-directional networks in which the authors defined various metrics for evaluation. It is also observed that the LSTM models are provided with more parameters for training than GRU, which performs with the highest median accuracy with minimal metrics. The major issues with the hybrid models are that the experimental results and evaluations are bound to a single dataset [10,28]. Automatic improper conversation (such as disgusting language and offensive chats) extraction is complicated and time consuming [32,49,50,51].

Moreover, the processing of huge knowledge bases on resource-constrained platforms such as on-boards and mobile devices is very difficult. Increasing the number features increases the difficulties in the process of feature extraction and selection. No current research uses a word’s meaning or semantics for cyberbullying detection [52,53,54]. Transformers, mentioned in a few papers stated above, often require high computing resources while training, especially at regular intervals. These models combined with different types of recurrent neural networks produce predictions with decent accuracy and precision. A novel approach is introduced to detect and restrict abusive textual content being exchanged in a network with better accuracy and precision. MaLang solves the challenge of detecting abusive text at both the edge and cloud levels, without consuming high computation resources and producing optimal predictions, while dealing with small influx of data at regular intervals.

Table 1. State-of-the-art research on cyber bullying detection.

Approach	Reference	Methods and Models	Accuracy
Text Analytics	[22]	Keyword matching	58.6%
Natural Language Processing	[7]	Sentiment Analysis	95%
	[53]		N/A
Machine Learning (ML)	[23]	C4.5 decision tree	78.5%
	[20]	SVM	77%
	[21]		77%
	[30]		78%
	[48]	Ensembles of ML classifiers	90.57%
	[12]		91%
Deep Learning	[35]	CNN	81%
	[33]		84%
	[34]		92.8%
	[50]		96%
	[53]	Deep Neural Network	73.2%
	[40]		86.6%
	[8]		93.97%
	[39]	LSTM	92.7%
	[38]		93.97%
	[55]	Bi-directional LSTM	52.6%
	[42]	Tree and LSTM	90.3%
	[45]	Gradient Boosting BERT and LASER	62%
	[56]	CNN-GRU and mBERT	95%
	[10]	CNN-LSTM and CNN-BiLSTM	92%
	[46]	LSTM and GRU	98%

provides a summary of the various state-of-the-art approaches and the accuracy of these models for detecting bullying language.

Major Contributions of MaLang

• MaLang uses two-stage abuse detection and restriction analysis as opposed to more traditional single-stage detection processing.
• At the edge level, MaLang uses Bi-LSTM-based CASE to detect toxic text, and at the Cloud level, KIPP utilizes a combination of Bi-LSTM- and Bi-GRU-based models for Entity-based recognition. This combination of machine learning approaches has not been used previously.
• MaLang proposes an architecture that keeps data private at the user level and the cloud level, without disclosing or transmitting the profile of the user involved in the system. Thus, data privacy and confidentiality are a central focus of the MaLang system design.
• Unlike simple RNN approaches, MaLang uses an actively monitored pipeline to detect and alert users and administrators of possible toxic textual content being exchanged.

3. Materials and Methods

To overcome issues encountered in the previous related work described above, this paper proposes a novel approach called Malang. Malang is a decentralized learning approach to detect abusive textual content and initiate managerial response to detected cyberbullying events. MaLang leverages the use of Bi-LSTM with a Max Pooling Layer for abusive content classification in system-level modules and the use of Bi-LSTM and Bi-GRU with a combination of Average and Max Pooling layers. The cloud-level modules are interconnected for different departments in the workplace when deployed in medium to very large organizations. When a tagged entity is received in one module, a copy is sent to other cloud modules in the network to check if a similar entity is present. In the case of a match, it is concluded that two or more people were involved in the sharing, and an alert is sent to the Human Resource representative and the cyber protocol enforcement department.

As shown in Figure 1, the automated business-level cyberbully detection framework is divided into two modules: System Level and Cloud Level. The System-level module consists of the deep learning model called ‘CASE’ that classifies abusive content from non-abusive content on the user’s device. The collected abusive content is transmitted to the cloud to detect the intensity of the abusive content. The Cloud-level module consists of a cloud-based deep learning model called ‘KIPP’ that classifies the level of Toxic Content of each sentence into one of four classes: ‘Toxic’, ‘Threat’, ‘Insult’, or ‘Hate Speech’. A sentence can contain multiple classes of Toxic content as presented in

Table 2. Toxic content and different classes.

Sentence	Toxic	Threat	Insult	Hate Speech
You are gay or antisemmitian?	Yes	No	Yes	Yes
C*******ER BEFORE YOU PISS AROUND ON MY WORK	Yes	Yes	Yes	Yes
Bye! Don’t look, come or think of comming back! Tosser.	No	No	Yes	No

, which shows examples of partial real-world organizational messages and their corresponding classifications (the second message has been modified for viewing purposes).

3.1. System-Level Module of MaLang

Figure 2 depicts the system module architecture of MaLang. The application is deployed as an extension to most of the messaging and communication applications. The application collects communication data such as conversations from messaging applications such as WhatsApp or Messenger as a text file. Every sentence is in raw textual format. To convert it into usable form, every sentence is tokenized into entities. The tokenized entities are cropped to uniform length to be converted into input vectors. These entities are represented as arrays of vectors using one-hot encoding. One-hot encoding simply converts categorical variables into unique features or numerical values that can be easily fed to a deep learning model. The obtained vectors are passed through the deep learning model CASE. CASE is built with a sequence of Bi-directional LSTM and fully connected neural network layers that classify the entities into Abusive and Non-abusive. The categorized Abusive entities are then pushed to the cloud to detect levels of toxicity. Subsequently, all the classified entities are used as a training data set for continuous learning to improve the model. Every entity received by the application is used to improve the model.

3.2. Cloud-Level Module of MaLang

Figure 3 depicts the Cloud module architecture of MaLang. The detected abusive entities containing the array of tokens are tagged with the user’s identification number and transmitted to the cloud module. The received tokens are converted into input vectors of 32 indices by mapping the word to the id included in the pre-trained GloVe vectors. This creates an embedding matrix that is passed through the cloud-based deep learning model ‘KIPP’. KIPP consists of a sequence of Bi-directional LSTM, Bidirectional GRU, pooling layers, and fully connected layers. The KIPP system classifies the level and type of abusive content present in the sentence as shown in Figure 4.

3.3. Recurrent Neural Networks

Many text classification operations have been performed using recurrent neural networks (RNN) [57,58]. They enable the modeling of sequence-based tasks; however, they are known to suffer from the problem of vanishing gradients that diminishes the learning of long sequences [59]. To avoid the problem of a vanishing gradient, an ensemble of Bi-directional LSTM and Bi-directional GRU are used.

Bidirectional LSTM or Bi-LSTM allows the neural networks to preserve both forward and backward information received from input about the sequence at every step with respect to time, making it time-dependent. Inputs are used in two ways, past to future and future to past, allowing the Bi-LSTM to better understand the context of the text. Figure 5 shows the design of a Bi-directional LSTM layer along with the operations happening in each cell. As seen in the figure, LSTM has three gates, input, output, and forget gates. Unlike RNN, the cell state derivative of cell state vectors prevent gradient vanishing since the long-term dependencies are encoded in them.

The operations that happen in LSTM are:

Previous state:

$$f_t=\sigma \left(W_f·\left[h_{t-1},x_t\right]+b_f\right)\dots$$

(1)

$$i_t=\sigma \left(W_i·\left[h_{t-1},x_t\right]+b_i\right)\dots$$

(2)

where $i_t$ input/update the gate’s activation vector.

$${\tilde{C}}_t=tanh\tanh \left(W_c·\left[h_{t-1},x_t\right]+b_c\right)\dots$$

(3)

$$C_t=f_t\ast C_{t-1}+i_t\ast {\tilde{C}}_t\dots$$

(4)

$$o_t=\sigma \left(W_0\left[h_{t-1},x_t\right]+b_0\right)\dots$$

(5)

$$h_t=o_t\ast tanh\tanh \left(C_t\right)\dots$$

(6)

where W is the weight matrix, b is the bias vector parameter, $o_t$ is the activation vector of output vector,${\tilde{C}}_t$ is the cell input activation vector, $C_t$ is the cell state vector, and $h_t$ is the hidden state vector [60].

Similar to BiLSTM, bidirectional GRUs predict the current state of a sequence using the information preserved from both the previous and successive time steps. Unlike LSTMs, GRUs are equipped with only two gates: an update gate and a reset gate. Information is transferred using the hidden state, and no memory unit is used. GRUs are more efficient and train faster on less amounts of data giving results on par with those of LSTMs. Figure 6 shows the architecture of the Bidirectional GRU or Bi-GRU and the operations happening in every cell.

The operations that happen inside a GRU cell are:

Reset Gate:

$$R_t=\sigma \left(X_t{W}_{xr}+H_{t-1}{W}_{hr}+b_r\right)\dots$$

(7)

Update Gate:

$${\mathit{z}}_{\mathit{t}}=\mathbf{\sigma }\left({\mathit{X}}_{\mathit{t}}{\mathit{W}}_{\mathit{x}\mathit{z}}+{\mathit{H}}_{\mathit{t}-\mathbf{1}}{\mathit{w}}_{\mathit{h}\mathit{z}}+{\mathit{b}}_{\mathit{z}}\right)\dots$$

(8)

where ${\mathit{W}}_{\mathit{x}\mathit{r}}$, ${\mathit{w}}_{\mathit{x}\mathit{z}}$ and ${\mathit{W}}_{\mathit{h}\mathit{r}}$, ${\mathit{w}}_{\mathit{h}\mathit{z}}$ are weight parameters, whereas b_r and b_z are biases.

Hidden State:

$${\tilde{H}}_t=\tanh \left(x_t{W}_{xh}+(R_t \odot H_{t-1}{W}_{hh})+b_h\right)\dots$$

(9)

Update State:

$$H_t=Z_t\odot H_{t-1}+\left(1-z_t\right)\odot {\tilde{H}}_t\dots$$

(10)

The Reset gates capture short-term sequences, whereas the update gates capture long-term sequences.

3.4. Architecture of the System-Level Module

3.4.1. Data Collection Unit

The application is deployed as an extension to messaging applications, which allows it to read and collect user data. For every 3 h, the user’s communication data are collected and stored as a simple text file with each sentence separated by a newline character. Non-alphabetic characters and HTML content are removed from each sentence of the collected data, thus obtaining clean sentences. The cleaned sentences are tokenized into an array of tokens or words and one-hot encoded into binary vectors. These vectors act as the input for the deep learning model CASE. CASE processes the input vectors and determines whether each sentence consists of abusive content. After classifying all the entities, they are initialized as a training dataset for the deep learning model to improve its classification metrics and update its weight matrix. Once detected, the respective abusive sentences are labelled with a corresponding category. If CASE predicts the presence of an abusive class of text with a probability of more than 0.8, it is fed into the dataset for training. The 0.8 cutoff for the collection of future training data is to ensure that clean data that have been correctly identified are used for training purposes. After the CASE model is trained, these entities are completely deleted from the MaLang application, making space for the next iteration of collected textual data. The entire process of cleaning, tokenizing, and training data with CASE takes an average time of 15 min.

3.4.2. Deep Learning Model for the Classification of Abusive Content

To identify abusive textual content, the Hate-speech dataset of [61] was considered. The dataset contains more than 28,000 sentences that were categorized into Normal, Offensive, and Identity-hate. The dataset was modified into two categories, i.e., abusive and non-abusive content. All the sentences were converted into lowercase and cleaned by removing non-alphabetic characters and the HTML content present in them; 25,000 sentences were used as the training set, and 3000 sentences were used for the testing set.

All the sentences were tokenized using the NLTK package [62] and were cropped to uniform length. These were converted into an array of words for each sentence called entities. Simultaneously, every word from each entity was one-hot encoded, thus obtaining a matrix representing categorical variables as binary vectors. The vectors act as an input to the embedding layer in the CASE deep learning model.

Figure 7 shows the architecture of the CASE deep learning model. The system-level deep learning model consists of one embedding layer with an input dimension of 21,000 and an output dimension of 300. The input length sequence is 53. The embedding layer is followed by a Bidirectional LSTM layer with 128 cells and a one-dimensional global Max Pooling layer. These are connected to two sets of a fully connected layer and Dropout layer. Finally, a SoftMax function is used as an output layer to predict the probabilities of either of the classes. The model is used to determine whether the sentences collected contain any abusive content. The usage of Bi-LSTM over a traditional RNN such as LSTM provides a solution to two traditional problems associated with abusive text detection:

• When dealing with abusive sentences, many phrases might not contain a foul word; however, the sentence might be abusive or offensive when its meaning is considered. Bi-LSTM was chosen as it accesses past information from past words and the future information in the reverse direction, therefore attempting to understand the context and overall meaning of the sentence.
• Because CASE is deployed locally on the user’s mobile device, it should only use minimum storage space and processing power. The current implementation requires 8 megabytes (MB) of storage space. Although transformers fulfill the purpose of understanding the context, they are known to occupy a huge amount of space and computation resources, which is practically infeasible for this purpose. The transformer type of deep learning used in prior research can use up to 200 MB of storage space.

The application generates a user identification tag. The tag consists of a randomly generated 5-digit text that consists of both alpha and numeric characters. The tag is unique to only one user. After the entities are classified into abusive and non-abusive, the abusive entities are linked with the user identification tag and sent to the cloud-level module. Similarly, all such entities from multiple devices or users are sent to the cloud-level module along with the tagged user’s unique identification number.

3.5. Architecture of the Cloud-Level Module

3.5.1. Data Collection Unit

The cloud-level module receives the tagged entities. These entities are received every 3 h from multiple users. All the entities that belong to a single tag are extracted into an array of tokens of uniform length for each sentence. These arrays of tokens are converted into input vectors of 32 indices, making them suitable as input to the embedding layer in the KIPP deep learning model.

3.5.2. Deep Learning Model for Identification of Toxicity of Abusive Content

The Jigsaw/Google Toxic Comment Dataset [61] is used to classify the toxicity in the text message content uploaded to KIPP into four classes: ‘Toxic’, ‘Threat’, ‘Insult’, and ‘Hate Speech’. The dataset contains labelled examples of Wikipedia comments that were manually rated by human raters based on the comment’s toxicity in six classes, which are ‘Toxic’, ‘Severely Toxic’, ‘Threat’, ‘Obscene’, ‘Insult’, and ‘Hate-speech’. Although the dataset consisted of six classes, classes such as ‘Toxic’ and ‘severely toxic’ were merged since severely toxic is a subset of Toxic text, and classifying both as a single set seemed conceptually convincing. Similarly, the class ‘obscene’ was merged with ‘Insult’.

The dataset consists of more than 157,000 sentences. To separate the dataset, a 60–20–20 spilt was considered for training–validation–testing, respectively. All the samples or sentences were cleaned by removing non-alphabetic characters and html content. The cleaned samples are tokenized using the Keras tokenizer, obtaining an array of tokens of unique words for each sentence. Subsequently, every token is cropped to uniform length in each array, obtaining an array of uniform tokens for each sample called entities.

The obtained entities are converted into input vectors of 32 indices. This is done by mapping tokens from entities to their corresponding vector representations present in pre-trained GloVe vectors [63]. Word embedding for each token is performed using the 300-dimension pre-trained global vector representation of word–word occurrences, producing 840 billion tokens from the Massive Common Crawl Corpus with a vocabulary size of 2.2 million.

Some abusive words may not be present in MaLang’s (or KIPP’s) recognized vocabulary. When any missing word is encountered, it is given a value of zero, or if any token belongs to the substring of the main word in the vocabulary, the token is mapped to the root or main string vector. Mapping the token from entities to pre-trained GloVe vectors results in input vectors that are passed through the embedding layer of the KIPP deep learning model.

Figure 8 shows the architecture of the deep learning model KIPP. The cloud-level model consists of an embedding layer followed by a bidirectional LSTM layer and a bidirectional GRU layer with 128 cells in each layer. These layers are connected to a combination of a one-dimensional Average Pooling Layer and a Max Pooling Layer. The output layer consists of a sigmoid function to represent the output in the form of probabilities to detect the toxicity of the content in four classes. The use of Bidirectional GRUs and Bidirectional LSTM is solely for the purpose of handling single labelled entities sent to KIPP from mobile devices. KIPP receives a very large amount of unstructured data, which GRUs are known to handle and produce better results without monopolizing computational resources.

KIPP predicts the level of Toxicity in each entity and returns the probability of presence of the kind of toxic content present as shown in Figure 4. After all the entities are analyzed, a report is generated stating the average toxicity score of the user’s content received at the cloud level of MaLang every 3 h.

3.6. Methodology to Calculate the Toxic Score of a User’s Content

After predicting the probability of the presence of different types of toxic content in each entity, the entity toxic score (ET) is obtained by calculating the mean of the score of the different types of classes as shown below:

$$U{T}_k=\frac{{{\displaystyle \sum }}_{i=1}^{n}E{T}_i}{n} \mathrm{where} E{T}_i=\frac{P\left(T\right)+P\left(Th\right)+P\left(I\right)+P\left(H\right)}{4}$$

(11)

where n is the number of text entities for user k, P(T) is the probability of the presence of Toxic content, P(Th) is the probability of the presence of Threat content, P(I) is the probability of the presence of Insulting content, and P(H) is the probability of hate speech content, in each entity. Similarly, an entity toxic score is calculated for all entities that belong to one user identification number. The highest entity scores are considered, and their average is calculated to obtain a User Toxic Score (UT).

Similarly, a User Toxic Score is calculated for all users in the network. A threshold of permitted User Toxic Score less than or equal to 0.4 is considered relatively benign and will not result in any user or administrator notification. When a user’s Toxic Score exceeds 0.4, the user is prompted with a warning. In addition, a flag is raised in the area’s cyber security patrol. Any abusive textual sentence, on average, scores an Entity Score of 0.6–0.8 when at least two classes of abusive text are present. The 0.4 value used to indicate toxic content is meant to allow for a certain level of profane language that may be used humorously or to relieve stress and is considered acceptable by the organization. The organization may elect to change this number, lowering it to make text monitoring more restrictive or raising the value in team cultures where a certain level of profanity is accepted if not expected.

The aim of MaLang is not only to detect such content but also to restrict and curb its use in the network among the users. Therefore, if the system detects a user crossing the threshold of 0.4 more than six times, a notification along with the report is sent to the nearby cyber forensic analyst to take any necessary action. The threshold of 6 is arbitrary and is meant to allow some level of inappropriate language, similar to the threshold value of 0.4 used to detect toxic text. Again, this value can be lowered to allow the organization to maintain much tighter control over potential abusive or toxic messaging and resulting cyberbullying, with a value of 0 being the most restrictive. The threshold counter resets every month. If MaLang is deployed in an organization, exchange of abusive content between two or more employees present in the network can easily be detected.

3.7. Methodology to Identify Users Involved in the Exchange of Identical Abusive Content

In an organization, every department or every team is connected to a dedicated cloud module with KIPP. As shown in Figure 9, various cloud modules are interconnected and are given access to share and compare the incoming abusive entities coming from users’ devices. When the entities tagged along with the user’s identification number are transmitted to the cloud module, the extracted arrays of tokens are compared with different arrays of tokens of other user’s entities in the same cloud module or connected cloud modules. When an exact match is found wherein an array of tokens exactly matches another entity’s array of tokens, it is concluded that there was an exchange of abusive content between the devices in the network. The toxicity is analyzed using KIPP, and a report is generated stating the level of toxicity in each sentence. In such cases, the organization’s Human Resource department is notified along with the involved user’s identification number to take necessary action. The developed methodology can potentially reduce workplace harassment.

3.8. Methodology to Update the Weight Matrix of CASE in Various Devices over the Network

After CASE has trained itself using user’s data for more than five iterations in the devices, the model is evaluated and results including accuracy and area under the receiver operating characteristics curve (AUROC) score are produced. These results along with CASE’s weight matrix are transmitted to the cloud module wherein a comparison is performed between the accuracies and AUROC scores of CASE received from different devices. The weight matrix that projects the highest accuracy and AUROC score is selected and sent to all the devices in the network to update their respective CASE’s weight matrix. Another set of two iterations of training is performed to compare CASE’s performance with the updated and previous weight matrix. In the case where no improvement is observed, the existing matrix is used as usual and the updated matrix is permanently deleted. This approach helps CASE learn the presence of new abusive content in one device and detect the same in other devices as well. Additionally, this strategy helps prevent the CASE learning models from becoming trapped in a local minima.

3.9. Metrics for Evaluating Deep Learning Models

The evaluation of the deep learning models CASE and KIPP is performed using metrics such as Precision, Recall, Accuracy, Loss, and the AUROC score and curve. To implement these metrics, the following basic concepts are defined:

True Positive (TP): Textual Content that belongs to the toxicity class and is predicted as belonging to the same class

True Negative (TN): Textual content that does not belong to the toxicity class and is predicted as not belonging to the same class.

False Positive (FP): Textual Content that does not belong to the toxicity class and is predicted as not belonging to the same class

False Negative (FN): Textual Content that belongs to the toxicity class and is predicted as not belonging to the same class.

The above defined concepts are used to compute the following metrics:

1.

Precision (or positive predictive value): The ability of the model to identify the possible textual content from the input.

$$\mathrm{P}=\left(\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}\right)$$

(12)

2.

Recall (or sensitivity or true positive rate (TPR)): The ability of the model to identify all the relevant abusive textual content from the predicted possible textual content.

$$\mathrm{R}=\left(\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\right)$$

(13)

3.

Accuracy: The ratio of correct predictions made by the model to the total number of predictions by the model.

$$\mathsf{\alpha }=\left(\frac{\mathrm{TP} + \mathrm{TN}}{\mathrm{TP} +\mathrm{TN}+\mathrm{FN}+\mathrm{FP}}\right)$$

(14)

4.

Selectivity (or specificity or true negative rate): The ability of the model to identify all non-abusive textual content from the predicted non-abusive content.

$$\mathrm{S}=\left(\frac{ \mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}\right)$$

5.

AUROC: measures the ability of the model to distinguish between classes whereas the ROC curve is a probability curve where TPR (R) is plotted against the false positive rate (1 − S).

AUROC determines the accuracy of the model being able to predict the true values as true and false values as false. As the AUROC increases, the MaLang model becomes better at recognizing different types of toxic content.

4. Results

4.1. Validation of the System-Level Deep Learning Module CASE

The rate of learning of CASE was 0.001. The optimizer and loss function used were Adam and a categorical cross-entropy function, respectively. The model achieved an average accuracy of 92.2%. The average precision and recall of CASE were 92% and 91%, respectively. The average AUROC score of CASE was 0.977. The pattern of maintaining accuracy and measuring loss in classifying the textual content based on the defined class is shown in Figure 10.

4.2. Validation of the Cloud-Level Deep Learning Module KIPP

Figure 11 shows the ROC curves that depict the performance of the KIPP deep learning model in predicting abusive text categorization values correctly. The rate of learning of KIPP was 0.001. The optimizer and loss function used were Adaptive Movement Estimation (Adam) and Binary Cross-entropy function, respectively. The KIPP model achieved an average accuracy of 98.21%. The average precision and recall of KIPP were 93% and 92%, respectively. The average AUROC score of KIPP was approximately 0.986. Figure 12 shows the KIPP toxic message type classification prediction accuracy and the corresponding loss function. Figure 13 shows the ROC curves that depict the performance of the KIPP model in accurately predicting the values of the different toxic message classes.

4.3. Validation of the Decentralized Approach of MaLang

The system-level module classifies more than 98% of the data into abusive and non-abusive text messages for smartphones or personal computers. Post classification, only the abusive content is pushed to the cloud in the form of entities that contain an array of tokens that describe the sentence. Therefore, all of the user’s data other than tokenized text messages are secure and are not transmitted to the cloud. Moreover, the tokenized version of data sent to the cloud, when accessed by a human, will not be able to make any sense, structurally or contextually, due to the lack of articles or stop words. Reframing the articles transmitted to the cloud back into a sentence is difficult and unlikely for a human. This would result in giving no exploitable meaning or value to the user’s data in the cloud module.

Since 98% of a user’s data are classified on the user’s device, the cloud-level module consumes less storage space and computing time to identify the toxicity in the text. It is estimated that the consumption of computing resources and time can be reduced by 35–40% over prior cyberbullying detection systems by deploying MaLang.

5. Discussion

5.1. Comparison of MaLang with Existing State-of-the-Art Models

Table 3. Performance comparison of MaLang with respect to existing state-of-the-art detection models.

Approach	Accuracy	Precision	Recall	F1-Measure	AUROC
MaLang	0.982	0.987	0.960	0.973	0.986
Text mining [22]	0.586
NLP [7]	0.950
Logistic Regression [64]	0.905	0.910	0.900
Decision Tree [23]	0.785
SVM [30]	0.780			0.780
LSTM [38]	0.9397
CNN [50]	0.960
Deep Learning Ensemble [46]	0.980

depicts the performance comparison of MaLang compared against the best existing state-of-the-art models taken from Table 1, where best performing is based on the accuracy of overall predictions. Logistic regression is added as a reference value since this is a common method for traditional research. Other values of precision, recall, F-score, and AUROC values are provided when available in the previously reported research. Prior research utilized different data sources for analyzing cyberbullying detection systems and as such, only the reported performance measures are used for comparison.

Tolba et al. [65] recently performed an independent comparison of various machine learning and deep learning approaches using various encoding schemas for the detection of online harassment (no age limitations specified). Tolba et al.’s findings report an optimal geometric mean, F1 score, and AUROC of: 0.8107, 0.8107, 0.7612; all for systems that utilized GloVe with either a Bi-LSTM or LSTM based deep learning classifier.

As may be seen in Table 3, the MaLang approach outperforms current existing natural language, parametric statistical, text analytics, and machine learning-based abusive language detection models. While ensemble deep learning methods come close to MaLang’s performance, the decentralized approach of MaLang has the additional benefit of preserving data privacy for employees. The decentralized methodology of MaLang also makes this approach scalable to very large organization data networks because the CASE component runs on separate individual devices.

5.2. Limitation

A limitation of the MaLang approach is that corporations must install and maintain CASE on all corporate computing devices that may be used for text communications and also requires that all employees install CASE on any of their personal devices that are brought to work. Fortunately, regardless of employee compliance on their personal devices, cyberbullying may still be detected via receipt of messages on corporate-owned communication endpoint devices that have CASE installed.

6. Conclusions

Electronic communications are becoming ever more ubiquitous in today’s organizations as well as in general life. The ever-increasing use of electronic communication technologies has enabled and is increasing the occurrence of workplace cyberbullying and the exchange of other hostile or abusive communications [3]. Various attempts were previously made to utilize machine learning to try and identify abusive language in textual communications but are not being widely deployed due to system overhead or other business factors. In this paper, a new method for identifying and reacting to abusive text messages, called MaLang, is developed and tested.

The MaLang approach ensures data privacy by not sharing sensitive or raw data being transmitted to the cloud. With the increasing global emphasis on data privacy, MaLang’s anonymization features could be critical adoption factors for organizations.

The Deep Learning models CASE and KIPP perform with an average accuracy of 92.2% and 98.21%, respectively. They achieved an average AUROC score of 0.977 and 0.986, respectively, in classifying abusive content. With the methodology presented in the current paper, it can be concluded that MaLang outperforms existing methods with respect to overall accuracy for detecting abusive textual content in a user’s electronic communications using their device and over the business’s network.

The MaLang framework can be deployed in an organization to detect the exchange of abusive content among employees, even across different departments. The methodology to improve the model by updating the weight matrix forms the basis of federated learning and helps multiple devices benefit from the mechanism of training one model on multiple datasets. The decentralized approach can be used to detect abusive textual content in the downloaded media on user’s device.

A potential future upgrade to the framework would scale down the functionalities of Cloud-level classification to an edge platform preserving data securely and performing classification on mobile devices without pushing any data to the cloud. The approach can be scaled up to perform real-time text prediction without suggesting any abusive content to the user to type in. This could potentially decrease the usage of abusive or toxic content.