Innovations in Urdu Sentiment Analysis Using Machine and Deep Learning Techniques for Two-Class Classification of Symmetric Datasets

Abstract

Many investigations have performed sentiment analysis to gauge public opinions in various languages, including English, French, Chinese, and others. The most spoken language in South Asia is Urdu. However, less work has been carried out on Urdu, as Roman Urdu is also used in social media (Urdu written in English alphabets); therefore, it is easy to use it in English language processing software. Lots of data in Urdu, as well as in Roman Urdu, are posted on social media sites such as Instagram, Twitter, Facebook, etc. This research focused on the collection of pure Urdu Language data and the preprocessing of the data, applying feature extraction, and innovative methods to perform sentiment analysis. After reviewing previous efforts, machine learning and deep learning algorithms were applied to the data. The obtained results were compared, and hybrid methods were also recommended in this research, enabling new avenues to conduct Urdu language data sentiment analysis.

1. Introduction

According to statistics derived from the internet, the top 10 most popular languages used on the Internet according to percentage share are English (25.9%), Chinese (19.4%), Spanish (7.9%), Arabic (5.2%), Portuguese (3.7%), Indonesian/Malaysian (4.3%), French (3.3%), Japanese (2.6%), Russian (2.5%), and German (2.0%), totaling 76.9%, while the rest of the languages constitute 23.1%. English’s share is only 25.9%, while non-English language users comprise 74.1% out of all users. Major research has been carried out on English language processing, but a lot of work is yet to be carried out on other languages, especially South Asian languages [1]. Non-English languages include Chinese, which is used by 19.4% of people worldwide. Work has been carried out on Chinese natural language processing (CNLP), in which assessments have been performed and resources have been built [2]. Text processing and speech processing are still underway.

In this paper, a detailed study was performed to investigate Urdu text data processing, the classification of data, and the application of machine learning as well as deep learning algorithms on Urdu text to perform Urdu sentiment analysis and compare the obtained results. The key contribution of this study is the use of a hybrid technique to analyze and process Urdu language text; hence, it suggests a new method for further improvement of results.

1.1. Sentiment Analysis/Attitude Scrutiny

Sentiment analysis determines the feelings of people about a certain product or service. In social networking, people write microblogs in Urdu or other languages because they express their feelings/reviews/expressions according to various classes. These posts may be posted on microblogs in real-time using various attributes on a myriad of subtopics and comprising discussion of contemporary issues and criticism, forming positive, negative, or neutral sentiments for objects/goods or services they use in their day-to-day routine. Manufacturing factories/companies of such products have started to collect and utilize these data to obtain a realization of the general sentiments for their products/objects or services. Microblogs are a shorter form of blogs that allow users to express their emotions in real-time. Sentiment analysis is the major aspect of the current NLP, as it is with Urdu Language Processing (ULP).

1.2. Attitude Scrutiny in Social Networks

Social media popularity impacts the preferences of customers, with Twitter and Facebook enjoying widespread popularity. This popularity not only enables people to express their opinions, but also provides guiding principles for others to follow and for companies/organizations to improve their products and services. Users write blogs as well as reviews on social media that are related to companies. Hence, customers’ views are readily available for the company’s use. Text mining is carried out to determine the opinions of customers. Website data and text are available and mined, providing a new dimension to the field of computer science. Sentiment analysis has recently emerged as a very important field with the development of social media. Public opinions are now available to study. Textual data are usually unstructured, and incomprehensible, vague, and unclear, making analysis difficult, but they contain a huge amount of valuable information if processed properly. Herein, we collected and processed Urdu text data on certain topics using the methods of NLP.

1.3. Research Inspiration

This work was carried out keeping in mind the following considerations:

• People provide their views on social media to express emotions about certain products/services, especially in the Urdu language.
• South Asian countries such as India, Pakistan, etc., use the Urdu language to express their feelings, comprising a huge online market for other countries.
• No work has previously been performed that combines the latest deep learning algorithms to conduct such an analysis.

1.4. Contributions and Obligations

Our contributions and obligations are the following:

• Elaborate the importance and utility of Urdu sentiments and perform an analysis thereof.
• Investigate methods to perform S.A. in Urdu.
• Apply machine learning and deep learning algorithms for S.A.
• Apply a combination of algorithms to propose a hybrid approach for S.A.
• Compare the applied algorithms to suggest the most effective ones for the Urdu language.
• Suggest future work.

1.5. Relation to Previous Work

Research was performed to apply sentiment analysis on an Urdu tweets dataset, and five algorithms were applied and the results were compared [3]. Another study was conducted to apply machine learning and deep learning approaches such as SVM, LR, LSTM, and CNN on Urdu sentences for the analysis of polarity, with the results outperforming previous ones [4]. Research was performed by extracting a small number of tweets, both in Urdu and Roman Urdu, and the WEKA tool was applied but was not able to give results on all applied algorithms in Urdu [5]. Research was conducted as Urdu sentences were extracted. LSTM was applied for sentiment analysis and results were obtained. [6]. Sentiments from social data were analyzed using Urdu pre-existing algorithms, attaining adequate results [7]. Urdu language data were collected and some pre-processing was applied using classical machine learning along with deep learning techniques to perform sentiment analysis [8]. The characteristic of context dependency of Urdu words was resolved to conduct Urdu sentiment analysis by assigning polarities to sentences; the accuracy of results therefore improved [9].

Sentiment analysis in Urdu was performed using a lexicon-based approach, a traditional method adopted using rules and results obtained in Urdu [9]. Multimodal datasets concerning text, audio, and videos of Urdu and comprising 1372 expressions were collected to identify context-aware sentiments, achieving a reasonable accuracy [10]. A review paper was written to examine the work conducted in Urdu and Roman Urdu for product reviews [11]. Urdu data were collected, annotated, and classified using PART, Naïve Bayes Multinomial Text, Lib SVM, decision tree (j48), and k-nearest neighbor; polarity prediction was also performed [12]. A corpus of Urdu Sentiments was developed comprising of tweets between two political parties, which were collected and tagged for polarity, and linguistics analysis was performed on them [13]. The polarity of sentences was calculated using a sub-opinion approach using orientation scores separately from sub-fragments [14].

Research was performed on Urdu language intensifiers by applying rules to assign polarities to them for S.A, and an Urdu sentiment analyzer was developed improving the accuracy [15]. People’s feelings and behavior toward a product/service or brand has gained importance in the field of opinion mining, therefore opinion mining on Urdu comments on websites was performed [16]. A rule-based approach was used to handle negations in Urdu sentences, thus improving the accuracy of S.A [17]. A word-level translation scheme was introduced to create an Urdu lexicon as well as polarity scores to assign sentiments and performed evaluation [18]. A review of previous work on sentiment analysis of Urdu was conducted using machine learning to investigate improvement [19]. Reviews from various channels on sports, food, software, politics, and entertainment were collected to create a corpus in Urdu, and some machine learning models were applied for S.A [20]. English sentiments were used to form Urdu sentiment lexicons and ML was used to compare their performance [21]. Urdu opinion mining was performed by first collecting Urdu tweets as a form of pre-processing and then forming a feature vector, identifying positive and negative words as well as using a decision tree used for classification [22]. Another study was performed to identify Senti-units from Urdu text using parsing; a lexicon-based approach was used and performance was evaluated on various texts [23]. Another study was performed to classify tweets data into three classes using a Markov Chain, applying prediction improving results from previously used lexicon and traditional machine learning approaches [24]. A large scale dataset was created in the Urdu language by applying pre-processing, and emoji’s were used to identify sentiments [25].

Songs were selected from YouTube and online reviews were compared for S.A [26]. A review was performed for this study using various ML algorithms, and their contribution toward opinion mining was evaluated [27]. An attempt was made to perform cross-domain S.A in Urdu using ML and DL classifiers [28]. An identification of the challenges and future directions of research to be performed in the Urdu language toward emotion detection was conducted [29]. A study was conducted during the Pakistan General Elections and during the promotion of political parties and public opinions on social media; we also observed the effects on the elections’ results [30]. Urdu poetry was analyzed using the work of many poets over a long time period, exploring the main features of Urdu ghazal popularity and public liking [31]. The identification of characteristic-based sentiment analysis was performed with the help of a dataset for this purpose [32]. Another study was conducted to develop Urdu language communication on Twitter and observe its use in opinion mining [33]. Research was conducted to detect sarcasm and S.A relationship by using a cognitive approach to perform classification [34]. Sentences in English were extracted and translated to Urdu, grammatical errors were corrected using natural language processing, and an emotional analysis was performed using ML approaches [35]. S.A was performed on Urdu Blog data using structural correspondence learning (SCL), POS tagging, and the approach was validated using the supervised learning method [36]. Urdu news text was classified using 12 approaches of machine learning and compared [37]. Another study was performed to study Urdu and English tweets/news data from websites in conjunction with the dengue epidemic, and S.A was performed to get insights using ML algorithms, with the data then being evaluated [38]. A hybrid feature selection approach was used to classify the Urdu text of news articles, giving improved results [39].

Efforts were made to develop an annotated corpus from the Urdu Nastalique Emotions Dataset (UNED), and machine learning and deep learning approaches were used to conduct emotion detection [40]. Using an Urdu dataset comprising of many labels, emotions were classified using machine learning [41]. Research was performed to reduce the gaps in pre-processing tools in Urdu by designing a stemmer, tokenizer, and preparing a stop-words list to perform Urdu news classification [42]. Urdu text document classification was performed using DL models in comparison with ML models in product manufacturing data [43]. The intent detection of users data in Urdu was performed after information retrieval using Bidirectional Encoder Representation from Transformers (BERT) [44]. To detect threatening content in the Urdu language, a stacking model was developed using Naïve Bayes and was applied for learning, while Logistic Regression was used for meta-learning; the stacked model compared well with other approaches [45].

The natural language processing and Urdu NLP tasks performed are shown in Figure 1 below:

2. Material and Methods

This research was conducted to explore the following research questions:

2.1. Research Questions

The survey was conducted using the following research questions.

• Research Question 1: Is the pre-processing required for Urdu text justified?
• Research Question 2: How can the ML and DL methods be applied on Urdu text for extracted data?
• Research Question 3: Can we apply more than one algorithm simultaneously on the Urdu data?
• Research Question 4: Which algorithm was found to be more effective and accurate for Urdu language processing?

2.2. Technique and Criteria for Acceptance and Rejection

To obtain the most relevant research articles, different research queries were given using keywords such as “sentiment analysis”,” Urdu sentiment analysis”, “opinion mining”, “Urdu opinion mining”, “Urdu opinion mining for review,” “Urdu tweets sentiment analysis ”, “Urdu preprocessing”, “sentiment classification in Urdu”, “Urdu part of speech tagging”, and “Urdu Classification method”.

The selection and rejection criteria of the papers is as follows:

• IP1: Inclusion Principle: All articles to be selected matching title or some words.
• IP2: Inclusion Principle: Papers matching in abstract used to classify Urdu sentiments.
• IP3: Inclusion Principle: Papers including methods used in Urdu sentiment analysis.
• IP4: Inclusion Principle: Papers using some preprocessing for Urdu sentiment analysis.

Exclusion Principles (EPs) are:

• EP1: Exclusion Principle: Remove all papers not following the above inclusion principles.
• EP2: Exclusion Principle: Do not include any Roman Urdu papers.

2.3. Study Quality Evaluation

The quality of the study was maintained by adopting only selected papers for the study. Additionally, research questions mentioned in Section 2.1 were used to assess and address the quality of research. Two researchers analyzed all selected papers and only those selected qualified by meeting the criteria.

2.4. Survey Execution

A reasonably sized Urdu dataset was developed and used in this research to perform sentiment analysis. Initially, some pre-processing was performed on the data, and then the algorithm’s application was studied and applied. Finally, the results were compared. The survey was conducted and categorized according to the research questions.

2.4.1. Data Collection

The data were collected from various blogs, websites, and tweets comprising of more than 10,000 sentences. A tedious exercise was conducted to obtain 10,009 sentences, which were manually annotated by 3 annotators in terms of being positive, neutral, or negative, as shown in Figure 2.

Nearly 7000 sentences were found to lie in the above three categories by the annotators. Finally, two classes of positive and negative sentences were selected for this work. Hence, a gold standard dataset was formed to perform the analysis. It was divided into 3737 negative labels and 2815 positive labels; remaining sentences were undecided to fit in these two criteria, as shown in the figure below. The data were trained for two-class classification; before this, the data needed cleaning.

2.4.2. Data Cleaning

Data cleaning is an important step towards pre-processing, hence the removal of URLs from the data was performed; then, the data ere tokenized, removal of emails was performed, new line characters were removed, distracting single quotes were also removed, all the punctuation symbols were removed, then detokenization of the data was performed; finally, the list of texts was converted into a ‘numpy’ array as this is the requirement for the application of machine learning to the sentence needs to be converted into a list, as shown below:

‘جہاں تک بہترین سوفٹویئر کی بات ہے تو بہترین سوفٹویئر وہ ہے جو آپ کو استعمال کرنا آتا ہو’

Then, the data were converted as follows:

’جہاں‘،’تک‘،’بہترین‘،’سوفٹویير‘،’کی‘،’بات‘،’ہے‘، ’تو‘،’بہترین‘,’سوفٹویير‘,’وہ‘,’ہے‘،’جو‘، ’اپ‘،’کو‘، ’استعمال‘، ’کرنا‘، ’اتا‘، ’ہو‘

After being detokenized, the data were as follows:

’جہاں تک بہترین سوفٹویير کی بات ہے تو بہترین سوفٹویير وہ ہے جو اپ کو استعمال کرنا اتا ہو‘

Initially, the machine learning algorithm Naïve Bayes and a support vector machine were applied. Naïve Bayes is a probabilistic-based machine classifier, while support vector machines also belong to a set of supervised machine learning methods for classification, with these methods being highly effective. Then, deep learning algorithms such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Bidirectional LTSM, Bidirectional GRU, and Attention-based Bidirectional GRU were applied on the data. Then, a combinational algorithm was also tested on the data.

3. Results

The following answers were found for the research questions stated above.

3.1. Research Question 1

• Is the preprocessing required for Urdu text justified?

The data were collected from various blogs, websites, and tweets comprising of more than 10,000 sentences. Urdu being a low-resourced language, this task proved to be a tedious one. Annotators were able to classify 7010 sentences into 3 classes: positive, neutral, and negative. A gold standard dataset was formed to perform the analysis. To perform two-class classification, two classes were selected, including 3737 negative labels and 2815 positive labels; remaining sentences were excluded, resulting in a balanced data.

Sentiment analysis cannot be performed before this, since the processing of text in any language, especially Urdu, contains many unavoidable words/symbols, etc., those used which have no significance in forming opinions. To process text in any language is difficult, but Urdu has its own challenges, which surpass all other languages, making this task more daunting. Urdu text/data preprocessing includes some steps to be followed. The processing of the Urdu dataset was performed as shown in Figure 3 below:

Urdu Text Pre-processing

• Data cleaning/noise removal;
• Data normalization;
• Tokenization and tagging.

3.1.1. Data Cleaning/Noise Removal

The data were to be trained for two-class classification, but before this the data needed to be cleaned for processing. The Urdu text from tweets comprised some additional ‘noise’ in the form of URLs, emails, new line characters, punctuation/signs; removing these from the data makes the data more suitable to use and more meaningful. Additionally, this improves the results in the classification and accuracy of the implementation of algorithms.

Typical steps identified and followed were as follows.

• Remove URLs from the tweets.
• Tokenize text.
• Remove email addresses.
• Remove new line characters.
• Remove all punctuation signs.
• Detokenize text.
• Convert list of text to ‘Numpy’ array.
• Apply text cleaning process to the data.

The steps involved, from data pre-processing until the application of the algorithms, are shown in Figure 4 as follows:

3.1.2. Normalization

The Urdu text data extracted required normalization to follow proper Urdu grammatical rules and to making the data meaningful as well. They required some normalization; grammatical and lexical rules also needed to be followed. This process must be applied before any algorithm to be applied for sentiment analysis in order to give better results.

3.1.3. Tokenization

Tokenization is performed using treebank tokenizer, as it is a must to break long sentences into understandable separate words for analysis; also, wordnet lemmatizer is used to convert words into their base forms, with this process being called stemming; stop-words are also removed from the text. The process of stemming in Urdu is shown below with the help of an example in Figure 5:

3.1.4. Tagging

The Urdu text is then tagged and labelled manually by 3 annotators in the form of positive or negative sentiments to perform two-class classification with 0 and 1 labels, thus improving the quality of data.

3.2. Research Question 2

• How can the ML and DL methods be applied on Urdu text for extracted data?

The extracted Urdu data needed to be converted into a readable form via the machine learning algorithms as well as deep learning, therefore the above-mentioned steps in Figure 3 were applied. Model building was performed using Keras API in Python. Initially, machine learning algorithms were applied to the Urdu dataset. Naïve Bayes was applied, and the model was trained, giving an accuracy result of 0.78265. The support vector machine applied on the same data gave an accuracy result of 0.56075.

Deep learning algorithms require all data to be converted to similar lengths, and so a maximum length of 139 words was selected based on the available data. A Convolutional Neural Network (CNN) was applied on the given data firstly by using sigmoid as the activation function and binary cross entropy as the loss model to train and test the model. A Simple Recurrent Neural Network (RNN) was applied on the data using the same perimeters; Single-Layered Long Short-Term Memory (LSTM) was also applied on the same data, and 20 epochs were processed for these models as the data were large, but the results obtained from the special GPU processing machine were used for model building. Bidirectional LSTM and a Bidirectional Gated Recurrent Unit (GRU) were also applied on the data. A complete depiction of Urdu text collection until the comparison of the sentiment analyses is shown in Figure 6 below:

3.3. Research Question 3

• Can we apply more than one algorithm simultaneously on the Urdu data?

An innovation in the application of models in this study was achieved, and a combination of models was also applied to the Urdu language data. LSTM was used in combination with GRU, resulting in the attention-based GRU (LSTM + GRU) algorithm; the results were then noted. Additionally, RNN was applied in combination with CNN resulting in a combined model (RNN + CNN), and the results for the Urdu data were noted. Hence, this showed that combinational models can also be applied on data, resulting in a Hybrid model that can be used for data analysis.

3.4. Research Question 4

• Which algorithm was found to be more effective and accurate for Urdu language processing?

In order to evaluate the effectiveness and accuracy of various algorithms applied in the Urdu language data, the measures used are accuracy, precision, and recall. The accuracy score in machine learning is an evaluation metric that measures the number of correct predictions made by a model in relation to the total number of predictions made. We calculate it by dividing the number of correct predictions by the total number of predictions, as shown in Equation (1):

$$\mathrm{Accuracy} =\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}} ,$$

(1)

where, TP = True Positive, TN = True Negative, FP = False Positive, and FN = False Negative.

The precision is calculated as the ratio between the number of positive samples correctly classified to the total number of samples classified as positive (either correctly or incorrectly). The precision measures the model’s accuracy in classifying a sample as positive, as shown in Equation (2):

$$\mathrm{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},$$

(2)

The recall is calculated as the ratio between the numbers of positive samples correctly classified as positive to the total number of positive samples. The recall measures the model’s ability to detect positive samples. The higher the recall, the more positive samples detected.

$$\mathrm{Recall} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},$$

(3)

Various machine learning and deep learning algorithms were applied on the data. Naïve Bayes performed better than SVM, while deep learning algorithms performed efficiently; simple RNN gave acceptable results; LSTM improved results further than GRU; attention-based GRU proved even better in this regard; CNN also gave good results; a combination of two algorithms, namely RNN plus CNN, outperformed all other applied algorithms. A detailed discussion of the algorithm results obtained is conducted in Section 4.

The following answers were found for the research questions stated above.

4. Discussion

The extracted Urdu Data needed to be converted into a readable form by the machine learning algorithms as well as deep learning, therefore the above-mentioned steps in Figure 3 were applied. Model building was performed using Keras API in Python. Initially, machine learning algorithms were applied to the Urdu dataset. Firstly, the Naïve Bayes algorithm was applied, and the model was trained, giving an accuracy result of 0.78265. Another machine learning algorithm support vector machine applied on the same data gave an accuracy result of 0.56075; this shows that SVM did not perform well on the Urdu dataset.

Deep learning algorithms require all data to be converted to similar lengths, and so a maximum length of 139 words was selected in each sentence based on the available data. A Convolutional Neural Network (CNN) was applied on the given data firstly by using sigmoid as the activation function and binary cross entropy as the loss model to train and test the model; 20 epochs were run using the GPU machine. The final obtained accuracy was 85.8%, with a precision of 0.8799 and a recall value 0.8901. A Simple Recurrent Neural Network (RNN) was applied on the same dataset using the same parameters, resulting in an accuracy of 82.8%, a precision of 0.9144, and a recall value of 0.7931. The dataset was further tested on Single-Layered Long Short-Term Memory (LSTM), resulting in an accuracy of 84.0%, with a precision of 0.8757 and a recall value of 0.8160. Bidirectional LSTM was also tested, obtaining an accuracy result of 82.99%, with a precision of 0.9187 and a recall value of 0.7927. A Bidirectional Gated Recurrent Unit (GRU) was also applied, giving an accuracy of 84.4%, with a precision of 0.8960 and a recall value 0.8412.

An innovation in the application of models in this study was achieved, and a combination of models were also applied to the Urdu language data. LSTM was used in combination with GRU resulting in the (LSTM + GRU) algorithm, achieving an accuracy of 84.5% with a precision of 0.8889 and recall value of 0.8593. Additionally, the RNN was applied in combination with the CNN resulting in a hybrid model (RNN + CNN), and the results showed an accuracy of 85.8% with a precision of 0.8799 and a recall value of 0.8901 for the dataset. Comparing these results shows that a combination of the two models can also be applied on Urdu datasets, resulting in a Hybrid model for data analysis. Additionally, the results have improved in terms of the accuracy being above 85%, and both precision and recall being above 90% makes them superior to the previous figures achieved, as shown in Figure 7 below:

5. Conclusions

With the growth in businesses and development of online strategies for e-commerce, it has become vital to understand public opinions and feel the pulse of customers in terms of products or services. Urdu, being a widely used language of South Asia, is of great importance and yet research on this language is limited, hence a lot of work is required. The primary objective of this work was to explore and implement the techniques for Urdu text preprocessing and develop a new source for the usage of data analysis. Various algorithms of machine and deep learning categories were implemented on the Urdu text data. One-class and two-class classification were applied on a large dataset, and the results were obtained. Machine learning algorithms showed good results, while the majority of deep learning algorithms improved the results further. A combination of machine learning algorithms was applied on the Urdu text data and led to the further improvement of results, opening a new avenue for researchers. It is recommended that other combinations of algorithms be tested on different datasets of Urdu in the hope of leading to even better results for sentiment analysis, which can be used to make software that responds toward customer sentiments better than the existing ones.

6. Future Work

Future work can be conducted by applying various combinations of algorithms on Urdu text data to products or service reviews. Additionally, similar work can be conducted on other languages such as the Sindhi, Panjabi, Pashto, and Balochi languages.

7. Human and Animal Rights

This study did not involve any experimental research on humans or animals.