Next Article in Journal
Time–Frequency Analysis of Diesel Engine Noise Using Biodiesel Fuel Blends
Next Article in Special Issue
Integrating Data-Based Strategies and Advanced Technologies with Efficient Air Pollution Management in Smart Cities
Previous Article in Journal
Banks’ Capital Requirements in Terms of Implementation of the Concept of Sustainable Finance
Previous Article in Special Issue
The Long-Run Impact of Information Security Breach Announcements on Investors’ Confidence: The Context of Efficient Market Hypothesis
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 13
Issue 6
10.3390/su13063497
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Framing Twitter Public Sentiment on Nigerian Government COVID-19 Palliatives Distribution Using Machine Learning

by
Hassan Adamu
1,
Syaheerah Lebai Lutfi
1,*,
Nurul Hashimah Ahamed Hassain Malim
1,
Rohail Hassan
2,
Assunta Di Vaio
3 and
Ahmad Sufril Azlan Mohamed
1
1
School of Computer Sciences, Universiti Sains Malaysia, Penang 11800, Malaysia
2
Othman Yeop Abdullah Graduate School of Business (OYAGSB), Universiti Utara Malaysia (UUM), Kuala Lumpur 50300, Malaysia
3
Department of Law, University of Naples “Parthenope”, 80132 Naples, Italy
*
Author to whom correspondence should be addressed.
Sustainability 2021, 13(6), 3497; https://doi.org/10.3390/su13063497
Submission received: 10 January 2021 / Revised: 9 March 2021 / Accepted: 13 March 2021 / Published: 22 March 2021
(This article belongs to the Special Issue Management and Innovation for Environmental Sustainability)
Browse Figures
Versions Notes

Abstract

:
Sustainable development plays a vital role in information and communication technology. In times of pandemics such as COVID-19, vulnerable people need help to survive. This help includes the distribution of relief packages and materials by the government with the primary objective of lessening the economic and psychological effects on the citizens affected by disasters such as the COVID-19 pandemic. However, there has not been an efficient way to monitor public funds’ accountability and transparency, especially in developing countries such as Nigeria. The understanding of public emotions by the government on distributed palliatives is important as it would indicate the reach and impact of the distribution exercise. Although several studies on English emotion classification have been conducted, these studies are not portable to a wider inclusive Nigerian case. This is because Informal Nigerian English (Pidgin), which Nigerians widely speak, has quite a different vocabulary from Standard English, thus limiting the applicability of the emotion classification of Standard English machine learning models. An Informal Nigerian English (Pidgin English) emotions dataset is constructed, pre-processed, and annotated. The dataset is then used to classify five emotion classes (anger, sadness, joy, fear, and disgust) on the COVID-19 palliatives and relief aid distribution in Nigeria using standard machine learning (ML) algorithms. Six ML algorithms are used in this study, and a comparative analysis of their performance is conducted. The algorithms are Multinomial Naïve Bayes (MNB), Support Vector Machine (SVM), Random Forest (RF), Logistics Regression (LR), K-Nearest Neighbor (KNN), and Decision Tree (DT). The conducted experiments reveal that Support Vector Machine outperforms the remaining classifiers with the highest accuracy of 88%. The “disgust” emotion class surpassed other emotion classes, i.e., sadness, joy, fear, and anger, with the highest number of counts from the classification conducted on the constructed dataset. Additionally, the conducted correlation analysis shows a significant relationship between the emotion classes of “Joy” and “Fear”, which implies that the public is excited about the palliatives’ distribution but afraid of inequality and transparency in the distribution process due to reasons such as corruption. Conclusively, the results from this experiment clearly show that the public emotions on COVID-19 support and relief aid packages’ distribution in Nigeria were not satisfactory, considering that the negative emotions from the public outnumbered the public happiness.

1. Introduction

Sustainability helps society in many ways by improving wellbeing and quality of life. Sustainable societies include many various sectors that encompass businesses, government agencies, environmentalists, and civic associations. Sustainable communities always seek to be innovative to boost their local economy for a healthy ecosystem [1]. Sustainable development goals (SGDs) play an essential role in shaping daily societal activities, especially in under-developed countries such as Nigeria. However, the impact of the COVID-19 pandemic has caused the reversal of the United Nations target of lifting millions of people out of poverty. A recent study [2] projected that there are 99 million people already pushed into poverty in 2020. There will be an additional 44 million people who will still live in extreme poverty by 2030 due to the impact of the COVID-19 pandemic. These would bring the total number of people globally living in poverty to 905 million by 2030.
Under sustainable development goals (SDGs), there is a need for countries such as Nigeria to come up with strategies and targeted interventions to reduce the number of vulnerable people that are currently living in extreme poverty, especially in remote areas that face difficulties in their daily lives with limited resources and poor infrastructure. Most of the under-developed countries, such as the Nigerian government, usually executed their services, including humanitarian relief aid, without seeking the opinion of the public, which is one-way governance. The sustainable development goal in information and communication technology (ICT) requires two-way governance where the government brings its citizens closer to them for better decision-making, especially during disaster management such as emergency response services including COVID-19 palliatives, relief aid, and logistics packages’ distribution to vulnerable people. Management and innovation for environmental sustainability meet the needs of the present without compromising the ability of future generations to meet their own needs. Sustainable development in the context of humanitarian support helps the government to reduce the number of vulnerable people living in severe pain caused by a disaster such as the COVID-19 pandemic.
COVID-19 is a disastrous disease that started spreading among people in December 2019 in Wuhan, China [3,4]. The virus started increasing and spreading on 31 December 2019, which began to draw public attention. In this regard, the World Health Organization (WHO) announced COVID-19 as a pandemic in March 2020 [5]. As of 8 January 2021, the number of new COVID-19 cases reached the total number of 88,222,239 and the total number of 1,909,910 reported by the Centre for Systems Science and Engineering, Johns Hopkins University [6]. Although not up to 1 percent of the Nigerian populace have been infected, out of the current approximate population of 200 million [7], 136 thousand people (approximately 0.068% of the country’s population) have been infected by the virus [8]. Similarly, the mortality rate has been low as currently 1630 deaths have been recorded, while the active cases count is 23,949 out of the total (136 thousand) detected cases so far [8]. Many countries forced their citizens to stay at home to curb the spread of the deadly virus, making businesses and social activities stop worldwide [9,10,11]. In times of disaster such as COVID-19, vulnerable people need help from the government to survive, and Nigeria, just like other countries, claimed to have spent billions of Naira distributing relief packages and COVID-19 palliatives to vulnerable citizens. The distribution of these palliatives is part of the measures the government has put in place in order to provide basic welfare to the masses as they have been restricted from conducting their daily business and means of earning income. However, Nigerians have expressed mixed feelings via social media, specifically on the billions of Naira claimed by the government on COVID-19 relief aid and palliatives [12]. Some of these reactions are based on citizens’ concerns about the government’s failure to enforce sustainability (human, social and fiscal) laws on shared ideas of equality and rights, which improves qualities such as transparency, accountability, equality, reciprocity, cohesion, and honesty, the promotion of relationships amongst people, and their wellbeing.
Several governance approaches include the public’s participation as customers, partners, or citizens and bringing sustainable life to the public [13]. Good governance requires improvement overtime to address critical issues concerning existing and emerging challenges to the governed. Part of such challenges is to improve the government’s ability to manage rapidly growing information demands on accountability and transparency of public funds; this requires the use of analytics to address governance challenges [14] proactively. Considering the current rising expectations of citizens on their government, there is a need for the government to be innovative in gathering and understanding the data about the expectations of its citizens. This means that social media should not be limited to communicating government achievements to the public but should be extended as a listening and information source for the government. Such information sources can be used to develop predictive models that enable decision support, which is a vital part of smart governance [15,16].
With tremendous advancements in Information and Communication Technology, Business Intelligence is an essential tool to improve business processes and an influential and successful instrument in shaping business objectives to target customer’s needs [17]. Similarly, Opinion Mining applies to businesses and other areas such as politics and governance. Opinion mining has become highly popular and interesting to researchers because of its application to several fields [18]. According to [19], 3.81 billion active social media users are depositing a massive amount of data on several social media platforms such as Facebook, Twitter, Instagram, WhatsApp, and others. Currently, Nigeria has more than 85 million internet users [20]. This study chose Twitter as the primary data source due to its popularity because, according to [21], Twitter accounts for 50 percent of the 25 million social media users in Nigeria. It is a popular microblog with 140 million active users posting more than 400 million tweets every day world-wide. Many users post information such as disaster damage reports and disaster preparedness situations during the disaster response period, making Twitter essential for updating and accessing data. Mining sentimental data efficiently will help better understand the emergency response, timely and efficiently [22,23].
Data mining plays a vital role in several research domains, such as artificial intelligence, statistics, and machine learning [24]. Data mining is used to discover important information from databases. Opinion mining and sentiment analysis in the domain of social media platforms, data mining, or computing at large, as stated by [25], refers to the identification, assessment, generating, and summarizing of an opinion and feelings of users about a different aspect of services rendered by governmental or private organizations, socio-economic and daily life activities. Additionally, for computation, sentiment analysis might be defined as written expressions of subjective mental states. Sentiment analysis and opinion mining in the context of emergency response help governmental and humanitarian agencies to identify sentiment and opinion that could bring the public closer to them by understanding their concerns and emotional feelings in times of emergency responses; hence, that would help in making an informed decision [26,27].
Several sentiment analysis studies on the English language have been conducted using different standard machine learning algorithms on social media regarding governmental issues using classifiers that have proven efficient in text classification, as illustrated by [28,29]. Despite the achieved efficiency, such studies are yet to be conducted on the citizens’ emotions regarding accountability and transparency of the Nigerian government spending on COVID-19 relief and aid response. Besides, such studies cannot be ported to informal (Nigerian Pidgin) English texts. Nigerian Pidgin English is estimated to have between 3 and 5 million people that use it as the primary communication medium in their daily interactions, and as the second language, around 75 million Nigerians [30]. Due to its difference in vocabulary to standard English, words such as “tank” are interpreted as an expression of gratitude, not as a container, while “ginger” is interpreted as motivation, not as a plant [31]. Therefore, this study aims to fill the existing research gap on emergency response-related studies and address the lack of text emotion datasets by constructing a Nigerian Local English Twitter Emotion dataset (Pidgin English). The dataset is experimented with using six Machine Learning algorithms to classify aggregated citizens’ emotions on the Nigerian government’s COVID-19 palliatives’ distribution. The remainder of this paper is structured as a review of related literature discussed in Section 2 of this paper. Section 3 discussed the methodology in terms of materials and methods used in conducting the experiments conducted in this paper. The results obtained from the experiments conducted are presented and discussed in Section 4 and Section 5, respectively. Finally, the conclusion of the study is presented in Section 6 of this paper.

2. Review of Related Literature

2.1. Pandemic Emergency

COVID-19 is an example of a pandemic disaster that ceased almost every social and economic activity across the world. People were forced to stay at home, regularly wash their hands, and maintain social distancing to reduce the virus’ spread. Doing so could be the only way to curtail and contain the deadly infectious disease [32]. During this pandemic, vulnerable people need help to survive; therefore, many countries distribute relief packages to vulnerable citizens as COVID-19 palliatives. Government agencies and media organizations provide information about the public’s daily cases [33]. However, individuals share their opinion and sentiment on social media regarding the coronavirus pandemic every day. A recent study by [34,35] shows how public opinion and sentiment on Twitter are analyzed to help government and humanitarian agencies predict and manage disasters and election processes.

2.2. Social Media Data

Twitter is now fourteen years old [17], and it is ranked as number four in terms of the most popular and highest in number with 81.47 million registered users. It is estimated that every second, users post about 600 tweets, resulting in 350,000 tweets per minute and 500 million per day. This shows that about 200 billion tweets are posted yearly. Despite the number of tweets posted by users, each tweet is grouped based on its topic; it could be about political matters, personal opinion, national economic issues, COVID-19 pandemic issues, or others. According to [36], social media topics and events are divided into stable and temporal topics. The stable topics are usually the user’s interest and day-to-day discussion, which is persistent and not time-based. In contrast, temporal topics are time-based topics such as recruitment advertisements or promotions.
Sentiment Analysis (SA) cuts across different domains of Information Technology [37]. Notably, it falls within the domains of machine learning and data mining. A study by [38] described sentiment analysis as the process of extracting an opinion, an individual’s emotion, attitude, and view from either speech or text from web contents such as tweets using Natural Language Processing (NLP). They further referred to sentiment analysis as opinion mining by categorizing the public opinion as positive, negative, or neutral. Their study differentiated sentiment analysis and opinion mining as sentiment is the feelings and emotions of an individual. In contrast, an opinion is the conclusion of how someone feels about the open disputed discussions, even though the words are interchangeable. These studies [39,40] narrated that sentiment analysis and opinion mining were initially regarded as a domain of research specifically on document classification into topics back in 2000. Researchers eventually realized the need to conduct it in an external domain by expanding the knowledge to sentiment lexicon and polarity classifications. With the emergence of social media platforms, many research types have been conducted in sentiment analysis, especially Twitter.

2.3. Related Research on Nigerian Pidgin English

Studies have been conducted on informal Nigerian English (Pidgin) to cover existing gaps in various research aspects. Although not many studies have been conducted on the Pidgin NLP domain, some of the NLP studies on Pidgin are semantics, the evaluation of lexical and syntactic standards, discourse analysis, anthropomorphisms, and automated speech processing. This section presents the summary of Pidgin English studies and their limitations in relation to the NLP domain, as shown in Table 1.
A study by [41] explored Pidgin English sentiment in an intrasentential code-mixing and switching context with significant word localization. They also contributed 300 VADER (Valence Aware Dictionary for Sentiment Reasoning) lexicon compatible Nigerian Pidgin sentiment tokens and their scores and 14,000 gold standard Nigerian Pidgin tweets and their sentiment labels. Similarly, [42] performed discourse analysis by analyzing the Pidgin English discourse on collected data through participation and anonymous observations and tape recording methods, using a synthesis of methods, principles, and approaches. A study by [36] used the theory of conceptual metaphor by Lakoff and Johnson with additional input by the linguistic and social components of diction, structure, audience, and response as the theoretical framework to analyze twenty (20) Nigerian Pidgin proverbs. Their findings reveal that human attributes are given to animals such as monkeys, elephants, tortoises, and other non-human concepts such as water, kettle, and “garri” (a local dish) to explain life issues. In a related study by [43], the researchers assessed the Pidgin English lexical and syntactic standards based on an empirical study. They concluded that Nigerian Pidgin is a fully developed language with rich lexico-semantics and syntax, which have evolved like any other language through contact and modification. Samples were collected spontaneously from formal settings, such as schools, markets, churches, and private homes, to explore the lexical and syntactic standards. A separate study by [44] built language resources for the Nigerian Pidgin English Tokenisation, and also, an automatic speech system for predicting the words’ pronunciation and segmentation for Nigerian Pidgin English was built.
The various studies presented in the above table show some of the research conducted on the Nigerian Pidgin English Language on the different NLP domains. However, at the time of writing this report, only one study was found on the sentiment analysis domain, which was performed by [24], as elaborated in the previous paragraph and the above table.

2.4. COVID-19 Research

Several studies have so far been conducted on COVID-19. Although none at the time of this writing focused on public perception or sentiment on COVID-19 distributed relief aid or palliatives, some of the studies on COVID-19 are presented in Table 2.

2.5. Sentiment and Emotion Classification

A recent study by [49] described the Naïve Bayes model as one of the Bayesian Theorems used for problem classification. It is chosen over other Bayesian methods that the Bayesian Theorem is the most common and popular [50]. Naïve Bayes performs very well, particularly if it assumes that the dataset function is independent. Naïve Bayes follows a probabilistic approach to document generation. However, the Support Vector Machine is one example of a supervised learning model used to position training data points in n-dimensional space, whereby n represents the number of functions and recognizes the support vectors [51]. A similar study by [52] described the Support Vector Machine (SVM) as a linear regression classifier that can reduce the empiric model complexity and increase the geometric margin.
A recent study by [53] employed five supervised machine learning algorithms: Decision Tree, Random Forest, Support Vector Machine, Logistics Regression, and Multinomial Naïve Bayes for Sentiment Analysis of the emotions of the public on product review and recommendations in e-commerce and social media. Their findings reveal that Random Forest outperforms other classifiers, even though their recommendations illustrated that the use of polarity in the review would be better for classification and analysis of the public emotions. Another recent study by [54] uses Random Forest and Decision Tree for sentiment analysis and sentiment classifications to detect the sarcasm of the public sentiment for decision making. Their findings reveal that Random Forest yields the best result within the Decision Tree family.
K-Nearest Neighbor (KNN) is one of the algorithms that focus on classifying data that are similar or close to each other. The model works typically based on the dataset class labels and feature vectors [55]. It stores all cases and identifies new cases with a similarity measure (nearest neighbors). A separate study by [56] described Logistic Regression as a model that classifies text into various forms of sentimental emotions using training and testing. It is expected that the text belongs to the labeled sentiment class. It is also said to be one of the fastest prediction models. A study by [57] on COVID-19 pandemic public opinion and emotions compared two machine learning algorithms to discover their accuracy and performances based on the different lengths of tweets on coronavirus. Naïve Bayes and Logistic Regression were used, and their findings showed that the Naïve Bayes (NB) classifier yielded 91% accuracy for a short tweet.
In comparison, Logistics Regression performed less than NB as it scored 74% accuracy. Additionally, both of them are weak in performance on longer tweets. A study by [58] examined public sentiment on natural disaster and disaster management responses for the South Carolina Flood in 2015. The TwiSA (Twitter Situational Awareness) framework was used. Their results successfully tracked the positive and negative responses from the public. The result was recommended to be used for disaster preparedness, recovery, and responses. This study [59] investigated public opinion and sentiment regarding the coronavirus pandemic in the Philippines from January 2020 to March 2020. The findings show that the Multinomial Naïve Bayes algorithm achieved 72% accuracy. A recent study by [60] utilizes different Machine Learning (ML) algorithms to analyze and predict the trend of COVID-19 infection in the most vulnerable age group. Using the COVID-19 benchmark dataset collected from Twitter, they used: Decision Tree, Random Forest, SVM, Multi-linear Regression, XGBoost Classifier, KNN, Random Forest Regressor, and Gaussian Naïve Bayes Classifier algorithms to conduct their experiments. Their results show that the Random Forest algorithm performed better amongst other experimented algorithms.
From the reviewed literature, it is clear that despite contributions to English language Text Emotion Detection (TED) by several researchers, the in-depth analysis and categorization of Nigerian Pidgin English beyond the surface of sentiment into emotional classes such as anger, disgust, fear, joy, sadness, and surprise is yet to be conducted. As a result, this leaves out a sizable number of texts from being accurately processed due to the absence and lack of consideration for Nigerian Pidgin English vocabulary. Therefore, to accurately classify emotions on COVID-19 palliative distribution in Nigeria, this study goes beyond formal English by constructing an emotion dataset for Pidgin English into the experiments conducted.

3. Materials and Methods

3.1. Cross-Industry Standard Platform for Data Mining (CRISP-DM)

The Cross-Industry Standard Platform for Data Mining (CRISP-DM) is a popular methodology that provides a structured approach to data mining projects. Made of six processes, it focuses on understanding and accomplishing objectives or requirements of a research or project. The methodology has been used by researchers such as [25] in the construction of a Building Information Model (BIM) using a hybrid supervised machine learning algorithm (Random Forests and Simple Linear Regression) to improve the accuracy of predicting labor cost in a construction company in Taiwan. Additionally, ref. [60] applied CRISP-DM in analyzing online customer behavior, while [61] applied the CRISP-DM methodology in classifying credit on Bank Customers. The six stages of CRISP-DM are business understanding, data understanding, data preparation, modeling, evaluation, and deployment. These stages were implemented as described in Figure 1. For this study, the CRISP-DM was adopted in defining several stages and activities conducted to accomplish the objectives of this study.

3.2. Dataset Description

An improved emotion dataset named Nigerian Local English Slang-Pidgin (NLES-P) was used in this study. The dataset was improved from an existing dataset known as the Nigerian Pidgin Twitter Dataset on sentiment (positive, negative, and neutral) by [30]. The improvement was converting the three emotion classes to five emotion classes (anger, sadness, joy, fear, and disgust), organized and carefully annotated. The total number of tweets or observations for the datasets is 9803. Three independent Nigerian Pidgin English speakers conducted the annotation based on the five emotion classes. Each annotator read one tweet at a time and scored it with the appropriate emotion class. After collating the three annotators’ emotion scoring, the majority rule was applied in deciding which emotion class to assign to each tweet. More description of the dataset is provided in the sections below, i.e., pre-processing, feature transformation, and annotation. The dataset and attributes descriptions are provided in Table 3 and Table 4, respectively.
This research aims to classify public emotions from Twitter data using machine learning techniques regarding COVID-19 palliatives and relief aid packages’ distribution in Nigeria; the different target emotion classes and their corresponding number of tweets are shown in Table 5.

3.3. Data Pre-Processing

Data pre-processing is very important in creating a dataset because a noisy dataset may cause inaccuracy. Therefore, before feeding the data into the modeling stage, we need to ensure that the dataset is clean to produce an accurate machine learning model. However, a raw tweet usually comes with unwanted data such as punctuations, HTML tags, web-links, and special characters. In this regard, all the unwanted information from the collected tweets was removed. Python libraries such as Natural Language Toolkit (NTLK) and Python Regular Expression Regex were used to remove unwanted characters. These Python libraries also help in converting words in uppercase to lowercase as well as removing special characters.

3.3.1. Feature Transformation

Count Vector and Term Frequency-Inverse Document Frequency (TF-IDF) techniques were selected in this study in order to identify and transform the Twitter text into feature engineering. Count vector is also referred to as a vocabulary of words, a common encoding scheme of a given word in a document. At the same time, TF-IDF is a numerical statistic that shows how important a word is in a document from a collection of corpora. Likewise, TF-IDF is a product of TF and IDF, as shown in Equation (1).
tf − idf (t,d) = tf (t,d) × idf (t,f)
From the above equation, tf (t,d) represent the term frequency, which shows the occurrences of term t in document d, meaning how many times t took place in a document, while the idf (t,d) indicate the inverse document frequency, which can be calculated as Equation (2).
idf (t,d) = log × nd/1+df (d,t)
From the above Equation (2), nd represents the total number of the documents while the df (d,t) represents the total number of documents d that contains the term t. Meanwhile, the addition of constant 1 to the denominator is ideal and serves the function of adding a non-zero value to words that appear in all training samples. Additionally, the Log is used to prevent overweight of low document frequencies.

3.3.2. Annotation and Holdout Split

The number of labeled emotions from Table 4 indicates that the tweet instances are balanced by allocating equal tweet counts across emotion classes. This shows that the labeled data are unbiased. Three independent annotators were assigned to annotate the dataset based on the five emotions classes (identified in Table 5). The dataset was then split into 80% for training and the remaining 20% for testing. Figure 2 shows the two most relevant attributes of the dataset: Y indicates the “tweets” and X represents the “target” emotion class, which is encoded from 1 to 5. Additionally, Table 6 shows some of the verified annotated datasets.

4. Results

This section presents and analytically discusses the results obtained from the conducted experiments. The evaluation metrics used are Accuracy percentage as well as Precision, Recall, and F-measure. However, while experimenting on the algorithms, K-Nearest Neighbor (KNN) was tested with values ranging from 1–50 for the “K” values, and for searching the K value, a Grid Search was used. Meanwhile, 1–200 ranges of values were selected for “number of estimators” in testing the Random Forest (RF) classifier. Therefore, the best “K” value chosen for KNN was 33, which returns the best accuracy among the tested values. Additionally, 200 values for the “number of estimators” was the best among the tested values for RF.

4.1. Results on Different Training and Testing Splits

This experiment was carried out to analyze the dataset used in training and testing all six models on different dataset splits. Figure 3 shows the results obtained from the experiment based on the algorithms’ accuracy. The algorithms are Multinomial Naïve Bayes (MNB), Logistics Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and K-Nearest Neighbor (KNN). The dataset split was carried out in two phases, training and testing, based on the total number of observations or instances (tweet).
The above result indicates no significant difference from the experimental results found between 50 and 90 percent training on the dataset. The percentage split’s accuracy values on the range (50 to 90 percent) yielded satisfactory performance, thus promising an acceptable result when implemented on the collected dataset. Therefore, the 80:20 training and testing split was used to create the models used in this study.

4.2. Experimental Results and Comparison

This section shows the overall comparison of the experimental results obtained on the six machine learning models shown in Figure 4 and Figure 5. Support Vector Machine outperforms the remaining classifiers with the highest accuracy of 88% from the experimental results. Random Forest (RF) and Logistics Regression (LR) emerge as the second-best classifiers amongst other models with 86% percent accuracy each and at 0.87 precision, 0.86 recall, and 0.86 F1-score on a weighted average. There is a slight difference in the macro average where Logistics Regression scores 0.87 precision, 0.86 recall, and 0.87 F1-score, while Random Forest scores 0.88 on precision, 0.85 recall, and 0.86 F1-score. This shows that LR outperforms RF on the macro average of the evaluation metrics. The third overall classifier with 81% accuracy is Decision Tree, while the fourth model is Multinomial Naïve Bayes with 77% accuracy. The worst performing classifier in terms of accuracy, precision, recall, and F1-score is the K-Nearest Neighbor with 70% accuracy.
These results indicate that SVM, LR, and RF achieved higher scores on weighted and average micro results from all models than MNB, DT, and KNN. Particularly on the weighted average, SVM, LR, and RF all scored the same percentage F1-measure of 86%. On the F1-measure for macro average, even though the scores are not the same for the three highest-scoring algorithms (SVM, RF, and LR), the difference is insignificant as SVM only surpasses LR with 0.02% and RF with 0.03%.
The results shown in Figure 6 are obtained based on the five emotions across all the six models. Additionally, we utilized the F1-score to show the models’ performance on each emotion class because the F1-score is the harmonic means of precision and recall.
Based on the results from Figure 6, the following was discovered:
  • “Disgust” is the emotion labeled with the highest number from the classification performed. Four out of the six models detected it as the highest emotion class: LR, SVM, NB, and KNN. DT and RF recorded a lower emotion class, “Disgust,” compared to the rest.
  • “Joy” is among the five emotion classes which achieved the best F1-score classification in two models, DT and RF, out of the six experimented models. This means that the two models surpass SVM, LR, NB, and KNN in classifying the emotion class “Joy.”
  • The three other emotions, i.e., “Anger, Sadness, and Fear,” scored varying counts that fall between that of “Disgust” and “Joy” across the six created models.
Figure 7 below indicates the average F1-score across all six models, SVM, RF, LR, DT, NB, and KNN, on each emotion class based on percentage.
The results shown in Figure 7 above reveal that the emotion label “Disgust” surpasses the remaining four emotions with an 87% average F1-score on all six models. The second highest emotion is “Joy” with an 85% F1-score, followed by “Sadness” with 83%. The last two emotions are “Anger” and “Fear” with 81% and 73%. However, based on the results indicated above, the highest classified emotion, “Disgust”, is part of the negative emotions, which shows that the public is not happy with the COVID-19 palliative distribution in Nigeria.

4.3. Correlation Analysis

Pearson’s Product-Moment correlation analysis was conducted to determine the significance level in terms of relationship on the outcome of the different classified emotions. This is based on the outcome of the six machine learning models, as illustrated in Section 3, i.e., methodology. The result from the correlation analysis, as shown in Table 7, reveals a strong, positive correlation between “joy” and “fear”, which was statistically significant (r = 0.870, n = 6, p = 0.05).
Based on the result from Table 7, the following was discovered:
  • The significant correlation between “Joy” and “Fear” implies that the public is happy and afraid at the same time. This means people are excited about the news of palliatives’ distribution but fear that there would not be equity and transparency in the distribution process due to corruption, which negatively affects both human and social sustainability.
  • The other three classes, i.e., sadness, disgust, and anger, all have their respective scores, but no significant correlation exists between them or other classes. This is despite the majority of classification results being “disgust” (which implies that the public is not happy with the distribution process), as illustrated in Figure 7.

4.4. Word Cloud

In this research, one of the content mining approaches was used to visualize the most frequent words used in the collected Twitter data on public opinion and sentiment on COVID-19 palliatives and relief aid packages’ distribution to citizens of Nigeria. This technique is called Word Cloud, a visual or graphical representation of the information in the dataset [62]. Additionally, a word cloud shows words in large, medium, and small font sizes. The largest word that appears in a word cloud represents the most frequent word in the dataset, while the smallest represents the least frequent word. Figure 8 shows the generated word cloud from the emotions of the collected dataset.
The word frequency (word cloud) of the collected tweets, as shown in Figure 8, reveals a high occurrence of the words “suffering” and “sadness” as the major feelings of the public on the COVID-19 palliatives’ distribution.

5. Discussion

This study aims to classify public emotions regarding COVID-19 relief aid distribution to vulnerable Nigerian citizens. The classification was conducted using six machine learning algorithms on the Nigerian Local English Slangs-Pidgin (NLES-P) dataset. Data pre-processing was conducted to refine the dataset and put it in order for experimentation. Various evaluation measures, including accuracy, precision, recall, and F-measure, were employed to evaluate the algorithms’ performance. The following paragraphs answer the posed research questions and objectives of the study.
Research Question 1: How to accommodate Nigerian Pidgin English in emotion classification from social media posts?
To answer this question, the researchers mapped this question with the research objective 1 (RO1), which is “to construct and annotate the relevant Nigerian Pidgin English dataset from Twitter post”. The approach to answer the question and achieve the objective is to construct Nigerian Pidgin English Twitter posts that contain an opinion and sentiment of Nigerians regarding COVID-19 relief aid packages’ distribution. Therefore, findings from this study reveal that the dataset was constructed with five emotion classes (anger, sadness, fear, joy and disgust).
Research Question 2: How can social media data and machine learning algorithms help identify public emotions associated with the distribution of COVID-19 relief aid to vulnerable Nigerian citizens?
To answer this research question, research objective 2 (RO2) was mapped with the research question, which is “to classify the emotions of the Twitter post using Naïve Bayes, Support Vector Machine, Random Forest, Logistics Regression, K-Nearest Neighbour and Decision Tree”. In this regard, the constructed emotion dataset (NLES-P) was utilized and experimented with the above-mentioned machine learning algorithms. Additionally, the basic aim of the algorithms is to predict appropriate emotion labels, namely, Joy, Sadness, Fear, Anger and Disgust. The experimental results obtained show that all the algorithms produced acceptable results, though there are some variations amongst the algorithms. The findings from this study revealed that the research question was answered, and the research objective was also achieved.
Research Question 3: What are the performance variations of the six standard machine learning algorithms?
To answer this research question, the researchers mapped research objective 3 (RO3) with research question 3, which is “to compare the accuracy of the machine learning models using standard performance evaluation metrics (accuracy, precision, recall and f1-score)”. Despite this, the researchers conducted a number of experiments to evaluate the performance of the six machine learning classifiers with respect to the emotion classification problem, as mentioned in the previous section. The approach to answering this question is aimed at inspecting and comparing the results obtained from the previous experiments conducted across all six ML classifiers. The experimental results revealed that SVM outperforms all the remaining five models with 88% accuracy with respect to the labeled emotions. This indicated that the research question was answered, and the objective was achieved based on the results obtained from the experiment.
However, there is no available work directly related to this study, but the researchers tried to compare the results with some previous studies conducted using the International Survey on Emotion Antecedents and Reactions (ISEAR) dataset. Table 8 below shows the comparison of this study with other studies conducted using the same emotion classes (anger, sadness, joy, fear, and disgust). The results reveal that despite the utilization of non-standard English in this study, it produced better results accuracy than other similar studies conducted with Standard English.
Varying scores on performance were recorded by the machine learning models created for the emotion classification. Based on the experiment conducted, the results clearly show that the public emotions on COVID-19 relief aid package distribution in Nigeria were not satisfactory because the negative emotions expressed by the public outnumbered the public happiness. In order to avoid the lack of accountability and transparency that caused so much disgust, sadness, and anger during the COVID-19 relief aid distribution, the following actions are suggested:
  • Online and physical registration centers for the needy should be opened. The online portal can be accessed by the needy that live within cities or can access the internet. Simultaneously, the physical registration centers can attend to the needy people who do not have access to the internet and live in rural communities. This would help the government know the exact number of needy based on their different locations, and thus ensure adequate provision based on numbers rather than random or equal distribution as the number of the needy varies based on regions.
  • Accessible distribution centers can be set up nationwide to invite registered nurses to come for their support pickup. This can be scheduled in order to maintain the COVID-19 social distancing safety protocol.

Contributions of the Study

The contributions of this study are six machine learning models for the emotion classification of informal Nigerian Local English Slangs-Pidgin (NLES-P). Although the different models achieved different performance scores, each model recorded an acceptable performance score used in emotion classification. Furthermore, the models in this study can be applied in the text classification of Pidgin in countries such as Ghana, Cameroon, and Equatorial Guinea, where the same Pidgin English is widely spoken as in Nigeria. Secondly, as this research is the first to perform emotion classification on Nigerian Pidgin English, it has contributed an emotion dataset to the Nigerian Pidgin English language resources used in performing emotion classification. Thirdly, the holdout modeling approach employed by this study in creating the models for emotion classification has proven to perform efficiently well in terms of factors such as accuracy. Even though the holdout approach was not compared with another approach, such as cross-validation, the performance recorded by the holdout approach is very satisfactory. Lastly, the text analytics results for the Nigerian government on the distribution of the COVID-19 palliatives to citizens have revealed varying emotions that inform the government about the masses’ emotions.

6. Conclusions

This research focused on the reach and perceived public sentiment on emergency responses, specifically COVID-19 palliatives and relief aid packages’ distribution to vulnerable Nigerian citizens. Twitter was selected as the source for data on public opinions and sentiments. Various machine learning algorithms were employed: Multinomial Naïve Bayes, Support Vector Machine, Random Forest, Decision Tree, K-Nearest Neighbor, and Logistics Regression. These algorithms were used to classify the aggregated opinion and sentiment of the public using the Nigerian Local English Slang-Pidgin (NLES-P) dataset. Some of this research’s contributions consist of: (1) The results from this study would help the government and other organizations in resource-oriented decisions by taking into account their citizens’ needs and opinions in times of disaster management; (2) The research produced a text emotion dataset named Nigerian Local English Slang-Pidgin (NLES-P) (containing the emotion classes: anger, sadness, joy, fear, and disgust) to further facilitate research on Nigerian Pidgin and the related aspects; (3) Performance comparison was conducted on the six standard machine learning classifiers for Twitter classification regarding COVID-19 palliatives and relief aid using the NLES-P dataset. Finally, the results from the conducted experiments reveal that Support Vector Machine outperforms the other models with the highest accuracy of 88%. Likewise, “Disgust”, as one of the five emotion classes, surpasses the other emotions with an 87% average F1-score across the six experimented models. Conclusively, the overall results of all the experiments conducted indicated that the level of unhappiness from Nigerians regarding the distribution of COVID-19 palliatives and relief aid packages by the government was very high, with little positive sentiment from the public. Furthermore, the correlation analysis conducted shows a significant correlation between “Joy” and “Fear”, implying that the public is happy and afraid at the same time, which implies that people are excited about the news of palliatives’ distribution but afraid that there would not be equity and transparency in the distribution process due to corruption.

Limitations and Future Work

This study currently has some limitations that need to be improved in future works. The suggested future works are:
  • Collect more data because some of the algorithms used in this study perform well on a large dataset. Other social media such as Facebook and Instagram should be used for data collection, creating a larger dataset.
  • The majority of Nigerians express their opinions and sentiments using their native languages. This is because the country consists of three major local languages: Hausa, Igbo, and Yoruba. Future research should use these languages to help in measuring the performance of the machine learning algorithms.
  • The machine learning algorithms could be used for real-time classification using Twitter social media.
  • The type of essentials, such as food, shelter, medical support, etc., should be used as features to classify the type of essentials needed by vulnerable people. Finally, future researchers should use different parameters and feature engineering techniques to experiment on these machine learning algorithms (MNB, RF, SVM, DT, LR, and KNN) and make comparative analysis because we noticed that some of the models have different parameters and factors to make them perform better.

Author Contributions

Conceptualization, H.A.; methodology, S.L.L.; Experiments, H.A.; validation, S.L.L.; analysis, A.S.A.M.; resources, H.A.; data curation, R.H. and A.D.V.; writing—original draft preparation, R.H., A.D.V.; writing—review and editing, R.H.; N.H.A.H.M.; visualization, N.H.A.H.M.; supervision, S.L.L.; project administration, A.S.A.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded jointly by RCMO, UNIVERSITI SAINS MALAYSIA under the, grants number 1001/PKOMP/8014001 and 304/PKOMP/6315137 and Research University Grant (1001/PKOMP/8014001).

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Abdul Hameed Siddiqui. A Sustainable Society: Its Meaning and Objectives. Int. J. Res. Sci. Innov. 2018, V, 128. Available online: www.rsisinternational.org (accessed on 6 March 2021).
  2. United Nations Development Programme; Frederick S. Pardee Center for International Futures. Impact of COVID-19 on the Sustainable Development Goals: Pursuing the Sustainable Development Goals (SDGs) in a World Reshaped by COVID-19; Joseph Korbel School of International Studies, University of Denver: Denver, CO, USA, 2020. [Google Scholar]
  3. Szabo, S.; Nhau, B.; Tsusaka, T.W.; Kadigi, R.M.J.; Payne, T.; Kangile, J.R.; Park, K.S.; Couto, M.; Runsten, L.; Burgess, N.D.; et al. Towards a Successful Post COVID-19 Transition of Monitoring, Evaluation, and Learning in Complex Sustainability Science Research-to-Policy Projects. Sustainability 2021, 13, 387. [Google Scholar] [CrossRef]
  4. Rutkowska, A.; Kacperak, K.; Rutkowski, S.; Cacciante, L.; Kiper, P.; Szczegielniak, J. The Impact of Isolation Due to COVID-19 on Physical Activity Levels in Adult Students. Sustainability 2021, 13, 446. [Google Scholar] [CrossRef]
  5. Vaz, E. COVID-19 in Toronto: A Spatial Exploratory Analysis. Sustainability 2021, 13, 498. [Google Scholar] [CrossRef]
  6. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). Available online: https://coronavirus.jhu.edu/map.html (accessed on 8 January 2021).
  7. Nigeria Centre for Disease Control. NCDC Coronavirus COVID-19 Microsite. Covid19.Ncdc.gov.ng; Nigeria Centre for Disease Control (NCDC). Available online: https://covid19.ncdc.gov.ng/ (accessed on 6 February 2021).
  8. Kemp, S. Digital 2020: Nigeria. DataReportal—Global Digital Insights. Available online: https://datareportal.com/reports/digital-2020-nigeria (accessed on 7 February 2021).
  9. Qiu, M.; Sha, J.; Utomo, S. Listening to Forests: Comparing the Perceived Restorative Characteristics of Natural Soundscapes before and after the COVID-19 Pandemic. Sustainability 2021, 13, 293. [Google Scholar] [CrossRef]
  10. Tetrevova, L.; Vavra, J.; Munzarova, S. Communication of Socially-Responsible Activities by Higher Education Institutions. Sustainability 2021, 13, 483. [Google Scholar] [CrossRef]
  11. Marinello, S.; Lolli, F.; Gamberini, R. The Impact of the COVID-19 Emergency on Local Vehicular Traffic and Its Consequences for the Environment: The Case of the City of Reggio Emilia (Italy). Sustainability 2021, 13, 118. [Google Scholar] [CrossRef]
  12. Human Right Watch. Available online: https://www.hrw.org/news/2020/04/14/nigeria-protect-most-vulnerable-covid-19-response (accessed on 24 September 2020).
  13. Ricciardelli, A. Governance, Local Communities, and Citizens Participation. In Global Encyclopedia of Public Administration, Public Policy, and Governance; Farazmand, A., Ed.; Springer: Cham, Switzerland, 2018. [Google Scholar] [CrossRef]
  14. Loshin, D. Text Data Analytics: In Service of Smart Government. Available online: https://papers.govtech.com/Text-Data-Analytics-In-Service-of-Smart-Government-49173.html (accessed on 16 March 2021).
  15. Barns, S. Smart cities and urban data platforms: Designing interfaces for smart governance. City Cult. Soc. 2018, 12, 5–12. [Google Scholar] [CrossRef]
  16. Kumar, A.; Sharma, A. Systematic literature review on opinion mining of big data for government intelligence. Webology 2017, 14, 6–47. [Google Scholar]
  17. Lennerholt, C.; van Laere, J.; Söderström, E. Implementation Challenges of Self Service Business Intelligence: A Literature Review. In Proceedings of the 51st Hawaii International Conference on System Sciences, Waikoloa Village, HI, USA, 3 January 2018; Volume 9, pp. 5055–5063. [Google Scholar] [CrossRef] [Green Version]
  18. Khan, K.; Baharudin, B.; Khan, A.; Ullah, A. Mining opinion components from unstructured reviews: A review. J. King Saud Univ. Comput. Inf. Sci. 2014, 26, 258–275. [Google Scholar] [CrossRef] [Green Version]
  19. Digital Users Worldwide | Statista. Available online: https://www.statista.com/statistics/617136/digital-population-worldwide/ (accessed on 7 October 2020).
  20. Joshi, S.; Deshpande, D. Twitter Sentiment Analysis System. Int. J. Comput. Appl. 2018, 180, 35–39. [Google Scholar] [CrossRef]
  21. Varrella, S. Nigeria: Leading Social Media Platforms. Statista. Available online: https://www.statista.com/statistics/1176101/leading-social-media-platforms-nigeria/ (accessed on 7 February 2021).
  22. BBC News. Pidgin—West African Lingua Franca. BBC News. 16 November 2016. Available online: https://www.bbc.com/news/world-africa-38000387 (accessed on 7 February 2021).
  23. Desai, R.D. Sentiment Analysis of Twitter Data. In Proceedings of the 2nd International Conference on Intelligent Computing and Control Systems, ICICCS, Madurai, India, 14–15 June 2018; pp. 114–117. [Google Scholar] [CrossRef]
  24. Huang, C.H.; Hsieh, S.H. Predicting BIM labor cost with random forest and simple linear regression. Autom. Constr. 2020, 118, 103280. [Google Scholar] [CrossRef]
  25. Reddy, D.M. Twitter Sentiment Analysis using Distributed Word and Sentence Representation. 2019. Available online: http://arxiv.org/abs/1904.12580 (accessed on 12 November 2020).
  26. Meng, L.; Dong, Z.S.; Christenson, L.; Fulton, L. Mining Public Opinion on Twitter about Natural Disaster Response Using Machine Learning Techniques. 2017. Available online: https://arxiv.org/ftp/arxiv/papers/2005/2005.07019.pdf (accessed on 17 October 2020).
  27. Squicciarini, A.; Tapia, A.; Stehle, S. Sentiment analysis during Hurricane Sandy in emergency response. Int. J. Disaster Risk Reduct. 2017, 21, 213–222. [Google Scholar] [CrossRef] [Green Version]
  28. Rathee, N.; Joshi, N.; Kaur, J. Sentiment Analysis Using Machine Learning Techniques on Python. In Proceedings of the 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India, 14–15 June 2018. [Google Scholar] [CrossRef]
  29. Kanish, S.; Henil, P.; Devanshi, S.; Manan, S. A Comparative Analysis of Logistic Regression, Random Forest and KNN Models for the Text Classification. Augment. Hum. Res. 2020, 5, 1. [Google Scholar] [CrossRef]
  30. Oyewusi, W.F.; Adekanmbi, O.; Akinsande, O. Semantic Enrichment of Nigerian Pidgin English for Contextual Sentiment Classification. arXiv 2020, arXiv:2003.12450v1. Available online: http://arxiv.org/abs/2003.12450?utm_source=researcher_app&utm_medium=referral&utm_campaign=RESR_MRKT_Researcher_inbound (accessed on 23 September 2020).
  31. Suh, A.; Li, M. Digital Tracing during the COVID-19 Pandemic: User Appraisal, Emotion, and Continuance Intention. Sustainability 2021, 13, 608. [Google Scholar] [CrossRef]
  32. Manguri, K.N.; Ramadhan, R.; Mohammed, A.P. Twitter Sentiment Analysis on Worldwide COVID-19 Outbreaks. Kurd. J. Appl. Res. 2020, 54–65. [Google Scholar] [CrossRef]
  33. Bento, A.I.; Thuy, N.; Coady, W.; Felipe, L.R.; Yong, Y.A.; Kosali, S. Evidence from internet search data shows information-seeking responses to news of local COVID-19 cases. Natl. Acad. Sci. 2020, 117, 11220–11222. [Google Scholar] [CrossRef]
  34. Hasan, A.; Moin, S.; Karim, A.; Shamshirband, S. Machine Learning-Based Sentiment Analysis for Twitter Accounts. Math. Comput. Appl. 2018, 23, 11. [Google Scholar] [CrossRef] [Green Version]
  35. Öztürk, N.; Ayvaz, S. Sentiment analysis on Twitter: A text mining approach to the Syrian refugee crisis. Telemat. Inform. 2018, 35, 136–147. [Google Scholar] [CrossRef]
  36. Yin, H.; Cui, B.; Lu, H.; Huang, Y.; Yao, J. A unified model for stable and temporal topic detection from social media data. In Proceedings of the International Conference Data Engineering, Brisbane, Australia, 8–11 April 2013; pp. 661–672. [Google Scholar] [CrossRef] [Green Version]
  37. Sidarenka, U. Sentiment Analysis of German Twitter. Ph.D. Thesis, Universität Potsdam, Berlin, Germany, 2019. [Google Scholar] [CrossRef]
  38. Sonawane, S.S. Sentiment Analysis of Twitter Data: A Survey of Techniques. Int. J. Comput. Appl. 2016, 139, 5–15. [Google Scholar] [CrossRef]
  39. Nakov, P.; Ritter, A.; Rosenthal, S.; Sebastiani, F.; Stoyanov, V. SemEval-2016 task 4: Sentiment analysis in twitter. In Proceedings of the SemEval 2016—10th International Workshop on Semantic Evaluation, San Diego, CA, USA, 16–17 June 2016; pp. 1–18. [Google Scholar] [CrossRef]
  40. Chakriswaran, P.; Vincent, D.R.; Srinivasan, K.; Sharma, V.; Chang, C.Y.; Reina, D.G. Emotion AI-driven sentiment analysis: A survey, future research directions, and open issues. Appl. Sci. 2019, 9, 462. [Google Scholar] [CrossRef] [Green Version]
  41. Balogun, T.A. In defense of Nigerian pidgin. J. Lang. Cult. 2013, 4, 90–98. [Google Scholar] [CrossRef]
  42. Osoba, J.B. Analysis of Discourse in Nigerian Pidgin. J. Univers. Lang. 2015, 16, 131–159. [Google Scholar] [CrossRef] [Green Version]
  43. Idegbekwe, D. Anthropomorphisms and the Nigerian Pidgin Proverbs: A Linguistic Conceptual Metaphorical Analysis. EBSU J. Soc. Sci. Rev. 2020, 10, 71–76. [Google Scholar]
  44. Bigi, B.; Caron, B.; Abiola, O. Developing Resources for Automated Speech Processing of the African Language Naija (Nigerian Pidgin). In Proceedings of the 8th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznan, Poland, 10 November 2017; pp. 441–445. [Google Scholar]
  45. Sung, Y.A.; Kim, K.W.; Kwon, H.J. Big Data Analysis of Korean Travelers’ Behavior in the Post-COVID-19 Era. Sustainability 2021, 13, 310. [Google Scholar] [CrossRef]
  46. Zhao, F.; Zhu, N.; Hämäläinen, J. Protection of Children in Difficulty in China during the COVID-19 Pandemic. Sustainability 2021, 13, 279. [Google Scholar] [CrossRef]
  47. Radulescu, C.V.; Ladaru, G.R.; Burlacu, S.; Constantin, F.; Ioanăș, C.; Petre, I.L. Impact of the COVID-19 Pandemic on the Romanian Labor Market. Sustainability 2021, 13, 271. [Google Scholar] [CrossRef]
  48. Awwalu, J.; Umar, N.A.; Ibrahim, M.S.; Nonyelum, O.F. A multinomial Naïve Bayes decision support system for COVID-19 detection. FUDMA J. Sci. 2020, 4, 704–711. [Google Scholar] [CrossRef]
  49. Kaklamanis, M.M.; Filippakis, M.; Touloupos, M.; Christodoulou, K. An experimental comparison of machine learning classification algorithms for breast cancer diagnosis. In Proceedings of the 16th European, Mediterranean, and Middle Eastern Conference, EMCIS 2019, Dubai, United Arab Emirates, 9–10 December 2019; Springer: Cham, Switzerland, 2020; Volume 381, pp. 18–30. [Google Scholar] [CrossRef]
  50. Jianqiang, Z.; Xiaolin, G. Comparison research on text pre-processing methods on twitter sentiment analysis. IEEE Access 2017, 5, 2870–2879. [Google Scholar] [CrossRef]
  51. Sahoo, D.; Liu, C.; Hoi, S.C.H. Malicious URL Detection using Machine Learning: A Survey. arXiv 2017, arXiv:1701.07179. [Google Scholar]
  52. Rohini, J.; Ambesh, B.; Animesh, S.; Abhishek, K.S. A Survey on Various Approaches for Sentiment Analysis and Performance Optimization. Int. J. Eng. Res. Technol. 2017, 6, 716–720. [Google Scholar] [CrossRef]
  53. Khanvilkar, G.; Deepali, V.P. Sentiment Analysis for Product Recommendation Using Random Forest. Int. J. Eng. Technol. 2018, 7, 87. [Google Scholar] [CrossRef] [Green Version]
  54. Joshi, A.M.; Prabhune, S. Random forest: A hybrid implementation for sarcasm detection in public opinion mining. Int. J. Innov. Technol. Explor. Eng. 2019, 8, 5022–5025. [Google Scholar] [CrossRef]
  55. Pathan, M.; Patel, N.; Yagnik, H.; Shah, M. Artificial cognition for applications in smart agriculture: A comprehensive review. Artif. Intell. Agric. 2020, 4, 81–95. [Google Scholar] [CrossRef]
  56. Varathan, K.D.; Anastasia, G.; Crestani, F. Comparative Opinion Mining: A Review. J. Assoc. Inf. Sci. Technol. 2017, 64, 811–829. [Google Scholar] [CrossRef] [Green Version]
  57. Samuel, J.; Ali, G.G.; Rahman, M.M.; Esawi, E.; Samuel, Y. COVID-19 public sentiment insights and machine learning for tweets classification. Information 2020, 11, 314. [Google Scholar] [CrossRef]
  58. Karami, A.; Shah, V.; Vaezi, R.; Bansal, A. Twitter speaks: A case of national disaster situational awareness. J. Inf. Sci. 2020, 46, 313–324. [Google Scholar] [CrossRef] [Green Version]
  59. Delizo, J.P.D.; Abisado, M.B.; De Los Trinos, M.I.P. Philippine twitter sentiments during COVID-19 Pandemic using Multinomial Naïve Bayes. Int. J. Adv. Trends Comput. Sci. Eng. 2020, 64, 408–412. [Google Scholar]
  60. Karisani, N.; Karisani, P. Mining Coronavirus (COVID-19) Posts in Social Media. arXiv 2020, arXiv:2004.06778. [Google Scholar]
  61. Emil, E.; Jozef, B. Analysis of Online Consumer Behavior—Design of CRISP-DM Process Model. Agris On-Line Pap. Econ. Inform. 2020, 9, 13–22. [Google Scholar]
  62. World Bank. Population, total—Nigeria | Data.; The World Bank Group. Available online: https://data.worldbank.org/indicator/SP.POP.TOTL?locations=NG (accessed on 6 February 2021).
  63. Kabir, A.I.; Karim, R.; Newaz, S.; Hossain, M.I. The power of social media analytics: Text analytics based on sentiment analysis and word clouds on R. J. Inform. Econ. 2018, 22, 25–38. [Google Scholar] [CrossRef]
  64. Danisman, T.; Alpkocak, A. Feeler: Emotion classification of text using vector space model. Conv. Commun. Interact. Soc. Intell. 2008, 1, 53. [Google Scholar]
  65. Thomas, B.; Vinod, P.; Dhanya, K.A. Multiclass emotion extraction from sentences. Int. J. Sci. Eng. Res. 2014, 5, 12–15. [Google Scholar]
Sustainability 13 03497 g001 550
Figure 1. Research Conceptual Approach.
Figure 1. Research Conceptual Approach.
Sustainability 13 03497 g001
Sustainability 13 03497 g002 550
Figure 2. Features of the tweets.
Figure 2. Features of the tweets.
Sustainability 13 03497 g002
Sustainability 13 03497 g003 550
Figure 3. Dataset Training and Testing Split.
Figure 3. Dataset Training and Testing Split.
Sustainability 13 03497 g003
Sustainability 13 03497 g004 550
Figure 4. Models comparison on weighted average.
Figure 4. Models comparison on weighted average.
Sustainability 13 03497 g004
Sustainability 13 03497 g005 550
Figure 5. Models comparison on macro average.
Figure 5. Models comparison on macro average.
Sustainability 13 03497 g005
Sustainability 13 03497 g006 550
Figure 6. Result of Emotion Classification.
Figure 6. Result of Emotion Classification.
Sustainability 13 03497 g006
Sustainability 13 03497 g007 550
Figure 7. Average F1-score across each model on emotion classes based on the percentage.
Figure 7. Average F1-score across each model on emotion classes based on the percentage.
Sustainability 13 03497 g007
Sustainability 13 03497 g008 550
Figure 8. Public opinion and sentiment (word frequency).
Figure 8. Public opinion and sentiment (word frequency).
Sustainability 13 03497 g008
Table
Table 1. Summary of Related Natural Language Processing (NLP) Studies on Nigerian Pidgin English.
Table 1. Summary of Related Natural Language Processing (NLP) Studies on Nigerian Pidgin English.
ResearchResearch DomainLimitations
Semantic Enrichment of Nigerian Pidgin English for Contextual Sentiment Classification [30]Sentiment Analysis The study was limited to sentiment classes, which are: positive, negative, and neutral. This means that it did not go deeper by classifying emotions in the collected text.
In defense of Nigerian pidgin [40]Lexical and Syntactic AnalysisThe study is limited to exploring the lexical and syntactic standards of Pidgin English, thus, leaving out any form of emotion classification.
Analysis of Discourse in Nigerian Pidgin [41]Discourse AnalysisIt is limited to discourse analysis only—no emotion classification.
Anthropomorphisms and the Nigerian Pidgin Proverbs: A Linguistic Conceptual Metaphorical Analysis [42] Metaphorical Analysis The study is limited to anthropomorphisms—no emotion classification of any kind.
Developing Resources for Automated Speech Processing of the African Language Naija (Nigerian Pidgin) [43]Speech ProcessingIt is limited to speech processing only—no emotion classification.
Table
Table 2. Related Studies on COVID-19.
Table 2. Related Studies on COVID-19.
ResearchDescriptionFindingsRelationship with This Study
Big Data Analysis of Korean Travelers’ Behavior in the Post-COVID-19 Era [45]To explore changes in Korean tourism behavior after COVID-19. Keywords were extracted from Korean social media data that were collected from the World Health Organization’s declaration of a public health emergency of international concern, from January 2020 to August 2020; the analyses conducted on the data are centrality analysis, CONCOR (CONvergent CORrelation) analysis, and semantic system network analysis.The study discovered that Korean tourists have increased their domestic travel preference and showed interest in government subsidies.The study by [45] relates to this study in terms of the nature of data used. Although their contexts are different, they both share the similarity of data from social media on COVID-19.
Protection of Children in Difficulty in China during the COVID-19 Pandemic [46]To investigate the resilience of the child protection system in China as it responds to the special needs of children in difficulty under the circumstances caused by the COVID-19 pandemic. Adopting the multiple mutual proof methodology, data were collected using document analysis, interviews, and a case study of 14 children.Results indicate that there are good policies in Chinese child protection services, but implementation is complicated by the organizational and functional fragmentation, thus necessitating the need for the development of bottom-up practices.Although this study is on COVID-19 relief aid distribution, the study by [46] is on children protection during the COVID-19 pandemic; both studies have the relationship of welfare and COVID-19 center- point and era during which both studies were conducted.
Impact of the COVID-19 Pandemic on the Romanian Labor Market [47]To assess the impact of the COVID-19 pandemic on the Romanian labor market. Data were collected by administering a questionnaire, while semantic differential and ordering rank methods were used to interpret the results from collected data.Findings reveal that the respondents claimed to have maintained a similar income, but in the event of changing jobs, they would consider it very important for the new employer to ensure the provision of adequate measures for preventing and combating COVID-19, as well as better health insurance.Unlike the study by [46] and [48], the study by [47] focused on the labor market, where it assessed COVID-19’s impact on workers. However, it relates to this study in terms of ordering and ranking methodology. This study ranks detected emotions based on counts, while [47] ranks user responses from the questionnaire.
A multinomial Naïve Bayes decision support system for COVID-19 detection [48]Introduced an interactive Artificial Intelligent web system using the Multinomial Naïve Bayes algorithm to detect warning COVID-19 symptoms and provide medical suggestions.Performance evaluation of the Multinomial Naïve Bayes experiments shows reliable accuracy on detection.Despite the difference in the primary aim of both studies, where [48] aims at healthcare while this study aims at public emotions on government welfare distribution, the relationship between the study by [48] and this study focuses on COVID-19 and usage of the Multinomial Naïve Bayes algorithm.
Digital Tracing during the COVID-19 Pandemic: User Appraisal, Emotion, and Continuance Intention [30]Examined the interplay among cognitive appraisal (threats and opportunities) and its association with user continuance intention and emotion on COVID-19 contact tracing apps by collecting data from 506 of the app users.The findings show that when users experience loss emotions, such as disgust, anger, and frustration, they are no longer willing to continue using the apps anymore. Both studies are on emotion classification on the impacts of COVID-19 on human lives.
Table
Table 3. Dataset Description.
Table 3. Dataset Description.
PropertiesNLES-P Dataset Description
Dataset CharacteristicMultivariate
Attribute CharacteristicMultiple
Missing ValuesNone
Number of Instances9803
Number of Attributes3
Table
Table 4. Attributes Description.
Table 4. Attributes Description.
NameTypeDescription
TweetsTextContents of the Tweet
TargetIntegerLabeled emotion classes from 1–5
LabelTextEmotion Class
Table
Table 5. Labelled Emotion Classes.
Table 5. Labelled Emotion Classes.
LabelEmotionTweets
1Anger1961
2Sadness1959
3Joy1962
4Fear1960
5Disgust1961
Table
Table 6. Cross-section of the annotated NLES-P dataset.
Table 6. Cross-section of the annotated NLES-P dataset.
S/No.Tweet (Nigerian Pidgin English)Tweet (Standard English)Emotion Label
1no mata wia you dey go juss no forget road wey go lead you go house back. This coronavirus na scam.No matter where you are going, make sure you do not forget your way back home. This coronavirus is a scam.Disgust
2na lie say FG dey share covid palliatives. I dey here dey wait for una infor but nothing nothing.It is a lie to say the government is sharing palliatives. I would be here waiting for information from you, but there is actually nothing.Anger
3na you talk dis one ooo. Anyway sha, nothing spoil. The pesin wey sell monkey buy dog, na still animal wey dey use nyash siddon na im the pesin buy. Make we jollificate with FG for their support.That is your opinion, but no problems. It still remains the same. Let us celebrate with FG (Federal Government) for their support.Joy
4I be sad oooo, as di info dey go round wia una say corona support dey go pesin hand, na wia u don see pesin who collect am.I am sad because of the information you circulated saying that corona support has been distributed. Where have you seen anyone, who has collected it?Sadness
5di way government dey pass una news for this covid make pesin fear to believe them ooo. Which kind wahala be dis na.The way the government is passing information on covid makes them afraid of believing them. What kind of stress is this?Fear
Table
Table 7. Pearson R Correlation Analysis between Emotions.
Table 7. Pearson R Correlation Analysis between Emotions.
AngerDisgustFearJoySadness
AngerPearson Correlation1−0.159−0.094−0.4820.120
Sig. (2-tailed) 0.7640.8590.3330.821
N66666
DisgustPearson Correlation−0.1591−0.712−0.357−0.168
Sig. (2-tailed)0.764 0.1120.4880.750
N66666
FearPearson Correlation−0.094−0.71210.870 *−0.554
Sig. (2-tailed)0.8590.112 0.0240.254
N66666
JoyPearson Correlation−0.482−0.3570.870 *1−0.735
Sig. (2-tailed)0.3330.4880.024 0.096
N66666
SadnessPearson Correlation0.120−0.168−0.554−0.7351
Sig. (2-tailed)0.8210.7500.2540.096
N66666
* Correlation is significant at the 0.05 level (2-tailed).
Table
Table 8. Comparison with similar emotion classes with different datasets.
Table 8. Comparison with similar emotion classes with different datasets.
AuthorsClassifier with Best ResultsExperimental Results
Accuracy (%)Precision (%)Recall (%)F-Score (%)
[63]Naïve Bayes65.1575959
[64]Naïve Bayes64.23616262
[65]Logistic Regression67676666.5
This studySVM88908889
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Adamu, H.; Lutfi, S.L.; Malim, N.H.A.H.; Hassan, R.; Di Vaio, A.; Mohamed, A.S.A. Framing Twitter Public Sentiment on Nigerian Government COVID-19 Palliatives Distribution Using Machine Learning. Sustainability 2021, 13, 3497. https://doi.org/10.3390/su13063497

AMA Style

Adamu H, Lutfi SL, Malim NHAH, Hassan R, Di Vaio A, Mohamed ASA. Framing Twitter Public Sentiment on Nigerian Government COVID-19 Palliatives Distribution Using Machine Learning. Sustainability. 2021; 13(6):3497. https://doi.org/10.3390/su13063497

Chicago/Turabian Style

Adamu, Hassan, Syaheerah Lebai Lutfi, Nurul Hashimah Ahamed Hassain Malim, Rohail Hassan, Assunta Di Vaio, and Ahmad Sufril Azlan Mohamed. 2021. "Framing Twitter Public Sentiment on Nigerian Government COVID-19 Palliatives Distribution Using Machine Learning" Sustainability 13, no. 6: 3497. https://doi.org/10.3390/su13063497

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop