Arabic Sentiment Analysis of YouTube Comments: NLP-Based Machine Learning Approaches for Content Evaluation

Round 1

Reviewer 1 Report

In this paper, the authors presented a data collection, pre-processing and modeling framework of sentiment analysis on Arabic YouTube comments. A total of six ML models were trained and benchmarked.

 

Suggestions and Questions:

1.     Given the popularity and recent development in the NLP field (RNN, transformer based neural network models, large language models), the authors decided to apply more traditional approach for sentiment analysis and classification. If the main purpose of this study is to benchmark the models for sentiment analysis and demonstrate the state-of-the-art performance, I suggest the author performing additional analysis by comparing transformer-based NLP models as well, such as leveraging the pretrained Arabic BERT and multilingual BERT (and finetune them for sentiment analysis task with the YouTube data in this study).

2.     Table 1, there are issues with the last 2 rows. Based on Line 114 to 128, Ombabi et al. study should correspond to the Google Play, App store data and KNN. Hadwan et al. study should correspond to the FastText-CNN-LSTM.

3.     Figures are quite blurry, please improve the resolution. Especially Figure 1, 7, 8, 9, 10, the text in the images is quite difficult to read.

4.     Figure 1, the diagram covers data collection, pre-processing and modeling, thus, the title is inaccurate.

5.     The information of Figure 1-3 is repeated. I suggest improving Figure 1 by adding the name of stages, and removing Figure 2 and 3.

6.     Section 3.2, I suggest the authors performing additional analysis to evaluate the reliability of agreement among the raters, such as reporting the Fleiss' kappa.

7.     Based on the description of Line 156-160, the original data seems to be relatively balanced. What’s the justification to perform underdamping to create a perfectly balanced data set?

8.     Table 2 is unnecessary, as the information is already clearly presented in Line 160.

9.     Section 3.3.2, please provide details on what script/software package you used to perform normalization.

10.  The order of section 3.3.3 and 3.3.4 is not the same as the order of pre-processing steps in Figure 1 and 3. Did the author remove stop words first or tokenize first? Please clarify and make the orders consistent.

11.  Section 3.4, what is the size of your feature set? i.e., how many N-grams did you use as features?

12.  The authors comprehensively described the ML models used in this study in section 3.5 (3 pages). As this is not a review paper, I suggest keep the model descriptions brief.

13.  Formula (1) is incorrect

14.  Formula (8) and (9), the left hand of the equation should be Recall and F1-Score.

15.  Section 5, did the authors perform feature selection or regularization for the ML models?

16.  The NB model performance is slightly lower when TF-IDF features are included, which is inconsistent with other ML models. Do the authors have a hypothesis on why this is the case?

17.  Figure 9 and 10 are not very informative for non-Arabic readers. Please consider modify these figures.

18.  What’s the difference between Figure 8 and 11? Figure 11 is not necessary as the information has been included in Figure 8.

19.  Figure 12 is difficult to read due to many points gathered in the low word count region, I suggest changing the figure to a boxplot or violin plot.

Minor editing of English language required.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Overall, this article is fairly organized and informative on the given topic. I would like to provide some personal suggestions for the author for improving the article if the author sees fit. The purpose of my suggestions is to help readers to get the essentials of this article sooner and easier. Even I read it through, I might be missing some points the author made

1.  Title: It could be more specific about what the author found about Arabic sentiment analysis, The author mentioned “ML,” but it seems not to be the focus of this article. There were not much talking about how the perspective of ML for Arabic sentiment analysis

2. Purpose of this article: I would assume that this article is revised from a thorough literature review for a proposed empirical study. Some sentences and paragraphs might not be necessary, and they need to be either removed or revised. And some claims could be over-promising, and this aim could be problematic since the discussion and conclusion could only provide a partial understanding of ML in Arabic sentiment Analysis.

3. Please narrow down on performance analysis section to be more details and clear description

4. Provide a section: "DISCUSSION" to deeply discuss your findings/performance analysis. this section is your critical thinking and proposed an idea

5. The systematics does not reflect the research report. Add Sections / sub-sections titles that are necessary.

6. Provide your research framework. As it requires as arguments that are supported by references and findings of your research

7. Demonstrate the problem statement at least in the abstract.

8. Emphasis on clarifying the objectives of the research.

9. lack the methodology to achieve the objectives of the research.  Therefore, the imperative is to choose a suitable methodology to achieve the research objectives.

10. Proofread your work

 

 

Proofread your work

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

The abstract needs quantification. section 1 needs to be improved. Table 1 enhanced with more previous papers. no need for theory of SVM NB, regression, DT, RF classifiers. Statistical tests may be included. Please check the equation for equation 8 and 9. Why entropy why not Gini index? MCC and Kappa may be included. The conclusion may be modified.

NIL

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

I thank the authors for making efforts to address all my comments. The revision improves with respect to all issues I have raised in my review. The following are some minor suggestions:

 

Line 397 and 399 “when using False TF-IDF”, please consider rephrase it to “without using TF-IDF”

 

Regarding my comments #7, my take of the major pros of creating a balanced data set is to mitigate the tendency of ML models to predict the samples to be majority class. However, in this study, the data set is relatively balanced. It less likely that the model will bias towards the majority class. As a result, I don’t see a need to performing under-sampling to create a perfectly balanced data set. In addition, there are also cons: when you perform down-sampling, some samples are removed from the data and there will be information loss. I acknowledge that it is not a major issue, but I would suggest avoid using the words, such as “it is important to ensure a balanced dataset for the next stage.” (Line 174)

Minor editing of English language required

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

All the corrections are included and there is no need for further review for this paper.

NIL

Author Response

Thank you, we sincerely appreciate your acknowledgment of the corrections we made. Your insightful feedback has been valuable in refining our manuscript. Thank you for your time and thorough review.