Machine Learning-Based Text Classification Comparison: Turkish Language Context

Round 1

Reviewer 1 Report

I think the subject of the manuscript is worthy of investigation and is appropriate for the journal. The authors used data obtained from customers' inquiries to train Support Vector Machine, Nave Bayes, Long Term-Short Memory, Random Forest, and Logistic Regression models and identified Long-Term-Short Memory model as the most effective technique in terms of accuracy. The authors also conducted extensive literature review on NLP and classification methods. 

The revealed mistakes do not allow a presentation of the manuscript in its current form. In this regard, I recommend that the authors revise the manuscript in response to the comments. 

Common comments 

Line 47, what is Nave Bayes? 

Line 72, SVM, NB, LTSM, RF, and LR cannot be used to train the dataset.  

The variables in equation 2, 4 must be defined. 

Line 378, “data training” is incorrect term. 

There is no need for such a detailed description of the python libraries. 

All variables should be formatted and defined correctly. 

Line 439, again, data and texts cannot be trained. 

Section 4.2 almost duplicates Section 4.1. 

What is the meaning of bigram and trigram data comparison for the approach? 

What are the units of measurement of y-axis in Figures 4-7?  

Line 567, what is the raining loss? 

Precision, F1-score, and recall are three metrics not two. 

Equation 9 is not correctly written. 

Figure 9, one variable “model” cannot be used for three different models. The first two will be lost. 

It would be appropriate to indicate the time spent on data preparation. Because a size of the prepared data is smaller that is why it takes shorter training. 

Line 627, “to learn an algorithm” is a wrong statement. An algorithm cannot be learned. 

What are the units of measurement of all axis in Figures 11-12?  

 

English needs improvements. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

First of all, congratulations on submitting the manuscript to the MDPI journal. To improve the quality of the manuscript the following suggestions should be taken into account:
1) The title of the manuscript needs to be more specific, lead to the data type, or something. Now it is a very general and typical topic, like many others.
2) In the abstract, some aspects or facts of novelty should be highlighted. Now I can see that authors used typical classification algorithms to classify Turkish texts.
3) In the abstract I can see listed some algorithms, in keywords just some of them are listed. In my opinion, no need to list methods that could be more representative keywords. By the way, the last keyword with a mistake should be Random Forest (authors: Ransom).
4) In my opinion, Table 1 is not necessary. The authors mentioned the abbreviations in the text, so why does it need to list in the table one more time? Also, the novelty needs to be highlighted in the Introduction as well. The authors could write something about the uniqueness of the Turkish language, the reasons why existing models do not fit their problem, or something other unique compared to the other research.
5) One of the most used nowadays methods is transformers, but the authors do not use it and do not mention it in the analysis as well. I believe the results would be more accurate even using multilanguage BERT if specific models for Turkish do not exist.
6) The title of section 3 is the proposed model, but the authors not proposed a model but just used typical steps of the training process. The title should be changed.
7) Is Something wrong in Figure 12 a), how the y-axis can be from 1-4?
8) The manuscript was described well, but the authors just performed classification tasks using the Turkish language without suggesting at least one or other novelty aspect.  The experimental investigation used classical algorithms, so at least transformers models need to be used and results compared with obtained results of this research.
9) How the dataset has been split into training/testing subsets. Does the hyperparameters optimization has been used? If the experimental investigation is typical, in this case, more attention could be taken to dataset pre-processing, parameters selection, etc.
10) Authors need to read their manuscript closely and fix many minor mistakes.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper presents a research where different ML techniques are used to classify Turkish documents. References are contemporary. The appliced methods are also contemporary and described in the detail. 

However, there is a lack of description of the dataset and and the classification purpose. Since I do not understand Turkish I can not judge the data presented in Figures 4-7. To simplify my remarks, I ask authors to describe in text the dataset used, classes into which text is going to be classified, how classes are determined and why is clastering needed. The method is well described but I do not understand what kind of classification is done. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Some suggestions should still have to be taken into account:
1) In some cases LSTM is a technique, in others - the algorithm, authors need to use the same definition.
2) Need to decide if it is Naive Bayes or Naïve Bayes.
3) In my opinion the keywords still are not good enough, maybe it should be: text classification, Turkish texts, machine learning, text preprocessing, and algorithms effectiveness.
4) Need to decide if it is K-MEANS or k-means. Still may same kind of minor mistakes.
5) Some methods are described much better, using some graphical presentation, etc., and some just short description is given. In my opinion, the authors do not change anything in the algorithms, just used them, so it would be enough just to mention the algorithms, without using deep descriptions. It is not necessary in the field of ML and text mining, everybody uses these algorithms too. But authors need to decide by them self it is ok or not. It is just a recommendation.
6) Need to decide if "word" or 'word'. Need to use the right symbol.
7) In Figure 5, the x-axis is not a term, is n-grams or something. Same in the rest of the Figures, except in Figure 4.
8) Authors need to choose the same formation for numbers as well. For example, 256000 or 256,000. The difference is in the text, the Figures, etc.
9) The quality of the Figures should be the same and in some cases need to improve, its blurring.
10) In Figure 12 it can be observed that the authors overfit the training of the model, so maybe the results are not so good.
11) I am not sure it is necessary to present the code of the research, it is just the implementation of standard algorithms. In my opinion, authors should review the manuscript and leave just the most important information, because the manuscript is 27 pages about natural processes in text mining. The only novelty according to the authors is Turkish language research. It is an applied science journal, so it is an application, so can't judge critically. But need to improve the presentation of the manuscript.

Good luck with the final submission.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf