Utilising Artificial Intelligence to Turn Reviews into Business Enhancements through Sentiment Analysis

Round 1

Reviewer 1 Report

Electronics – 2643389

I think the paper tells a good story, I would love to see the authors to address the following issues before it get published:

1) I will start with the major concern. From line 331 – 338, the authors mentioned that they use MonkeyLearn to get their positive, neutral, negative labels. Any academia references for this monkeyLearn, such as describing what the technical detail of their algorithm? How confident we are for those positive, neutral, negative feeling labels. Any paper/document describe the accuracy of their algorithm ? 

2) in line 21 and 447 , the authors mentioned industry 5.0. It will be great the authors can give a brief one sentence definition somewhere what industry 5.0 is . 

3) in table 4 &5 , its recommended that the authors also provide explanation what the relevance is and how they calculated it . 

4) in line 381, for regression analysis, we usually recommend the authors to report all the estimates. If the authors believe those are irrelevant to the rest of the story, I will recommend the authors to put the results in the appendix or supplement material. 

5) Proof reading is recommended: 

            - check verb tenses

            - check punctuation

            - avoid very long sentence (e.g. the very first sentence in introduction)  

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

In this paper, the authors demonstrated the usage of artificial intelligence techniques in analyzing reviews collected from online stores part of E-commerce Europe Trustmark for the purpose of business promotion. Specifically, sentiment analysis is employed based on natural language processing and machine learning.

The topic of the paper is of practical value. My comments are as follows.

Figure 1 depicts the methodology used in the paper. The second box “mixed-combined qua(n)titative research” includes two methods “case study” and “content analysis”. The former is used to collect more comprehensive samples and the latter is then adopted to analyze them. As actually they are conducted in sequence, it might be improper to set them to be parallel in the diagram, which is a little bit misleading. Besides, why are some contents not enclosed in any box? What is the difference between contents in or not in a box? 

Table 1 lists 18 national associations for the countries registered in the program, which doesn't make much sense, but it takes up almost a whole page.

The paragraph from Line 331 to 338 concerns sentiment analysis, where the authors mentioned: “artificial intelligence is used to decode context”, and “By classifying technique through natural language processing (NLP) and machine learning (ML), the software …” Then, what is the exact technique that the term “Artificial Intelligence (AI)” indicates? As is known, AI is a research field with a broad range of aspects and related techniques. Furthermore, what kinds of NLP and ML models or methods are employed to perform sentiment analysis? What are the functions of them? what kind of data can be the input and what is the form of the output data? These issues related to techniques employed in this paper are not clearly explained. 

In Table 2, what do the terms “most relevant” and “most recent” mean? A precise definition or explanation for each of them is needed.

In Table 2, why do the authors only choose the reviews with one or five stars, that is, why are reviews with two, three, or four stars not applicable for the objectives? In common sense, for example, reviews with two stars can also be viewed as negative. Please provide some explanations on this point.

Notice that in Table 3, adding up the numbers of the fourth column, yields 101%. So, when counting the percentages, using numbers with a decimal could be better.

In tables 4 and 5, what does "Relevance" mean? and how the values of "Relevance" for the categories are obtained?

In Equation 1, how to determine the values of the coefficients βs?

A few typos or syntactical mistakes in text, listed below:

-Line 17: “… must be pay to them …” -> “… paid to …”;

-Line 27: “… technique recognises the need of …”, please reconstruct the sentence;

-Line 322: The subject of the sentence should be put in front of “were”; besides, the second “end” is redundant;

-Line 327: The number in text is different from that of Table 2.

-Line 345-346: it seems that “nor” and “also” are redundant;

-Line 414: In the “if” clause, the subject is missing;

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper is a mess.  It purports to be about one thing, responsible use of AI, but has almost nothing to do with that topic.  The paper is very poorly written.  The English grammar and many of the phrases make it nearly impossible to get the full meaning.  There seems to be a lot of repetition in stating something and restating it in several other places.  Without a rewrite, this paper should not be published.

The authors need to separate out the literature review from the introductory section.  In addition, their three hypotheses and the conclusions they draw throughout the lit review should not be in that section.  They can list their hypotheses in the introduction, but a lit review should strictly inform the reader of what research has gone on before that the authors are building upon. 

In section 2, I found their "explanation" of figure 1 inadequate.  They provided no detail for how they were going to perform either the case study or the content analysis.  This section should provide full details including a description of all algorithms and tools used.

In section 3, under some of the tables, the source is listed as "authors' conceptualization based on analysis results".  This is a meaningless statement without an explanation for how the authors reached their conclusions for the values of relevance.  Again, more detail is required to support their findings. 

As a researcher in AI, I found this paper to be woefully inadequate.  The conclusions provided cannot be confirmed by experimental results without far more details.  The bigger issue for me though is to claim that they have an argument for the responsible use of AI.  I found nothing in the article to support that other than some vague conclusions.  

My recommendation to the authors is to rewrite the paper with four changes:

1.  Have a native English speaker help you rewrite this

2.  Move the lit review into its own section and remove any "conclusions" you draw from the lit review

3.  Rewrite the methods section to provide more detail

4.  Remove all mention of responsible use of AI as I do not believe you showed anything that supports it

There were so many issues that I couldn't write them down.  Just three things I noted: 

Line 59:  change "5 stars" to "five stars" to be consistent with "one star"

Lines 111-113:  you spell out HCD twice

Section 1.2:  you are inconsistent between Google Reviews and Google reviews.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The aim of the work is to use customer reviews from e-commerce to induce improvements in the business management of e-commerce companies. While the title introduces the concept of responsible use of Artificial Intelligence, in the document, it is limited to the confidentiality of reviews. Although this is important, it does not appear to be a central objective of the work.

Overall, the document suffers from a lack of necessary details and a lack of argumentation or analysis of metrics. It uses natural language processing, data mining, and statistical regression as instruments, as shown in formula (1).

In general, the document is not well-organized, and the sequence does not seem to be the best for readers to follow the development properly.

In the abstract, the main or secondary objectives of the work are not clearly defined. It does not highlight the contribution that the authors make in the article. In short, reading the abstract does not provide a clear overall view of the work.

The introduction is correct, although it could be strengthened with some current references in some sections.

The Materials and Methods section needs substantial improvement. Firstly, the Case Study requires a better statistical description and some metrics to understand the data structure. Regarding the tools used for the proposed methods, it is necessary to know which ones have been specifically used and why the chosen ones have been chosen. SaaS is a cloud service platform. To interpret the results obtained, it is necessary to define the metrics that can be used.

The Results section requires, first of all, a better and more detailed description of the results obtained. This is important to follow the subsequent reasoning leading to the conclusions.

In this section, quantitative information (equation 1) is used for which there is no statistical information available. This is a consequence of a previous comment about the need for an adequate description. Also, the obtained results, the regression method used, and appropriate statistical tests are not known. This limits confidence in the indicated results.

The Discussion section focuses on comments on its results and the fulfillment of proposed objectives. It does not interact with other results from the referenced literature or with others who may have carried out related work.

Finally, the Conclusions require a rephrasing after incorporating the considerations made, reflecting those supported by the results of this document.

As a result, I consider that the document requires substantial modifications to be considered for publication.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Thanks for the response. Most of the issues have been addressed except the first one. The 72% accuracy is still quite concerning at this point, making people wonder how trustworthy the ground truth (those negative, positive, neutral labels) is.  If the ground truth is questionable, can we trust the conclusions drawn from that. I only have access to the abstract of the reference [71], so not 100% sure how they get the accuracy of 72%. In an extreme example as follows, if there is 70% chance the algorithm gets positive labels correctly in a binary classification (positive/non-positive), same for negative and neutral. Assuming they are all independent, then the chance to get all labels correctly is .7^3 ~ .34 . Of course in reality they are not like this , but the main point here is that this accuracy make the validity of the labels questionable.   

That being said, I do notice that the authors mentioned MonkeyLearn also provide confidence level of the label, does the confidence level here means how confident the algorithm is to correctly classify the labels?  If it is , how the confidence scores look like for those labels(positive/negative/neutral) ?  

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors addressed all my concerns. Now, I suggest one more revision: In Table 3, to keep consistency, all entries, rather than only those in the third column, need to be revised by adding decimals.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This is my second review of this paper and while this is an improvement, the paper still has two significant flaws.  The larger issue is the emphasis on demonstrating responsible use of AI.  While using AI to help assist in their conclusions about the use of sentiment analysis is fine, they fail to demonstrate how or why their approach is a responsible use of AI.  If this was to be the main topic of the paper, they should demonstrate irresponsible uses of AI and use experiments to prove that their approach is a responsible use.  Instead, they throw this in seemingly as an afterthought and yet based on the abstract, introduction and title, this should be a significant topic of the paper.  It is my recommendation that they leave "responsible use of AI" out of the title and discard the portions of the abstract/intro and conclusion that state this.  I could see keeping a paragraph in the conclusion that addresses how they feel they used AI responsibly and perhaps others could follow their methodology.

The other issue is in the writing.  Some of the problem is in the use of English - poor grammar, incorrectly chosen words, etc, but some of this is in an inability to clearly explain topics.  In some cases, statements are vague, in others they seem inaccurate.  There also seems to be some significant repetition where the authors state something and then repeat it in later paragraphs or sections.  The entire paper needs another rewrite to be more readable and understandable.  I have specific comments below of items I found the hardest to understand and items that need correcting.

Intro:  Your three questions are poorly worded.  I really have no idea what "manifest themselves in reviews" means, what you mean by "reviews category" or what you mean by "what are the best reviews about? but the worst?"  I don't understand objective 2 and objective 3 seems like it will vary company-to-company.  You then list three hypotheses but these need fuller explanation.  For instance, how does a negative review positively influence a user's action?  I think what you mean is that a user who has a negative experience and thus writes a negative review is more likely to leave a review than one who has a good experience.  You should more clearly define what positive influence means in all three of these.  In the last paragraph of the intro, you mention four sub-sections but only describe three.  

Lit Rev:  I don't get what trademarking has to do with sentiment analysis.  This is not explained.  On lines 117-118 you have "click or tap here to enter text", I'm not sure what that is.  Do you have a reference regarding gamification?  I'm really unclear on why you have included the second and third paragraphs of 2.1, they seem to have no contribution toward the role of consumer evaluations/ratings especially since many of these comments are woven into or repeated in later paragraphs.  Your conclusions on lines 151-165 seem to repeat what you just discussed earlier.  What is social proof? (line 180)  You keep mentioning how reviews can help a business and cite surveys but you don't offer any details on those surveys.  You recommend a proactive approach for users to leave reviews but you don't specify how to go about this outside of one suggestion.  Really, almost of section 2.2 seems to repeat parts of 2.1 and itself.  This is poorly written and lacks focus.  In 2.3, how does a trademark impact trust?  I don't understand the relationship here, unless you mean name brand recognition as trademark.  Perhaps this section (2.3) should start by explaining what exactly you mean by trademarking.  You later mention that a study showed a company that trademarked would be less likely a victim of counterfeiting.  Like most of what you say, this is too vague.  Share some of the findings that led to this conclusion.

Materials/Methods:  lines 367-368:  "combination of" but then you only use one method, and even at that you don't say what method you used to analyze the text.  Line 393:  you abbreviate relative term frequency as IFD and later say IFD is inverse document frequency.  This is confusing.

Results:  table 4's caption is on a separate page from the table.  You regularly use "authors' conceptualization based on sentiment analysis results" but you don't explain how the authors came up with the relevance values.  You should explain this in a paragraph before you present the tables.

Discussion:  you have this section labeled as 4 but it should be 5.  

Conclusions:  should be section 6. 

I described those in the previous box.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The new version of the article submitted includes the most relevant aspects suggested in the first revision. Although there are still some aspects to be included, I consider that they are not substantial and, therefore, I consider the current text to be adequate for publication. I only require one minor issue and that is that in table A2 the current generic names of the variables should be replaced by the names used in equation 4. In this same equation, the meaning of the upper bar above the variable names should be indicated (although it can be easily recognized).

In any case, I have recognized a great and effective effort to adapt the original text to a much more suitable version for publication.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 3

Reviewer 1 Report

All the issues have been addressed

Author Response

Thank you very much for your contribution!

Reviewer 3 Report

This version is a dramatic improvement over the first.  I still am not convinced about the authors' conclusions and this form of research is not like that which I do (I do AI research where we build systems and test them out for proof), so I cannot really state how well the authors proved their points nor whether the approach taken is particularly useful for other researchers in the field.  But with that said, they addressed just about all of my comments and improved the paper well enough to be published.  Although I don't have any specific grammar/English issues, there are a few mistakes that the authors should fix, which I list in the next box.  I appreciate the authors taking my first review seriously and working out the issues I noted.  I wish this was more in my area of expertise so that I could provide a stronger recommendation but again, my area of AI research is on implementation of AI algorithms and not the use of AI in business settings.  I suggest that the authors do one more minor rewrite as noted in the next box and then the paper should be publishable.

lines 27-35:  I really don't understand the reason you included all of this.  The emphasis of the paper is not to convince people of the value of AI but to show how it can be positive used for sentiment analysis.  

Line 37:  change "(Reference 8)" to "[8]".
In this opening paragraph you keep using "artificial intelligence (AI)", after the first instance, just use AI.

It seems to me that the text starting "In the realm..." (line 45) should start a new paragraph.  You might take a look at some of your length paragraphs (of which you have many) to see if you can divide them into multiple paragraphs.

You should explain what sentiment analysis is in the introduction (perhaps around line 60).

Line 65 & 69:  you are again repeating spelling out artificial intelligence (AI) and now natural language processing.  Once you have shown its abbreviation, just use the abbreviation.

Line 86-92:  this is still confusing about the "negative" experience but "favourable" impact.  The added sentence about "positively influences" doesn't really help.  

Line 116:  you should be explicit that the subsections you refer to are part of the lit review section.

Line 120:  spell out BP the first time you use it.

Line 390:  the caption for figure 1 is on page 9 whereas the figure is on page 8, make sure the caption is directly beneath the figure.

Line 464:  try to keep this with equation 2 (that is, on the same page, whether thats page 10 or 11)

Line 599:  you say "by the authors", do you mean yourselves or all the authors you referenced?  I assume you mean everyone you referenced and if so, you might want to express that differently.  I found this entire paragraph to be a bit unclear. 

Line 604:  no, the use of AI does not require the adoption of tools, many AI researchers are developing systems from scratch; I think you should be more specific and say that in business settings, the use of AI usually requires the adoption of tools.

Author Response

Please, see the attachment.

Author Response File: Author Response.pdf