Natural Language Processing in Knowledge-Based Support for Operator Assistance

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The presented research, while providing quite limited value of scientific novelty, still might be important for specific context (manufacturing) and provide interesting applied results, given the language specificity and structure in this particular context. However, I still find some issues which should be addressed in order to provide more complete and clear study

My first and most important concern is the selection of the NLP tools for research: there are almost no recent deep-learning based tools present in evaluation. While tools listed here (NLTK, Spacy, Stanford) are still very important in NLP research, the superiority of DL-based tools such as BERT, RoBERTa, ELMO, Flair, T5, XL-Net, etc. (or any transformer-based model) has been demonstrated for almost every task, hence, they should be present and compared in this study as well. The tools listed are usually taken as baselines for modern research. I suggest adding results or at least 2-3 most prominent approaches (or language models, which is currently the popular term for them), like BERT, DistilBERT (which should be an interesting option in your case, concerning the amount of resources required for deployment) or other from the list I enumerated previously which you consider preferable. GPT-like or recent large models might be considered as out-of-scope, as they require large amount of resources of deployment and might be difficult to be applied in assembly settings.

Figure 8 - I would consider it as rather vague (it is rather difficult to view correlation relations in these figures). Consider providing actual correlation values in the form of a table or other structured means.

Author Response

• Thank you for your insightful suggestions. To make Figure 8 clearer, we added Table 5 (line 551), mentioning the correlation value for each library and each category.

We acknowledge the efficacy of transformer-based techniques like BERT~\cite{devlin2018bert}, as evidenced by recent studies in entity recognition, information extraction, and semantic analysis~\cite{danenas2022exploring,phan2022ner2ques,forth2023calculation}. Notably, \cite{chantrapornchai2021information} evaluated BERT and SpaCy for named entity recognition and text classification across various datasets in the close domain. BERT appears to be more sensitive to the clean data set and the annotation method. The accuracy of BERT is a little higher than that of SpaCy for a clean data set~\cite{chantrapornchai2021information}.

However, in the preliminary step of pre-processing, particularly part-of-speech (POS) tagging, traditional NLP techniques demonstrate notable efficiency~\cite{das2024extracting,schmitt2019replicable}.
In \cite{nemes2021information}, the BERT method is already used for sentiment analysis while considering information extraction and named entity recognition. However, for the POS tagging, they used the methods of the SpaCy and NLTK libraries to perform the analyses. The reason for choosing these libraries, based on \cite{nemes2021information}, is the efficient ability to make predictions about which tag or label most likely applies in this context. 

Traditional NLP libraries offer pre-trained models optimized for POS tagging tasks, rendering them adept at efficiently processing large volumes of text. Their interpretability enhances understanding and analysis of POS tagging results, an advantage over deep-learning-based models.

Many types of research analyzing NLP tasks consider traditional and deep-learning pre-trained libraries and compare them in different aspects~\cite{lin2023personal,incollection}. Considering our research scope, constrained timelines, and specific objectives, we opted for traditional libraries to assess their suitability for handling the intricacies of assembly-related concepts during the initial NLP analysis. We positioned the newly added references in the manuscript in blue in the Literature Review (lines 176 to 193) and Methodology sections (lines 412 to 415).
However, we acknowledge the relevance of transformer-based methods for subsequent stages, such as featurizing, entity recognition, and entity mapping. We plan to incorporate them accordingly for the development of recommender systems in future research endeavors.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

1. The abstract lacks a clear structure and seems to jump between different topics without a smooth transition. Consider reorganizing the abstract to follow a logical flow, starting from the background and motivation, followed by objectives, methodology, results, and conclusions.

2. It would be helpful to provide a more detailed rationale for why addressing the challenges in operator assistance systems is crucial for the manufacturing industry.
3. The Introduction briefly mentions Industry 4.0 technologies and their impact on assembly tasks, but it could benefit from providing more context on the specific technologies involved and how they contribute to the complexity of tasks. This would help readers unfamiliar with Industry 4.0 understand the relevance of the study in the broader context of technological advancements in manufacturing.

4. Clearly articulating how the proposed domain-specific dataset, analysis of incomplete sentences, and evaluation of NLP libraries advance the current state of knowledge in the field would strengthen the significance of the study.

Author Response

1.  Thank you for your feedback on our work. We appreciate your concern about the clearance of the abstract. I modified the abstract based on your recommendation. We have highlighted the modified section in blue in the main manuscript as the abstract section (lines 1 to 15).
2.  Regarding the second and third comments, We agree with the reviewer’s assessment related to adding more explanation about the importance of the operator assistance system in manufacturing and the connection between Industry 4.0 and the complexity of the tasks. We modified the introduction's first section and added the contexts related to the requested explanation. The modified part can be found in the manuscript in red color font. Thank you for this suggestion. (lines 18 to 44)
3.   Thank you for your feedback on our work. We appreciate your perspective on making the contribution of the paper more clear. As mentioned in the manuscript, the main challenge in operator smart assistance lies in the efficient interaction between computers and operators. The first step in this regard is that the computer understands the different types of questions and sentences from the operator's side. This is where NLP technology enters. NLP can act as a bridge between the computer and the operator. However, in the close domain of assembly, there is limited existing research on the application of NLP techniques. So, there is a lack of corpus in this area. In our work, the proposed domain-specific dataset, analysis of incomplete sentences, and evaluation of NLP libraries collectively contribute to advancing our knowledge of how NLP techniques can be effectively applied to address the unique challenges of the assembly domain. By addressing these challenges, the study lays the groundwork for the development of intelligent and efficient operator assistance systems that can improve productivity, efficiency, and safety in manufacturing environments.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

1) The performance of platforms (NLTK, etc.) on NLP tasks depends on the target-language. Here, only implicitly it is deduced that the target-language is English. Somewhere in the text (choose the appropriate place) it should be stated which the target-language of your research was.

2) The handling of incomplete sentences has a few methodologies that are proposed, like Hole Semantics, that are not referenced at all. There is no need to elaborate on that, but just a couple of papers mentioned would suffice, to show that you are aware of this issue (google it).

3) Table 2 stands rather far from its 1st mention (line 295). Maybe this sentence would be better positioned on line 283 (I understand that is necessary for Fig. 2, so, it could be placed before the sentence of Fig. 2, on line 282).

4) On page 8, the difference between Manual documents and Work instructions is not very clear. Maybe you should add that the former mainly refer to equipment, while the latter to processes (in general).

5) The list of references lack their label/heading.

 

Author Response

1. Thanks for mentioning the target language. We agree with the comment. We added the target language as  English in the last paragraph of the introduction in line 101.
2.  Thanks for mentioning this. I added some references and a short paragraph in the literature review section (lines 246 to 251).
3.  If we understand the comment correctly, the main point is introducing the 8 different POS tags before the example of the output for the POS tagger. I positioned it in the manuscript based on your recommendation. Thanks for your suggestion (lines 316 and 317).
4.  Thanks for your concern. For the manual, I already mentioned that "they provide detailed instructions on how to operate, maintain, and repair various types of equipment." So, as you mentioned, it is on the level of equipment. Also, for the work instruction, "documents that provide step-by-step guidance to operators, technicians, or assembly line workers on how to perform a specific task or operation," you are correct; it is on the level of the process. I added two short sentences for each of them to put stress on this(lines 357 and 378). 
5. We added the heading for reference and double-checked the reference to ensure the format is correct for all the references (line 601).

Author Response File: Author Response.pdf