A Semantic Classification Approach for Indoor Robot Navigation

Round 1

Reviewer 1 Report

The authors presented a study for developing a machine learning-driven classification approach for the indoor navigation system. The dataset was collected using a 2D LIDAR scanner. Different machine learning algorithms were used for classification including decision tree, CatBoost, Random Forest, Light Gradient Boosting, Naïve Bayes, and Support Vector Machine. Four different classes were created for training including door, hall, room, and hall. Various evaluating metrics were used including Confusion matrix, classification accuracy, precision, recall, F1, and ROC. The authors reported that the SVM model offers the best results for classification accuracy, precision, recall, and F1-Score, followed by the Random Forest. Autonomous navigation with low-power low-cost devices is a stimulating topic and would be of interest to the broader scientific community. Overall, the manuscript is good but lacks major details that must be added before the manuscript can be considered for publication. My comments are below:

1.      The authors presented section 3 as a navigation system, however relevant results are not presented. E.g. which navigation or path planning algorithms were used, criteria for path testing, map creation, map length, test environment, success to avoid the obstacle, etc., A lot of details are missing in the paper. I suggest giving the detailed execution process of the path algorithms and the pseudo-codes. What was the step size for path planning, path planning time, and target deflection probability, and how it was chosen? I believe the authors need to define the scope of the work and revise the paper accordingly. The scientific contributions of the study should be clearly presented to avoid confusion. Current results are just for classification, not motion planning and navigation, therefore the experimental results are not sufficiently persuasive and there should be more detailed explanations of the obtained results.

2.      The authors mentioned that multiple ML algorithms were used, however, no details are presented in the paper. Defining the algorithm is not enough for a scientific paper, how you have deployed it in your study is important. Additionally, pseudocode should be added, as well as for the navigation implementation.

3.      The results & discussion section of the paper is too weak. More information must be added to improve the quality of the manuscript. What results are obtained is important, but how these are obtained, what are the sources of errors, observations during the experiment, and limitations of your work, and discussion on those aspects make a quality paper. The author must provide a more in-depth discussion of the results obtained, and future research directions based on this study, all of this information should be included in the paper. The authors mentioned using the online Lidar dataset and in conclusion mention this as future work, the scope of the paper must be revised.  

4.      What was the resolution of the LiDAR sensor, the format of data, the size of the data, and the processing time to generate the maps? The methodology section needs to be revised with more technical details and data processing methods. Add some figures for the integrated robot navigating indoors, add figures for the lidar data and the maps generated, and data processed from 2d lidar to 3d environment, how was the data is been seen by the robot? The figure will be useful here.

5.      The author should add images of the developed robotic system, testing environment, the example frames of the dataset, and processed frames, for better visualization of the system and dataset for the readers. Additionally, the quality of the figures should be improved. For example Figure 4, if a real robot is used for data collection and validation, add the figure or the CAD model of your robot with lidar mounted, rather than using an internet source figure.

6.      For all the hardware (off-the-shelf) and software used in this study, add the trade names in a standard format. For example, using the generic description “a LiDAR scanner (product model, company name, City, State or Country).

7.      Please read the manuscript careful for necessary grammar and spelling corrections, e.g. check line 28 “distention”, Figure 6 label “flidar rames” and line 437 “wok”

 

8.      In scientific writing the term “Significant” is used when the statistical analysis is performed, if you have done some statistical test, provide the details, else revise the term. E.g. line 244

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper compares the performance of several commonly used classifiers in traditional machine learning in different scenarios, and selects the classifier with the best performance according to different evaluation indexes. However, the following issues should be well addressed before the paper can be considered for publication.

1.      The algorithm selected in this paper is a traditional machine learning algorithm, which is prone to interference from the external environment in natural scenes and has poor stability, and its performance is insufficient compared with common deep learning algorithms.

2.      This paper belongs to the field of image classification, and the existing image classification algorithm based on deep learning has already been deployed in mobile terminals.

3.      In order to express each evaluation index clearly and intuitively, please use the formula.

4.      The paper does not give specific model training details and process, please further supplement and complete.

5.      The authors should improve the bibliography proposed in the paper. Several relevant examples that shall be cited are listed below.

Scene perception based visual navigation of mobile robot in indoor environment.

ISA Transactions, 2021, 109, 389-400.

Localization, obstacle avoidance planning and control of a cooperative cable parallel robot for multiple mobile cranes. Robotics and Computer-Integrated Manufacturing, 2015, 34, 105-123.

Empirical study of future image prediction for image-based mobile robot navigation. Robotics and Autonomous Systems, 2022, 150,104018.

 

6.      The paper emphasizes that the image classification method used can enhance navigation tasks and human-robot interaction, but does not clarify the relationship between the two. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

General comments:

The main innovation point of the article is to use the semantic information to assist the robot to navigate indoors, so that the robot no longer only relies on simple geometric information, but can be like a person with a deep understanding of the environment, and thus realizes the intelligent navigation. The semantic information used in this article is mainly the environment information, such as corridors, rooms, etc. The article mainly introduces how to implement the environmental classification, and does not further discuss how to use such information for navigation. The main idea is to help robot navigate by establishing the semantic model of the scene. The scenario semantic model should be a hierarchical model --- The top layer is the connection between scene categories, such as the bedroom connecting the corridor, and the bedroom is near the study-room, the living room may be adjacent to the kitchen; Then, the next layer is the connection between the object and the scene, such as the sofa in the living room, the bed in the bedroom, etc. Finally, the association between objects, such as chairs, will be placed next to the table, which will be placed on the desk or bookshelf. Using this scenario semantic model, you can identify the location of the room, the location of the room, and the location of it in the room, in order to achieve efficient and intelligent navigation in order to confirm the location of the room that it may exist, the location of the room, and its location in the room.

The second point of the paper is based on laser data, using a simpler machine learning algorithm to realize the classification of scenarios, which is more efficient and fast than the low performance embedded platform, compared to the image data that is usually based on image data.

The third point is that this article divides the process of scene classification into offline and online forms. The offline model trains the field view classification, and then online implements classification. But the flaw in this approach is that it can only identify the type of scene that it contains in the training model, such as the room, the corridor, the door, and the hall. There is nothing to do with the type of scenario that is not included in the training model. Whether you can achieve online learning, according to the information collected by robot sensors, keep learning and optimize the semantic model?

Minor comments:

This article is a classification of the scene using a laser dot cloud. The information is collected by the laser point cloud. This is time-consuming to handle the cloud data。

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

1. The authors present an interesting and simple exploration into machine learning algorithms for detecting the various semantic class, to assist in navigation.  The language use is also good. It was a good read.

2. The flow in Figure-1 might be best represented as a flowchart instead of what seems like a block diagram. 

3. The block diagram in Figure-3 seems to be unnecessary as it is seems to simple. Consider omitting or if the authors insist, make it as a flowchart. 

4. It is the reviewer's opinion that Figure-6 might not be necessary as detailed information is already presented in Table-2. 

5. It is not clear whether each of the confusion matrices are for which set. It is the training set, or test set? 

6. Table-10 shows the testing accuracy... and it seems that SVM is performing best. However, it is this reviewer's opinion that this SVM is the 'best' SVM (in terms of configuration) compared to the other SVM configurations that the authors have tested with. Perhaps, it would be best to mention all the configuration hyperparameters and parameters of the best SVM.

7. The authors report the ROC and AUC. The reviewer believes that, since in the previous section, the SVM was superior, it might make more sense to look at the AUC for each class for the SVM. This is because it was indirectly concluded in the previous section, that the SVM (along with its configuration) is the overall classifier of choice, and it would be of interest to see what its multiclass classification accuracy is for each of the classes. It makes little sense to compare with other classifiers at this point.

8. As an extension to the comment in (7), it might also be wise for the authors to actually include the ROC (as a Figure) to visualize the AUC for each class. For example, it would be good to see the ROC of the SVM for ROOM vs others, and then for CORRIDOR vs others, and DOOR vs others and finally HALL vs others. 

9. The DISCUSSION section is not a discussion about the results of the work proposed. Nor does it give any insights on, for example, why some of the results are as they are. The first 3 paragraphs seem to give an overview of what's done before. This reads more like a literature summary. The fourth paragraph is a summary of what was developed by the authors.... Please revise this section so that it becomes a more proper discussion section.

10. The conclusion section mentions .... FASTER. Time was not measured in this paper. How can the authors conclude that what was proposed is faster? 

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 5 Report

I would like to appreciate the author's effort in presenting their work in this article.

Although this article seems to have made some contributions superficially,  there are some severe concerns to be addressed. 

Please consider the comments below to improve the article further. 

1. Authors shall consider thoroughly checking the article to avoid typo mistakes. For instance, refer to line 119, "The developed system has been developed" and line 226, Y-axis label of figure 6, "# of flidar rames". These kinds of mistakes shall be avoided. 

2. The accuracy of 97.31% claimed in the abstract is nowhere substantiated in the paper. In fact, It seems to be inappropriately claimed.  

3. Authors did not mention the online/offline semantic navigation model anywhere in the Introduction section. There seems no logical order in the explanation. 

4. With the technical specifications of the Lidar and other hardware setup used, It raises concerns about the low-cost and low-storage claim. This is rather written fancily. This needs to be substantiated thoroughly. 

5. Figure 1, 2, 3, 4, and 5 has no significance and seems like just placeholder images. Authors shall consider using much more effective figures captured during the experimentation phase. I would suggest using more detailed images of the mobile-robot system, navigation setup/plan, and the environments used. This would create a good interest among the readers. 

6. The algorithm 1 used is again of less significance and does not highlight any key or novel contribution. 

7. The similar work cited [6-12] is more or less having a multisensor setup which is against the single lidar usage in this work. I would suggest narrowing down further and using appropriate related work. 

8.  The classification section (4.3) looks completely debilitated. Why are these many models used which is against the claim as in the abstract? Moreover, the existing models are used straightaway and do not bring any new contribution. 

9. The rest of the paper seems to have less impact without clarifying the key contributions. Also, it is good to be very careful with the accuracy aclimed.  I was not able to find "97.31%" accuracy anywhere in the paper apart from the abstract. 

 

 

 

 

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have provided reasonable responses to my comments and revised the paper accordingly. I have a couple of minor comments:

1. Figure 8 & 9 should be added as a table since it shows tabular data, not graphical data. 

2. Figure 3 quality (integrated system) should be improved. Add a high-resolution image.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

The revised contents of the paper are perfect. 

Author Response

Thanks for your comments.

Reviewer 3 Report

The authors have revised their paper in accordance to the my comments.

Author Response

Thanks for your comments.

Reviewer 5 Report

How was the accuracy misrepresented? there seems to be no conclusive evidence to prove the achieved results and accuracy of the proposed methods. 

The authors did not provide a convincing response to previous comments. Rather there are only a few textual changes that will not bring any significant value additions. 

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 3

Reviewer 5 Report

I have no further comments.