Design and Implementation of an Ontology for Semantic Labeling and Testing: Automotive Global Ontology (AGO)
Round 1
Reviewer 1 Report
- I believe the problem you identified in your research is valid.
- Since your solution is to reach a consensus on an ontology that can be used by various industries, I suggest in your "future works" or even in this iteration if possible, contact the experts within the main contributors in the field you identified in your paper including those collaborated in nuScenes, Lyft Level 5, H3D Honda 40 Dataset, Waymo, Audi A2-D2, Berkeley Deep-Dive (BDD), Apolloscape, or Mapillary 41 Vistas and ask their opinion (survey) about your method and possibly collaborating in your paper (or the next one). This will take your method one step closer to be actually used in the real world.
- In Section 3, Terminology, the order in which the terms are presented is confusing. It must be either in alphabetical order or in the way that the basic terms that are used in more advanced ones come first.
Author Response
Suggestion #1: “2. Since your solution is to reach a consensus on an ontology that can be used by various industries, I suggest in your "future works" or even in this iteration if possible, contact the experts within the main contributors in the field you identified in your paper including those collaborated in nuScenes, Lyft Level 5, H3D Honda 40 Dataset, Waymo, Audi A2-D2, Berkeley Deep-Dive (BDD), Apolloscape, or Mapillary 41 Vistas and ask their opinion (survey) about your method and possibly collaborating in your paper (or the next one). This will take your method one step closer to be actually used in the real world.”
Author response:
I would like to thank the reviewer for the suggestion. However, taking into account the deadline established to present the article with the changes, it is not possible for me to consult with the experts before the submission deadline. Nevertheless, I believe the suggestion is interesting and will be taken into account in future developments.
Suggestion #2: “3. In Section 3, Terminology, the order in which the terms are presented is confusing. It must be either in alphabetical order or in the way that the basic terms that are used in more advanced ones come first.”
Author response:
I would like to thank the reviewer for stating that the order of the terms presented in Section 3 is confusing. Hence, considering the suggested options, it has been decided to reorder the terms following the alphabetical order.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
While most of this is grammatically correct, in many cases the use of the words don't flow together well. A rewrite with the help of a native English speaker would be most useful.
Labeling is not a new problem in AI, it dates back to early attempts at supervised and semi-supervised learning. Change "recently identified as one of the" to "an on-going" in line 29. Its only more of an issue now that we are trying to train with millions of pieces of data.
Seems to me that section 1, 2nd paragraph should have more than 1 reference when it comes the new challenges.
Spell out KOS in section 1 (not everyone will read the abstract). Also, you reference AD on line 56 and then spell it out on line 69. You need to spell it out first.
Line 92: Geographic can be lower cased, also a reference for GDF and OWL would be useful.
You keep using the term scenarios. I think you should define what you mean by a scenario. I think you mean domain whereas scenario is usually an instance of a problem.
Line 120 you have an ( but no ).
Sections 3 and 4 have the same name (Terminology). If you truly mean for both sections to introduce terminology, I suggest you combine the sections, otherwise change the title of section 4. It seems to me this is more a section describing the components of an ontology so you might call it Defining an Ontology or Defining a Geospatial Ontology.
Section 5.1 could probably be reduced to a single paragraph and Table 1. I didn't see much in that subsection worth discussing.
Section 5.2.1: this whole subsection on the semantic web should be moved to the introduction (section 1, or maybe section 2).
Line 291: Should probably be "A domain ontology..." instead of "The".
Line 305: have some added spaces here between "the" and "simpler".
Seems like the first part of section 5.2.3 should be moved into section 4 (up until figure 4).
You might want to add some text explaining figure 6 (say at the end of line 345).
Table 3: under URI you have a link rather than a description of what a URI is, you might want to replace this URI with a description of a URI and then have the link available as an example, either in the table or in the text.
5.4: this subsection is useless since you cover this in section 6. I suggest a minor rewrite of section 5 to not include consumption. Just my opinion.
Figures 7 and 9 need to be explained rather than just referenced. Figure 8 elaborates on an earlier glimpse of AGO's ontology and probably deserves a little space in the text for some analysis.
There is something wrong near the beginning of section 6.2 (paragraph starting on line 444) as you say "continuing with the scenario" of fig 11, but you haven't introduced figures 10 or 11 yet. That happens after figure 10.
Line 491 has a ( but no ).
This is a fine paper and should be published. However, I was disappointed because the introduction made it sound like you were going to offer a means of automated labeling of data sets through an ontology. Instead, this is just another ontology with a revised methodology for building the ontology. You don't offer too much insight into how to use the ontology for automated labeling. You might want to tone that aspect of the paper down in that it seems to me that it is misleading. If you insist on including that, the first use case should probably be more descriptive in how AGO can be used for this purpose.
Other than the small changes I suggest, and the disappointment I had with the emphasis on automated labeling, this paper is a good paper that describes a new ontology and a revised approach to building an ontology. It isn't particularly novel but adds to the knowledge base of ontology construction with some suggestions for concrete applications for the ontology. To truly be novel, I think you need to go further into these applications areas and/or provide more detail on the graphical nature of the construction of the ontology. It is those areas where readers probably would gain the most insight.
Author Response
I would like to thank the reviewer for the detailed revision and for suggesting that the article should be published despite not having met your expectations after reading the introduction. Below I try to respond to all the comments and suggestions presented.
Suggestion #1: “Labeling is not a new problem in AI, it dates back to early attempts at supervised and semi-supervised learning. Change "recently identified as one of the" to "an on-going" in line 29. Its only more of an issue now that we are trying to train with millions of pieces of data.”
Author response:
As suggested, the sentence in line 29 has been modified. However, it has not been chosen to include the suggested words “an on-going” to the sentence, instead "recently identified as one of the” has been changed with “is one”. This action was done to try to respect the phrase of the original text.
Suggestion #2: “Seems to me that section 1, 2nd paragraph should have more than 1 reference when it comes the new challenges.”
Author response:
I believe that the reader is provided with sufficient information for understanding the context of the presented work. A list of several challenges is presented and since the aim is to focus the reader on the idea of representing and working with ontology-linked datasets a reference related with the area of interest has been procured. That is why, it has been chosen with all due respect not to modify this part.
Suggestion #3: “Spell out KOS in section 1 (not everyone will read the abstract). Also, you reference AD on line 56 and then spell it out on line 69. You need to spell it out first.”
Author response:
On the one hand, as suggested KOS has been spelled in line 53. On the other hand, I noticed that AD was first mentioned and spelled in line 40. Therefore, line 69 was fixed by deleting the duplicated lettering of the term.
Suggestion #4: “Line 92: Geographic can be lower cased, also a reference for GDF and OWL would be useful.”
Author response:
‘Geographic’ in line 92 has been changed to be lower case since this was an error.
As for the suggested references, as far as GDF is concerned, it is a relevant term for the work presented in Reference [7] (included in line 93), therefore I believe that if the reader is interested in gaining more insight around this, should have a look at the presented Reference [7] where this term is explained and referenced. In case of including the reference for GDF in the summary included in the article, I understand that it would be like duplicating the reference.
OWL instead is referenced in section 5.2.1 where the technologies and tools considered for building the ontology are mentioned and further explained. I believe it could be more helpful for the reader to have the reference there since it is where it is more relevant.
Suggestion #5: “You keep using the term scenarios. I think you should define what you mean by a scenario. I think you mean domain whereas scenario is usually an instance of a problem.”
Author response:
The reviewer may have been overlooked, but the term ‘Scenario’ is defined in the original manuscript as part of the terminology presented in Section 3.
Suggestion #6: “Line 120 you have an ( but no ).”, Suggestion #11: “Line 305: have some added spaces here between "the" and "simpler".” and Suggestion #18: “Line 491 has a ( but no ).”
Author response:
These errors have been fixed in the new submission.
Suggestion #7: “Sections 3 and 4 have the same name (Terminology). If you truly mean for both sections to introduce terminology, I suggest you combine the sections, otherwise change the title of section 4. It seems to me this is more a section describing the components of an ontology so you might call it Defining an Ontology or Defining a Geospatial Ontology.”
Author response:
As the reviewer has noticed, these two sections should not have the same name, originally section 4 had been named as 'The AGO Domain Model', but when passing the text to the template I made the mistake of not changing the title. This has been corrected taking into account the original name of the section.
Suggestion #8: “Section 5.1 could probably be reduced to a single paragraph and Table 1. I didn't see much in that subsection worth discussing.”
Author response:
I agree with the reviewer when suggested that subsection 5.1 can be summarized as the content can be a bit redundant. But on the other hand, I have to respectfully disagree when stated that the subsection has not much information that is worth discussing. I believe it is important to emphasize that current datasets practically do not contemplate terms related to 'Actions' and/or 'Events'. In addition, if mentioned they do not provide a semantically meaningful definition of these terms. As mentioned in the manuscript terms describing the movement and maneuvers of the automotive objects should be carefully defined and related with other elements. This is part of the contributions made by defining ‘AGO’ and we believe that will be vital for the future development of ontology-based applications in the automotive area and the use cases presented.
In consequence, the first two paragraphs of the subsection have been collapsed into one and the rest has been kept as originally presented.
Suggestion #9: “Section 5.2.1: this whole subsection on the semantic web should be moved to the introduction (section 1, or maybe section 2).”
Author response:
After revising this comment, I may state that I considerately differ. I think I understand why this suggestion is done, since the first part of the subsection seems a little theoretical and generic. However, I believe that this section is meaningfully placed as a subsection of section 5.2 since the concrete tools used for the construction of AGO are mentioned along with the included metadata and features. Hence, I trust that moving that section would make it lose meaning in the text.
Suggestion #10: “Line 291: Should probably be "A domain ontology..." instead of "The".”
Author response:
When saying ‘The domain ontology’ the idea was to refer to the presented ontology, therefore instead of changing ‘The’ with ‘A’, the ontology name (AGO) has been included in the phrase.
Suggestion #12: “Seems like the first part of section 5.2.3 should be moved into section 4 (up until figure 4).”
Author response:
Following the suggestion, part of subsection 5.2.3 has been included as part as ‘the AGO domain model’ section since as the reviewer has notices it describes part of the general characteristics of the defined domain ontology. Concretely, this part of the text presents the RDF/OWL principles that have been adopted for the knowledge representation as an ontology, which would be independent of the use case. This way, the revised section 5.2.3 only explains the more specific characteristics of the ontology representation for the automotive domain.
Suggestion #13: “You might want to add some text explaining figure 6 (say at the end of line 345).”
Author response:
A little explanation has been added when presenting figure 6. Hence, the included lines in the revised manuscript are:
“Using n10s Release 4.1.0 a Cypher query can be used to import the OWL ontology from a local directory or an url. Besides, the query could be passed with some specific parameters that will determine the names given to the elements in the database. The specific query used to structure the ontology is presented in Figure 6.”
Suggestion #14: “Table 3: under URI you have a link rather than a description of what a URI is, you might want to replace this URI with a description of a URI and then have the link available as an example, either in the table or in the text.”
Author response:
As suggested table 3 has been modified by adding a description for URI:
“URIs are used in OWL as object identifiers. Its structure is formalized by the ontology language to be a valid URL where the complete ontology is available and unique name for each class. In the case of AGO the URI looks like: http://vcd.vicomtech.org/ontology/automotive/# + {name}”
Suggestion #15: “5.4: this subsection is useless since you cover this in section 6. I suggest a minor rewrite of section 5 to not include consumption. Just my opinion.”
Author response:
I believe the reviewer have made a valid point here, thus section 5 has been modified as suggested by deleting consumption as one of the phases of the methodology. The actions performed to do so have been:
- Modify Figure 2 with a new methodology pipeline without the last phase.
- Delete section 5.4.
Suggestion #16: “Figures 7 and 9 need to be explained rather than just referenced. Figure 8 elaborates on an earlier glimpse of AGO's ontology and probably deserves a little space in the text for some analysis.”
Author response:
With the intention of giving meaning to the presentation of the figures mentioned in the reviewer's comment, the following explanation have been included along the text and also a new reference:
- For Figure 7: “During the development of the work, all the datasets have been represented graphically and it has been verified that the labels of the elements can be mapped to their respective synonyms in the ontology. This translation function requires the definition of advanced search queries. The result could be used to feed training models that need to be adapted for new tests that use different datasets from those used for training. A diagram of the process is presented in Figure 7.”
- For Figure 8: “As an example of the above-mentioned process, Figure 8 depicts a graphical representation of the classes defined in ‘Waymo’ dataset mapped to their equivalent AGO classes. The equivalence among elements is defined by the ‘owl:sameAs’ relationship, which means that one can be replaced by the other without altering the meaning and vice versa.”
- A new reference to the Figure 9 has been done in the same paragraph since the explanations given there are related to the diagram presented in the figure.
Suggestion #17: “There is something wrong near the beginning of section 6.2 (paragraph starting on line 444) as you say "continuing with the scenario" of fig 11, but you haven't introduced figures 10 or 11 yet. That happens after figure 10.”
Author response:
I apologize for this confusion; I have noticed that the order has been altered at some point when adapting the text to the template. I have adapted the revised manuscript taking into account the order of the original text and I hope that this way it will be understood correctly.
Suggestion #19: “This is a fine paper and should be published. However, I was disappointed because the introduction made it sound like you were going to offer a means of automated labeling of data sets through an ontology. Instead, this is just another ontology with a revised methodology for building the ontology. You don't offer too much insight into how to use the ontology for automated labeling. You might want to tone that aspect of the paper down in that it seems to me that it is misleading. If you insist on including that, the first use case should probably be more descriptive in how AGO can be used for this purpose.
Other than the small changes I suggest, and the disappointment I had with the emphasis on automated labeling, this paper is a good paper that describes a new ontology and a revised approach to building an ontology. It isn't particularly novel but adds to the knowledge base of ontology construction with some suggestions for concrete applications for the ontology. To truly be novel, I think you need to go further into these applications areas and/or provide more detail on the graphical nature of the construction of the ontology. It is those areas where readers probably would gain the most insight.”
Author response:
I'm sorry to read that even though you consider that the work should be published, it has not met the expectations you had when reading the introduction. I understand your interest in seeing a more complete development around ontology-based automated labeling, but the work presented was aimed at presenting the first part of the pipeline of a labeling or testing system. Thus, it is intended to present the knowledge-base and the characteristics that must be supplied so that in future developments it is possible to present a complete and automated pipeline of the presented use cases. Furthermore, as far as I know, no work has been published that presents an open-source ontology that can cover the needs of the use cases in the automotive area analyzed in this work. As far as I know, ​​semantic labeling and testing through simulations are areas that are currently being worked on and this manuscript presents part of the work that we are doing in these domains. I believe it is important to emphasize the areas where the presented ontology will be essential to understand the reason why the characteristics and terms mentioned along the article are important. Nonetheless, these use cases are not the main contribution of the presented work. We are aiming to present developments in the future with a complete pipeline for each of the use cases, since they will be of vital importance for future works in the area.
Author Response File: Author Response.pdf
