Skeptical Learning—An Algorithm and a Platform for Dealing with Mislabeling in Personal Context Recognition
Round 1
Reviewer 1 Report
When one reads the title of this paper, one expects a new paradigm of learning. However, what is described is a system, which in particular is focused on dealing with noisily-labelled data; actually, what is learning in the traditional sense results to be secondary in this work. The shift of the focus on a system instead of a new learning model makes the main contribution of the paper of a different nature. In this regard, I would suggest modifying the title of the paper to reduce the potential confusion of the reader with the paper/research contents.
In any case, I consider of enough interest the research problem addressed in this paper, i.e. dealing with noisy labellings of data.
As for the context of the research, I would like to draw the attention of the authors to the fact that, among the 36 works referenced along the paper, only 7 correspond to less than 5 years old publications, and also the fact that 6 out of these 7 publications involve part of the authors of this paper.
As for the presentation of the system/platform, certainly it can be improved: the level of presentation along the paper is too abstract, it is difficult to follow the discourse. In this respect, I would suggest the authors to make an effort to illustrate somehow the different funcitonalities, maybe with e.g. real cases. On the other side, all of a sudden, the smartphone turns out to be the main protagonist, but one learns that at the middle of the reading. It would be a good idea to emphasize more the relevance of mobile devices at the beginning of the paper.
At the English level, the paper can be followed without major problem, although the English can be improved, and this refers specifically to some minor English mistakes that appear here and there. Some particular mistakes/typos follow:
* line 121: Here is is => Here is
* line 125: acronym SK appears in this line for the first time but it is not expanded anywhere throughout the paper
* line 145: sets c^u and c^p appear for the firs time at this line but are not defined anywhere
* line 157: Notice that, the => Notice that the
* equation 1 (lines 162-163): function f_PARENT(y) [x,y] has not been defined
* equation 2 (lines 171-172): the meaning of E[.] is not defined
* equation 3 (lines 186-187): same as before
* line 242: "the concept behind SKEL is to empower ordinary citizens", I would say this is a weird sentence for a scientific paper
* line 245: "The former is a type of question sent at fixed time interval, which answers are ..." => " The former is a type of question sent at fixed time INSTANTS, WHOSE answers are ..."
* line 263: doesn't => does not (try to avoid contractions in technical communications)
* line 277: it's => it is (try to avoid contractions in technical communications)
* lines 313-318: please, redo this paragraph it is full of errors and difficult to understand
* line 453: as as follows => as follows
* line 494: please, provide the parameters employed for the DBSCAN clustering algorithm
Finally, I would like to ask the authors whether the user of this system is always intended to be a girl/woman, given the way how the user is referenced along the paper.
Author Response
Please see the attachment.Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper title is simple and not attractive, should be extended in order to be more expressive.
Abstract
“The proposed solution is evaluated in university student life scenario where the goal is to recognize locations and transportation modes.”
It is not very clear what you want to say. Must be slightly extended the explanation.
“…advantages provided by Skeptical Learning….”
Should be mentioned the advantages. Comparatively to what are considered the advantages?
“The results highlight an unexpectedly high pervasiveness of user mistakes…”
This also depends on the specificity of the scenario. In another scenario, it could happen to be low. Based on this the scenario presentation must be slightly extended.
The problem proposed for solving is very interesting.
“2. The algorithm”
The title is crisp, it should be extended to be more expensive. Can be included for instance the name of the algorithm and the fact that is novel.
“2.1. The Prior Knowledge”
The same observation as above. You should extend to explain what kind of prior knowledge refer.
Algorithm 1 should be mentioned explicitly the input and output of the algorithm.
“2.3. The Predictor”
Must be extended to mention what kind of predictor do you refer.
The same observation also for: “3. The platform”, “3.1. i-Log”, “3.1.1. askUser”, “3.1.2. sensorReading” ….”4. The experiment” and the others (for example 5.2. The algorithm), to slightly extend them to be more illustrative to the content.
Algorithm 2 should be mentioned explicitly the input and output of the algorithm.
Algorithm 3 should be mentioned explicitly the input and output of the algorithm.
It is used the term “oracle label” but is not explained clearly to the reader its significance.
“6. Related Work”
It is better to change the title of the subsection to “6. Discussions”
Include in the bibliographic study
Zamfirescu, C.B., Duta, L., Iantovics, L.B.: On investigating the cognitive complexity of designing the group decision process, Studies in Informatics and Control, 19(3), 2010, 263-270.
II-Learn-A Novel Metric for Measuring the Intelligence Increase and Evolution of Artificial Learning Systems, International Journal of Computational Intelligence Systems, 12(2), 2019, 1323-1338. https://doi.org/10.2991/ijcis.d.191101.001
The reference list.
The references must be formatted according to the paper format requests. They should include all the details. For instance, “Giunchiglia, F.; Khuyagbaatar, B.; Gabor, B. Understanding and exploiting language diversity. International Joint Conference of Artificial 643 Intelligence (IJCAI17), 2017” – is not complete, does not contains details like the page numbers…..
Author Response
Please see the attachment.Author Response File: Author Response.pdf
Reviewer 3 Report
The paper is well written and easy to follow.
It presents an interesting approach skeptical learning, in which the feedback of the user is checked (and fixed when a problem arises). The authors present an application of this method in an university campus, where they deal with user location and transportation mode from sensor data.
The main issue with the manuscript is the fact that it relies a lot on the paper "Dealing with Mislabeling via Interactive Machine Learning" (which is not cited in the paper). Even the plagiarism checker detected 17% similarity between the papers.
Author Response
Please see the attachment.Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
The paper has been improved according to the revision requests. Now is appropriate for publication.
