Optimizing Crowdsourced Land Use and Land Cover Data Collection: A Two-Stage Approach

Round 1

Reviewer 1 Report

The paper presents the results of research on classification tasks involving visual interpretation of images are one of the most important areas of application are classification tasks involving visual interpretation of images on the example of creating land cover and land use maps. The object of this paper is to achieve accuracy in these classification tasks at minimum cost. The authors used a Bayesian approach, which provides an intuitive and fairly simple solution to achieve the research objective.

This paper presents a two-step approach for collecting the additional information needed in the Bayesian approach. Its application is demonstrated on a hypothetical two-class example, and then applied to a real dataset of 5 classes, which comes from a previous Geo-Wiki crowdsourcing campaign on identifying crop field sizes from very high resolution satellite images. The results obtained confirm the hypotheses of the paper and are a useful solution not only for Geo-Wiki campaigns.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

The revised version by Moltchanova et al. provides a number of improvements of the previous manuscript submission. During the first round, I suggested a major review not because of lack of a scientific basis but because I felt that the communication of relevance (in particular with respect to the readership) needs to be improved and that this work needs to be put into a larger context.

While still in favour of seeing this paper being published, I do not see how any of the issues I mentioned were addressed. I see improvements regarding the description of experiment conduct, I see some modifications regarding formalism, but i neither see the larger context nor a discussion of relevance (and thus potential), when, e.g., also compared with other classification approaches. It is a work very much focussed on itself and I still do not believe that this suffices in itself. I therefore recommend major revisions and I would in fact appreciate to see a author's response letter for an upcoming revision -- if there is any.

For reference, I paste my previous review down here along with some marks:

"thank you for the interesting read that you have provided with this manuscript. While the relevance of the topic is evident, I do have a number of issues regarding its communication and would therefore suggest a major revision.

That suggestion is not related to the methodological approach, as I have a sound understanding but I am not an expert in statistics. I rather refer to two aspects: (a) communication of approach to a wider readership, (b) relevance for the journals' focus. Both aspects are somewhat related.

(a) While the exposition is clear and the need to investigate that issue has been communicated quite well, the methodological approach lacks a bit of transparency and a wording that is appropriate for the readership. I would strongly suggest to review section 2 in particular and see if you can provide a few more explanations and references that allow the reader to follow your explanations -- beyond the mathematical rigor. Why i mention this, is because (b) remains unclear.

The data you are using for investigation has only marginally something to do with topics of land. It could be any other type of object and your approach would be the same. Since that is the case, I would expect a more detailed discussion about the relevance and implications for the readership.

I was expecting a detailed introduction into the data acquisition and the resulting data from the survey but I just saw myself confronted with a few dry statistics and a table. Not even a figure. It cannot get more abstract than this. Line 109 lists the results but -- although familiar with that kind of data -- I feel lost as I do not know what you are referring to. The reader can certainly also read the paper by Lesiv et al., but I feel the whole experiment and its results should be better presented here in your manuscript to make it complete and coherent. I also find half a page of description not overwhelming given that any sort of background information is missing.

It might also be a good idea to provide readers and interested users with a few instructions of how to make use of your code by introducing a small use case perhaps -- a use case perhaps related to topics of land? (...)".

Kind regards.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

The text is very well written and brings new and interesting perspectives on the use of citizen science. However, I think its inclusion in Land is questionable - it would be more suited to one of the special issues of Sustainability, for which these special issues are still open. The thematic inclusion in Land journal is atypical.

On the theoretical anchoring side, I have no comments; it is evident that the authors are well oriented in the recent literature. However, I would also recommend that the authors work more with The science of citizen science book (the authors cite 1 chapter from it) - in particular, the chapters on machine learning and conceptual model for participants are very useful and would complement the text appropriately.

Separation of the Discussion and Conclusion sections should be considered - this way the polemics of the discussion merge with the conclusions reached by the authors.

Overall, I recommend the text for publication after incorporating minor changes and considering whether to redirect it to another journal closer in topic.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

I here refer to the first revision of a re-submission of the manuscript "Optimizing Crowdsourced Data Collection: a Two-Stage Approach" by Moltchanova et al. submitted to mdpi/Land.

It seems that this revision round has been a cycle of re-sending old comments on both sides. Rather than dissecting the origin of that issue, I'd rather suggest to pick up the lose ends and I here refer to the comments that have now been re-sent by the authors, including their recent additions.

I suggested to improve two aspects
(a) communication of approach, and
(b) relevance with respect to the journal's focus and the journal's readership.

I agree that these two comments in particular might seem unspecific. That is because I could not establish the connection between the need and the provided solution. The intention of course is clear, but the case -- at this point -- has not well been established for me. Hence, my comment lacks specification. The question I would like to see answered is why the authors have chosen this approach. It is not about the derivation of equations, it is about the overarching framework and the thoughts behind this.

I did notice some modifications and the authors also added more context/motivation in L40+ which I appreciate. It might seem all obvious to them but I think the reader will appreciate the more explicit connection that is establishing now. At the end, however, it all points back towards the initial lack of communicating why the authors have chosen to address this topic by this method. If that is clearly communicated, the level of maths and potential other use cases remain of secondary importance.

I will be very specific here and phrase what I would expect from authors making a contribution to a broader audience with readers from the social and natural sciences alike.

I would expect from authors to establish the motivation by providing a literature background on what has been done in that particular field, and what are the limitations and challenges. And by that I am not referring to making some general statements, but to establish and explain the issues without assuming that all readers have read all papers being cited. There is probably no second opinion on the fact that the authors have not provided any decent literature background. They jump into medias res and while now -- in the second revision -- some background has been established, the methodological background is missing.

Up to line 49 it is all well now. Then the authors start listing some papers and putting them into some topical drawers without any background ("subject of intensive research", "including comparisons in accuracy between experts and volunteers", "Other papers have outlined"). Then they state, that there are basically two approaches but do not provide background. They list the issues related to simple majority, but they do not provide any background to "A Bayesian approach provides an intuitive and fairly straightforward solution to the above problems." How is it addressed, in which cases, what are some of the key ideas? "However, its application requires additional information such as the relative frequency of classes and the user accuracy". Why? When do they want to show that? How has this issue been overcome in other approaches if "... Bayesian approaches are the most popular methods"? Clearly, there must be something?

Further, they find "A Bayesian approach provides an intuitive and fairly straightforward solution to the above problems" though the statement remains unsubstantiated.
They then focus on the general approach by stating "To overcome this challenge, we present a two-stage approach in which a subset of the data are collected for calculating the user accuracy followed by a second classification stage that covers the full data set." but it remains unanswered why this is a feasible (or perhaps the only feasible approach)? They hint at the reason within the first paragraph of the methodology, but again, I would expect background in the introduction.

The description of methods is clear although I believe the setup and, again, the reasoning, could benefit from some background about the concept (which would be easily covered in the introduction) -- but I still wonder how referring to a text book and communicating expectations about the math level of a readership feel appropriate during a review process...

The other issue is how it related to land and how it can be of interest for the readership.
I acknowledge that they do not agree with my argument, and that is fine. Perhaps we are not talking about the same point here and I will again, try to be specific, as requested:

The classification as exemplified here is related to land. I agree. LULC is related to land and the authors have established that connection now, I also agree. Half of the paper, however, provides the application/derivation of how the Bayesian approach could be used for overcoming common problems in classification, i.e. seven pages of methods centered on the approach. The results are not even one page about accuracy and the discussion is about one page on approach and accuracy. After two review rounds the authors squeezed out some general connections to how this approach could be used perhaps in a larger context. That is fine by me. The first paragraph of the discussion reads more like a literature review, while the last paragraph does not even provide references (one, that is) to _discuss_ results in a wider context and with comparisons to results obtained by other researchers. After all, the general approach is not novel as stated in the introduction, so I would expect some comparisons and discussions about the gain and novelty after all. And the conclusions, half a page in length, again focus on the approach only.

This is why I made the point that the relevance for the readership is still lacking. Just because the method takes land-use classification as an example, does not mean it is a paper that is centered on land-use issues (with all its implications) if the connection is not established. At least 5/6 of the manuscript is focused on the method, there is
(a) barely some background information,
(b) no in-depth discussion about concrete implications and comparison against the published literature,
(c) nothing said about how to practically work with the approach for the standard LULC researcher or in land management in general.

To me, while accomplished to communicate a detailed methodological approach, the research remains method-centric and is largely dissociated from the reader. As I stated earlier several times, I would like to see that manuscript being published, so I am neither doubting its value nor its originality... but I do also have some issues with its presentation.

I would ask the authors to discuss this matter with the editors as I do not believe we will reach consensus here otherwise.

Kind regards.

Author Response

Please see the attachment

Author Response File: Author Response.docx