Accuracy Assessment of Digital Elevation Models (DEMs): A Critical Review of Practices of the Past Three Decades

Round 1

Reviewer 1 Report

This manuscript gives a very good review on the accuracy assessement of DEMs. It’s a very detailed and high-quality work. The review is performed very critically as it should be. It is well written and well-structured, and discussing all the actual issues related to the quality and accuracy assessment of DEMs. The issue considered in the manuscript is very important to the scientific community, therefore, this work can be of great use for the geospatial community.

The manuscript is scientifically sound and it deserves to be published, in present form, in Remote Sensing. Overall, the paper is well written and organized, though I have tried to point out some comments and grammatical /typo errors, (see comments).

 

Comments:

>> Line 28 to 31: DEM is a generic term and can be referred to as DTM and/or DSM. According to Oksanen (2006), the terms, DEM, DTM, and DSM have a context-dependent implication and there may be differences in meaning from one country to another. It will be preferable if you will be more specific about these terms.

>> Line 33: …mesh(lattice… => …mesh (lattice…

>> Line 308 to 321: The authors gave a good explanation for their choice of classes, however, the class “Remote Sensing” can be changed to “Photogrammetry” as it refers to the used technology such as other classes “InSAR”, “LiDAR”, etc. Also, the class “Topography” can be renamed to “Ground Surveying”.

>> Line 345 to 346: is UAV-based DEMs are included? Because the recent years have witnessed an increase in the use of these DEMs because they give a very good alternative to generate very high-resolution DEMs. Therefore, this technology should be added to the analysis.  

>> Line 420: The authors add a new class; “GEDTF”, into their analysis without any explanation, try to explain it in the text and add the full name.

>> Line 488: D, 2D+1D… => 1D, 2D+1D

>> Line 588: forested and built-up areas)

>> Line 597: 960 Km2 => 960 Km²

>> Line 703: AS shown => As shown

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript is well structured and written, it only requires some improvements, as suggested in the pdf.

I believe that the article highlights very important problems related to the use of DEM models: the lack of standards to define accuracy and to obtain useful and comparable results. It is a current topic that needs to be addressed by the scientific community and the standardization of procedures, both by producers and users, would improve the quality of research.
Despite this, I believe that the manuscript has more the aspect of a review, as the discussion highlights what emerges from the analysed literature and ends with suggestions (line 834).

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear authors, it is visible you put a large effort in drafting this manuscript. However, I still think there is room for the improvement and that it should be significantly better to be published. Please see my comments below.

• Keywords, maybe add: Quantitative analysis, standardized procedures, …
• Title, maybe change: A critical review of current practices
• Section Titles typically do not have dot at the end
• When enumerating the equations, common is to use notation “(1)”, “(2)”,… and not “Eq.1”, … Also it is common to have an introductory sentence before introducing the equations ending with colon symbol
• Avoid very long sentences. E.g. lines 172-174 should be split in two.
• Too long: there are many sentences which are completely unnecessary or redundant (very similar ones appear directly before or after). The text can be reduced for 30% without losing any relevant content or quality. E.g. remove lines 212-216, line 117-188, lines 238-242 or 191-192 & 208-211, 247, Section 3.3,305-306,…., 708-711,…
• General remark: As a scientific writer, your main goal/aim is to present a data or a message with a minimal number of words as clear as possible. Not to present data or a message as nice as possible in a literate way. Keep that in mind (I am also personally struggling with this).
• Please explain me in short why should someone read the content of pages 6-15? How can it be helpful to someone? How can it contribute to the scientific community? Please give at least 3 reasons why is this data relevant, especially in the connection with the main aim of the article (DEM accuracy assessment). Please for each Figure/paragraph you are not sure how to answer to this, remove it from the article. (I mean, it is nice to know that Germany cares the most in Europe about this stuff, but this is material for job-searching website).
• Just a thought: Lines 247-254 – your own sentences say that maybe you should have better spent your time by analyzing new measures for fitness for use, because simply RMSE or standard deviation of the pixelwise differences in heights are becoming less relevant. Also lines 84-85 say “to the best of our knowledge, there are no methods for the quality assessment centered on the fitness for use”, but now you say there are some since 2012? Do you want to say that you just did a bad literature review?
• I do not see it as acceptable to separate the origin on “DEM” by different sensors and here include “DEMs” given from 3^rd parties as a separate class. You should investigate which sensors are used from the 3^rd party or exclude such instances from the analysis (at least from the analyses for which this is a relevant information).
• Line 42: relation with the precision, not necessarily to accuracy, if the accuracy is conformity with the true value
• Line 43-44: missing some references, this is a review article in the end
• Lines 44-47: confusing sentence
• Line 49: everything behind …remote sensing [5]… can be removed from the brackets, all can be considered part of the remote sensing
• Line 108: “which is nothing more than a particular case of positional accuracy” -> either this part of the sentence is redundant or whole Section 2 is redundant. But do not try to explain something in a sentence that you will be explaining over a whole page.
• Lines 107-112: Make references to sections as Section 1, Section 2, Section 3…. This paragraph should serve as a very short table of content.
• Line 149 – atypical errors -> more commonly used terms are blunders or outliers
• Figure 2 and following paragraph -> I can hardly see a potential value of this data for the readers. Can you please elaborate this, give an example or simply remove it.
• Figure 3 and surrounding text – not understandable:
  • How can source of DEM be DEM?
  • Do you have 0 articles on ICESat (I would prefer “Satellite altimetry” as a class…) and TLS?
  • What classes Remote Sensing, Topographic Maps, DEM, Topography really stand for (all methods they comprise).
• If: Topography and GNSS techniques are not very representative, then why are they separate classes?
• What is the difference between topography and topographic maps?
• Figure 5 -> you need to explain why it is there and what is it about. The explanation needs to be positioned next to the figure. Give it a separate section, or place the figure (with explanation) at the beginning of section 4. Most of the entries are not self explanatory. Also, what “ground cover” has to do with DEMs?
• Section 4.2 -> haven’t you already stated that you will be analyzing only grids ? I do not understand the purpose of this section.
• It is recommended to avoid such phrases in academic writing is something that seems quite obvious
• Figure 7 and corresponding text – unnecessary, also quite obvious as you say + how the hack you have “mesh” here? I thought we already agreed few pages back (twice) that you will be concentrated only on DEMs represented with regular grids or rasters.
• Chapter 4.12. Is the most interesting and the most relevant part of the article and it should be expanded.
• Lines 690-691: Interpolations are necessary to calculate the discrepancies between elevations of the same location of the DEM product and the reference. -> This is true, but this is the less relevant interpolation step. The more relevant one that you should also mention is filling up missing data for the nodes in the grid when creating DEMs. This is the one that is analyzed in majority of the articles. Your sentences leave different impression.
• In our study an exhaustive analysis of these metadata has not been carried out, but the feeling we obtain from what we have analyzed is that the quality part of these metadata either does not exist or is very deficient. -> Having a feeling is not acceptable in scientific writing. You are writing a critical review, and this way of reporting is not sufficient. I recommend that you further investigate this metadata to the further extent possible before resubmitting the article. Ideally, the class you have in many of your figures, 3^rd party generated DEMS, should disappear.
• Line 831: something is wrong with the citation
• Lines 855-856: you probably mean evaluating/documenting/… the quality of DEMs is a challenge for the…
• English should be re-checked, preferably by a native speaker. It is mostly good, but there are some instances where it really needs to be better. Some examples:
  • L47: analysis of algorithms? – not sure if correct
  • L51: great relevance for, not on
  • L70-71 unclear sentence
  • L74: Remove semicolon, generally avoid using semicolon. That symbol should be used for a very special cases (preferably never), not as frequent as colon or dot.
  • L100: replace vision with overview
  • L101: remove ´s from DEM`s
  • L185-187: Unclear sentence
  • Line 201-204: Unclear, too long.
  • Line 521: superficial is a wrong term for this

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

Dear authors,

Thank you for considering most of my comments. Now, I am recommending minor revision before the final publishing, as there were a few points that were unclear in my first review. I apologies for that.

1. I actually wanted you to remove dots at the end of the section titles. Sorry for the confusion. The thing is I have never seen an article with dots at the end of the section titles.
2. Lines 708 - 711 (talk about simulations). My question for you is: do you think that any potential reader of your scientific article does not know what is simulation? I would recommend you removing these two sentences. On some way, you are a bit offending us.
3. Regarding my comment about the content of pages 8-15. I am still not convinced about this material being so detailed in this article. Your reason number 1 is that you will need this in your future work. Then save it for you future work. Then your future work will have more new things to show + readers will read directly relevant info. in one article and they will not need to search for it in this article. Your reason number 2 is more like a political then scientific answer. Those are nice sentences, but from there I have no clear picture (clear picture == example(s) ) how producers could actually benefit e.g. from knowing that most DEMs have resolution of 30 meters. I politely ask you again, please consider shrinking this part, or at least give a few sentences with a hint how this can help someone (and not the political ones, but concrete hints how someone can use these values/numbers).
4. Based on your comment I would definitely change "Remote Sensing" class with "Stereophotogrammetry". Remote Sensing is very wide term that contains most of your classes: LiDAR, Stereo images, Satelite measurements and TLS.
5. THE BIG ISSUE:

This is your answer to my last review: Although, in this work we intend to review the methods of assessment the quality of the DEMs, taking into account all the most relevant aspects. -> You say here finally quality, and then I agree: the resolution and area covered with the DEM are related to quality of the DEMs. However, your Tilte, Abstract, Sections 1 and 2 are strongly focusing on accuracy, and not the overall quality. Hence, your fist part of the article is contradicting the content of the 2^nd part. Please, for the sake of the quality (not accuracy) of your own article consider this. Either change first part of the article to speak more about the overall quality and not strictly accuracy or substantially reduce content of sections 8-15.

6. On comment starting with: “Lines 690-691…” I really liked your answer. I think you should include this somewhere in the instroduction.
7. Regarding the last sentence of your conclusion. I think it is too unclear. “the quality of DEMs is a chellange…” can be interpreted as we cannot achieve sufficient quality, hence we need to be working on increasing quality and accuracy. And I do not think you meant that. So please try to rephrase in a sense that adequately documenting and assessing quality is the big chellange.

That’s it. After this you are ready for publishing a really nice review article. I wish you a lot of citations and good luck.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf