A Field Method for Landscape Conservation Surveying: The Landscape Assessment Protocol (LAP)

Round 1

Reviewer 1 Report

The paper addresses an interesting topic, still there are several issues that need to be strengthened and/or reviewed prior to publication.

 

Literature review misses several seminal works and important advances crossing landscape assessment and evaluation. Read for example (Burley et. al., Loures et al., or Panagopoulos, regarding this subject).

 

Material and methods presents several flaws, considering not only that it is hard to understand the used methodology, but also the fact that research steps are not adequately described, neither is the identification of the strategy used to define/establish the 15 metrics of landscape evaluation. 

Additionally, I would seriously recommend the introduction of a phased methodological diagram.

 

The results are also not adequately presented. For example figure 2 presents little information regarding the objectives of the research. Further information is needed and landscape evaluation metrics need to be supported on further data.

 

Conclusions are very broad highlighting not only the limitations of the research but also a considerable lack of scientific soundness.

Author Response

Reviewer #1

We thank this reviewer for the helpful comments, and especially since he/she considered ours an interesting topic. He/she recommends extensive English editing; we have given the paper to a fluent native English speaker, Dr. James Manolis (The Nature Conservancy, Minnesota, USA) and have made corrections throughout this revised MS.

We respond to each comment:

1. Literature review: As requested by the Reviewer, we have carefully presented the reviewed literature and included many important papers, both in the introduction and discussion;  including papers by some of the authors he/she has suggested (LINES: 55).

2. Materials and Methods. The methods section has been completely revised as proposed by this Reviewer; especially the research steps and details of development; enhanced with a new table (LINES: 115).  The strategy/procedure to define the 15 metrics is expressed within the new table and this replaces the “thematic categories list” presented under the “2.1. Protocol philosophy” part. A phased methodological workflow diagram has also been produced; something also suggested by Reviewer #2 (LINES: 103).

3. Concerning the results section, major changes have been made as suggested by this reviewer. In terms of the comment on Figure 2 (which is a map with the applied LAP Conservation Index) we cannot provide “information regarding the objectives of the research” in this figure. However, we have included a better description of “objectives” within the methodology and this may help better orientate the reader. Following a similar approach, the landscape evaluation metrics are better described in the methods section which should help orientate the reader.

4. The discussion is not commented on by this Reviewer, but there is a comment on the conclusion. The conclusions in our opinion can only really be made up of broad generalizations, they do refer again to the problems and weaknesses; and the need for this tool to develop through further testing since it is at an early stage of development. And we are clear about the need for further scientific validation; this is not something that can be fully completed at this stage of development (and this is now made clearer due to the suggestions made by this Reviewer). We have slightly re-worded aspects of the conclusion to attempt to clarify and make the points sharper, as suggested by this reviewer. 

Reviewer 2 Report

Landscape assessment analysis is always a crucial point of studies: how to quantify, what to quantify and what is expected as a result. Independent and dependent variables are produced and finally, we have an outcome and some thesis-point. This study tried to provide a guidance for quick filed assessment, which can be used to assess landscape values, issues or degradation with a special protocol card. Authors tried to make an objective methodology and provided a justification of their surveys.

My opinion is that the paper is well written and has good research design. The description of results is also good, but I have some notes and questions for the authors.

I suggest to provide a better explanation for the assessment, some points were not clear for me:

-        a workflow figure would help to see the process

-        the testing is not clear (it is also suggested to put into workflow): if 35 sites were assesses, how the final result contain 170 completed protocols? three tutors, one undergraduate student and two PhD student were six, I understand that it means 5+1, but please make it clear;

-        I am missing the description of the study site, and some kind of outer justification of the scores, such as number of visitors

-        if there were 35 sites with altogether 6 evaluations, a PCA can be applied where a biplot figure can represent the correlation among the assessors’ points (alternatively, a Spearman correlation also can so it; i.e. the larger the correlation, the more stable the structure with higher reliability; furthermore, if authors consider this testing as an experiment with 5 repetitions, they can identify whether there is a consistent under or above scoring)

-        now the applied statistical test was a regression analysis with expert choice vs the average(?) of the five assessors – later the authors remarked that it is better to use collaborative group decision than the mean of the 5 scores, accordingly, my suggestion is to prove that: a regression with collaborative scores and the means can help in it … however, I also understand, if this option now is gone with summer students, and if this idea came too late, it cannot be produced;

-         a sensitive question: who can be an expert of assessing? why can his opinion be better than of 5 persons? Sorry, it is just a provocation, but also worth to think about: the more assessor has individual opinion, and provide a score, the more reliable and objective is the final result; we are able to think that we know better something, but when landscape is assessed by “eyes”, maybe the subjectivity arise; let me explain what I am thinking: a good point is to declare that what is the aim, who are supposed to be the users, and whose opinion counts; just highlight my concern, an engineer thinks that a power plant is nice (as a magnificent creation of planning), while others my think that is a disturbing feature of the landscape, such as a wound … maybe these thoughts can arise some new research topics only …

-        accordingly, the hardest question is that there is no reference, thus, the final scoring is hard to be confirmed … I suggested previously the number of visitors, or tourist nights at the accommodations, hotels, or simply the number beds at accommodations … these my reflect the popularity of the places except these place are closed from the tourists …

-        although the authors state that it is the best to keep the score calculation method as simple as possible, if they split the scoring into 2-3 parts, the partial results can help to identify the natural and the anthropogenic features, and the summation of the scores will not influence these scores; furthermore, the final score can still be true: if there are disturbing factors in one direction, the natural environment can counterpoint to another direction

Overall, although the paper is well written, I suggest to think over these requests and to rephrase the signed parts.

Author Response

Reviewer #2

This reviewer recommends only a fine/minor English language check and suggests that the introduction section is sufficient and includes all relevant references (in contrast to Reviewer #1). Despite his/hers comment that the paper is “well-written” we have re-checked English in-depth and extensive re-phrasing has taken place (see response to Reviewer #1).

Reviewer #2 acknowledges our efforts to develop an objective methodology and that we provided justification for this survey set-up. Specific points/suggestions made are addressed here.

1-A detailed workflow figure is incorporated as suggested by this Reviewer (LINES: 103).

2-Testing aspects are now made clearer (and also included into the new workflow diagram). And aspects of the trial have been incorporated in the diagram as well. Corrections have been made to the methods section, as revealed by careful scrutiny of the data (we had 5 participants; two tutors, 3 students that participated in the assessing; and another tutor who did not participate at all sites and did not contribute protocols). In total we have complied 173 protocols (two were not completed due to accessibility reasons). All this is made clear now in the revised methods and results.

3-A description of the study area is included (LINES: 155-165) and new literature is inserted about the study site.

4- As mentioned, there are only 5 assessors providing protocols (the sixth person, is a tutor who did not contribute protocols to the analyses but worked with the team; this mix-up created this silly arithmetic confusion in the first draft of the manuscript). New statistical approaches have also been incorporated and many points have been ameliorated; discussed below.

5- In order to improve our statistical evaluation of the scores, we calculated p-values between the designated expert and other assessor scores per site. In this way, we also address the comments of Reviewer 3. Table 3 (LINE: 211) shows number of sites per LAP quality class based on expert scores and number of sites where majority of other assessors (3/4) significantly differ in their scoring. The p-values between expert and other assessors’ scores showed significant similarities (p-value > 0.2) in 69% of all assessments. At locations where expert LAP quality class was excellent or good, the majority of other assessors’ scores were significantly similar to the expert’s score. The significant differences between expert and other assessors’ scores were recorded at locations where the expert gave lower scores (poor or bad) than the others. These sites comparably had lower scores from all assessors for road network, modern anthropogenic interferences and buildings. However, the expert gave lower scores for the other metrics as well in comparison to the scores by the other assessors. This might be due to the fact that other assessors underestimate the effects of anthropogenic inferences to other metrics (This interpretation is made on LINES: 202-209).

6-The "expert" title was bestowed upon one tutor that the others agree is most experienced and skilled in this work. As directed by this reviewer we make clear statements on how this expert was selected etc. (LINES: 170-177). This procedure is consistent with other rapid assessment testing circumstances (and we refer to Esselman 2001; who developed a new Stream Visual Assessment Protocol for Belize). Finally, in landscape assessment - the specific landscape vista is really difficult to compare to a background standard. Sometimes context can be more important than content in the landscape, so in this case the granting of expert-judgment to refer to a base-line is one way we see some kind of comparison to a base-line. Also, qualitative comparisons based on former landscape assessments and expert opinion among the 3 tutors also assisted in our assessment of reliability of the scores.

7-Yes, we agree completely about this, there is no strict base-line reference (no objective reference (as expressed above)). In our opinion the Reviewers suggestion of using tourist-accommodation in a small island is not a good comparative reference of attractiveness or other major landscape quality/conservation traits. Tourism development on Samothraki has to do with the idiosyncrasies of this particular island’s geography and its peculiar tourism development history (initially confined to the port and near the thermal spa on the north side of the island; later spreading in increments).  Again, we feel expert based ranking of sites is the best possible comparison under the circumstances; we propose more rigorous testing and validation.

8-Yes this is an important point and we have carefully crafted this point in the revision. The issue of eclipsing metrics(LINES:312-318); noted also here by this reviewer, i.e. natural environment can “counterpoint to another direction” is further explained in this revision in the discussion.

There are many approaches to analyze the data and some of these may or may-not be meaningful within the framework of such a paper. Ameliorations have been made in the paper (i.e. new Figure 3, use of p-values, new tables, distinctions clearly explained etc). Many analyses were not incorporated.  As an example, the comparison of p-values (metric-by-metric). A paired t-test was conducted to analyze significant differences and similarities between expert assessment results with the results of other assessors as well as between metrics (the significance level was set to 20%) - see table below. We have done several such analyses but chose not to exhibit them within this paper; if the Editor feels otherwise, we can re-consider.

Table. Metric to metric comparison p-values greater than 0.2.

Metric	Compared metric   where p-value is >0.2	p-value
Land Use   Pattern	Road   Network	0.46
	Modern   Antropogenic Int.	0.43
	Buildings	0.45
Vegetation	Flora	0.22
	Road   Network	0.43
Modern Antropogenic   Int.	Hydorological   Alteration	0.57
	Shorelines   &/or Riparian Cond.	0.22
	Wildlife	0.31
	Buildings	0.93
Garbage   & Debris	Agriculture	0.49
	Soundscape   Quality	0.63
	Wildlife	0.66
Agriculture	Soundscape   Quality	0.97
Hydorological   Alternation	Shorelines   &/or Riparian Cond.	0.37
	Wildlife	0.53
	Buildings	0.21
Shorelines   &/or Riparian Cond.	Landscape   Attractiveness	0.36
	Wildlife	0.57
	Buildings	0.94
Landscape   Attractiveness	Wildlife	0.73
Wildlife	Buildings	0.69

*The greater the p-value means the greater the similarities between metrics are. Please note that only significantly similar metrics (>0.2) were presented here. It is notable that for example "grazing" is scored significantly different than vegetation and flora (<0.2 not in the table) where one can expect that more grazing deteriorates vegetation and flora. This may be over-interpreting the behavior of the assessors, but it shows the difficulty of consistently assessing in anthropogenic cultural landscapes; something we do clearly mention this in the discussion.

Reviewer 3 Report

A field method for landscape conservation surveying:

The Landscape Assessment Protocol (LAP)

by Vlami et al.,

This study investigates a new field survey method to assess the conservation condition of landscapes. To reach this goal, the six conductors, a combination of three experienced and three inexperienced, assessed the protocol which has 15 metrics at 35 locations in an island, Samathraki, in the Northern Aegean Sea. One of the experienced persons is assumed as an expert; and his/her responses were compared with the mean of the rest.

An interesting study, especially, the simplicity of the questionnaire may ask help from community, and it will open new avenues in citizen science.

MAJOR POINTS:

1.      L156 Confusion about significance! I think the authors mention here the relationship between responses strong as seen R2 is ~0.81. But is it statistically significant? They need to report p value! Similar problem is seen in L164, the relationship is ‘weak’, but this weak relationship can be statistically significant. Please check your ‘significant’ usage. And if possible give their statistically significance level at least in L156.

2.      L183-5. Using mean is not good, may be distorted with extreme within the group. OK, did you try the median?

3.      I am also curious how did you choose one of the experts out of three? Did you check your response relations if you pick other two persons as an ‘expert’? I highly recommend commenting and comparing this what-if scenario!

FIGURES

L153. Figure 2. The resolution is poor! I recommended using better/higher one! Also Figure 4 at L 186.

L158. Figure 3. Resolution is poor! Please use higher one! Also add 1:1 line for better comparison. It looks overestimation at low values and underestimation at higher values in respect to ‘expert one’!

TABLES

L173. Replace commas with dots to identify one tenths.

MINOR POINTS:

Mostly Oxford commas, commas, subject verb agreement, and typos.

L19. Punctuation. ….; however,

L24. Plural verb. ….further X and Y are….

L50. Please check, I did not get ‘developed up’? Or did you mean ‘until’?

L51. Add comma after ‘procedures’.

L54. Add comma after ‘[21-22]’.

L55-6. Typo. Replace (14, 21) with ‘[14, 21]’.

L71. Add comma after ‘students’.

L71. Add comma after ‘acoustic’.

L96. Typo. Delete ‘_’.

L100. Add comma after ‘landscape’.

L109. Add comma after ‘simple’.

L112. Add comma after ‘percent’.

L126. I recommend following for defining countries: ….Vancouver, Canada; Jalisco and Nayarit, Mexico; Attiki, Greece; and Pafos and Lemessos, Cyprus……

L135. Comma after ‘student’.

L136. Add ‘an’ before ‘expert’.

L142. Add comma after ‘landscapes’.

L144. Add comma after ‘condition’.

L146. Add comma after ‘island’.

L161. Be careful about this sentence! Subject-verb-agreement is not ensured! I recommend following:  …the unscored metrics were used as the indicators of the….

L177. Add comma after ‘abondonment’.

L203. Add comma after ‘[7, 12]’.

L218-220. This sentence is a bit tricky. Decide where to put ‘comma’. Please check!

L223. Add comma after ‘(Table 1, Fig.2)’.

L224. Delete comma after ‘project’.

L233. Add comma after ‘degradation’.

L245. Add commas…… ‘, however,’.

L250. Add comma after ‘Samothraki’.

L278. Typo. Replace ‘(36)’ with [36].

L282. Add comma after ‘[30, 35]’.

L288. The verb should be singular?! …depends on….

L291. Add comma after ‘awareness’.

L295. I recommend hyphen between ‘local-scale’.

L323. Add comma after ‘Knoll’.

Author Response

Reviewer #3

This reviewer recommends only moderate English language changes and suggests that the introduction section is sufficient and includes all relevant references. The reviewer also recognizes this work as an “interesting study” and suggests that this work “will open new avenues in citizen science”- we appreciate these encouraging words.

Major points:

1- We thank the reviewer for this comment about significance. We have added a Table with p-values and set the significance level to 0.2.  (LINES: 211).

2-We have also incorporated the median (along with the mean) in our analyses in revised graphs (LINES: 218, 250). We have updated the Figures with additional data and trend line showing correlation with median values and the r^2 did not improve in Figure 4 (LINE 218). However, median values in Figure 4 correlates better with the expert scores therefore median values can be used for sites where overall scores are widespread. 

3-About choosing one "expert"; this was done simply by coming to a consensus among us as to the most experienced and skilled individual in this domain. We have added a description of this procedure show that the decision selected the person with deep background knowledge in index development and local knowledge of the study area as well (LINES: 170-177).

Figs and Tables: All figures will be provided in high resolution and the minor issue on the table former L173 (Current LINES 237) has been corrected (dots instead of commas). If there is a specific  resolution of figure definition required, please inform us and we will ameliorate this.

All minor points made by Reviewer #3 have been corrected (in blue highlight in the text).

Round 2

Reviewer 1 Report

The introduced changes contributed to strengthen the overall quality fo the paper.

There are still some english grammar mistakes and some typos that should be corrected. 

Reviewer 2 Report

I checked the revised version and the replies for my comments, too. I find this version much better than the previous and accept the submission in its current form.  I accept the anwsers, too, as most of my comments were to improve the scientific soundness and clarity of the content and to rise some questions which can be a basis of new submissions.

Reviewer 3 Report

Thanks to the authors for their effort to address my comments.