Decision Trees to Forecast Risks of Strawberry Powdery Mildew Caused by Podosphaera aphanis

Round 1

Reviewer 1 Report

Review

Decision trees to forecast risks of strawberry powdery mildew caused by Podosphaera aphanis

General

The manuscript summarises a study about factors that are needed to measure and use in order to get an accurate prediction of pesticide application needs. Timely fungicide applications is the goal, to find the economic threshold, to avoid spraying if there is no risk and to spray if there risk that the level of the disease will pass the economic injury level. Having collected an enormous amount of factors (94) the authors used a pattern centered statistical technique, a Decision tree classification technique to evaluate which factors could be used. The main results are not clear and some work is needed to enhance the new findings in their work and to more clearly express/discuss the consequences for the decisions in the field.

The aim was not entirely clear, yet I understand it as to improve the current decision methods and to be able to use only weather or weather, host, and inoculum data for prediction. Authors state there is a problem with current methods because they:

- require assessing disease severity on leaves

- does not reduce fungicide application

- do not take into account the interactions between factors.

Hence, I expected the current study to avoid these problems, yet I am not sure that was the result.

With more clear statistical explanations on how the accuracy was evaluated, some clarification, reduced result part, and the manuscript can be published.

Intoduction

Line 88: What is the problem with SafeBerry system?

Method and Results

Maybe good to make it more clear that host predictors are = “Host susceptibility was assessed based on the number of susceptible leaves per 1 meter of row”. It is not that easy to understand.

As the same 25 plants were re-examined many times, it means that you need to take that into account when you do the statistical analysis. How would it have been if you checked the same field on other 25 plants every time? How will it be, using the decision tree if different plants are checked?

Line 208: I do not understand “Did you not have all the data for all the sites?

Evaluation of the classification results was done using training and validation data sets. Which data was that? It is not fully clear to me.

Training, internal validation and external validation. It is not clear for me why this 3 classification metods were used and if any of them were more usefull or more accurate.

Which of the Reliability measures should be used? Can it be analysed what is best?

Line 211: The reliability of the trees in assessing classes of SPM severity was used to assess the preformace of the trees. Why was that important?

Line 356: There are some explanation of why the use of SPM class (1-4). Yet I would recommend to clarify it better. Is it so that when you reach class 4 (PLAD>15%), that you need to act?

The explanation why it is important to separate the data into classes needs explanation. Why is it importatnt to have risk classes? Why can it not be continuous?

Line: What does it mean in practice that the tree had lower sensitivity. What will that lead to in the field?

Line 415: You conclude that using inoulum, host and weather, you get the best results. However, then you both need to measure inoculum AND go out in the field and observe, no? Is that not what you wanted to avoid?

What does accurately mean? How is it evaluated/tested statistically?

Line 406-409. If the specificity was high (0.81-0.99 (are they different?)), 

I would highlight more in the article that the availability of inoculum data will always be giving the most accurate predictions. That is one of the main results no?

Figures do not have letters (A, B, C) as indicated in the text.

The result part can be much more condensed as much of the data is already presented in the table.

It might not be too important for the objective of the study, but it would have been interesting to see if there were differences between Classes of strawberry powdery mildew severity as shown in the figures, eg. Figure 1,2, and 3.

Format:

The number of the references does not fit. 14 should be 15 and 21 should be 22 etc.

Italic of species names

Author Response

With more clear statistical explanations on how the accuracy was evaluated, some clarification, reduced result part, and the manuscript can be published. Response: Done, see specific responses below

Intoduction

Line 88: What is the problem with SafeBerry system? Response: New information was added

Method and Results

Maybe good to make it more clear that host predictors are = “Host susceptibility was assessed based on the number of susceptible leaves per 1 meter of row”. It is not that easy to understand. Response: Done; host predictors were changed for host susceptibility predictors

As the same 25 plants were re-examined many times, it means that you need to take that into account when you do the statistical analysis. How would it have been if you checked the same field on other 25 plants every time? How will it be, using the decision tree if different plants are checked?  Indeed, the same plants were evaluated at each sampling. In order to have a large number of cases, as mentioned in the M&M, we used observations collected as part of another study and new observations collected specifically for this study. o properly use classification trees, it is important to analyze large datasets. In this way several conditions (values of the predictors) are considered. Also, disease severity was assessed only on the three youngest leaves, hence different leaves were assessed at each sampling.

Line 208: I do not understand “Did you not have all the data for all the sites? Response: all cases were used to build all trees, the differences between trees was in the predictors used (see M&M section2.3. Description of the response variable and classification trees predictors. )

Evaluation of the classification results was done using training and validation data sets. Which data was that? It is not fully clear to me. Training, internal validation and external validation. It is not clear for me why this 3 classification metods were used and if any of them were more usefull or more accurate. Response: classification tree is a supervised learning method, hence bootstrap validation (internal validation) method was used to validate the models (trees), and the 136 independent cases collected in 2015, 2016, and 2018 were used to evaluate their prediction accuracy (external validation). In all supervised learning method the data set is split into training and validation data sub-sets, here in addition, we added external data to have independent validation with data not used to develop the trees. Please note that this approach is common to all supervised learning methods, and use of external validation data acknowledged by reviewer 2 and 3.

Which of the Reliability measures should be used? Can it be analysed what is best? Response: it depends on the objective of the end-users. In all classification methods there are only four possible outcomes: true positive (TP) and true negative (TN) calculated as the number of cases and controls correctly classified and false positive (FP) and false negative (FN) calculated as the number of cases and controls incorrectly classified. If the objective is to protect the crops regardless of the cost, the user will consider mostly the true positives.  If the objective is to reduce the cost of control than the user may considerer mostly the true negatives. In general we are looking at a tradeoff between the sensitivity (true positive proportion), specificity (true negative proportion), and overall accuracy (proportion of good classification, TP+ TN) (see Carisse, O., Caffi, T., and Rossi, V. 2014. How to develop and validate plant disease forecasting systems. In Exercises in Plant Disease Epidemiology, Eds K. L. Stevenson and M. J. Jeger, 2nd edition, American Phytopathological Society APS-Press, St-Paul Minnesota, ISBN 978-0-89054-440-2). Practical meaning of the reliability measures was added in the discussion.

Line 211: The reliability of the trees in assessing classes of SPM severity was used to assess the preformace of the trees. Why was that important? Response: The objective of building classification trees is to find the tree with the highest reliability expressed as proportion of good classifications. In other words when the disease severity is in class 1, the tree predicts that it is in class 1, when disease severity is in class 2, the tree predicts that it is in class 2, and so on. Obviously it is impossible to achieve 100% correct classifications (accuracy) this is why we used reliability expressed as sensitivity, specificity, and overall accuracy as recommended by statisticians (see Hughes, G. 2012. Applications of information theory to epidemiology. American Phytopathological Society APS-Press, St-Paul Minnesota).

Line 356: There are some explanation of why the use of SPM class (1-4). Yet I would recommend to clarify it better. Is it so that when you reach class 4 (PLAD>15%), that you need to act? Response: done, new information was added.

The explanation why it is important to separate the data into classes needs explanation. Why is it importatnt to have risk classes? Why can it not be continuous? Response: information on the meaning of these classes was added in the introduction and in the discussion. The objective was not to predict the exact strawberry mildew severity but rather to predict risk. From a disease management standpoint predicting the exact % disease severity (continuous disease variable) is not required to decide if an action is required.

Line: What does it mean in practice that the tree had lower sensitivity. What will that lead to in the field? Please see previous comments on reliability. The reliability expressed as sensitivity, specificity. Response: the practical meaning of sensitivity, specificity and accuracy in relation to disease management objective was added in the discussion.

Line 415: You conclude that using inoulum, host and weather, you get the best results. However, then you both need to measure inoculum AND go out in the field and observe, no? Is that not what you wanted to avoid? Response: nowhere in the manuscript it is mentioned that our objectives was to avoid going in the field. Our objective was to use a different approach to assess SPM risk (see end of introduction).

What does accurately mean? How is it evaluated/tested statistically? Response: there is a complete paragraph in the M&M describing how the sensitivity, specificity and accuracy was calculated, and again in the table footnotes. The practical significance is presented in the discussion and we added few sentences to explain how it can be used to select trees.

Line 406-409. If the specificity was high (0.81-0.99 (are they different?)), Response: Indeed, it is possible to perform a statistical test to determine whether there is a significant difference between the reliability of classification trees, such as the area under the ROC curves (Receiving operating curve), however, it is not possible to perform a statistical test to compare measures of accuracy, sensitivity, or specificity for the different trees because there is no formal repetitions.

I would highlight more in the article that the availability of inoculum data will always be giving the most accurate predictions. That is one of the main results no? Response: yes it is discussed in the last paragraph of the discussion

Figures do not have letters (A, B, C) as indicated in the text. Response: done

The result part can be much more condensed as much of the data is already presented in the table. Response: done

It might not be too important for the objective of the study, but it would have been interesting to see if there were differences between Classes of strawberry powdery mildew severity as shown in the figures, eg. Figure 1,2, and 3. Response not done because the classes were delimited based on significance in terms of disease management thresholds, hence whether or not there are significantly different is not important, really.

Format:

The number of the references does not fit. 14 should be 15 and 21 should be 22 etc. Response: the numbering was ok but during the conversion from the word to the PDF document the numbering included the line with ‘references’, hence modifying all numbers.

Italic of species names. Response: done

Reviewer 2 Report

A very good study in which the authors used the CART algorithm to build a decision tree model using many of the factors affecting the severity of powdery mildew of strawberry. They included not only weather factors but also inoculum concentrations and host tissue age, which greatly affects susceptibility. They used earlier data to build decision trees; then used more recent data to validate the model. Thus, a large amount of data were included, which greatly increased the reliability of the study. The decision tree should allow the development of predictive models useful to growers. They evaluated some factors that are easy to measure, i.e., weather, and more difficult ones such as inoculum. Regardless of the predictors used, the authors found that the specificity (proportion of true negative) was high. This is a significant finding since it would allow growers to predict the absence of disease and avoid unnecessary sprays. I found the first paragraph of the Results difficult to follow, but I am not sure that can be improved.

Overall, the paper is well written, but I have included a few corrections and comments as Sticky Notes on a pdf copy.

Comments for author File: Comments.pdf

Author Response

Response: All comments made as Sticky Notes directly in the PDF were considered and appropriate modifications made.

Reviewer 3 Report

The paper is a very well written and designed study of disease forecasting in strawberries. The authors explain things well including their methodology and having a sound validation. I especially like the fact that the authors used data from separate years for the validation that they made.

I have a few suggestions to improve the quality of the manuscript.

First, can you please explain how the leaf area was measured? Was it on the 25 plants? Later, in the manuscript you make mention that a scout could collect these observations. If so, could you please describe how long this observation takes to collect?

Please include figures for all four trees. Use the supplemental if needed.

In table 1, I found the description of the variables confusing. For example (Daily, day and night mean, minimum and maximum temperature; previous 6-day means) = 3 x 3= 9 variables.   Or is it daily and 6 day means? How does it equal 12 variables? Can you explicitly define all the variables even if it is in the supplemental.

In the results lines 286 to 301 should be presented as a table. It is very hard to read in text format!

In addition, lines 388 to 414 belong in the discussion since this is description of the results. This text could be summarized in the discussion leaving more room for comparison of your techniques with others in the literature. For example, how do your techniques compare to other approaches for strawberry disease forecasting, for example the system used in Florida? How does this technique compare to machine learning approaches? 

Since airborne inoculum is difficult to measure, can it be measured at one site and extrapolated to others?  That is what is the inter-site variation in airborne inoculum?

One of a question should the scientific name be italicized? Please check the journal requirements.

Overall I compliment the authors for a great manuscript.

Author Response

First, can you please explain how the leaf area was measured? Was it on the 25 plants? Later, in the manuscript you make mention that a scout could collect these observations. If so, could you please describe how long this observation takes to collect? Response: leaf area was not measured it was the percent leaf area diseased which was estimated using a using a diagrammatic scale with 5% steps (0%, 5%, 10%, 15% … 100%).

Please include figures for all four trees. Use the supplemental if needed. Response: partially done

In table 1, I found the description of the variables confusing. For example (Daily, day and night mean, minimum and maximum temperature; previous 6-day means) = 3 x 3= 9 variables.   Or is it daily and 6 day means? How does it equal 12 variables? Can you explicitly define all the variables even if it is in the supplemental. Response: DONE, Table 1 was rearranged

In the results lines 286 to 301 should be presented as a table. It is very hard to read in text format! Response: DONE, the selected weather variables are highlighted in bold in Table 1 and consequently the description was removed from the results section.

In addition, lines 388 to 414 belong in the discussion since this is description of the results. This text could be summarized in the discussion leaving more room for comparison of your techniques with others in the literature. For example, how do your techniques compare to other approaches for strawberry disease forecasting, for example the system used in Florida? How does this technique compare to machine learning approaches? Response: Done the discussion was reorganized, the paragraphs which repeated the results section were deleted and more information on strawberry forecasting systems was added.

Since airborne inoculum is difficult to measure, can it be measured at one site and extrapolated to others?  That is what is the inter-site variation in airborne inoculum?

Response: the spatial variation in airborne inoculum was studied and published reference (ref 43). From this study it was concluded that airborne inoculum concentration was uniform and hence only one sampler would be needed. This was added in the discussion.

One of a question should the scientific name be italicized? Please check the journal requirements. Response: DONE

Overall I compliment the authors for a great manuscript. Response: thank!