Enhancing Smallholder Wheat Yield Prediction through Sensor Fusion and Phenology with Machine Learning and Deep Learning Methods

Round 1

Reviewer 1 Report

1.     I cannot agree with that field-scaled prediction based on remote sensing is rare. Lots of effort has been done in this field, and this authors should rearrange their motivations.

2.     How to define the smallholder? It appear in the topic, but failed to be discussed.

3.     The language is confused and not friendly with readers, such as line 10-11, what is ‘their’ and what ‘yield prediction methods’; and in smallholder scale, what is the specific challenge? What is the predictors? And hidden layers? Line 22-25. This paper is full of these issues that I cannot get the detailed meanings of the authors

4.     The study purposes (135-140) have not been relied well in the conclusions and abstract.

5.     The major concern is how is the unique contribution of this study by using this method? And I cannot find detailed validation procedure (based on what data to validate so many small fields?)

Author Response

Reviewer one

# 1. I cannot agree with that field-scaled prediction based on remote sensing is rare. Lots of effort has been done in this field, and this authors should rearrange their motivations

Response:

We accept the comment that several efforts have done in developing field-scale prediction methods. However, in comparison to efforts made at regional, national and international scales, field-scale approaches are yet seldom. This is due to the well-archived medium and low resolution sensors such as Landsat and MODIS favors and enabled more number of researches at regional scales than field-scale. This was the point that we tried to communicate, but, failed to do so. In this regard, we accept the comment and make the point more explicit that the narration is made in relative terms.

#2. How to define the smallholder? It appears in the topic, but failed to be discussed.

Response:

We directly accept and present the widely used definition and characteristics of smallholder system within the first paragraph of the introduction. I.e., “Farm units less than 2 ha area and managed by family labor are considered a smallholder system (Rapsomanikis, 2015).”

And we added “In Africa, the median size of a crop farm is between 1 and 2 hectares and the vast majority of farms are less than 5 hectares (Dercon and Gollin, 2014)” within the third paragraph of the introduction.

# 3. The language is confused and not friendly with readers, such as line 10-11, what is ‘their’ and what ‘yield prediction methods’; and in smallholder scale, what is the specific challenge? What is the predictors? And hidden layers? Line 22-25. This paper is full of these issues that I cannot get the detailed meanings of the authors

Response: concerning issues from line 10-11”

We replace the sentence “Despite their significance in many major global projects, remote sensing-based field-scale 11 yield prediction methods are seldom available” by “Field scale prediction methods using remote sensing are significant in many global projects, however, the existing methods have several limitations ”.  This will avoid the usage of too many noun strings and add clarity.

Concerning issues from line 22-25

We replace the “A deep neural network of three hidden layers” by “A deep neural network with three hidden layers”. This will explain more the meaning of the sentence.

Concerning “This paper is full of these issues that I cannot get the detailed meanings of the authors”

We appreciate and kindly accept the concern by the reviewer. In this revision, throughout the paper, we rigorously address the usage of too many nouns together, reduce the number of sentences with passive voices and try to make reader friendly sentences reducing and replacing confusing words.

# 4. The study purposes (135-140) have not been relied well in the conclusions and abstract.

Response:

We partially accept this comment.

The study set three objectives: “First, to evaluate the potential of selected vegetation indices derived from Sentinel-2, representative of an optical sensor, as a wheat yield predictor. Second, to evaluate the potential of selected SAR indices derived from S1 as a wheat yield predictor. Third, to apply fast, reproducible, and open-source statistical, machine learning and deep learning algorithms for wheat yield prediction under a small dataset domain.”

In the abstract, as it has to be concise, we present a general objective summarizing the three objectives. However, in the current revision, we add some words to increase conformity. In the conclusion, we add additional sentences which describe the specific objectives, i.e.,” This study was set to evaluate the potential of selected vegetation indices derived from Sentinel-2 and Sentinel-1, as a wheat yield predictor. Besides, it applies fast, reproducible, and open-source statistical, machine learning and deep learning algorithms for wheat yield prediction under a small dataset domain.”

#5 The major concern is how is the unique contribution of this study by using this method? And I cannot find detailed validation procedure (based on what data to validate so many small fields?)

Response:

Concerning unique contribution

We kindly try to explain what the unique contribution of the manuscript as follows:

1. The study implemented satellite based yield prediction approach using purposively selected statistical, machine learning and deep learning methods under a small dataset domain. In other disciplines, for instance, in material science, machine learning and deep learning models were applied under a small data set context for prediction problem (regression) and promising results were reported. This paper takes such studies as foundation and applied for crop yield prediction using a small dataset, i.e., yield data collected from 165 wheat fields, and good prediction results were obtained. In doing so, the study is the pioneer (to the best of our knowledge) to implement knowledge transfer from other disciplines to agriculture.
2. The application of machine learning and deep learning methods is challenged by tiresome process of repeated training, optimum parameters search and validation. In particular, the application of these methods under a small dataset domain further increases the challenge, demanding the design of reliable and robust approach. To this end, in this study, a combined method, AutoML with GLM hyper-parameters showed higher performance over the rest methods. The AutoML sub-method is found to be a straight forward and less complex application of machine learning compared to the conventional algorithms. It complements well with the GLM algorithm and revealed high performance. As the development of automated machine learning models in agriculture is currently a trending topic, the current finding contributes to the knowledge gap in the topic.

Concerning detailed validation procedure

Response: we refute this point as we implemented rigors validation methods and kindly present our response as follows:

The design of machine learning and deep learning under a small dataset domain requires primarily to develop a robust validation approach that explicitly assess with higher reliability the major problems encountered during the learning process such as, bias-variance trade off and controlling over fitting. In this regard, the study extensively assessed various validation approaches and implemented three groups of validation methods where they are appropriate to the specific algorithm.

1. For nlsLM model, a leave-one-out cross-validation (LOOCV) technique was applied. LOOCV is a special case of k-fold cross-validation technique in which the number of folds is the same as the number of observations. It reduces bias and randomness and controls over fitting and it offers a comprehensive evaluation as it uses all samples for validation. In this study, the model goodness of fit was assessed using the root mean square error (RMSE) and leave-one-out cross-validation root mean square error (LOO RMSE). (see figures 3-5)
2. For GLM model a fivefold cross-validation method is used to validate model performances. Besides, as the stochastic nature of the machine learning algorithms constrain yielding reliable validation outputs, the study applies a robust method of validation by determining the confidence interval for the population mean.(tables of 4, 7 and 8)
3. In addition to metric-based validation, model performance was assessed using scatter plot analysis. The scatter plot analysis between measured yield and estimated yield was implemented for several randomly selected seeds on test dataset. (figures 4 and 11)

Concerning the small field size vs impact on validation

Response: we refute and kindly present our response as follows:

The area of the study farms spans from 0.12 ha to 2.13 ha with an average area of 0.53 ha and most of the farm areas, as shown in the histogram of figure 1c, have an area less than the average farm size. The effect of farm size is associated with the spatial resolution of the sensor, i.e., for instance, in using sentinel-2, a 0.53 ha farm is expected to be represented by 53 numbers of observations (a pixel has 100 square meters area). And for the smallest farm size, 0.12 ha, 12 numbers of observations are obtained. As presented under section 2.5 an average value is used to aggregate such observations per a farm boundary.

Rather, the small number of observation fields (total of 165) is the one which influences the validation process which is addressed in the previous response.

# 6 The reviewer also ordered major revisions and improvements in the following areas:

• Extensive editing of English language and style required
• Does the introduction provide sufficient background and include all relevant references
• Are all the cited references relevant to the research
• Is the research design appropriate
• Are the methods adequately described
• Are the results clearly presented
• Are the conclusions supported by the results

Accordingly, we did extensive improvements to address each of the points so that the standard of the manuscript in terms of presentation, language and scientific quality is improved.

Reviewer 2 Report

General Comments

1. Although the authors use the H2O machine learning platform to realize the algorithm, but the theory of the algorithm  needs to be explained in method and note the advantage and disadvantage of the algorithm.

2.  The precision comparison or validation section of method should be added.

3. Line 299, Why repeat 30 times，but not 20 times or 40 times. Please explain the reason or describe the the principle for determinating the  30 times.

4. the disccusion should be revised deeply,  include but are not limited to  the comparation between the algorithm and compared with other research and uncertain analyze.

 

Specific Comments

 

1. The VI in table 1 should add the cited reference one by one.

2. End of the line 195 should add a end mark.

3. The VI in table 2 should add the cited reference one by one.

4. Line 240, Figure 2 , S2 third box "Indices and" delete the "and".

5. Figure 2 "nlsLM" is the first shown in manuscirpt,  I suggest to change to "Nonliner Regression" will be better.

6.figure 3,4,5 RMSE add the unit. Figure 11 add the r (correlation coefficient).

Author Response

General comments

# 1. Although the authors use the H2O machine learning platform to realize the algorithm, but the theory of the algorithm needs to be explained in method and note the advantage and disadvantage of the algorithm.

Response:

We accept. In this revision we presented a concise theoretical description of the an automated machine learning, GLM and Deep learning under sections 2.6.2 and 2.6.3.

#2.  The precision comparison or validation section of method should be added.

Response: we refute this point as we implemented rigors validation methods and kindly present our response as follows:

The design of machine learning and deep learning under a small dataset domain requires primarily to develop a robust validation approach that explicitly assess with higher reliability the major problems encountered during the learning process such as, bias-variance trade off and controlling over fitting. In this regard, the study extensively assessed various validation approaches and implemented three groups of validation methods where they are appropriate to the specific algorithm.

1. For nlsLM model, a leave-one-out cross-validation (LOOCV) technique was applied. LOOCV is a special case of k-fold cross-validation technique in which the number of folds is the same as the number of observations. It reduces bias and randomness and controls over fitting and it offers a comprehensive evaluation as it uses all samples for validation. In this study, the model goodness of fit was assessed using the root mean square error (RMSE) and leave-one-out cross-validation root mean square error (LOO RMSE). (see figures 3-5)
2. For GLM model a fivefold cross-validation method is used to validate model performances. Besides, as the stochastic nature of the machine learning algorithms constrain yielding reliable validation outputs, the study applies a robust method of validation by determining the confidence interval for the population mean.(tables of 4, 7 and 8)
3. In addition to metric-based validation, model performance was assessed using scatter plot analysis. The scatter plot analysis between measured yield and estimated yield was implemented for several randomly selected seeds on test dataset. (figures 4 and 11)

General Comments:

#3. Line 299, Why repeat 30 times but not 20 times or 40 times. Please explain the reason or describe the principle for determinating the 30 times.

Response:

In this study, 30 numbers of repetitions was used two times: to pick the best hyperparameters during for instance AutoML optimization, to determine the confidence interval for an average value of model performance.

In the first case, identification of the best hyperparameters, 30 was the number of times the hyperparameters were searched. Finally, values of hyperparameters that appear more frequently both on the training and test dataset were picked as the best one for each of the hyperparameters. The number of repetitions was kept at 30 because, in general (since for some parameters it was possible to determine earlier), at which it was possible to determine the most frequently appearing best values across the tested hyperparameter.

In the second case, the selection of 30 was related to t-interval statistics. In general, many machine learning algorithms have stochastic properties. Stochasticity refers to a variable process where the outcome involves some randomness and has some uncertainty. The underlying causes of stochasticity are associated with random point generation (seeding), and the number of cores of the machine (computer) used among others. Therefore, in this study we repeat both the training and validation process to control the uncertainty in the values reported. This is achieved by computing the confidence interval using t-interval. T-interval is used as the nature of the population distribution and population standard deviations are unknown. Therefore, 30 is the minimum sample size required to apply the t-interval test.

#4. the disccusion should be revised deeply,  include but are not limited to  the comparation between the algorithm and compared with other research and uncertain analyze.

Response:

We accept and we revised accordingly.

Specific Comments

#1. The VI in table 1 should add the cited reference one by one.

Response:

Accepted and incorporated.

#2. End of the line 195 should add an end mark.

Response:

Accepted and revised

#3. The VI in table 2 should add the cited reference one by one.

Response:

Accepted and revised

#4. Line 240, Figure 2 , S2 third box "Indices and" delete the "and".

Response:

Accepted but there was a missing word next “and” and now corrected

5. Figure 2 "nlsLM" is the first shown in manuscirpt,  I suggest to change to "Nonliner Regression" will be better.

Response:

Accepted and now corrected

6.figure 3,4,5 RMSE add the unit. Figure 11 add thei r (correlation coefficient).

Response:

All are accepted and addressed

 

 

Author Response File: Author Response.docx

Reviewer 3 Report

The article seemed to me interesting and of scientific significance. The authors describe in detail the approach to the use of AutoML technologies to solve a specific problem of yield forecasting. When analyzing the work, some points raised questions that I would like to get explanations for.

1. Figure 2. The "Induces and" block. Is the end of the phrase definitely not missing here? It is not very clear what this block does in the current formulation.

2. Figure 3-5 require a more detailed description of the letters (a), (b), etc., as well as what the black graphs indicate on them. For example, as you gave an explanation to graph 6 regarding the red line.

3. Is it worth adding designations in the text for CI – "confidence interval" and CV – "cross-validation" before Table 4.

4. Figure 9. Is this figure a refinement of Figure 8? Does it make sense? If Figure 10 is used to analyze a three-layer neural network, then why don't you provide a graph for a two-layer network?

5. Line 473 – "... for each of the two hidden layers..." one, two or three layers? Please check the text, because Table 7 mentions the parameters of a single-layer network. Double-check the designations of neural networks in the tables and references to them in the text so that there are no contradictions.

Author Response

# 1. 1. Figure 2. The "Induces and" block. Is the end of the phrase definitely not missing here? It is not very clear what this block does in the current formulation.

Response:

We accept the comment.  There is a missing word that happens during resizing the figure. The correct phrase is indices and predictors. The figure shows the overall methodology the study followed, and we find it more appropriate to present at this section. Nonetheless, in the current revision, we add a sentence that shortly explains the purpose of the figure.

# 2. Figure 3-5 require a more detailed description of the letters (a), (b), etc., as well as what the black graphs indicate on them. For example, as you gave an explanation to graph 6 regarding the red line.

Response:

We accepted the comment and add additional explanations referring to representations of the letters and the black fitting lines in the plots (see section 3.1).

#3. Is it worth adding designations in the text for CI – "confidence interval" and CV – "cross-validation" before Table 4.

Response:

Yes it will be worthy but we address it differently. As the word confidence interval appears for the first time on page 8 at section 2.6.2, there, we added the Confidence Interval (CI) designation and on the subsequent appearances we use CI. The same is applied for other repeatedly appearing terminologies including CV.

# 4 . Figure 9. Is this figure a refinement of Figure 8? Does it make sense? If Figure 10 is used to analyze a three-layer neural network, then why don't you provide a graph for a two-layer network?

Response:

Yes figure 9 is presented as a refinement to figure 8. In figure 8 (the gap was 1000 neurons) the range from 0-1000 could reveal the global minima but this is a wider range and yet difficult to get the global minima. Thus, using figure 9 (the gap was 200 neurons) we confirm the global minima is expected between 1-200 number of neurons.  Finally, as presented in table 5, much narrower ranges were tested to arrive at the global minima value. Therefore, the presentation of figure 9 gives meaning as it confirms the range(0-200 neurons) that gives the global minima values.

Concerning adding a graph for two number of hidden layer network

We decided to present the graphs of a one-layer and a three-layer network as the two groups were competent and outperforms a two-layer network. Besides, as we present the result of a two-layer network under table 5 for comparison to the two groups, for the interest of limiting the size of the manuscript, we determine to drop the graph for a two-layer network.

# 5. Line 473 – "... for each of the two hidden layers..." one, two or three layers? Please check the text, because Table 7 mentions the parameters of a single-layer network. Double-check the designations of neural networks in the tables and references to them in the text so that there are no contradictions.

Response:

Accepted and correction is made that table 7 presented values for one number of hidden layer while table 8 present the case for three number of hidden layers. Accordingly, the following revision is made (see the bold sentence below) that guides correct reference of tables and the corresponding texts.

“Finally, for each of the two hidden layers, the best hyperparameters were applied and the confidence interval is computed. Thus, table 7 and table 8 revealed mean RMSE of wheat yield at a 99% CI for one and three number of hidden layers respectively.”

# 6. Moderate English revision is ordered.

Response:

We accept the point and we did an extensive language editing to further add clarity and increase the overall readability of the manuscript.

 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

ok 

Reviewer 2 Report

There is no other comments.