Estimating Crop Seed Composition Using Machine Learning from Multisensory UAV Data

Round 1

Reviewer 1 Report (Previous Reviewer 2)

The comments have been addressed.

Author Response

Thank you very much for reviewing our draft!

Reviewer 2 Report (New Reviewer)

This article carries out a very detailed study to determine the potential use of high resolution hyperspectral and LiDAR data captured by unmanned UAV platforms at the R5 reproductive stage, for the assessment of protein and oil content in soybean and maize seeds.
Based on hyperspectral images and lidar data captured with state-of-the-art sensors (possibly the most suitable at present), the aim is to determine the protein and oil composition of the seeds of these crops using as variables parameters obtained from the flights (vegetation indices, texture parameters and parameters derived from the point cloud). Machine learning algorithms (H2O-AutoML) were used for their evaluation and regression models were built comparing the results obtained.
In my opinion it is a very well designed and executed work with high scientific and technological solvency and it is in my point of view ready for publication. However, a number of minor corrections should be made, which are detailed below.

Line # 187, Figure 1. As it is a map it should have its coordinates (geographical location) and scale. It should be redesigned so that the scales of each of the orthophotos are clear.
Line #190, "with a small-plot combine 190 and data from 91 plots were used for this study (Figure 1)", the explanation of the plots is not clear, because Figure 1 is not well designed

Line # 203. Same argument as above for soybeans. It is not clear as the figure is not explanatory.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The authors discuss the possibility to estimate the crop seed composition using multisensory data. The work is well written, apart from a few typos, but the results do not provide the 'superior performance' stated by the authors when compared with other techniques. Thus, for the work to be accepted, the authors must improve the discussion of the results.

The authors present a workflow able to perform data fusion on hyperspectral and LIDAR data, then fed to an ML engine to predict the crop seed composition. The advantage of using data fusion techniques should be considered negligible, in my opinion, with respect to using hyperspectral or LIDAR data only, looking at the data presented in this work.

The arising question is: from an economic and technological viewpoint, is it worth it? On the other side, the authors provide a full analysis of the potential advantage that combined use of hyper and LIDAR data can offer, proving that the combined use should be considered only in some cases, such as the need for larger accuracy despite the costs.

The authors should better elaborate on this and avoid stating that data fusion can provide 'superior' performance for such an application, thus revising the presentation of the results and the conclusions. 

Some additional minor comments:

• the ground GPS station in fig. 3 is an RTK station, correct the label in the subfigure
• correct the typo 'precicted' in fig. 7 and the following ones

Author Response

Dear reviewer,

Thank you very much for your valuable time and feedback on our paper; the comments were informative and beneficial to improve our result and discussion quality. To your comments ( in bold ), we have prepared our response :

1. The authors discuss the possibility of estimating the crop seed composition using multisensory data. The work is well written, apart from a few typos, but the results do not provide the 'superior performance' stated by the authors when compared with other techniques. Thus, for the work to be accepted, the authors must improve the discussion of the results.

The UAV method is superior in performance when considering the overall time and costs to currently obtain simultaneous estimates of seed concentrations of protein, starch, or oil.   Current best practices would be to hand-harvest the cobs/pods, thresh the seeds, and then scan the seeds for their near-infrared reflectance using specialized instruments and calibration algorithms that are costly to create.  Although less accurate, the UAV estimations can be performed for the entire field site in a few hours, compared to certain days and likely weeks using current best practices and much lower overall cost.  Furthermore, the UAV information is collected approximately four weeks prior to harvest. The UAV method may be sufficient to ensure that a field was likely to be above minimum compositional requirements for marketability (e.g. >8% corn grain protein)  or in a breeding program to identify varieties with extreme differences in seed composition. 

Another point we would like to clarify is that the term " superior performance" is used repeatedly in the paper; in the revised version, we carefully selected phrases to alternate; It was used:1 to express the better performance of the gradient boosting model than the other four models in the Automated Machine learning framework;2. To express the better performance of Hypersepctural and Data fusion compared to single sensor Data;  and 3, to express the better version of canopy spectral and texture features than structural performances. According to the validation statistics provided in tables 6,7,  8 and 9, among the five different machine learning models:1, the gradient boosting model provided the highest results on eight prediction tests out of a total of 12 tests; 2, combined features from hyper and lidar sensors showed higher results than the single sensor-based features;3, last but not least, spectral features yielded higher results than structural features when used to predict seed composition estimation. The  "superior performance" refers to the best results within this study's scope as no previous research was conducted to predict seed compositions from Hyperspectral and LiDAR remote sensing imagery. Thus we presented our research as the first of its kind.

2.The authors present a workflow able to perform data fusion on hyperspectral and LIDAR data, then fed to an ML engine to predict the crop seed composition. The advantage of using data fusion techniques should be considered negligible, in my opinion, with respect to using hyperspectral or LIDAR data only, looking at the data presented in this work.

On lines 105, 521 and 530  we have brought up the optical saturation issue when the plant canopy has a high leaf area index (during R5 stage canopy LAI >3.0) remote sensing imagery-based spectral information misses data of three _dimensional canopy vertical profile. The following is taken from our paper: "Moreover, mainly due to the weakened optical saturation issues, along with information complementary, a combination of canopy information obtained from multispectral/hyperspectral imagery (i.e., spectral and texture features) with point cloud-based 3D canopy structure features has been proved to yield more robust models with improved accuracies in terms of plant traits estimation and grain yield prediction in many cases [26,39,42]. Nonetheless, multisensory data fusion/combination, especially fusing UAV hyperspectral and LiDAR-derived features, has not been examined in estimating crop seed composition. As such, the fusion of structural and spectral information data available spatially resolved manner promises to enhance quality, precision, accuracy, and the applicability of predictions/ model performance [26,30,31]. "

The potential physiological explanations for using both hyperspectral and LiDAR-based features are explained in response to comment one as well. Furthermore, Data fusion is one of the main objects of our research that we would like to highlight and recommend. We got comparatively higher prediction results when using structural and spectral features together than using them separately, based on the results presented in tables 6,7,8 and 9. 

3.The arising question is: from an economic and technological viewpoint, is it worth it? On the other side, the authors provide a full analysis of the potential advantage that combined use of hyper and LIDAR data can offer, proving that the combined use should be considered only in some cases, such as the need for larger accuracy despite the costs.The authors should better elaborate on this and avoid stating that data fusion can provide 'superior' performance for such an application, thus revising the presentation of the results and the conclusions.

While it costs more to obtain LiDAR ad Hyperspectral sensors, it also takes more time and technological background to process the imagery than multispectral and RGB images because they are more advanced and accurate sensors. In the long run, comparing the whole life cycle of the production process, this methodology can still be a cost-effective choice. Our research methods and results can be used if and when there is a need for speedy, accurate and less time-consuming seed composition prediction. Our research aims to contribute to the advancement of scientific research. The UAV method is superior in performance when considering the overall time and costs to currently obtain simultaneous estimates of seed concentrations of protein, starch, or oil.  

4. Some additional minor comments:

• the ground GPS station in fig. 3 is an RTK station, /correct the label in the subfigure
• correct the typo 'precicted' in fig. 7 and the following ones

 The typos in the figures 7,8,9,10 updated; The label on the subfigure in Figure 3 was corrected.

A thorough spell-check conducted to prevent further typos and/or misspellings

Reviewer 2 Report

The manuscript illustrate that UAV remote sensing equipped with hyperspectral imager and LiDAR can provide a rapid method for pre-harvest estimation of crop seed protein and oil. However, there's a critical defect. Why the structure information derived from LiDAR can be used for estimating crop seed concentration? Why the canopy spectral can be used for estimating the seed composition in soybean pod and the seed in corn cob at the middle of plant as lack of direct detect of spectral information?

As we know, based on the difference of pigment in leaves or other physiological characteristics, many phenotypic traits can be estimated by hyperspectral spectroscopy. Unfortunately, there's evidence that convince the reasonability of using density points to estimate seed concentration, which must be supplemented according to the accumulation law of crop seed components. In my advice, the author can think what are the differences of crop seed components under different growth status (such as plant height or biomass).

Even the estimation accuracy can be improved with the fusion of hyperspectral and LiDAR, however, it's like ill-posed problem to some extent as there's lack of direct relationship between structure information and seed biochemical concentration.

Moreover, the seed of soybean was in pod, which is hard to directly detect. The mechanism or reason for using hyperspectral or LiDAR to estimate seed composition should be supplemented. For the corn, using UAV hyperspectral imager also can't provide direct spectral information of corn cob, which lack of sufficient evidence to convince its rationality and principle. 

Author Response

Dear reviewer:

Thank you very much for your valuable time and feedback on our paper; they were very informative and beneficial to improving our results and discussion quality. To your comments (Bold ), we have prepared our response in :

1.The manuscript illustrate that UAV remote sensing equipped with hyperspectral imager and LiDAR can provide a rapid method for pre-harvest estimation of crop seed protein and oil. However, there's a critical defect. Why the structure information derived from LiDAR can be used for estimating crop seed concentration? Why the canopy spectral can be used for estimating the seed composition in soybean pod and the seed in corn cob at the middle of plant as lack of direct detect of spectral information?

The composition of both corn and soybean seeds is determined in part by the physiology of both the above-ground vegetative tissues and the activities of roots underground in acquiring water and nutrients.  Some physiological processes occurring in non-seed tissues can directly impact seed composition; for example, plants that accumulate more nitrogen in leaves and stem during vegetative growth (detectable as hyperspectral changes) often also have higher protein concentration in seeds.  There are also indirect effects, such as variation in the duration of photosynthesis or senescence in leaves.  Similarly, LIDAR is effective at estimating plant height, which is correlated with grain yield, and varieties with higher grain yields typically have lower seed nitrogen concentrations.  So, the combination of hyperspectral and LIDAR may provide information on distinct physiological processes that contribute to final seed composition.

2.As we know, based on the difference of pigment in leaves or other physiological characteristics, many phenotypic traits can be estimated by hyperspectral spectroscopy. Unfortunately, there's evidence that convince the reasonability of using density points to estimate seed concentration, which must be supplemented according to the accumulation law of crop seed components. In my advice, the author can think what are the differences of crop seed components under different growth status (such as plant height or biomass).

Even the estimation accuracy can be improved with the fusion of hyperspectral and LiDAR, however, it's like ill-posed problem to some extent as there's lack of direct relationship between structure information and seed biochemical concentration.Moreover, the seed of soybean was in pod, which is hard to directly detect. The mechanism or reason for using hyperspectral or LiDAR to estimate seed composition should be supplemented. For the corn, using UAV hyperspectral imager also can't provide direct spectral information of corn cob, which lack of sufficient evidence to convince its rationality and principle. 

Regarding different growth stage comments, the imagery used in this study is collected from each crop's reproductive stage five (R5). Under the R5 stage, crop seeds are starting to fill, and with the further linkage between seed chemical composition and canopy signals, we would like to refer to the explanation we gave for comment #1 above.

The UAV data is not directly sampling the seed's features that are covered by husks or pods.  Instead, it is detecting features in the vegetative tissues at a stage of peak grain-filling that influence final seed composition, which can then be associated with machine learning.

Round 2

Reviewer 1 Report

I thank the authors for addressing my comments.

I would strongly recommend improving the paper by adding the text provided as responses to my questions. I am not sure this has been already carried out because revisions are not highlighted in the resubmitted document. 

The paper can be then accepted.

Author Response

Dear reviewer,

We appreciate your effort and comments on our manuscript, which helped us to substantially improve its quality. In the previous first-round revision, the responses to your questions were actually included in the revised manuscript; however, I mistakenly accepted all the changes before submitting the revised draft, making revisions harder to differentiate. I avoided that mistake this time around. We also ran spell checks to prevent further spelling-related mistakes. Thank you again for your constructive comments.

Reviewer 2 Report

I can fully understand that the crop seed quality parameters are correlated with vegetative growth. The nitrogen fertilizer application perform positive impact on the protein content or biomass accumulation. However, unlike the biomass accumulation by canopy photosynthesis, if there exist quantitive relationship or qualitative mechanism between protein/oil accumulation with vegetative growth?  The physiological processes of vegetative growth that can influence protein and oil content should be specified. Now I can't be convinced by the author's reply.  

Author Response

Revision Report (Reviewer 2, Round 2)

Reviewer's comments are greatly appreciated. Our responses start with “RESPONSE:” and are highlighted in blue color( displayed blue in  the attached word document.

Manuscript ID:  Remotesensing-1710664
Title: Estimating crop seed composition using automated machine learning from multisensory UAV data
Journal: Remote Sensing

comments from Reviewers and Authors’ Responses:

 

Reviewer #2

 I can fully understand that the crop seed quality parameters are correlated with vegetative growth.

RESPONSE: Thank you very much for your valuable comments. Our research hypothesis was developed on the correlation between vegetation growth and crop seed qualities.

The nitrogen fertilizer application perform positive impact on the protein content or biomass accumulation.

 RESPONSE: While the seed nitrogen concentration impacts the protein and oil content,  the scope of our study does not include the N fertilizer impact.

 However, unlike the biomass accumulation by canopy photosynthesis, if there exist quantitive relationship or qualitative mechanism between protein/oil accumulation with vegetative growth? The physiological processes of vegetative growth that can influence protein and oil content should be specified. Now I can't be convinced by the author's reply.  

RESPONSE : Numerous agronomic and breeding experiments have established an inverse relationship between grain yield and protein concentrations in cereal seeds such as maize(Uribelarrea et al. 2009; Zörb et al. 2018). Imaging of the vegetative canopy at the R5 stage could reveal features associated with two distinct physiological processes that increase seed N concentration, continued late-season nitrogen uptake and/or more efficient remobilization of vegetative N to grain during reproductive development (Mueller et al. 2019) Conversely, canopy imaging may detect features generally associated with faster overall growth(Olmedo Pico and Vyn 2021), leading to larger grains with more starch, thus reducing seed nitrogen concentration via the “dilution effect”.

The imagery data we utilized in this paper is limited to R5 (reproductive five) growth stage of the crops discussed in this research. The capacity of the plant canopy to capture light energy and store nitrogen determines crop yield from which compositions result (Sinclair and Sheehy 1999)which in turn depends mainly on the photosynthetic rate per se and canopy architecture of the plant (Ren et al. 2020)

 

 

 The formation of seed composition depends on multiple factors, including biotic and abiotic conditions such as sunshine hours and water and nutrient availability. Sunlight is the most crucial element that catalyzes photosynthetic activity, and canopy architecture (closure, plant height, and leaf angle distribution) plays a vital role in light absorption by plants. Although LiDAR point clouds do not directly relate to photosynthetic activities, it characterizes canopy architecture, affecting seed composition. We used percentile height at different levels, which is a way to quantify canopy architecture in digital form. The LiDAR point clouds features used in seed composition estimation represent the computer-readable canopy architecture that largely dictates canopy and light interactions. 

 

Citation:

Mueller, S.M., Messina, C.D., & Vyn, T.J. (2019). Simultaneous gains in grain yield and nitrogen efficiency over 70 years of maize genetic improvement. Scientific Reports, 9, 1-8

Olmedo Pico, L.B., & Vyn, T.J. (2021). Dry Matter Gains in Maize Kernels Are Dependent on Their Nitrogen Accumulation Rates and Duration during Grain Filling. Plants, 10, 1222

Ren, Z., Wu, L., Ku, L., Wang, H., Zeng, H., Su, H., Wei, L., Dou, D., Liu, H., & Cao, Y. (2020). ZmILI1 regulates leaf angle by directly affecting liguleless1 expression in maize. Plant Biotechnology Journal, 18, 881

Sinclair, T., & Sheehy, J. (1999). Physician-scientists at risk. Science, 283, 1456-1456

Uribelarrea, M., Crafts-Brandner, S.J., & Below, F.E. (2009). Physiological N response of field-grown maize hybrids (Zea mays L.) with divergent yield potential and grain protein concentration. Plant and soil, 316, 151-160

Zörb, C., Ludewig, U., & Hawkesford, M.J. (2018). Perspective on wheat yield and quality with reduced nitrogen supply. Trends in plant science, 23, 1029-1037