A Multimodal and Temporal Network-Based Yield Assessment Method for Different Heat-Tolerant Genotypes of Wheat

Abstract

Large-scale yield estimation in the field or plot during wheat grain filling can contribute to high-throughput plant phenotyping and precision agriculture. To overcome the challenges of poor yield estimation at a large scale and for multiple species, this study employed a combination of multispectral and RGB drones to capture images and generation of time-series data on vegetation indices and canopy structure information during the wheat grubbing period. Five machine learning methods, partial least squares, random forest, support vector regression machine, BP neural networks, and long and short-term memory networks were used. The yield estimation of wheat grain filling period data was executed using a long and short-term memory network based on the preferred machine learning model, with a particular focus on distinguishing different heat-tolerant genotypes of wheat. The results unveiled a declining trend in the spectral reflectance characteristics of vegetation indices as the filling period progressed. Among the time-series data of the wheat filling period, the long and short-term memory network exhibited the highest estimation effectiveness, surpassing the BP neural network, which displayed the weakest estimation performance, by an impressive improvement in R² of 0.21. The three genotypes of wheat were categorized into heat-tolerant genotype, moderate heat-tolerant genotype, and heat-sensitive genotype. Subsequently, the long and short-term memory network, which exhibited the most accurate yield estimation effect, was selected for regression prediction. The results indicate that the yield estimation effect was notably better than that achieved without distinguishing genotypes. Among the wheat genotypes, the heat-sensitive genotype demonstrated the most accurate prediction with an R² of 0.91 and RMSE% of 3.25%. Moreover, by fusing the vegetation index with canopy structure information, the yield prediction accuracy (R²) witnessed an overall enhancement of about 0.07 compared to using the vegetation index alone. This approach also displayed enhanced adaptability to spatial variation. In conclusion, this study successfully utilized a cost-effective UAV for data fusion, enabling the extraction of canopy parameters and the application of a long and short-term memory network for yield estimation in wheat with different heat-tolerant genotypes. These findings have significant implications for informed crop management decisions, including harvesting and contingency forecasting, particularly for vast wheat areas.

1. Introduction

A crucial crop in agricultural production systems is wheat (Triticum aestivum L.), with the highest commercial rate and widespread distribution among cereals [1]. As it is the source of staple food for a large part of the world’s population [2], ensuring wheat yield research remains timely and of high quality is imperative. To address the challenge of predicting yields for a large number of wheat varieties, genomic selection has emerged as a promising prediction method. Agriculture stands as one of the most weather-sensitive and dependent industries. Climate change has been observed to primarily exert negative effects on crop productivity and quality across diverse regions. The proper temperature plays a pivotal role in all critical growth phases of crops, including wheat. Empirical evidence points to heat stress as the predominant causal factor for yield loss in winter wheat [3]. For filling wheat grains, an ideal temperature range lies between 20 °C to 22 °C. Any exposure to temperatures beyond 24 °C during the plant’s reproductive stage results in a considerable reduction in wheat yield. Furthermore, the duration of exposure to high temperatures directly correlates with the magnitude of yield decline [4]. In addition to reducing yield, increased temperatures also lead to the shortening of the wheat filling time, thereby impeding the accumulation of dry matter in the grains [4]. Subsequently, wheat yields may ultimately experience a reduction of 6 to 51% under conditions where the average daytime temperature exceeds 30 °C or when the highest recorded temperature rises above 34 °C during the critical filling phase [3]. Timeous, reliable, spatial, and temporally specific, large-scale winter wheat yield estimates during wheat grain filling are required during the wheat filling period. This is essential for effective agricultural management, food security monitoring, and optimal resource allocation [5,6].

In recent times, Unmanned Aerial Vehicles (UAVs) have shown great potential due to their significantly higher temporal and spatial resolution compared to airborne and satellite platforms [7,8,9]. Thanks to advancements in sensor technology, unmanned aerial vehicles platforms have become cost-effective and versatile tools, particularly in high-throughput plant phenotyping. Field phenotyping systems should be more simple to use and as high-quality as possible to save people labor costs [10,11,12,13]. The simplicity of use and cost effectiveness of unmanned aerial vehicles have proven instrumental in predicting crop yields accurately over vast areas without causing damage [14]. In space and time, high accuracy was obtained for the detection of biomass [15], total yield [16], crop growth monitoring [17], quality detection [18], and crop disease detection [19] by unmanned aerial vehicle technology, making it invaluable for large-scale spatial and high temporal resolution data collection. Such capabilities are crucial for assessing the phenological stages of specific crops and selecting high-performing varieties [20,21].

Quantitative remote sensing techniques have shown promising results in predicting yields during the filling stage by analyzing canopy spectral characteristics [22,23]. Integrating canopy spectral and structural data from multi-unmanned aerial vehicles sensor systems have significantly enhanced crop trait estimation, including grain yield, across various agricultural applications [24,25]. Among the remote sensing technologies used for predicting crop production, spectral data stand out due to their stable and superior performance. Vegetation indices, with their inherent memory capabilities, not only enhance the vegetation data signal but also mitigate the influence of confounding factors such as soil background, canopy structure, solar irradiance, and environmental stressors [26]. Sustained high temperatures can have detrimental effects on plants, including scorching of plant branches and leaves, sunburn, leaf and stem senescence, growth inhibition, fruit and leaf discoloration, and alterations in spectral reflectance characteristics [27]. The generation and estimation of VIs play a crucial role in effective crop management, especially in the face of unpredictable climatic conditions like drought and heat stress, which increase production uncertainty [17]. Based on the spectral reflectance at different wavelengths, several VI were derived by various mathematical combinations, offering a valuable proxy for plant physiological traits and enabling accurate wheat yield predictions [17,28,29]. Numerous studies have highlighted the superior accuracy of VI in predicting wheat production during the filling stage compared to other growth stages [17,29].

Technologies for high-throughput phenotyping play a crucial role in rapidly and precisely characterizing the phenotypic traits of thousands of plants, without causing additional damage or invasiveness [30,31,32,33]. To enhance experimental capacity, it is vital to monitor the same plot throughout a specific life cycle [34]. Unmanned aerial vehicles offer several advantages in this context, with their wider field of view and ability to avoid physical contact with crops, minimizing environmental impact and mechanical disturbance to the growing crop. Unmanned aerial vehicles enable high-frequency measurements during wheat filling, allowing for multiple measurements to capture the dynamic changes in plant phenology [35,36]. At different developmental stages, plants respond uniquely to various spectral properties, resulting in corresponding changes in spectral information relevant to yield estimation [37]. Hence, many measurements enable the computation of some characteristics that are less dependent on phenology. In yield prediction studies, models utilizing time series data outperform those using only single-period data, resulting in improved prediction accuracy and precision [35,36]. However, in the context of precision agriculture management and sustainable agricultural production, the canopy spectrum and structure information generated from multispectral and RTK drones are combined, there is limited knowledge regarding the contribution of time-series yield prediction for multiple wheat varieties during the filling period.

Many regression techniques use deep learning (DL) techniques such as artificial neural networks (ANN) [38], deep convolutional neural networks (CNN) [39], and deep neural networks (DNN) [40]; these models have all been used in various plots to forecast yield and detect crops. DL’s multi-layer framework allows for the discovery of sophisticated nonlinear functions and automatic analysis and expression of applications through data [41]. The accuracy of DL may be further enhanced with a big amount of data, and data optimization, collection advantages, and technical means are all becoming more and more feasible. However, the time component also plays a crucial role in determining the link between predictor variables and response, in addition to the spatial impact on yield prediction [42,43]. In this study, five time series models will be selected for time-series yield assessment in wheat during the filling period.

The motivation behind this study lies in the need for accurate yield estimation of numerous wheat varieties across a wide cultivation area. The combined system and time series model, utilizing multispectral and RTK drones, aims to enhance the yield prediction capability for multi-species wheat. The specific objectives include: (I) Quantifying yield differences due to temperature and varietal genotype variations in a typical breeding field trial setup. (II) Assessing the dynamic changes in image features collected from wheat at different filling periods. (III) Generating high-resolution yield prediction maps for a multi-variety wheat collection.

2. Materials and Methods

2.1. Experimental Materials and Design

The test location is situated at 117°19′ E, 31°85′ N, 37 m above sea level in Hefei High-tech Agricultural Park, Anhui Province, China, as shown in Figure 1. The experimental area belongs to a subtropical humid monsoon climate. The annual sunshine duration is 2100 h, and the rainfall and temperature conditions are ideal for the growth and development of winter wheat. Winter wheat is grown and harvested only once a year; winter wheat is sown on 10 November 2021, and harvested on 25 May 2022.

A total population of 20 wheat varieties was selected for the experiment including representative varieties grown from the HuangHuaiHai region (

Table 1. Wheat variety.

Wheat Variety
V1	Zhongmai 895	V11	Luohan 19
V2	Xinkemai 169	V12	Bainong 207
V3	Zhongmai 175	V13	Luomai 26
V4	Zhengmai 136	V14	Xinmai 36
V5	Liangxing 99	V15	Fengdecunmai 5
V6	Huaimai 33	V16	Fengdecunmai 21
V7	Annong 0711	V17	Fengdecunmai 1
V8	Huacheng 3366	V18	Zhengmai 369
V9	Zhoumai 27	V19	Zhoumai 36
V10	Luohan 22	V20	Zhengmai 366

). The winter wheat was grown with different temperature treatments, temperature processing sections, and placement of specially designed field heat tents on established plots [44]. The maximum temperature in the shed was guaranteed to be no higher than 40 °C, and 20 winter wheat varieties were randomly arranged and with multiple replicates. The plot area is 6 m² (2 m × 3 m), with row spacing of 0.2 m. Edge-setting protection lines, fertilizer, and irrigation were applied in the same way in all sample plots and were consistent with the local routine.

2.2. Data Collection

To gain a comprehensive understanding of the temperature dynamics during the wheat filling period, Figure 2 presents the highest and lowest temperatures recorded throughout the entire grouting period. Notably, the maximum temperature during the filling phase frequently surpasses the optimal temperature range required for wheat grain filling [3]. As such, accurate and representative yield prediction for the wheat filling period necessitates a focus on this specific time point.

2.2.1. Field Data Collection

The experiments were conducted in early grain fill (EGF) (22 April 2022), mid grain fill (MGF) (9 May 2022), and late grain fill (LGF) (21 May 2022). In-ground data included plant height and yield. When 50% of the plots reached a specific period, we used a ruler to measure the height of the plant [45]. To increase the unmanned aerial vehicle ability to accurately measure crop height, during the same day as the unmanned aerial vehicle flight, a manual field height measurement was used to establish the ground live data. In each plot, each point was sampled repeatedly according to the 5-point sampling method; a total of 25 representative plants were randomly taken. We avoided uneven mounds and soil fractures when measuring the length of wheat in its natural condition from the surface of the ground to the apex of the spike, omitting the awn [46]. As the real ground measurement of plant height in each plot, the average of 25 representative plants’ values was used. Winter wheat yields were collected from each plot, grain wheat was gathered from a randomly chosen 1 m² area in each plot (excluding border plants), and manual threshing was used for seed yield determination. The moisture content of seed yield in this study was 13%. In each sample, 1000 seeds were randomly selected in three replications, threshed, dried, weighed, and evaluated for quality as the final yield [47].

2.2.2. UAV Image Acquisition

According to Figure 3A, DJI Phantom 4 Multispectral and DJI Phantom 4 RTK unmanned aerial vehicles (SZ DJI Technology Co., Ltd., Shenzhen, China) were used to capture multispectral and RGB photographs in the pre-filling, mid-filling, and late filling of wheat. Centimeter-level navigation and positioning systems were present aboard the unmanned aerial vehicles. The multispectral unmanned aerial vehicles recorded images in five spectral bands at wavelengths of blue (450 nm ± 16 nm), green (560 nm ± 16 nm), red (650 nm ± 16 nm), red edge (730 nm ± 16 nm), and near-infrared (840 nm ± 26 nm) at a 1600 × 1300 pixel resolution. The photos had an 80% average forward overlap and an 80% average side lap. The FC6310R lens on the DJI Phantom 4 RTK camera has an aperture of 8.8 mm and a resolution of 4864 × 3648 pixels. The mean forward overlap was 80%, and the mean side-lap was 90%. The drone combination flew at a height of 15 m without causing any disruption to the wheat with a speed of 2 m/s and photo interval of 2 s. The drone images were taken during the day in clear, cloudless weather.

2.3. Image Preprocessing

Raw data stitching, localization point correction, reflection correction, and radiation correction are all part of the unmanned aerial vehicle image processing workflow. Both RGB and multispectral photos were analyzed in the same way. The unmanned aerial vehicles’ multispectral and RGB data were combined using the DJI Terra software 4.1.0 (SZ DJI Technology Co., Ltd., Shenzhen, China). We used a standard diffuse reflector with a fixed reflectance for radiation correction in the DJI Terra software. The digital number (DN) value of the photograph was then changed into a reflectance value using the empirical line approach (Figure 3A). Orthorectified mosaic images and unmanned aerial vehicle canopy data were created using RTK unmanned aerial vehicle images and geographic coordinates.

2.4. Canopy Information Extraction

2.4.1. Canopy Spectral Information

Unmanned aerial vehicle remote sensing technology plays a significant role in agricultural applications as it enables the detection of various spectral properties related to plant absorption, reflection, and radiation. Among the methods used to obtain crop data through remote sensing, the estimation of vegetation indices (VIs) stands out as the most frequently employed approach. These indices are crucial for inferring agronomic traits, and their correlation values were found to be comparable to, if not stronger than, those obtained through ground assessments [48]. A collection of vegetation indices that had previously been frequently employed for predicting grain yield were calculated in this study. These indices were calculated and generated in five bands (Figure 3B), namely blue, red, Nir, red-edge, and green, widely utilized for plant vegetation detection and crop productivity monitoring, with specific vegetation indices shown in Figure 3C; the formula is in

Table 2. Selected optical indices in this study.

Features	Formulation	Reference
Difference spectral index	DSI = NIR − R	[49]
Enhanced vegetation index	EVI = 2.5 × (NIR − R)/(1 + NIR − 6 × R − 7.5 × B)	[50]
Two-band enhanced vegetation index	EVI2 = 2.5 × (NIR − R)/(NIR + 2.4 × R + 1)	[51]
Green difference vegetation index	GDVI = NIR − G	[49]
Green optimal soil adjusted vegetation index	GOSAVI = (1 + 0.16)(NIR − G)/(NIR + G + 0.16)	[52]
Green re-normalized different vegetation index	GRDVI = (NIR − G)/SQRT(NIR + G)	[53]
Green soil adjusted vegetation index	GSAVI = 1.5(NIR − G)/(NIR + G + 0.5)	[54]
Modified chlorophyll absorption in reflectance index 1	MCARI1 = [(NIR − RE) − 0.2 × (NIR − G)](NIR/RE)	[55]
Modified enhanced vegetation index	MEVI = 2.5 × (NIR − RE)/(NIR + 6 × RE − 7.5 × G + 1)	[56]
Normalized difference vegetation index	NDVI = (NIR − R)/(NIR + R)	[57]
Optimized soil-adjusted vegetation index	OSAVI = 1.16 (NIR − R)/(NIR + R + 0.16)	[52]
Red edge difference vegetation index	REDVI = NIR − RE	[49]
Red edge soil adjusted vegetation index	RESAVI = 1.5 × [(NIR − RE)/(NIR + RE + 0.5)]	[54]

. In addition, to eliminate boundary effects and sample areas, the region of interest (ROI) for each plot in the orthophotography was determined using the ENVI. To depict the values of each plot, the averaged VI of each ROI was retrieved.

2.4.2. Canopy Structure Information

By watching the environment in which the plants reside, it is possible to assess the distinctive features of plant height, which enables the estimation of plant height estimates for various crop growth stages [58]. In the realm of data acquisition for plant canopy information, manual field ground measurements have been utilized. However, this method is limited in its applicability, as it is only practical for a small number of plants in small plots. Moreover, the data obtained through manual measurements may suffer from inconsistencies due to the lack of uniformity. Unmanned aerial vehicles offer a more efficient and standardized approach for acquiring plant canopy data, mitigating potential errors and discrepancies [59].

In this experiment, a digital surface model (DSM) was created by identifying the canopy height from point clouds obtained from RTK unmanned aerial vehicle photos and incorporating it as a canopy characteristic for yield structure, as shown in Figure 3D, and pictures captured in the air on 10 November 2021, following planting, were used to calculate the Digital Elevation Model (DEM) over the whole filling period. The crop surface model (CSM) is determined by the difference here between DSM and the DEM for each unmanned aerial vehicle flight, excluding outliers that occur by chance. ERIS ArcMap produced the CSM minus the DEM from the DSM [25,60].

2.5. Building the Dataset

In order to prepare the dataset required for training the network model, image blocks need to be extracted from the pre-processed vegetation index images and canopy images for each training area. Therefore, multiple ROI is created and each plot is segmented into 0.5 m × 0.5 m rectangles using MATLAB R2016a [61], and the size of the multispectral image and RGB image cut image blocks are 65 × 65 pixels and 58 × 58 pixels, respectively. The dataset for each training map consists of individual pixel image blocks and yield values. To annotate each image block, the corresponding yield data are employed as labels for ground truth. The dataset was further curated by removing blocks representing weed and sampled areas, and the 640 labeled picture blocks were split into training, validation, and test sets at random in the ratios of 7:1:2.

The test results were taken as a whole, and the different treatments (high-temperature treatment) did not affect them. By incorporating various treatments in the dataset, the model was exposed to a diverse range of scenarios, covering most possible outcomes. As a result, the model can provide reliable and trustworthy yield estimates under different conditions.

2.6. LSTM Network Model

LSTM, as a type of recurrent neural network (RNN), holds distinct advantages in modeling serial data, particularly in the context of remote sensing [62,63]. Unlike traditional RNNs that employ repeated hidden nodes, LSTM utilizes memory cells to capture both short- and long-term connections between neurons. A crucial feature of LSTM lies in the inclusion of a cell, which acts as an arbiter to evaluate the relevance and usefulness of incoming information. As data enter the LSTM network, they undergo an assessment based on predefined rules. Only information deemed valuable according to these rules is retained, while irrelevant data are discarded through a forgetting mechanism. This underlying structure of LSTM efficiently manages data transmission statuses, ensuring retention of pertinent information for an extended period while discarding unnecessary data [64]. This dynamic process enables LSTM to maintain system state and dynamics over time. Compared to other neural network architectures, LSTM’s recurrent state space effectively leverages spectral and spatial features, resulting in higher prediction accuracy.

The LSTM model employs four layers to perform the regression task: a time series input layer, an LSTM layer, a fully-connected layer, and a regression layer. The quantitative information for the input and output variables is represented by the input layer and the fully linked layer, respectively. The input layer is in charge of supplying the network with the time series information from the filling period, as shown in Figure 3F. The coefficient of determination (R²), root mean square error (RMSE), and relative RMSE (RMSE%) were computed to assess the efficacy of the yield prediction model.

$$R^2={\displaystyle \frac{{\sum }_{i=1}^n{(x_i-\tilde{x})}^2\times {(y_i-\overline{y})}^2}{{\sum }_{i=1}^n{(x_i-\tilde{x})}^2\times {\sum }_{i=1}^n{(y_i-\overline{y})}^2}}$$

(1)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n(y_i-{\widehat{y_i})}^2, \in [0,+\infty )}$$

(2)

$$RMSE\%={\displaystyle \frac{RMSE}{\overline{x}}}$$

(3)

3. Results

3.1. Obtaining Yield Information

After obtaining the weighted average of wheat yields, the final results are shown in

Table 3. Yield of wheat varieties under different treatments.

No.	Cultivar	Control Yield (kg/ha)	High Temperature Yield (kg/ha)
V1	Zhongmai 895	8139	7734
V2	Xinkemai 169	9049	8428
V3	Zhongmai 175	9581	8730
V4	Zhengmai 136	8774	8659
V5	Liangxing 99	8669	7808
V6	Huaimai 33	8976	8550
V7	Annong 0711	8527	8016
V8	Huacheng 3366	7243	6396
V9	Zhoumai 27	7959	7016
V10	Luohan 22	8869	8120
V11	Luohan 19	8424	8131
V12	Bainong 207	8985	8440
V13	Luomai 26	9270	8565
V14	Xinmai 36	8989	8134
V15	Fengdecunmai 5	8418	7698
V16	Fengdecunmai 21	7639	7361
V17	Fengdecunmai 1	7517	7202
V18	Zhengmai 369	7514	7185
V19	Zhoumai 36	7920	7307
V20	Zhengmai 366	7445	7123

.

3.2. Canopy Information Acquisition

3.2.1. Spectral Index

Advancements in drone imaging technology have revolutionized crop monitoring by enabling large-scale image capture and more accurate extraction of various phenotypic traits. The three developmental stages selected in this study, early grain fill (EGF), mid grain fill (MGF), and late grain fill (LGF), are the critical grouting stages before full maturity. Combining unmanned aerial vehicle remote sensing data from multiple developmental stages yields superior accuracy compared to single-stage crop production prediction [65]. During these stages, wheat exhibits less bare soil and high leaf cover, with crop nutrient growth nearing completion, making these periods highly correlated with yield traits [17,29]. Numerous studies have supported the notion that the filling stage is the optimal time for crop production forecasting using unmanned aerial vehicle remote sensing techniques.

During the three stages of wheat filling, vegetation indices (VIs) were measured for both the normal treatment (H0) and heat stress treatment (H1) components, with the results depicted in Figure 4. VI values exhibited a continuous decline during wheat filling, with a notable gap observed during seed filling. The trend of VI changes over time at different temperature treatments showed that: in early grain fill, the difference between heat stress treatment (H1) and the normal treatment (H0) was not significant and basically coincided; mid grain fill, the two produced differences, but heat stress treatment (H1) vegetation index values were generally lower than those of the normal treatment (H0); and late grain fill tended to level off, but generally speaking, heat stress treatment (H1) vegetation index values were still lower than those of the normal treatment (H0). The occurrence of high-temperature stress during the heat treatment led to specific physiological changes in wheat, including a reduction in flag leaf chlorophyll [66], source content, and hindered assimilate transport and distribution processes, ultimately resulting in limited grain filling and grain weight loss [3]. These physiological alterations are the primary reasons behind the weaker VI observed in the heat stress treatment compared to the normal treatment in this experiment. This is the reason why the VI in the heat stress treatment part of this experiment was weaker than the VI in the normal treatment part. The distinct changes in spectra observed among different wheat varieties, wheat filling stages, and temperature treatments corroborate the feasibility of using spectral indices as input variables for predicting wheat yield.

3.2.2. Canopy Height Observation

The distribution of canopy height serves as a critical indicator, reflecting the heterogeneity and genotypes of wheat communities. It plays a pivotal role in characterizing the vertical structure of wheat development and growth. Throughout various developmental phases, the spectral features of the wheat canopy undergo significant changes, which in turn affect the spectral data related to yield estimation. Notably, data on plant height become more important during the late growth stage compared to the early growth stage, and the contribution of canopy data to yield estimation varies significantly across different stages of wheat growth [67]. Thus, extracting canopy information specifically for the wheat filling stage holds greater significance for accurate yield assessment.

Based on different varieties, different filling stages, and different temperature treatments of wheat, the obtained crop surface model (CSM) was compared with the true canopy height, and the observed correlation R² between Measured Crop Height and Estimated Crop Height were 0.91, 0.90, and 0.92, respectively, as shown in Figure 5. This demonstrates the effectiveness and accuracy of our approach in extracting canopy information based on Real-Time Kinematic (RTK) drones. The RTK drones consistently predicted canopy height well, and the regression prediction results aligned closely with the actual canopy height data.

Analyzing the actual plant height data in Figure 5 reveals that during the filling phase, there is little discernible change in the height of wheat plants, with only a slight decrease in plant height observed during the late filling period. This slight decrease can be attributed to factors such as the proximity of plants to maturity, the influence of wind, seed weight, and a slight drooping of the wheat awns, leading to a slightly lower measured plant height. According to the data, CSM-derived plant heights are marginally less than those determined by field measurements, and a similar underestimation was reported by [25]. There are two possible reasons for this underestimation. Firstly, the positions of the same ears may shift in overlapping photos due to various factors, including random winds in the field that may displace the wheat awns and alter the positions of the wheat ears. Secondly, during the filling stage, wheat is nearly mature, and the tops of wheat plants become sharp and affected by the wheat awning, making it challenging to capture precise canopy top information at high resolution in unmanned aerial vehicle images.

3.3. Multi-Species Yield Estimates

Partial least squares regression (PLSR) [68], random forest (RF) [69], support vector machine (SVM) [40,70], BP neural network (BP NN) [71], and LSTM [62] were used to predict wheat seed yield manner using canopy spectral information extracted from unmanned aerial vehicle multispectral. As seen in

Table 4. Machine learning model regression effect.

Machine Learning Models	R2	RMSE	RMSE%
PLSR	0.2109	636.4563	7.8241
RF	0.2884	604.3907	7.4299
SVR	0.2685	612.7971	7.5333
BP NN	0.1882	645.5558	7.9360
LSTM	0.3951	557.2632	6.8506

, the yield prediction models constructed by the five machine learning algorithms had validation accuracies R² ranging from 0.19–0.40 and RMSE and RMSE% ranging from 557.26–645.56 and 6.85–7.94%, respectively.

The experimental long and short-term memory networks outperformed machine learning networks such as partial least squares, random forests, support vector regression machines, and BP neural networks in the modeling task, with an R² improvement of 0.21 over BP neural networks, which were the least effective in estimation, probably because traditional machine learning algorithms such as partial least squares, random forests, support vector regression machines, and BP neural networks are usually used for non-sequential data modeling tasks and cannot effectively capture the temporal dependencies in sequential data. For the wheat-filling period data, the LSTM model’s capacity to retain temporal memory enabled it to effectively process time series data, preserving relevant information while disregarding irrelevant elements [72]. To evaluate the yield estimation accuracy, a scatter plot was employed to compare the obtained grain yield estimates with corresponding observations, as depicted in Figure 6. By integrating the vegetation index with the canopy information as input for yield regression, the LSTM model achieved an R² value of 0.47. This integration of data from multiple sensors resulted in improved forecast accuracy compared to using a single sensor. However, in this regression, it was not able to estimate the yield of wheat close to harvest, probably because of the influence of multiple varieties and treatments, and the farmland information was too complicated for uniform regression processing, so this study explored categorizing wheat for separate yield regression predictions.

3.4. Species Classification Validation

High temperatures during the filling period of wheat significantly impact grain quality, particularly the accumulation and synthesis of starch and protein. This effect is primarily attributed to the functioning of related synthetic enzymes. Studies have demonstrated that under drought conditions, the activity of these enzymes decreases, directly affecting wheat yield [73]. Similarly, under high-temperature stress, proteins may suffer damage, leading to the disruption of their synthesis, inactivation of major enzymes, and destruction of cell membranes. Cell division is significantly affected by heat stress, ultimately influencing wheat yield [74].

Heat stress also affects thylakoid membranes, leading to chlorophyll degradation and crop senescence. Reduced plant photosynthetic capacity occurs due to metabolic restriction and oxidative damage to chloroplasts, resulting in decreased grain yield and dry matter accumulation [75]. High-temperature stress severely damages photosynthetic organs, causing a slowdown in leaves’ photosynthetic activity and accelerated leaf senescence. These factors collectively contribute to a slower buildup of dry mass and a reduction in the time taken for grain filling, ultimately leading to a decrease in grain weight and yield [76].

Heat resistance in wheat cultivars can be assessed by their ability to synthesize assimilates under heat stress [77]. Developing heat-tolerant is essential for cereal plants, like wheat, to improve grain quality and yield under high-temperature conditions [78]. Based on the above, a two-year physiological experiment was carried out [79]. Based on SOD activity, grain weight per spike, CAT activity, 1000-grain weight, canopy temperature, MDA content, and yield, 20 wheat varieties were divided into different heat-tolerant genotypes [79], which were the heat-tolerant genotype (HTG), moderate heat-tolerant genotype (MHTG) and heat-sensitive genotype (HSG) in

Table 5. Wheat heat-tolerant genotype classification table.

No.	Cultivar	Drought Tolerance	No.	Cultivar	Drought Tolerance
V1	Zhongmai 895	HTG	V11	Luohan 19	HTG
V2	Xinkemai 169	HSG	V12	Bainong 207	MHTG
V3	Zhongmai 175	HTG	V13	Luomai 26	MHTG
V4	Zhengmai 136	HTG	V14	Xinmai 36	HSG
V5	Liangxing 99	MHTG	V15	Fengdecunmai 5	HTG
V6	Huaimai 33	HTG	V16	Fengdecunmai 21	HTG
V7	Annong 0711	MHTG	V17	Fengdecunmai 1	HSG
V8	Huacheng 3366	HSG	V18	Zhengmai 369	MHTG
V9	Zhoumai 27	HSG	V19	Zhoumai 36	MHTG
V10	Luohan 22	HTG	V20	Zhengmai 366	MHTG

.

With the extension of the filling stage, the phenotypic changes of three heat-tolerant genotypes of wheat were extracted (Figure 7). It can be obtained that there are great differences in the traits and image characteristics of different genotypes in time. This disparity can be attributed to the heat-tolerant genotype’s inability to maintain high carbohydrate assimilation during heat stress [80], the heat stress treatment employed in the experiment further accentuated the divergences between the different wheat varieties. To facilitate efficient and large-scale assessment, unmanned aerial vehicles platforms were utilized. These unmanned aerial vehicle platforms demonstrated their capability to rapidly segregate genotypes and differentiate between various wheat varieties [81]. For the same genotype, there is a strong association between spectral indices assessed within a single environment and those measured across multiple environments [35,82]. It means that the spectral indices and canopy information generated by the unmanned aerial vehicles can be employed to forecast how much a genotype will produce in various circumstances, so in this paper, three genotypes of wheat, HTG, MHTG, and HSG, were distinguished and all treatments of the same genotypes were integrated and yield regressions were estimated separately.

3.5. Multi-Genotype-Based Yield Estimation

3.5.1. Spectral-Index-Based Yield Regression

The long and short-term memory network with the best prediction performance was chosen to predict grain yield for three different heat-tolerant genotypes of wheat. The vegetation index served as an input variable, and the obtained yield estimates were compared with corresponding observations using scatter plots (Figure 8A). The results indicated significant variations in yield prediction for different heat-tolerant wheat types. Where the heat-tolerant wheat predicted poorly: R² = 0.71 and RMSE = 312.35 kg/ha; The moderate heat-tolerant genotype predicted better than the heat-tolerant genotype: R² = 0.77 and RMSE = 329.13 kg/ha; The heat-sensitive genotype predicted best: R² = 0.84, RMSE = 345.03 kg/ha.

Figure 6A were split according to three wheat genotypes, obtaining a scatter plot (Figure 8B). Figure 8B was compared with the regression prediction results (Figure 8A) for the three genotypes. The R² of heat-tolerant wheat increased by 0.56, the R² of moderate heat-tolerant wheat increased by 0.40, and the R² of heat-sensitive genotype increased by 0.35. Distinguishing different heat-tolerant genotypes of wheat significantly influenced the yield estimation effect, resulting in a notable increase in the regression effect. The yield estimation results for all genotypes were excellent, with overall R² exceeding 0.7, demonstrating the strong predictive performance of the LSTM model for different genotypes of wheat.

The differences in the regression effect of different genotypes of wheat can be attributed to the spectral information’s ability to better reflect the physiological characteristics of wheat and express the spatial characteristics of the varieties [83] and better express the spatial characteristics of the varieties. Among them, heat-sensitive genotype yield prediction was better, probably because the phenotype of heat-sensitive genotype differed too much in different filling stages and the spectral index of response differed too much, resulting in better yield prediction of heat-sensitive genotype. Physiological characteristics may better capture the differences between heat-tolerant genotypes compared to appearance-based features. From the spectral expression of the three heat-tolerant genotypes (Figure 9), it can be seen that with the time change of the filling period, there is a great difference in the spectral information of different heat-tolerant genotypes of wheat, where the coefficient of variation (CV) of heat-tolerant wheat CV = 31.25%, moderate heat-tolerant CV = 34.32%, and sensitive CV = 36.52%. The increasing CV values indicate that the differences between different heat-tolerant genotypes of wheat grew as the filling period extended, significantly affecting the yield estimation results with notable variations among genotypes, all of which were superior to the yield regression estimation of multi-species wheat.

Notably, for instances with grain yields exceeding 8800 kg/ha, the regression results for all genotypes underestimated grain yields, as evidenced by the data points encircled in blue circles in Figure 8A. This phenomenon could be linked to the optical remote sensing asymptotic saturation issue, particularly at moderate to high yields [84].

3.5.2. Yield Regression Based on Spectral Index Combined with Canopy

The grain yield estimates obtained were compared with the corresponding observations using scatter plots to assess the performance of adding canopy information in improving estimation accuracy. The yield prediction results varied significantly for wheat with different heat-tolerant genotypes. It was observed that the combination of vegetation index (VI) and canopy structure information outperformed the use of VI data alone in wheat yield estimation, as shown in Figure 10. The best performance was found for the heat-sensitive genotype, R² = 0.91, RMSE = 255.63 kg/ha; the heat-tolerant genotype for wheat was predicted less well, R² = 0.79, RMSE = 268.17 kg/ha; and the moderate heat-tolerant genotype was predicted better than the heat-tolerant genotype R² = 0.85, RMSE = 268.70 kg/ha.

Figure 6B is split according to three wheat genotypes in a scatter plot (Figure 10B). Figure 10B compares the regression prediction results (Figure 10A) for the three genotypes. The R² of heat-tolerant wheat improved more, by 0.56; the R² of moderate heat-tolerant wheat by 0.38; and the R² of the heat-sensitive genotype by 0.35.

The integration of multispectral and RGB structure information resulted in a significant enhancement in prediction accuracy compared to utilizing a single sensor. Moreover, the combination of spectral indices and canopy information further improved the estimation of heat-tolerant genotype, achieving an overall improvement of approximately 0.07. This improvement can be attributed to the complementary aspects of canopy structure obtained from RGB. The independent information about canopy development and structure derived from RGB, such as CH, contributes valuable insights that are not attainable from spectral features alone [85]. Due to the differences in different genotypes of wheat varieties, the farm plots form different canopy structures and the spectra show different characteristic reflection curves. Therefore, the inclusion of canopy information is more expressive of spatial features and will improve the regression effect.

The canopy structures formed by different genotypes of wheat varieties exhibit distinct characteristics in their reflection curves. This variability in spectra and canopy structures necessitates the inclusion of canopy information to better capture spatial features and enhance the regression performance. Addressing the issue of under-prediction in the high-yield region, the utilization of the blue circle section has shown promising results. This integration of spectral and structural data helps overcome the asymptotic saturation problem that may arise in spectral characteristics [50]. The complementary nature of structural and spectral information is a key factor contributing to this improvement [86]. Numerous prior investigations have also validated the potential of combining spectral and structural data for accurate grain yield estimation [24,25].

3.6. Comparison and Application of Yield Forecasting

It is obvious from the VI regression analysis plot (Figure 11A) that the spatial prediction error of heat-sensitive wheat is large, RMSE% = 4.38%; the spatial prediction error of moderate heat-tolerant wheat is average, RMSE% = 4.06%; and the spatial prediction error of high-tolerant wheat is better, RMSE% = 3.76%; From the random distribution of the spatial prediction error of the model in Figure 9, it is evident that the model has indeed been modified to forecast the yield among many genotypes that are heat-tolerant, and the prediction error of different genotypes is different.

The VI combined with the canopy regression analysis plot (Figure 11B) clearly shows that the spatial prediction error of heat-tolerant wheat is large, RMSE% = 3.22%; the spatial prediction error of moderate heat-tolerant wheat is average, RMSE% = 3.31%; the spatial prediction error of heat-sensitive wheat is better, RMSE% = 3.25%. As can be seen from the stochastic distribution of errors (Figure 10), the errors in the combined canopy yield predictions are also reduced to some extent, and the regression predictions for all three wheat genotypes are reduced and more homogeneous than the predictions using a single sensor, demonstrating that combining spectral and structural data to predict grain yields is excellent.

The error analysis plot presented in Figure 11A,B indicates a noticeable reduction in prediction errors of heat stress treatment (H1) in comparison to the normal treatment (H0). This decrease can be attributed to the distinct developmental gaps observed in wheat with various heat-tolerant genotypes. These differences influence crop reflectance, consequently leading to varying prediction errors for wheat subjected to different temperature treatments. Notably, in wheat, seed weight correlates with assimilates derived from photosynthesis or reactivated from nutritive tissue to the growing reproductive tissue (seeds) [87]. Under high temperatures, grain yield reduction per plant during the filling stage can be attributed to increased growth and/or leaf senescence caused by a slowed photosynthetic rate [88]. Due to the significant variability in the sensitivity of wheat to high temperatures, this phenomenon may explain the model’s accurate prediction and low error rate.

In addition to the error analysis, a spatial distribution map of yields was generated based on the predictions of the above model, as shown in Figure 11C. This map demonstrates good agreement with the real yield distribution map (Figure 11D). The use of multispectral and RGB sensor data for calculating vegetation indices and plant canopy information has proven sufficient for accurately predicting wheat yields.

4. Discussion

In this paper, a temperature stress experiment in a field environment was utilized to conduct a unmanned aerial vehicle remote-sensing-based yield estimation study, and a representative natural population of wheat varieties in the Huanghuaihai wheat region was selected as the research object. Spectral indices and canopy structure were generated based on unmanned aerial vehicle multispectral image data of wheat canopies under high temperature stress, unmanned aerial vehicle RGB image data, wheat height data collected on the ground, and wheat yield data. Taking the three key periods of the irrigation period, pre-irrigation, mid-irrigation, and late-irrigation data as input features, five representative machine learning data analysis modeling methods, namely, partial least squares regression (PLSR) [68], random forest (RF) [69], support vector machine (SVM) [70], BP neural network model (BP NN) [71], and long-short-term memory network (LSTM) [72], were used to explore the yield prediction models of the five machine learning algorithms in different irrigation periods of wheat, and the accuracy and fusion of multi-source data were investigated. The potential of the five machine learning algorithm yield prediction models in different wheat sizing periods was explored in terms of prediction accuracy and multi-source data fusion for yield prediction. In order to explore higher prediction accuracy, a new method for wheat yield prediction is proposed, in which wheat is differentiated according to different heat-tolerant genotypes, and the LSTM algorithm with the optimal prediction effect is selected to establish single-sensor and multi-sensor remote sensing data fusion yield estimation models for different genotypes of wheat, and compare the accuracy of the estimation and assess the prediction accuracy of different heat-tolerant types of wheat in the large-field environment based on the spectral remote sensing imagery of unmanned aircraft. environment with prediction accuracy. The main conclusions are as follows:

The changes in spectral reflectance of wheat canopy were different for different varieties of wheat under different temperature treatments and at different critical grouting periods. For the image bands acquired by the multispectral drone, ratio or difference operations were performed to generate 13 vegetation indices, and it was found that the wheat canopy vegetation index showed a decreasing trend with the increase of the extension of the time of the grouting period, and this trend became more obvious as the wheat grew. For the wheat canopy vegetation indices under different temperature treatments, they were lower than the standardized wheat canopy vegetation indices of the experimental control in both the middle and late stages of the wheat grouting period. It is precisely because the spectra reflect different changes in different varieties, wheat grouting periods, and temperature treatments that the feasibility of the spectral indices as an input variable for predicting the yield of wheat is demonstrated [17,28].

For the canopy height observation of wheat, based on different varieties of wheat, different irrigating periods, different temperature treatments, the obtained CHM was compared with the real wheat canopy height, and the correlation R² values between the observed crop surface model and the real canopy height were 0.906, 0.90, 0.923, respectively. It can be concluded that the applicability of extracting canopy information based on the RGB unmanned aerial vehicle is real and effective, the RGB unmanned aerial vehicles are able to predict the canopy height well, and the regression prediction results compared with the real canopy information are basically consistent.

For the time-series data of wheat during the filling period, the vegetation indices generated by the multispectral sensor were used as input parameters to the five machine learning models. The wheat yield estimation models were built to predict the wheat yield. The validation accuracies of the yield prediction models constructed by the five machine learning algorithms ranged from 0.1882 to 0.3951 for R² and from 645.5558 to 557.2632 kg-hm² and 7.9360 to 6.8506% for RMSE and RMSE%, respectively. Among them, the multi-species wheat yield prediction model established by LSTM algorithm during the filling period showed better prediction accuracy. The LSTM model with the optimal prediction effect was selected, and the unmanned aerial vehicle sensor system was established by combining the RGB unmanned aerial vehicle with the vegetation index combined with the wheat canopy information to establish a multimodal wheat prediction model, in which the R² was improved by 0.07 compared with the single sensor, and the RMSE and RMSE% were decreased by 33.92 and 0.42 compared with the multispectral sensor, respectively. However, this may be due to the effects of multi-species and multi-treatment, and the farmland information is too complicated, with the result that the wheat yield prediction effect is not close to the harvested wheat yield. Therefore, a new method for wheat yield prediction is proposed in the subsequent study, using LSTM model to regressively predict different categories of wheat.

Twenty wheat varieties were categorized according to different heat-tolerant genotypes into heat-tolerant genotypes, moderate heat-tolerant genotypes, and high-sensitive genotypes. The LSTM model, which has the best prediction effect among the above machine learning models, was selected, and the vegetation index extracted based on multispectral unmanned aerial vehicle was used as the input variable to predict the yield regression of the three genotypes of wheat, in which the R² of heat-tolerant genotypes, moderate heat-tolerant genotypes, and high-sensitive genotypes ranged from 0.71 to 0.84 and the RMSE ranged from 312.35 to 345.03. Comparing with the multi-species wheat yield prediction, the estimation accuracies of the model predictions were all improved, ranging from 0.35 to 0.56. It can be proved that the LSTM model performs well in predicting the yield of different genotypes of wheat. This indicates that the LSTM model can handle the genotypic differences of wheat well with good accuracy and stability. By comparing the coefficients of variation between the spectra of different heat-tolerant wheat, it is found that the difference is gradually increasing with the extension of time, which is also the reason for the good effect of regression for distinguishing the yield of different genotypes of wheat.

Multi-sensor data fusion by combining spectral indices with wheat canopy information to build a multimodal wheat prediction model significantly improved the prediction accuracy compared to using only a single sensor, with an overall improvement of about 7% and an R² of up to 0.91. Multi-sensor data fusion shows great potential for unmanned aerial vehicle yield estimation, and the added wheat canopy structural information helped to increase the model’s spatial adaptability and better representation of spatial features, which will ultimately improve the regression effect of wheat yield, confirming the potential of combining spectral and structural information in cereal yield prediction.

Yield regression prediction error maps were established to compare the differences in prediction errors among the three genotypes, obtaining a large spatial prediction error for heat-tolerant wheat with RMSE% = 4.40%, an average spatial prediction error for medium heat-tolerant wheat with RMSE% = 3.63%, and a better spatial prediction error for heat-tolerant wheat with RMSE% = 3.76%. The final generation of randomly distributed spatial distribution maps of yield provides more possibilities to help farmers and agricultural scientists to make crop management and marketing decisions based on the predicted yield potential.

In agricultural production, numerous plots are involved, and in the future, as the scale of prediction expands to encompass thousands of acres of wheat, differentiating these plots based on their heat-tolerant genotypes and selecting an appropriate yield prediction model can significantly enhance prediction accuracy. This approach offers benefits to breeders, who can utilize more precise predictive models without the necessity of harvesting every single plot to calculate yield. Consequently, it becomes possible to reduce trial costs and data collection time, while obtaining yield distributions rapidly. The efficiency of the unmanned aerial vehicle phenotypic analysis makes it possible to monitor dynamics with high temporal resolution. Farmers and agricultural scientists stand to gain from accurate yield forecasts, providing them with additional options for crop management and sales, based on the projected yield potential. These advancements present new ideas and solutions to the agricultural sector.

The main problems and prospects of the study are as follows: (I) This study’s prediction analysis focused on 20 representative wheat varieties in the Yellow and Huaihai regions. However, to further improve the yield estimation models and broaden their applicability, future research should encompass a more diverse set of plots and incorporate a greater variety of wheat genotypes for prediction analysis. By expanding the scope of study to include more plots and varieties, researchers can refine and develop a more comprehensive understanding of yield prediction, build more variety of yield estimation models. (II) This paper employs remote sensing time-series data during the irrigation period to develop a yield prediction model. Additionally, it explores the potential of machine learning algorithms to enhance the precision of crop yield prediction through multimodal data fusion. While machine learning models show promise in leveraging remotely sensed data for crop yield estimation, further investigation is required to integrate this data with other pertinent perturbation factors, such as soil conditions, weather patterns, and management information. The combination of remote sensing data with these factors is crucial to achieve more accurate crop yield estimation. (III) The current unmanned aerial vehicle remote sensing platform solely relies on RGB and multispectral drones, lacking the capture of hyperspectral reflectance and thermal information provided by hyperspectral and thermal infrared sensors for fusion analysis. Further exploration can be based on different high-throughput phenotyping platforms, field phenotyping vehicles, etc., equipped with LIDAR, hyperspectral sensors, thermal infrared cameras, and other sensors to obtain more multi-modal, multi-temporal phenotyping data. Collecting such multi-modal, multi-temporal phenotyping data will enable the establishment of more comprehensive yield estimation models, resulting in prediction models of superior accuracy.

5. Conclusions

This study explores the potential of predicting wheat crop yields for different heat-tolerant genotypes during the wheat filling period, utilizing a low-cost multispectral and RTK unmanned aerial vehicle combination in conjunction with the LSTM model. The main findings indicate that the LSTM model can effectively regress crop yield for wheat data during the filling period, yielding an impressive R² value of up to 0.91. Furthermore, differentiating between various heat-tolerant genotypes of wheat for yield assessment outperforms the combined assessment approach, particularly excelling in the regression of high temperature-sensitive wheat. The integration of spectral information with canopy data proves to be a valuable enhancement, contributing to a notable improvement in prediction accuracy, approximately by 0.07. This fusion of data aids in establishing a spatially adaptive model, reducing spatial dependence and variability. As a result, the generated field yield forecast maps exhibit outstanding spatial resolution, enabling the provision of yield distributions for multi-strain wheat collections. In summary, this study successfully demonstrates a robust strategy for yield prediction, achieved through the differentiation of various wheat heat-tolerant genotypes to facilitate accurate yield estimation. The precise prediction of wheat yield holds substantial potential in supporting informed agricultural management decisions, especially concerning emergency forecasting for different wheat genotypes within vast agricultural landscapes comprising thousands of plots with diverse varieties.