Performance of Neural Networks in the Prediction of Nitrogen Nutrition in Strawberry Plants

Abstract

Among the technological tools used in precision agriculture, the convolutional neural network (CNN) has shown promise in determining the nutritional status of plants, reducing the time required to obtain results and optimizing the variable application rates of fertilizers. Not knowing the appropriate amount of nitrogen to apply can cause environmental damage and increase production costs; thus, technological tools are required that identify the plant’s real nutritional demands, and that are subject to evaluation and improvement, considering the variability of agricultural environments. The objective of this study was to evaluate and compare the performance of two convolutional neural networks in classifying leaf nitrogen in strawberry plants by using RGB images. The experiment was carried out in randomized blocks with three treatments (T1: 50%, T2: 100%, and T3: 150% of recommended nitrogen fertilization), two plots and five replications. The leaves were collected in the phenological phase of floral induction and digitized on a flatbed scanner; this was followed by processing and analysis of the models. ResNet-50 proved to be superior compared to the personalized CNN, achieving accuracy rates of 78% and 48% and AUC of 76%, respectively, increasing classification accuracy by 38.5%. The importance of this technique in different cultures and environments is highlighted to consolidate this approach.

1. Introduction

Strawberry production (Fragaria × ananassa Duch., Rosaceae), worldwide, is led by China and the United States, but Brazil stands out as the 8th largest producer, with production reaching 197 thousand tons [1]. From 2011 to 2021, global acreage grew by 20%, and production increased by 44%. In Brazil, the strawberry area grew to 6.280 ha in the 2022/23 crop year, up from 2.500 ha in 2012 [2]. This rise highlights Brazil’s contribution to the global strawberry production scenario. Investments in research and new technologies have been made to meet this growing demand, with the aim of developing more productive varieties.

Nutritional management plays a fundamental role in crop development, making it a widely studied topic due to its direct influence on increasing productivity and fruit quality. Nitrogen is an essential element for plants, being the main constituent of chlorophyll, which is directly involved in photosynthetic activity. Therefore, the greenness of leaves is determined by the concentration of chlorophyll molecules [3]. Variations in green hues due to foliar nitrogen (N) content can result in class separation in an RGB image bank when it comes to computer vision. Thus, this pigment becomes essential in the context of precision agriculture because it serves as a basis for quantifying foliar N content [4,5,6]. A practical and quick way to determine the existence of nutrient deficiency in plants is through visual diagnosis. However, its accuracy is limited by the experience of the technician and it also requires a significant amount work in generating a prescription map to be used in localized management. As a result, new research is constantly emerging to estimate the ideal nutrient level in crops [7,8].

Several scientific studies and research efforts have highlighted the proven relationship between the concentration of green in the leaves, analyzed through computer vision, and the nitrogen content in plants [9,10,11,12]. In this context, nitrogen exerts a significant influence on plant growth, productivity and soil fertility [13]. Agricultural production, which is essential for global food security, relies largely on synthetic fertilizers that provide this key element [14]. Thus, studies on nitrogen strive to increase its efficiency of use.

With regard to the assessment of the nutritional status of plants, conventional approaches to assessing the nutritional status of plants are generally destructive, labor-intensive and require specific resources [15]. Considering the importance of strawberry cultivation and the need for more accessible, quick and efficient methods to determine nutritional status, there is an interest in the development of more effective methods [16].

In the field of precision agriculture, the application of predictive techniques, such as the use of nutrient sensors in plants, stands out [17], and remote sensing with drones in urban forest management has been developed [18]. In addition, there are approaches based on image processing and artificial intelligence, which stand out for their ability to identify nutritional deficiencies in crops in a non-invasive way. However, it is essential to improve N prediction techniques in order to ensure the necessary accuracy in agricultural decision-making, considering the complexity of the relationships between the availability of this element, plant growth, and fruit quality.

Among the AI (artificial intelligence) tools used for prediction and classification, deep learning has been applied in different areas, presenting satisfactory results in solving various problems. According to Ferentinos et al., [19], this feature refers to the use of artificial neural network architectures with multiple processing layers, distinguishing them from the more superficial approaches of traditional neural networks. Among the former class, the convolutional neural network (CNN) stands out among the deep learning algorithms and is widely used to extract features from images [20,21]. During image processing, CNNs allow the extraction of several characteristics, especially those related to color-based indices, which are relevant to nutritional applications. In the face of the challenge of accurately identifying nitrogen levels and the growing importance of precision agriculture for sustainable development, investing in innovative technologies is essential. Deep learning tools capable of handling RGB images can assist in nitrogen fertilization management in strawberry crops by classifying foliar nutritional status. This approach involves applying CNNs to process images and classify nitrogen levels in plants.

Therefore, the present study aimed to evaluate and compare the performance of two convolutional neural networks in the classification of leaf nitrogen levels in strawberry crops, based on RGB images. The analysis focused on performance metrics of the generated models, aiming to contribute to the advancement of knowledge and the practical implementation of more effective tools in the nutritional management of plantations.

2. Materials and Methods

An experiment was carried out to create an image bank, with strawberry cultivation as its theme. The experiment was carried out from August to December 2022 at the Faculty of Animal Science and Food Engineering (FZEA/USP), with strawberries (Fragaria × ananassa Duch., Rosaceae) of the San Andreas cultivar. The experiment took place in a 100 m² greenhouse, with climate control provided by an evaporative air system, located at the “Fernando Costa” campus, in Pirassununga. The geographic coordinates of the location are approximately 21°57′25.2″ S, 47°27′13.7″ W, with an average altitude of 610 m above sea level. The region has a CWA climate, with well-defined seasons, one, rainy, from October to March, and the other, dry, from April to September, with an average annual rainfall of 1298 mm and an average temperature of 20.8 °C. The strawberry seedlings were transplanted in October 2022 to 30 pots, comprising three treatments and five replications, as each replication was divided into two plots. The treatments consisted of applying different doses of nitrogen fertilizers as follows: 50% of the recommended fertilization (FR), by Passos et al., 2013 (T1); 100% FR (T2), control; and 150% FR (T3) [22]. This methodology has been used by different researchers [23,24].

The image database was generated and organized to serve as a resource of extractable characteristics for classification, and these characteristics were used as input to the network and their vectors used in the training of computational classifiers based on deep learning methods, having the classes of the different N doses as the output or target attribute of the model.

To obtain parameters for comparison of the model, leaf chemical analysis was performed at the Laboratory of Agrarian/Soil Sciences of the Faculty of Animal Science and Food Engineering of the University of São Paulo, according to the methodology described in the manual of chemical analysis of soils, plants and fertilizers published by EMBRAPA [25].

2.1. Obtaining and Processing the Images

The indicative leaves of strawberries were collected on the third expanded leaf, and were selected for uniformity of age and stage of development in all treatments and replications, to ensure consistency in the comparative analyses. After collection, at the Laboratory of Agricultural Machinery and Precision Agriculture (LAMAP), of the Department of Biosystems Engineering of the University of São Paulo, the leaves were prepared to obtain the images. This step involved removing any excess dust or dirt by using a paper towel. With a high-resolution flatbed scanner (HP Scanjet 3800), the images of the sheets were scanned in RGB (red–green–blue) color, in JPG (Joint Photographic Experts Group) format and with a resolution of 1200 DPI (dots per inch).

Using a script developed in the Matlab© R2021b software, the scanned images of the strawberry leaves were cropped, automatically extracting blocks of 224 × 224 pixels and thus generating an image bank with specific areas of interest [26] for further processing. This procedure aimed to obtain images that would provide an ideal visualization of the patterns generated by the doses on the leaves, as exemplified in Figure 1. The image blocks were scaled to a standardized size, as required in the input of the ResNet architecture models [27].

After the completion of the automatic cropping stage, the manual selection of image blocks was performed to compose the database. During this process, each image block was checked to ensure the quality and integrity of the data to be used. The selection followed specific criteria, requiring that the image of each block be completely filled by a region of the leaf, i.e., without characteristics of visible leaf damage or injury. This approach ensured the presence of only the areas relevant for analysis in each block. In addition, blocks that presented noise, such as white background or dirt on the leaf, were excluded, since such imperfections could interfere with the subsequent classification process. After manual selection, 50 blocks of images were chosen for each vessel, totaling 500 blocks per treatment. With three treatments analyzed, the final dataset consisted of 1500 blocks of images.

With the definition of the dataset, a script was developed in the Matlab© R2021b software to generate the classifier models of the fertilized nitrogen dosage, using two different CNN architectures, one customized and the other based on ResNet-50, respectively, as described in

Table 1. Personalized CNN architecture for leaf nitrogen level prediction.

Layers	Settings
2D Convolutional	256 filters with size 5 × 5, activation by ReLU
Batch Normalization	Default
2D MaxPooling	Size pooling = 2 × 2, stride = 2
2D Convolutional	128 filters with size 3 × 7, activation by ReLU
Batch Normalization	Default
2D MaxPooling	Size pooling = 2 × 2, stride = 2
2D Convolutional	128 filters with size 7 × 3, activation by ReLU
Batch Normalization	Default
2D Convolutional	32 filters with size 3 × 3, activation by ReLU
Batch Normalization	Default
Fully Connected	3 neurons, activation by softmax

and

Table 2. CNN ResNet-50 architecture for leaf nitrogen level prediction.

Layer Name	Output Size	50 Layers
conv1	112 × 112	7 × 7, 64, stride 2
conv2_x	56 × 56	3 × 3 max pool, stride 2
		$\left[\begin{array}{cc}1\times 1& , 64\\ 3\times 3& , 64\\ 1\times 1& , 256\end{array}\right]\times 3$
conv3_x	28 × 28	$\left[\begin{array}{cc}1\times 1& , 128\\ 3\times 3& , 128\\ 1\times 1& , 512\end{array}\right]\times 4$
conv4_x	14 × 14	$\left[\begin{array}{cc}1\times 1& , 256\\ 3\times 3& ,256\\ 1\times 1& ,1024\end{array}\right]\times 6$
conv5_x	7 × 7	$\left[\begin{array}{cc}1\times 1& , 512\\ 3\times 3& , 512\\ 1\times 1& , 2048\end{array}\right]\times 3$
	1 × 1	Average pool, 3-d fc, softmax

.

The personalized CNN architecture is designed as a sequential model, with four layers of two-dimensional convolution (2D convolutional) using different numbers and sizes of filters. These layers form the basis of a CNN model, being responsible for extracting characteristics from the input data [28]. In the case of images, convolution layers allow one to learn visual patterns (such as colors, shapes, and textures), from the simplest to the most complex, successively [29]. Using the Rectified Linear Unit activation function in the convolution layers allows you to better deal with non-linearity issues at a lower computing cost [28]. The adoption of rectangular filters (3 × 7 and 7 × 3) in the second and third convolution layers aimed to capture possible vertical and horizontal patterns. Following each convolution layer is a normalization layer (Batch Normalization), which speeds up training, decreases overfitting and optimizes the learning process [30]. After the first and second normalization layers is a subsampling layer (2D MaxPooling), which is used to reduce the dimensionality of the activation maps, saving computing resources while preserving the most relevant data [28]. Finally, there is the output (Fully Connected), or classification layer. The softmax activation function used in this layer transforms its neurons into probabilities assigned to each class, effectively turning the model into a classifier.

The other CNN architecture adopted for this study was based on ResNet-50, as described in Table 2. A convolutional neural network proposed by He et al. [31], it consists of 50 layers, of which 49 are convolutional. Addressing the complexity of deep network training, ResNet-50 introduces the “identity shortcut connection”, allowing one to skip one or more layers to form residual blocks. In addition, to optimize computational efficiency, the ResNet-50 utilizes a “bottleneck design” structure, reducing the number of parameters and operations on each residual block [31].

The ResNet-50 architecture allows the loading of a pre-trained version with more than one million images from the ImageNet database. This makes it possible to classify images into 1000 categories. With an image input of 224 × 224 pixels, this pre-trained neural network provides comprehensive representations for a variety of natural images, allowing you to reuse learned visual patterns in applications involving other categories of images. This technique of reusing learned knowledge, called learning transfer, has become a common practice, as it accelerates the training process, saves computational resources, and helps in the generalization and convergence capacity of new models, being especially useful in situations where the database of the problem addressed is insufficient for learning [32]. Table 2 shows the elements of the 50-layer ResNet architecture.

2.2. Analysis of the Models

To evaluate the reliability of the model, the K-fold cross-validation technique was used, in which the dataset is divided into distinct groups [33]. In this study, we opted for a five-fold cross-validation (K = 5), which entails dividing the dataset into five different groups. Each of these groups served as a test set in five independent iterations. This means that in each iteration, a different group was used for validation, while the other four (or K − 1) were employed for model training. At the end of the five iterations, the average precision was calculated, which then served as the final measure of the model’s performance. During the cross-validation process, 80% of the images were used for training, which is equivalent to 1200 blocks of images. The remaining group, consisting of 20% of the images (300 images) was designated as a validation and test set to evaluate the performance of the model. This training and testing procedure was repeated five times, ensuring that each group would be used once as a test set. This strategy ensures that all images in the dataset are used for both training and testing, providing a more robust and reliable assessment of the model’s performance.

For the model, we took into account the configurations, as shown in

Table 3. Training options used in the convolutional model.

Number of times	10
Mini-lot size	8
Option for data scrambling	every-epoch
Positive scaling of the initial learning rate	0.0001

, in training the neural networks in which the hyperparameters are inserted, since they are responsible for defining how the model will be trained, and, consequently, how it will be able to perform the task for which it was designed [34].

For both the personalized convolutional neural network and ResNet-50, the number of epochs used was 10. In a comparative study, Too et al. [35] considered the fine-tuning of deep learning models for plant disease identification, and compared the behaviors for 10 and 30 epochs, observing that ResNet-50 and ResNet 101 work properly with fewer iterations, and, in addition, after fine-tuning, the models using 10 epochs obtained an accuracy above 90%.

Table 4. Description of performance evaluation metrics.

Metric	Formula	Description
Accuracy	$A_{cc}=\frac{TP+TN}{TP+TN+FP+FN}$	The overall efficiency of a model.
Sensitivity or Recall	$Se=\frac{TP}{TP+FN}$	The efficiency of a model as to positive samples.
Specificity	$Sp=\frac{TN}{TN+FP}$	The efficiency of a model as to negative samples.
Precision	$Pre=\frac{TP}{TP+FP}$	The proportion of actual positives out of all positives predicted by the model.
F1-score	${F1}_{score}=\frac{2 \ast Pre \ast Se}{Pre+Se}$	The harmonic mean between precision and sensitivity.
G-mean	$G_{mean}=\sqrt{recall\ast specificity}$	Maximum accuracy in each of the classes.

TP: true positive; TN: true negative; FN: false negative; FP: false positive; Pre: accuracy; Se: sensitivity.

summarizes the formulas of the performance metrics and their expressions used in the present study.

The area under the curve (AUC) reflects the performance of a classifier, illustrating how the true positive rate (recall/sensitivity) varies as the false positive rate (inverted specificity) changes [36]. This analysis provides insights into the model’s ability to distinguish between classes in different decision-making contexts. Models with an AUC close to 1.0 are considered highly effective in classification tasks [37], thus highlighting the fundamental relevance of this metric in evaluating model performance.

3. Results and Discussion

3.1. Foliar Chemical Analysis

Figure 2 shows the boxplot test for the values of leaf nitrogen concentration in strawberry leaves in the flower induction stage. N concentrations ranged from 19.4 g kg⁻¹ to 33.3 g kg⁻¹. The T1 (50% of the N dose) and T2 (100% of the N dose) treatments exhibit quite similar median and mean values, while the T3 treatment (150% of the N dose) shows a higher trend.

3.2. Analysis of the Models

The confusion matrix evaluates the correspondence between the output predicted by the model and the desired output, and then calculates the classification accuracy for each class in a specific class, thus revealing the strengths and weaknesses of the model [38]. The rating rate using personalized CNN was 48.1%, while for ResNet-50 it was 78.1%. Analysis of the matrices for the personalized CNN and ResNet-50 models, as described in Figure 3, reveals distinct patterns in the classification capacity of the treatments (T1, T2, and T3). For personalized CNN, the performance was not satisfactory in view of the fact that the model got more wrong than it got right. Strategies could be explored to improve the outcome of the model, such as optimizing the hyperparameters, including filter size, number of layers and learning rate [39]. As for ResNet-50, it performed better, as it extracts patterns from more complex spectral features [40]; the rate of correct answers was higher in this model, reaching 78% accuracy. For both networks, T1 was the one with the highest rate of correct answers, with 60.2% and 84.4%, respectively, for the personalized CNN and ResNet-50; this was possibly due to the symptom of nutritional deficiency, expressed as a uniform chlorosis, which allowed the model to classify more assertively.

Table 5. Personalized CNN convolutional neural networks and ResNet-50 performance metrics.

	Personalized CNN (%)			ResNet-50 (%)
Metrics	T1	T2	T3	T1	T2	T3
Accuracy	48	48	48	78	78	78
Sensitivity	60	44	40	84	73	77
Specificity	42	50	52	75	81	79
Precision	34	31	29	63	65	64
Recall	60	44	40	84	73	77
F1-score	44	36	34	72	69	70
G-mean	50	47	46	80	77	78

shows the performance of the classifiers in discriminating the different N doses (T1 = 50%, T2 = 100%, and T3 = 150%) in the phenological stage of floral induction based on neural networks. These metrics provide different perspectives on the model’s performance and help in the overall understanding of its effectiveness.

The generated models used 1200 excerpts from the database for training. The customized CNN model achieved a mean accuracy of 48%, an unsatisfactory performance in distinguishing between the 50%, 100% and 150% treatments, as confirmed by the lower recall values and precision values of these classes. Possibly, the personalized CNN had an unsatisfactory performance due to the complexity of the standards expressed by the treatments, making categorization difficult, given this structure and the configuration of the CNN, in addition to the lack of generalization in relation to negative samples.

The best performance in the classification presented was with the use of ResNet-50 for the dose of 50% N, with an accuracy of 84%. For the T2 and T3 treatments, the accuracy, recall and F1-score metrics showed better results than those of the personalized CNN, but they are similar to each other. This suggests that the leaves in these treatments exhibited less variation in color intensity and distribution, making the classification of models more costly.

A comparison of the AUC analyses is shown in Figure 4. First, for the customized CNN, the result was not satisfactory for floral induction, remaining within a range of 0.57 to 0.64, except for T1. As the phenological stage of the plant advances, an increase in leaf chlorosis can be observed due to differences in nitrogen concentration between the treatments. The coloration of the plant’s leaf can be influenced in a variety of ways, depending on the availability of nitrogen. In nitrogen deficiency, leaves tend to become paler, yellowish, or light greenish, due to the reduction in chlorophyll synthesis, which is responsible for the green coloration of leaves, and excess nitrogen can lead to a very dark green color of leaves and excessive vegetative growth [41].

The analysis of the metrics highlighted that ResNet-50 had higher values in all metrics, especially in specificity, ranging from 64% to 69%, while personalized CNN had lower specificity, ranging from 41% to 44%.

The comparison between the architectures revealed that ResNet-50 achieved superior accuracy, standing out for its ability to identify distinct patterns, in contrast to the custom CNN, which had lower performance due to its lower capacity for learning and generalization. These results corroborate the findings of previous studies, such as those of Costa et al. [42], which achieved 86.6% recall when applying ResNet-50 for detection of defects in tomatoes, reinforcing the efficiency of this architecture in predicting adequate fertilization rates in strawberry plants.

The superior classification performance of the model using ResNet-50 may be related to the characteristics presented in this convnet. In this architecture, there are blocks with convolutional layers in sequence and a separate parallel identity layer [43], which can result in a more precise extraction of color or texture features from plant leaf images [44]. In the ResNet-50 used, the technique of transfer learning, combined with pre-training on large datasets, was applied, which conferred an essential generalization capacity for classifying different nitrogen concentrations in strawberry leaves [45]. In addition, the application of residuals to the convolutions, a differential of ResNet-50, allows the network to learn residual differences in the image features, optimizing the model’s training.

4. Conclusions

The study focused on comparing the performance of convolutional neural network architectures in classifying the nitrogen status of strawberry leaves. The results highlighted the better performance of the Resnet-50 architecture compared to the custom CNN in the specific task of classifying nitrogen status in strawberry leaves by handling RGB images. Evaluating the two main performance analysis metrics of the models, it was found that Resnet-50, with total accuracy = 78% and AUC T1 = 0.76, T2 = 0.63 and T3 = 0.61, outperformed the custom CNN with total accuracy = 48% and AUC T1 = 0.96, T2 = 0.90 and T3 = 0.91. An increase of 38.5% was achieved with ResNet-50 when compared to the custom CNN. By offering a more accurate and efficient approach to nutritional status detection, Resnet-50 demonstrated its potential for improving monitoring processes, standing out in this work as an important tool in precision agriculture. Additionally, it is suggested that further studies explore image preprocessing techniques and fine-tuning, such as adjustments to model hyperparameters, regularization and data augmentation techniques, to further optimize the accuracy and effectiveness of the classification model. These initiatives have the potential to elevate Resnet-50 to a new level of performance, consolidating its role as an essential tool in precision agriculture. Thus, this research offers important practical knowledge for the efficient management of agricultural crops, promoting a more sustainable approach to modern agriculture. Finally, further field research involving other crops is recommended for the validation of this approach.