Recent Trends and Advances in Utilizing Digital Image Processing for Crop Nitrogen Management

Abstract

Crop nitrogen (N) management in agricultural fields is crucial in preventing various environmental and socio-economic issues arising from excess N use. However, precise crop N management (PNM) is hindered by its intensive data requirements, high cost, and time requirements. Digital image processing (DIP) offers a promising approach to overcoming these challenges, and numerous studies have explored its application in N management. This review aims to analyze research trends in applying DIP for N management over the past 5 years, summarize the most recent studies, and identify challenges and opportunities. Web of Science, Scopus, IEEE Xplore, and Engineering Village were referred to for literature searches. A total of 95 articles remained after the screening and selection process. Interest in integrating machine learning and deep learning algorithms with DIP has increased, with the frequently used algorithms—Random Forest, Support Vector Machine, Extreme Gradient Boost, and Convolutional Neural Networks—achieving higher prediction accuracy levels. In addition, image data using more variables as model inputs, including agriculture sensors and meteorological data, have increased prediction accuracy. Nonetheless, several challenges associated with DIP, including obtaining high-quality datasets, complex image processing steps, costly infrastructure, and a user-unfriendly technical environment, still need to be addressed.

1. Introduction

Nitrogen (N) is one of the essential macro-nutrients required by plants for their optimum growth, particularly for protein and chlorophyll synthesis [1,2,3]. Chlorophyll, which is central to photosynthesis, directly impacts the quantity of crop yield, whereas the yield quality is influenced by phytoproteins derived from amino acids. Despite its importance, N is not abundantly available for plant uptake, making it the most limiting nutrient in crop production [4,5]. Consequently, using N fertilizers in crop cultivation has become an indispensable necessity in commercial agriculture to meet the rising global demand for food.

Since the introduction of the Haber–Bosch process, the use of inorganic N fertilizers in commercial agriculture has significantly increased [6]. But N fertilizer use efficiency (NUE) remains below 54% [7,8,9,10], indicating that the remaining half of applied N is not utilized by plants. This could potentially lead to several unintended negative impacts on the environment and cost of production [8,9,11]. Unutilized N could leach or runoff to ground and surface water bodies, leading to water pollution and eutrophication, while gaseous N losses due to denitrification contribute to global warming. The United Nations identified sustainable N management as an essential aspect of agriculture. Out of the 17 Sustainable Development Goals set out by the United Nations, at least 9 are closely related to N management [9].

In this context, precision N management (PNM) has been recognized to increase the NUE in agriculture, which mainly focuses on supplying N fertilizer based on crop demand [2,8,12]. The primary challenge is real-time, accurate, and continuous crop N demand quantification across large spatial crop coverages. Traditional laboratory analysis of soil and plant samples is more accurate in determining crop N demand. As this approach is highly labor-intensive, time-consuming, and often provides limited spatial coverage [13,14,15,16,17,18], it has become less attractive and discourages farmers from adopting PNM. In contrast, remote sensing accompanied by digital image processing (DIP) offers a non-destructive and real-time approach for monitoring crop N with reliable accuracy [1,19,20,21].

Digital image processing is a rapidly advancing technology that has impacted various areas of agriculture. For crop N management, ground-based, airborne, and spaceborne imaging technologies, employing RGB, multispectral (MS), or hyperspectral (HS) sensors, offer diverse methods for estimating N levels. Key features related to crop N status (e.g., leaf color, texture, vegetation indices, chlorophyll content) are extracted from these images leveraging both physical-based and data-driven models to predict and map N variability in the field [22].

Figure 1 illustrates how DIP has been used for PNM in recent years compared to other agricultural applications. It shows the number of articles indexed in the Web of Science database based on separate keyword searches: ‘nitrogen’, ‘disease’, ‘yield’, ‘counting’, ‘pests’, and ‘weed’ with ‘agriculture’ and ‘digital image processing’. Most of the agriculture-related DIP has been focused on crop yield prediction (36%) and crop disease recognition (25%). Out of the total, only 11% have been published on crop N management using DIP in the last 5 years. The relatively limited focus on N management underscores a critical gap in literature.

It is important to identify current trends for researchers to stay ahead of the curve, ensuring their work remains relevant and impactful. There is limited research summarizing recent trends, challenges, and opportunities in using DIP for crop N management. Existing studies often focus on isolated applications or technologies, without integrating the broader context of recent advancements, socio-economic factors, or regional variability.

The main objectives of this review are to summarize and critically analyze the trends and advancements in DIP over the recent 5 years for crop N management, to recognize challenges in the practical application of DIP, including technical and socio-economic factors, and to propose future directions to enhance DIP applications in N management. To achieve these objectives, this paper discusses recent trends highlighting frequently co-occurring terms with agricultural N management, regions where this technology has been mostly studied, and the progression of statistical, ML, and DL algorithm applications in DIP. Then, it identifies key challenges in image acquisition, digital image processing, and socio-economic factors, along with future directions in the field.

This paper aims to bridge existing knowledge gaps and provide a guide for future research, ensuring the continued relevance and impact of DIP technologies in PNM.

2. Materials and Methods

A literature search was conducted using four databases: Web of Science, Scopus, IEEE Xplore, and Engineering Village, using four combinations of keywords.

• ‘Nitrogen’, ‘Machine Learning’, and ‘Computer Vision’;
• ‘Nitrogen’, ‘Deep Learning’, and ‘RGB images’;
• ‘Nitrogen’, Agriculture’, and ‘Digital Image Processing’;
• ‘Nitrogen’, ‘Artificial Intelligence’, and ‘Images’.

Documents, excluding review articles, retracted publications, books, and book chapters, were selected. The search was limited to materials published within the last five years (2020–2024 October). Only publications in English were selected, and all articles were closely screened to eliminate articles irrelevant to the review’s main focus. A summary of the literature selection criteria is given in

Table 1. A summary of the literature selection criteria for the review.

Keywords	Document Type	Database	Number of Results	After Screening and Selection
‘Nitrogen’, ‘Machine Learning’, and ‘Computer Vision’	Articles, proceeding papers, and early access	WoS	26	9
		SCP	26	9
		IEX	24	5
		EV	27	5
‘Nitrogen’, ‘Deep Learning’, and ‘RGB images’	Articles, conference papers, and early access	WoS	30	12
		SCP	25	12
		IEX	7	2
		EV	14	8
‘Nitrogen’, Agriculture’, and ‘Digital Image Processing’	Conference articles, proceedings, dissertations, preprints, and journals	WoS	61	24
		SCP	7	3
		IEX	9	1
		EV	40	13
‘Nitrogen’, ‘Artificial Intelligence’, and ‘Images’	Journal articles, conference articles, proceedings, preprints, and dissertations	Wos	94	14
		SCP	48	7
		IEX	42	19
		EV	91	36

Wos: Web of Science; SCP: Scopus; IEX: IEEE Explore; EV: Engineering Village.

. Books, review papers, and web page articles describing DIP technology that were published before 2020 were referred to for the third section, Overview of Digital Image Processing.

3. Overview of Digital Image Processing

DIP technology has advanced since Russell Kirsch developed the first digital image in 1957, which contained a mere 176 pixels. A pixel, or picture element, is a point sample of an image in which each pixel has a spatial coordinate and a value [22,23,24].

3.1. Digital Image Processing System

During DIP, interested image features are enhanced, and/or useful information is extracted [23]. DIP relies on a system comprising computer hardware and specialized software, with various factors influencing its efficiency. These factors include computer CPU number and speed, RAM capacity, mass storage capacity, monitor display spatial and color resolution, operating system, software compilers, image processing system application software, etc. [25]. Significant advancements in computer technology over the past millennium have resulted in substantial improvements in all these aspects.

There are more than 75 open-source and commercial image-processing software options available, many of which feature a “click and run” capability that offers limited designated options [26]. In contrast, software with a programming module allows for the customization and automation of image-processing tasks. This capability enables the sequential application of customized operations on batches of images, generating meaningful outputs [26]. Such software is frequently used in agricultural image processing applications.

DIP involves a series of sequential stages, and the specific pathway can vary slightly based on the intended objective of the image processing. Among these steps, key stages include image acquisition, initial image pre-processing, image enhancement, color image manipulation, image segmentation, image compression, data representation, information extraction, recognition, and interpretation [25,26,27,28,29]. The necessary stages and their sequence are decided according to the objective of the image processing.

3.2. Image Acquisition and Preprocessing

To obtain a digital image, it is essential to utilize a physical apparatus that is responsive to a specific range within the electromagnetic energy spectrum [27]. This apparatus could be a passive sensor, an active sensor mounted on an aerial vehicle or static platform, or it could be a scanner that scans an existing picture [30]. A passive sensor, which is commonly used in agricultural applications, is an apparatus that records the reflected electromagnetic waves from the interested element. Aerial cameras, video cameras, gamma-ray spectrometers, imaging spectrometers, and thermal scanners are some of the passive sensors that could be used in agricultural applications [30]. Active sensors can emit a characteristic wavelength, which measures the radiant flux coming back to the sensor. Both passive and active sensors perform the conversion of light data into a corresponding electrical signal, which is subsequently transformed into a digital signal or image by the digitization process [25,27].

Image pre-processing is carried out to improve the quality of the image. This step prepares the image for further processing and analysis. During this pre-processing stage, it is imperative to perform image denoising, along with the correction of radiometric and geometric errors, which are acknowledged as pivotal steps within this procedure.

3.2.1. Radiometric and Geometric Error Correction

Radiometric errors refer to incorrect brightness values assigned to pixels, which can occur due to imperfections in the sensor, atmospheric conditions like absorption and scattering, changes in scanning angles, fluctuations in the scene’s lighting, etc. [29,31]. Geometric errors occur whenspatial relationships and geometric properties within an image do not align with each other. Geometric errors are specially observed in remote sensing data, because of the earth’s rotation, the curvature characteristics of the earth, the movement of the earth and satellite/ sensor in different planes, and the velocity of the sensor [29,30]. Both of these errors could be systematic or non-systematic, and it is essential to understand the source of errors well to apply the appropriate correction methods effectively.

3.2.2. Noise Reduction

Noise refers to unwanted or random variations in the brightness or color of pixels. Such interference can originate during the image acquisition or transmission process. Inadequate light levels and other environmental conditions that could affect the imaging sensor, interferences in the transmission channel, and dust particles on the scanner are the main sources of digital image noise [32]. Digital image noises could be classified into different types: impulse noise (salt and pepper noise), amplifier noise (Gaussian noise), shot noise, quantization noise (uniform noise), film grain, on-isotropic noise, multiplicative noise (speckle noise), and periodic noise [32]. Depending on the type of noise, denoising techniques vary.

The goal of noise reduction is to reduce unwanted distortions in natural images while preserving essential image details and enhancing the signal-to-noise ratio [33]. Denoising is a challenging step as noises are diverse and complex and may not be uniform throughout the image. The denoising process should not alter the edges and texture of the image while ensuring no artifacts are introduced to the image. Another challenge is that the quality of the denoised image could be subjective, which could affect the final objective.

3.3. Image Enhancement

Image enhancement intensifies the interested image features by providing a better contrast between the target features and the background. For example, subtle differences in leaf color could be enhanced for clear and contrasty visualization for agricultural N management. Typically, applying one or two enhancement processes to the image suffices to meet the analyst’s needs, although the quality of the resulting image is subjective [30]. Several image enhancement techniques exist, like contrast enhancement, linear enhancement, non-linear enhancement, band rationing, and spatial filtering [26,28]. The choice of technique depends on the specific objectives of the image-processing task.

3.4. Thematic Information Extraction

Thematic information extraction in DIP is significant as it facilitates the retrieval of actionable insights from images essential for decision-making in various contexts. In agriculture, applications often rely on extracting critical data like color, patterns, and texture from digital images. This extraction typically involves the use of algorithms and mathematical models.

4. Digital Image Processing Trends in Crop Nitrogen Management

Numerous studies have explored the use of computer vision for the early detection of N deficiencies in crops, employing innovative and diverse approaches. Figure 2 illustrates the number of articles appeared in Web of Science, Scopus, IEEE Xplore, and Engineering Village over the past 5 years (2020–2024 October) related to the application of DIP in agricultural N management. After screening and removing duplicates across the databases, 95 publications were retained. . There is an increasing trend in the total number of studies, indicating promising progress in advancing this field of research.

A co-occurrence term map was developed to visualize frequently appearing terms related to DIP in agricultural N management (Figure 3) to visualize closely related research trends. This term map was developed using VOSviewer version 1.6.20 software. ‘Nitrogen’, ‘agriculture’, and ‘digital image processing’ keywords were searched in the Web of Science database for the past 5 years, and a bibliographic text file was created. The map was constructed based on the most frequently co-occurring terms found in the titles and abstracts of the retrieved bibliographic information. In this map, larger and central nodes indicate terms that appear more frequently, thicker links indicate stronger co-occurrence relationships and warm colors indicate regions with a high concentration of terms.

The map indicates that the use of digital images is strongly and closely associated with N management. Winter wheat is the predominant crop featured, suggesting it has been the subject of numerous studies on N management. According to the map, smartphones are the most commonly used image-capturing tool, while UAVs (Unmanned Aerial Vehicles) have also been used comparatively in lesser amounts. The figure highlights that texture features in RGB images have been frequently studied, and the NDVI (Normalized Difference Vegetation Index) is the most commonly studied vegetation index in these studies.

The global agricultural landscape is characterized by significant diversity in terms of environments, climates, and farming practices, highlighting the need for localized N managing approaches with DIP. Understanding different DIP trends followed by different agricultural regions is important for identifying research gaps and planning future research. Figure 4 indicates how DIP-related research on agricultural N management has been distributed across the globe. Most research on applying DIP in agricultural N management is from Asia, representing both tropical and subtropical environmental conditions. Studies conducted in temperate regions are at a comparatively lower level.

Employing ML models has demonstrated more robust and accurate results in most of the recent studies [34,35,36]. In particular, DL, which is a subset of ML, has demonstrated even more potential due to its ability to handle much larger and unstructured datasets, learn from past mistakes, require less human intervention for algorithm correction, recognize complex and non-linear correlations, and uncover hidden patterns without needing explicit programming [34,37,38,39,40,41,42].

The most commonly used ML algorithms are Random Forest [8,34,43,44,45,46,47], Support Vector Machine [44,48,49,50,51], Gradient Boosting Machine [44,52,53,54], K-Nearest Neighbor [52,54,55,56], K-Means Clustering [16,49], Naïve Bayes [48], and decision trees [17,51,57], mainly for classification and regression. DL algorithms leverage neural networks with multiple layers and the most commonly used are Convolutional Neural Networks (CNNs) [57,58,59,60,61,62,63], Artificial Neural Networks (ANNs) [64,65,66,67], and Recurrent Neural Networks (RNNs) [65].

Pre-trained neural networks were preferred in many studies, as they work well with inadequate datasets for DL, there is no need to train the model from scratch, and they have a proven architecture [68,69]. These are known as learning transfer models, and Dense Convolutional Networks (DenseNet) [59,70,71,72,73], Alex Convolutional Network (AlexNet) [71,74,75], Residual Neural Network (ResNet) [69,70,71,76,77], Inception-ResNet, Efficient Convolutional Network (EfficientNet) [69,78], YOLO (You Only Look Once) [17], and Mobile Convolutional Network (MobileNet) [59,72,73,79] are commonly used [66].

Automated Machine Learning (AML) systems, which have emerged in the recent past, address the complexity of applying ML in image feature extraction, selection, and classification [76]. AML can autonomously select the best ML pipeline for a given task and dataset, which saves time for ML experts [76]. As mentioned by Radocaj et al. [77], numerous researchers have placed significant emphasis on developing fully automated remote sensing techniques that enable the complete automation of satellite image processing. This approach allows end-users, even those without specialized expertise in remote sensing, to effortlessly generate final outputs like maps automatically and nearly in real-time [77].

Figure 5 shows how the use of different algorithms (statistical, conventional ML, and DL algorithms) has changed over time.

Table 2. Summary of articles that were published in 2024.

Crop	Image	Algorithm/s	Best Performance	Reference
Winter wheat	RGB	SVM, Classification and Regression Trees, ANN, KNN, RF	R2 = 0.62 (SVM)	[54]
Paddy	MS and RGB	Genetic CNN	Accuracy = 99%	[79]
Sugarcane	MS	CNN	Accuracy = 0.47 (N stress classifier)	[55]
Lettuce	RGB	CNN, RAN, and VGG-16	Accuracy = 96%	[80]
Paddy	RGB	Pre-trained CNN models (InceptionV3, VGG16, VGG19, ResNet50, and ResNet152), SVM	Accuracy = 0.9952 (ResNET152+SVM)	[81]
Basil	RGB	Curriculum by Smoothing technique with CNN (CS) and ResNet50V2	Accuracy = 91.17% (CS)	[5]
Maize	RGB	ResNet50, EfficientNetB0, InceptionV3,DenseNet121, Vision Transformer model	Accuracy = 97% (EfficientNetB0)	[78]
Pinapple	MS	LR, SVM, Decision tree, RF, XGBoost, AdaBoost, Lasso regressor, Ridge Regressor, and MLP regressor	R2 = 68% (XGBoost)	[50]
Broccoli	HS and CIELAB	Pycaret, PLS-DA	Accuracy = 1.00 (pycaret)	[76]
Cucumber seedling	RGB	Mask R-CNN, ReNet50, FPN, RPN, GLCM	Precision = 93% (Mask R-CNN)	[82]
Lettuce	RGBD and RGB	SVM, NCA, DTC, and linear model with SGD	Accuracy = 90.87% (DTC)	[17]
Ash gourd, bitter gourd, and snake gourd	RGB	EfficientNetB0, MobileNetV2, and DenseNet121	Accuracy = 90.62% (Nutrispace; a new colour space)	[70]
Paddy	RGB	ResNet50	Accuracy = 97.22%	[67]
Paddy	RGB	Xception, Vision transformer, and MLP mixer	Accuracy = 95.14% (Xception)	[39]
Apple	RGBD	K-means clustering, gradient boost	R2 = 0.72 (gradient boost)	[16]
Potato	MS	RF, XGBoost, and stacking ensembled (KNN, PLSR, SVR, RF, GPR)	R2 = 0.74 (stacking ensembled model)	[51]

SVM: Support Vector Machine, ANN: Artificial Neural Network, KNN: K-nearest Neighbor, RF: Random Forest, CNN: Convolutional Neural Network, VGG: Visual Geometry Group, ResNet: Residual Network, XGBoost: Extreme Gradient Boosting, LR: Linear Regression, AdaBoost: Adaptive Boost, MLP: Multilayer Perceptron, PLS-DA: Partial Least Squares Discriminant Analysis, R-CNN: region based CNN, FPN: Feature Pyramid Network; RPN: Regional Proposal Network, GLCM: Grey Level Co-occurrence Metrix, SGD: Stochastic Gradient Descent, NCA: Neighborhood Components Analysis, DTC: Decision Tree Classifier; RAN: Residual Attention Network, MS: Multispectral, HS: Hyperspectral.

summarizes the articles that were published in the year 2024 (until October), emphasizing the main algorithms used and their best performance.

5. Challenges and Future Directions

Digital image processing holds significant potential to address the increasing global food demand by reducing agricultural production costs, enhancing automation, and minimizing environmental impacts. Despite these benefits, its adoption for crop N prediction remains limited due to various challenges.

5.1. Image Acquisition

The image acquisition stage presents several challenges. The complexity of image data retrieval can vary based on the number of spectral bands a sensor can capture, the altitude and angle of image acquisition, and camera settings. RGB cameras provide less information compared to MS (3–10 bands) and HS sensors (hundreds of narrow bands) [12]. The altitude and method of photography—whether ground-based, UAV-assisted, aircraft-assisted, or satellite—influence the image quality, with ground-based photos providing more spatial resolution but less coverage compared to high-altitude images. Camera settings, including exposure time and aperture, also have an impact on the final accuracy of the prediction model [83]. Most importantly, having an adequate amount of images for training and validating prediction models is vital; small datasets typically result in less reliable predictions [84].

Environmental conditions like variable light quality and intensity, illumination, canopy shading, and leaf surface reflectance introduce noise and reduce image homogeneity across the image-capturing period [83,85]. Background elements like weeds, soil, and twigs often appear in field images, complicating image processing and introducing errors in the final model [84,86].

To address these challenges, researchers have followed different methods. Avoiding sunny conditions that cause shadows and streaks, as well as avoiding areas with crop residues, weeds, and wet and tilled soil during image capturing [84], improves image quality for DIP. In several studies, digital images have been captured followed by leaf sampling [52,71,87,88], or have used a background panel to avoid background components that are effective in maintaining image homogeneity throughout imaging. These methods can be tedious, especially for large-scale field applications.

To reduce illumination errors, Banerjee et al. developed a novel illumination adjustment protocol for HS imaging by using a cabinet lined with a reflective material which produces a closely uniform illumination in images [4]. During manual image capture, practices such as holding the camera parallel to the ground and maintaining uniform height, potentially with a tripod, can prevent image distortion [76,83,89]. The choice of image-acquiring tool and its altitude depends on the crop type, field area, objective, and technical and financial feasibility.

5.2. Digital Image Processing

The primary challenge during the image processing stage lies in having a thorough understanding of the procedures and technical skills involved [64,65]. As mentioned by Malounas et al., integrating multiple components, namely feature extraction, feature selection, and classification, is mandatory for building an effective ML prediction model [76]. Depending on the objective, image-acquiring conditions, type of image, image-acquiring tool, etc., image-processing steps need to be customized [49,54]. Rather than relying only on image features to predict N status, incorporating multi-sensor/multi-source data to develop a decision support system could enhance the prediction accuracy [16,34,40,50,86,90,91,92]. Integrating climate and agronomy data into the model has shown improved prediction accuracies [93,94]. NITREOS (Nitrogen Fertilization, Irrigation, and Crop Growth Monitoring using Earth Observation Systems) is such farm management information system that recommends N fertilizer rates using both satellite images and agro-meteorological data [94].

5.3. Socio-Economic Barriers

Socio-economic barriers could hinder the adoption of DIP technology in agriculture. One major issue is farmers’ lack of knowledge and qualified personnel to implement DIP in their fields effectively [95]. DIP demands skills in mathematics, image analysis, coding, and expertise in selecting model architectures [76]. This knowledge gap is compounded by inadequate financial resources and infrastructure, which further deter farmers from embracing this technology [48,51,74]. To address the lack of knowledge, developing smartphone-based crop N prediction applications or web applications that are easily accessible to farmers and farm advisors is suggested [69,83,95,96,97,98,99]. Such applications can provide user-friendly interfaces and guidance, making DIP more approachable. NITREOS [94], CropSAT [100], and N-Time (N-Time Fertigation Management System) [101] are software developed as precise N fertilization decision support systems. In addition to software development, Sandra et al. followed a different approach by developing a hand-held N fertilizer-predicting device using a TCS3200 color sensor and Arduino UNO data processor, which resulted in a 90% prediction success rate [102].

6. Conclusions

Digital image processing holds great potential for enhancing PNM in agriculture. A total of 95 publications were retained after the literature search and screening. Much research has been conducted covering diverse agricultural regions, mainly the tropics and subtropics of Asia. The use of machine learning (ML) and deep learning (DL) algorithms has notably increased, and the most used algorithms were Random Forest, Support Vector Machine, Extreme Gradient Boost, and Convolutional Neural Networks, showing strong performances in N prediction tasks. Pre-trained models and Automated Machine Learning (AML) systems are gaining traction due to their ability to handle large datasets and reduce the need for extensive human intervention. Challenges remain at various stages of DIP application, such as image acquisition, processing, and socio-economic factors. Environmental conditions, image quality, and the technical expertise required for image analysis pose significant barriers to widespread adoption. The integration of multi-sensor data into models and decision support systems has been proven to boost accuracy. Socio-economic constraints, including a lack of skilled personnel and financial resources, hinder the implementation of DIP in the field. Future research should aim to develop more user-friendly and cost-effective solutions to make DIP technology more accessible to farmers.