Attention-Driven and Hierarchical Feature Fusion Network for Crop and Weed Segmentation with Fractal Dimension Estimation

Abstract

In precision agriculture, semantic segmentation enhances the crop yield by enabling precise disease monitoring, targeted herbicide application, and accurate crop–weed differentiation. This enhances yield; reduces the overuse of herbicides, water, and fertilizers; lowers labor costs; and promotes sustainable farming. Deep-learning-based methods are particularly effective for crop and weed segmentation, and achieve potential results. Typically, segmentation is performed using homogeneous data (the same dataset is used for training and testing). However, previous studies, such as crop and weed segmentation in a heterogeneous data environment, using heterogeneous data (i.e., different datasets for training and testing) remain inaccurate. The proposed framework uses patch-based augmented limited training data within a heterogeneous environment to resolve the problems of degraded accuracy and the use of extensive data for training. We propose an attention-driven and hierarchical feature fusion network (AHFF-Net) comprising a flow-constrained convolutional block, hierarchical multi-stage fusion block, and attention-driven feature enhancement block. These blocks independently extract diverse fine-grained features and enhance the learning capabilities of the network. AHFF-Net is also combined with an open-source large language model (LLM)-based pesticide recommendation system made by large language model Meta AI (LLaMA). Additionally, a fractal dimension estimation method is incorporated into the system that provides valuable insights into the spatial distribution characteristics of crops and weeds. We conducted experiments using three publicly available datasets: BoniRob, Crop/Weed Field Image Dataset (CWFID), and Sunflower. For each experiment, we trained on one dataset and tested on another by reversing the process of the second experiment. The highest mean intersection of union (mIOU) of 65.3% and F1 score of 78.7% were achieved when training on the BoniRob dataset and testing on CWFID. This demonstrated that our method outperforms other state-of-the-art approaches.

1. Introduction

The global increase in food demand is driven by the increase in the global population, which is projected to exceed nine billion by 2050 [1]. This growth necessitates efforts to enhance the food supply by increasing crop productivity. In recent years, deep learning and computer vision techniques have advanced rapidly, playing key roles in applications such as medical imaging, image enhancement, building monitoring, biomedical engineering, and precision agriculture [2,3,4,5,6,7]. In parallel, fractal-related studies have also gained attention in various deep-learning-based computer vision tasks. Although many studies employ similar fractal dimension (FD) estimation techniques, their implementation and interpretation vary significantly across domains such as medical image analysis and agricultural applications [8,9,10,11]. Currently, field phenotyping systems for precision agriculture constitute a prominent method to address these challenges and increase crop yields [12]. Field phenotyping uses many advanced tools such as wearable sensors [13], ground robots [14], and computer vision technologies such as semantic segmentation [15]. Semantic-segmentation-based plant phenotyping and semantic segmentation of crops and weeds have been performed in previous studies [16,17]. Earlier, conventional methods like extensive manpower, expensive labor costs, overuse of fertilizers and herbicides, and conventional spraying methods were deemed ineffective in satisfying the growing demand for crops. Some prior deep learning methods like CG-Net and Unet++ also showed low performance. To overcome these problems, it is important to decrease manpower and utilize advanced methods of spraying herbicides [18].

In precision agriculture, robots and wearable sensor devices are used to spray herbicides on targeted weeds, monitor farms, and collect data to mitigate the factors that affect crop yield. Such farming problems are mainly resolved by accurately detecting crops and weeds [19]. Two methods are widely used for accurately detecting crops and weeds. The first is box-based detection [20,21], which has the drawback of overlooking certain regions of crops and weeds. The other is pixel-based detection and is commonly known as semantic segmentation [22,23]. It precisely detects regions of crop and weed objects at the pixel level. Therefore, it is necessary to correctly differentiate the segments of crops and weeds at the pixel level because these crop and weed objects have irregular shapes. Figure 1 compares the shapes of “sugar beets” (the BoniRob dataset) [24], the Crop/Weed Field Image Dataset (CWFID) [25], and the Sunflower dataset [26].

Many machine learning (ML)- and convolutional neural network (CNN)-based methods have been used for semantic segmentation. These typically require large amounts of training data for efficient training. When considering a large training dataset, a significant number of annotations are necessary. These should be generated by agricultural experts. This process requires considerable time and effort. Owing to these constraints, sufficient training data may not be available in certain cases. This can result in a decrease in semantic segmentation accuracy. To address this issue, researchers have used small training datasets with potential outcomes. In ref. [27,28], classification and segmentation problems in the field of healthcare were resolved. This yielded satisfactory results with limited training data. In the agricultural sector, previous research [22] has employed a similar concept of small training data. In general, better testing results can be achieved with homogeneous data (using the same dataset for training and testing). However, the performance tends to decrease significantly in heterogeneous environments (using different datasets for training and testing). Furthermore, they demonstrated that the concept of small training data effectively contributed to achieving good testing performance even in heterogeneous environments for crops and weeds. In imagery data, factors like complex backgrounds, lighting, and crop–weed density can affect the method’s performance, leading to poor feature extraction and lower accuracy [29]. These challenges emphasize the need to develop a model that can effectively segregate crops and weeds even in the presence of complex backgrounds and heterogeneous environments. We propose a novel approach, the attention-driven hierarchical feature fusion network (AHFF-Net), to segregate crops and weeds using the concept of patch-based augmented small training data. This method effectively detected regions of crops and weeds in complex backgrounds and heterogeneous environments.

1.1. Considering Homogenous Dataset Environments

The methods in this category typically exhibit a higher performance because the data distribution for training and testing originates from the same dataset. Numerous studies have been conducted across various domains using homogeneous datasets, although not specifically for the agricultural sector. In this subsection, we classify crop and weed semantic segmentation into three categories: ML-based methods, single-stage CNN-based methods, and multi-stage CNN-based methods.

1.1.1. ML-Based Methods

The conventional ML-based methods were popular prior to the advancements and widespread use of deep learning for crop and weed segmentation. In ref. [30], principal component analysis (PCA) and a random forest (RF) algorithm were used to optimize feature selection for effective weed detection and maize crop differentiation. A vegetation detection system using Markov random fields (MRFs) for smoothing and random forest classification (RFC) was proposed in [31], enabling precise weed identification and robotic spraying. Another study [32] detected four common weed species in sugar beet fields using shape features evaluated with an artificial neural network (ANN) and a support vector machine (SVM). An automated method [33] used support vector data description (SVDD) and color indices to segregate maize crops and weeds, applying excess green [34] and Otsu thresholding for binary mask generation. Morphological operations and feature extraction methods like local binary patterns and Gabor filters [35] further improved accuracy through classifier fusion. Zhao et al. [36] also developed an accurate SVM-based system for cabbage–weed identification with real-time spraying, achieving efficient pesticide usage.

1.1.2. Single-Stage CNN-Based Methods

The single-stage CNN method proposed in [37] utilizes data augmentation by extracting patches, normalizing these, and feeding these into a CNN. Furthermore, using the segmentation results from a ResNet-50-based CNN, the study [38] proposed a blob extraction and region of interest (ROI) technique. These ROIs were then classified as crops or weeds using a VGG-16 [39] two-class classifier and achieved better results. A modified U-Net with diverse augmentation strategies enhanced pixel-level classification (PLC) in [40]. This approach effectively addressed the challenges of limited training data and multiple data combinations for augmentation methods to achieve a higher performance. Another study introduced DeepVeg [41], which segmented healthy and damaged crops and weeds using semantic masks, effectively learning from class imbalance and overlap. In ref. [42], the authors modified Enet [43] and SegNet [44] to develop a deep learning model for crop and weed segmentation with high accuracy. They replaced SegNet convolutional layers (CLs) with residual blocks, which improved both precision and efficiency by reducing inference time. A U-Net with ResNet-50 and conditional random field (CRF) refined segmentation from multispectral images in [45], achieving robustness but with high computational cost. It combined RGB and filtered near-infrared (NIR) inputs and applied the normalized difference vegetation index (NDVI) for normalization. In the study [46], U-Net [47] and Unet++ [48] were employed for early-stage weed detection; however, challenges such as missed detection of small objects, low contrast, and high computational cost persisted. To address these issues, the multi-stage CNN-based methods have been proposed.

1.1.3. Multi-Stage CNN-Based Methods

A multi-stage MTS-CNN method [49] uses stage-wise training with two models based on the U-Net architecture [47] and its modified version [50], resulting in improved segmentation performance. The cascaded encoder–decoder network (CED-Net) [51], another U-Net-based multi-stage model, separates weed and crop segmentation into different stages to improve performance. A two-stage model in [52] uses a lightweight U-Net for object-background separation and a VGG-16 backbone for detailed multiclass segmentation. Although these methods exhibit high accuracies in homogeneous dataset environments, these experience performance degradation in heterogeneous dataset environments.

1.2. Considering Heterogeneous Dataset Environments

In heterogeneous dataset environments, crop and weed segmentation is grouped further into two types: CNN with image-based augmentation (IMBA) methods and CNN with patch-based augmentation methods. To our knowledge, only one study has utilized a CNN with the IMBA method to address crop and weed semantic segmentation using heterogeneous data in the agricultural sector.

1.2.1. CNN with Image-Based Augmentation Methods

In ref. [22], researchers proposed a framework for heterogeneous data environments by utilizing IMBA methods for CNNs. Initially, Dataset-1 was used, with 70% of its data allocated for training using a conventional semantic segmentation model. After the model was trained and saved, Dataset-2 was pre-processed using the Reinhard transformation to adjust its visual properties to resemble those of Dataset-1. The transformed Dataset-2 was then split into two equal parts: 50% for the small training data and 50% for testing. They selected one training dataset from a small training data split, applied IMBA, and fine-tuned the model using augmented training data. Finally, semantic segmentation of crops and weeds was performed using the testing split of Dataset-2. This study demonstrated the applicability of the framework to heterogeneous datasets.

1.2.2. CNN with Patch-Based Augmentation Methods

Unlike ref. [22], our study employs patch-based augmentation and proposes AHFF-Net. It is a deep-learning-based modified version of U-Net designed to improve the semantic segmentation results for crops and weeds in heterogeneous environments. The strengths and weaknesses of previous studies and those of the proposed model are outlined in Table 1. The key contributions of this study are as follows:

• This study proposes AHFF-Net, which incorporates patch-based augmentation to improve the accuracy compared to previous methods. This method effectively addresses the challenges of a heterogeneous dataset environment for semantic segmentation of crops and weeds.
• AHFF-Net includes a progressive encoder-stage refinement block (PERB) as part of the encoder. The PERB enhances feature extraction by capturing more sophisticated details such as the edges and textures of crops and weeds through deeper convolutions while preserving the feature map size. Additionally, the PERB improves low-level feature retention and helps prevent the loss of critical region-level information on crops and weeds. This enables the progressive refinement of spatial features before downsampling at each encoder stage. This is particularly favorable in heterogeneous field environments, where the high intraclass variability (e.g., varying crop textures) and low interclass separability (e.g., weeds replicating crop morphology) challenge conventional encoders. By enriching the representation before resolution reduction, PERB preserves the fine-grained cues essential for distinguishing visually similar classes. The symmetrical design also facilitates a smoother gradient flow and stabilizes the training. Thereby, it improves the generalization of the model to unseen or complex conditions that agricultural segmentation tasks generally involve.
• AHFF-Net employs a hierarchical multi-stage fusion block (HMFB) to capture and combine diverse features across multiple semantic levels. Each stage within the HMFB independently learns unique representations by focusing on fine-grained spatial details and high-level contextual information. These features are combined hierarchically. This results in a rich multiscale representation that strengthens the model’s capability to handle variations in object size, shape, and appearance during the semantic segmentation of crops and weeds in heterogeneous environments. Moreover, the HMFB includes residual connections that help preserve low-level spatial details that are generally lost in deeper layers. These low-level details are important for accurately identifying the small or thin structures of crops and weeds, particularly when their visual patterns differ across heterogeneous environments.
• The proposed attention-driven feature enhancement block (AFEB) in the decoder of AHFF-Net uses an attention mechanism to focus on critical regions of crops and weeds. It thereby enhances the segmentation performance. This functionality is particularly advantageous in heterogeneous environments because the attention mechanism adaptively highlights the most informative spatial features while suppressing irrelevant or noisy background information. By selectively emphasizing discriminative features, the AFEB contributes to more accurate semantic segmentation across field conditions. Furthermore, this framework integrates an FD estimation component to extract critical information on the spatial distribution patterns of crops and weeds. AHFF-Net is also combined with open-source large language model (LLM)-based pesticide recommendation systems by large language model Meta AI (LLaMA). To support transparency and reproducibility, the implementation of AHFF-Net has been made publicly available on GitHub (https://github.com/) [53].

Table 1. Summarized comparisons of the proposed and related works on crop and weed segmentation.

Category		Method	Strengths	Limitations
Considering homogeneous dataset	ML-based	RF algorithm with PCA was used to distinguish maize crop from weeds [30]	Effective use of hyperspectral data by leveraging rich spectral information, particularly red-edge NIR bands, to improve weed–crop discrimination	This method was validated only under controlled conditions, not in real field scenarios
		This method used both RFC- and MRF-based vegetation detection [31]	It detected both local and object base features, and was also evaluated on real farm of sugar beet fields	For image acquisition, the system requires a specific setup (e.g., a four-channel RGB+NIR camera and artificial halogen lighting), which limits its flexibility in varying field environments
		ANN and SVM were evaluated based on shape features [32]	Focusing on shapes and patterns, it precisely detected the weeds and crops	In a sugar beet field, it was evaluated on only four types of common weeds
		Background was separated from crops and weeds, and further classified using SVDD [33]	Reduced computation time by using color index features instead of shape-based features	Threshold values may change and become inaccurate if the colors of crops and weeds vary from green
		This approach was based on multiple classifier system (MCS) [35]	It showed better performance with selection-based MCS, which utilized multiple classifiers instead of a single classifier	The model became too heavy due to the combination of multiple classifiers
		SVM with radial basis function (RBF) kernel [36]	Real-time applicability achieved through integrated image acquisition, decision-making, and nozzle control for in-field operation.	Tiny cabbage plants were often missed by the targeted pesticide spraying mechanism, reducing overall detection accuracy
	Single-stage CNN-based	CNN model with ROI and two-class classifiers [37]	ROI-based accurate segmentation method for crops and weeds	Light variations across different datasets can reduce overall accuracy
		A modified U-Net with data augmentations [40]	This study compared various input sizes for models and augmentation techniques to identify the optimal learning methods for PLC	Crops and weeds which are too small may not be detected in the augmented patches
		DeepVeg segmentation focused on damaged crops alongside healthy crops and weeds [41]	This lightweight model effectively addresses complex backgrounds and imbalanced classes	Image segmentation is challenging due to unclear boundaries, poor lighting, low contrast, and limited boundary information, resulting in errors and imprecise region definitions
		A modified Enet- and SegNet-based model taking 14 input channels [42]	14 transformed images were concatenated as input to improve segmentation performance and reduce environmental impact	Generating 14 RGB-based channels increased computational costs and risked overfitting due to isolated features
		U-Net with a ResNet-50 backbone and CRF as post-processing [45]	The U-Net with a ResNet-50 backbone and CRF method exhibited enhanced performance across diverse environmental images	Processing took longer because the relationships between all pixels needed to be analyzed
		This method utilizes the conventional U-Net and U-Net++ architectures with resize [46]	Weeds were detected during their early growth stages to enable timely and effective intervention	The study utilized a small dataset, limiting its representation of the diversity and complexity of real-world agricultural environments
	Multi-stage CNN-based	A multi-stage MTS-CNN method for crop and weed segmentation [49]	It reduced the disparity between crop and weed segmentation accuracy while enhancing overall segmentation performance	The second stage depends on the first stage; poor first-stage training hampers second-stage performance
		Modified U-Net based on conventional U-Net [50]	To address the challenges of data labeling and limited data availability, augmentation is employed, which enables accurate segmentation, even in complex backgrounds	The algorithm is limited to green bristlegrass, restricting its general applicability to other weeds
		A novel four-stage model, called CED-Net [51], is proposed, with each stage based on a modified U-Net encoder–decoder structure	A lightweight multi-stage model was developed, where training for each stage focused on a specific weed or crop separately	An error in any of the four stages impacted the overall performance
		Two stages encode a decoder-based semantic segmentation model [52]	It focused on objects rather than the background and delivered good results	Including the background yields varying results across different datasets
Considering heterogeneous dataset	CNN with image-based augmentation	A framework for crop and weed semantic segmentation in heterogeneous environments using conventional CNN methods [22]	Limited training data was used for segmentation and achieved good results on heterogeneous datasets	Overlapping of crop and weed plants causes misclassification and lowers the performance
	CNN with patch-based augmentation	AHFF-Net (proposed method)	It improves crop and weed segmentation accuracy across diverse shapes and heterogeneous data environments using small training data	Segregating very small crops and weeds poses a significant challenge in heterogeneous data environments

Table 1. Summarized comparisons of the proposed and related works on crop and weed segmentation.

Category		Method	Strengths	Limitations
Considering homogeneous dataset	ML-based	RF algorithm with PCA was used to distinguish maize crop from weeds [30]	Effective use of hyperspectral data by leveraging rich spectral information, particularly red-edge NIR bands, to improve weed–crop discrimination	This method was validated only under controlled conditions, not in real field scenarios
		This method used both RFC- and MRF-based vegetation detection [31]	It detected both local and object base features, and was also evaluated on real farm of sugar beet fields	For image acquisition, the system requires a specific setup (e.g., a four-channel RGB+NIR camera and artificial halogen lighting), which limits its flexibility in varying field environments
		ANN and SVM were evaluated based on shape features [32]	Focusing on shapes and patterns, it precisely detected the weeds and crops	In a sugar beet field, it was evaluated on only four types of common weeds
		Background was separated from crops and weeds, and further classified using SVDD [33]	Reduced computation time by using color index features instead of shape-based features	Threshold values may change and become inaccurate if the colors of crops and weeds vary from green
		This approach was based on multiple classifier system (MCS) [35]	It showed better performance with selection-based MCS, which utilized multiple classifiers instead of a single classifier	The model became too heavy due to the combination of multiple classifiers
		SVM with radial basis function (RBF) kernel [36]	Real-time applicability achieved through integrated image acquisition, decision-making, and nozzle control for in-field operation.	Tiny cabbage plants were often missed by the targeted pesticide spraying mechanism, reducing overall detection accuracy
	Single-stage CNN-based	CNN model with ROI and two-class classifiers [37]	ROI-based accurate segmentation method for crops and weeds	Light variations across different datasets can reduce overall accuracy
		A modified U-Net with data augmentations [40]	This study compared various input sizes for models and augmentation techniques to identify the optimal learning methods for PLC	Crops and weeds which are too small may not be detected in the augmented patches
		DeepVeg segmentation focused on damaged crops alongside healthy crops and weeds [41]	This lightweight model effectively addresses complex backgrounds and imbalanced classes	Image segmentation is challenging due to unclear boundaries, poor lighting, low contrast, and limited boundary information, resulting in errors and imprecise region definitions
		A modified Enet- and SegNet-based model taking 14 input channels [42]	14 transformed images were concatenated as input to improve segmentation performance and reduce environmental impact	Generating 14 RGB-based channels increased computational costs and risked overfitting due to isolated features
		U-Net with a ResNet-50 backbone and CRF as post-processing [45]	The U-Net with a ResNet-50 backbone and CRF method exhibited enhanced performance across diverse environmental images	Processing took longer because the relationships between all pixels needed to be analyzed
		This method utilizes the conventional U-Net and U-Net++ architectures with resize [46]	Weeds were detected during their early growth stages to enable timely and effective intervention	The study utilized a small dataset, limiting its representation of the diversity and complexity of real-world agricultural environments
	Multi-stage CNN-based	A multi-stage MTS-CNN method for crop and weed segmentation [49]	It reduced the disparity between crop and weed segmentation accuracy while enhancing overall segmentation performance	The second stage depends on the first stage; poor first-stage training hampers second-stage performance
		Modified U-Net based on conventional U-Net [50]	To address the challenges of data labeling and limited data availability, augmentation is employed, which enables accurate segmentation, even in complex backgrounds	The algorithm is limited to green bristlegrass, restricting its general applicability to other weeds
		A novel four-stage model, called CED-Net [51], is proposed, with each stage based on a modified U-Net encoder–decoder structure	A lightweight multi-stage model was developed, where training for each stage focused on a specific weed or crop separately	An error in any of the four stages impacted the overall performance
		Two stages encode a decoder-based semantic segmentation model [52]	It focused on objects rather than the background and delivered good results	Including the background yields varying results across different datasets
Considering heterogeneous dataset	CNN with image-based augmentation	A framework for crop and weed semantic segmentation in heterogeneous environments using conventional CNN methods [22]	Limited training data was used for segmentation and achieved good results on heterogeneous datasets	Overlapping of crop and weed plants causes misclassification and lowers the performance
	CNN with patch-based augmentation	AHFF-Net (proposed method)	It improves crop and weed segmentation accuracy across diverse shapes and heterogeneous data environments using small training data	Segregating very small crops and weeds poses a significant challenge in heterogeneous data environments

The remainder of this paper is organized as follows: Section 2 outlines the proposed method, Section 3 presents the experimental results, and Section 4 provides the discussion. Section 5 summarizes the study and presents future work directions.

2. Material and Methods

2.1. Experimental Setup

We conducted our experiments using three publicly available datasets: the BoniRob dataset [24], Crop/Weed Field Image Dataset (CWFID) [25], and the Sunflower dataset [26]. These are commonly used for crop and weed segmentation tasks because these provide images in conjunction with their corresponding annotated masks. The BoniRob dataset consists of 496 images captured by a camera on a farming robot, with a pixel resolution of 1296 × 966. The CWFID contains 60 images captured using a JAI AD-130GE camera (JAI A/S, Kushima, Japan) with a resolution of 1296 × 966 pixels. The Sunflower dataset includes 172 images collected during the emergency phase with a resolution of 1296 × 964 pixels. Sample images from these datasets, in conjunction with their corresponding masks, are shown in Figure 1. We started with a single image and used data augmentation to generate additional images. These two images were used to create 512 × 512 patches. We selected five patches based on high SSIM and used these for patch-based augmentation. This is explained in Section 2.2.2. To train the AHFF-Net, all the datasets were resized to an input image resolution of 512 × 512 by bilinear interpolation. The experiments were performed on a desktop system running Windows 10 and equipped with a Core i5-2320 Intel CPU (Intel Corporation, Santa Clara, CA, USA) with a 3.00 GHz processor, 16 GB of RAM, and a GeForce GTX 1070 graphical processing unit (GPU) with 8 GB of memory manufactured by NVIDIA corporation (Santa Clara, CA, USA). We developed the model using PyTorch version 1.13.0 and Python version 3.8.

2.2. Overview of the Proposed Method

This subsection provides an overview of the proposed method. It details the steps involved, including dataset pre-processing and patch-based augmentation of small training data for crop and weed segmentation, as shown in Figure 2. For the training, we employed the proposed AHFF-Net. It was initially trained on Dataset-1. Upon completing the training, we performed a pre-processing step (discussed in detail in the next subsection). This pre-processing stage involves adjusting the visual properties in Dataset-2 to align these with those of Dataset-1. This ensures consistency and enhances the generalizability of the model across datasets. After pre-processing Dataset-2, we augmented a training image by an in-plane rotation of 180°. Furthermore, these two images were selected and utilized to generate patches. Five patches with a high structural similarity index measure (SSIM) were selected and used to fine-tune the AHFF-Net on Dataset-2, which had been trained from scratch using Dataset-1. This patch-based augmentation approach enabled us to maximize the learning potential from the limited training data of crops and weeds. Thus, it effectively addressed the challenges posed by a heterogeneous dataset environment. Finally, the fine-tuned AHFF-Net was evaluated using the transformed testing data of crops and weeds to calculate the segmentation results. This systematic framework optimized the model performance through pre-processing, patch-based augmentation, and fine-tuning within heterogeneous datasets.

2.2.1. Pre-Processing

The variations in visual attributes such as contrast, intensity, and illumination across different datasets generally result in a lower performance, particularly in heterogeneous environments. This issue is addressed using color transfer techniques that adjust the colors of one image (input image) to match those of the other (target image). Various transformation methods have been developed to enhance the visual attributes of the data. In our study, we used the Reinhard transformation (RHT) [54]. It uses the Ɩαβ color space [55] by adjusting the standard deviation and mean of the image. In the Ɩαβ color space, “Ɩ” represents the brightness channel. It captures the intensity independent of color. The “α” channel captures the balance between yellow and blue hues. Meanwhile, the “β” channel reflects the relationship between red and green shades. The RHT uses the µ mapping function with π parameters to transform $dataA$ into pre-processed ${dataA}^{*}$, as shown in Equation (1). These π parameters include the standard deviation and mean values of input and target images. The pre-processing approach used in our study was inspired by a previously used technique explained in detail in [22]. This study [22] also shows that Reinhard gives better results using the same datasets. Reinhard may incur additional processing time, but according to this study, it provides better performance.

$${dataA}^{*}=\mathsf{\mu }(dataA, \pi )$$

(1)

Sample pre-processed images with input and target images are depicted in Figure 3.

2.2.2. Patch-Based Augmentation

However, data augmentation is a technique widely used in deep learning to enhance the diversity of training data, reduce overfitting, and improve model generalization. By introducing variations such as in-plane rotations, flips, and intensity variations, data augmentation enables models to better learn invariant and robust features. These data augmentation methods ultimately yield an improved performance. Building on this principle, patch-based augmentation provides additional advantages by breaking down images into smaller overlapping patches. This approach enables the models to focus on localized details and better capture spatial variations.

In our study, we used patch-based augmentation to extract the patches. We selected different step sizes for the height and width to extract patches that matched the 512 × 512 input size of the network. We evaded padding (which adds additional pixels) and cropping (which removes important information) because both the methods can adversely impact the performance. To address this issue, the step size was adjusted to ensure that 512 × 512 patches were extracted without altering or losing any part of the input image. Sample input images of the extracted patches are shown in Figure 4.

After extracting the patches, we calculated the SSIM [56] of each patch and selected those with high SSIM scores to ensure superior structural similarity. The mathematical expression for SSIM is given in Equation (2). Herein, ${\sigma }_{xy}$, ${\mu }_x$, ${\sigma }_x$, ${\mu }_y$, and ${\sigma }_y$ indicate the covariance of x and y, mean of x, standard deviation x, mean of y, and standard deviation y, respectively. The constants p and q are used to prevent the instability caused by division with near-zero denominators. This approach increases the amount of training data and preserves important features of the input image. This, in turn, yields an improved model accuracy. Leveraging patch-based augmentation, we effectively enhance the capability of the model to learn from high-quality localized patterns, thereby contributing to its overall performance.

$$SSIM(x,y)={\displaystyle \frac{({2\mu }_x{\mu }_y+p^2)({2\sigma }_{xy}+q^2)}{({\mu }_x^2+{\mu }_y^2+p^2)({\sigma }_x^2+{\sigma }_y^2+q^2)}}$$

(2)

2.3. AHFF-Net Architecture

The proposed AHFF-Net has encoder and decoder architecture similar to that of U shape. It is based on U-Net as the backbone with three new blocks: the PERB, HMFB, and AFEB. An architectural representation of the proposed AHFF-Net with a combination of blocks is shown in Figure 5.

We consider an input image of size 512 × 512 × 3 for the encoder part. We process it with a convolutional layer (CL) with a kernel size of 3 × 3, followed by a rectified linear unit (ReLU) activation function, batch normalization (BN), and the PERB. The output of the PERB follows the max pool layer (MPL) with a kernel size of 2 × 2 and stride of two. This is repeated four times in the encoder. The bottleneck starts with two CLs, followed by ReLU and BN. The output of these two CLs is passed through the HMFB, and the resultant is upsampled by a 2 × 2 up-convolutional layer (UCL) as the start of the decoder.

In the decoder, the upsampled feature map is concatenated with the feature map from the encoder, followed by two CLs, each with ReLU and BN. This is repeated until the final stage of the decoder. In the final stage of the decoder, the resultant of these two CLs is followed by the AFEB and a 1 × 1 CL as the final layer, with a size of 512 × 512 × 3. This is similar to the input of the encoder part. Furthermore, the mathematical representation of AHFF-Net, including the first encoder stage (${End}_1$, representing before and with the first MPL), first decoder stage (${Dec}_1$, representing the first UCL with subsequent layers), and bottleneck layer (Bottleneck), is shown in Equations (3)–(5):

$${End}_1= M_1\left(B_1\left({CN}_1\right)\right) where \left\{\begin{array}{c}{CN}_1=Conv\left(Y,X_1\right)\\ B_1=PERB\left({CN}_1\right)\\ M_1=Max pool\left(B_1\right)\end{array}\right.$$

(3)

$${Dec}_1={DC}_1\left(C_{concat}\left({DM}_{up1}\left({UP}_1\right),ES\right)\right) ,$$

(4)

$$where \left\{\begin{array}{c}{UP}_1=Up conv\left(D, Z_{up}\right)\\ {DM}_{up1}=Match\left(size\left(ES\right),size\left({UP}_1\right)\right)\\ C_{concat}=Concatenation\left({DM}_{up1},ES\right)\\ {DC}_1=Conv\left(Conv\left(C_{concat},N_{conv}\right)\right)\end{array}\right.$$

$$Bottleneck=BT\left(BC\right) where \left\{\begin{array}{c}BC=Conv\left(Conv\left(Y,F_{conv}\right)\right)\\ BT=HMFB\left(BC\right)\end{array}\right.$$

(5)

In ${End}_1$, Y denotes the input feature and $X_1$ denotes the convolutional operation weight tensor. After convolution, the PERB is applied. This is followed by ReLU and BN, which are applied after each convolution operation in all the encoder stages. In ${Dec}_1$, $D$ represents the tensor from the previous layer, and $Z_{up}$ in $Upconv\left(D,Z_{up}\right)$ represents the weight tensor used with the UCL operation. $ES$ in $Match\left(size\left(ES\right),size\left({UP}_1\right)\right)$ is the encoder stage skip connection with the size defined as $size\left(ES\right)$. The size of ${UP}_1$ is adjusted to match the size of $ES$ to ensure a smooth further operation. $C_{concat}$ represents the concatenation of ${DM}_{up1}$ and $ES$. It forms the final tensor from the previous layers. $N_{conv}$ denotes the weight tensor with convolution operations. Finally, after two convolutional operations in the decoder, the output is fed into ${DC}_1$. The remaining stages of the decoder operations persist using a similar approach. For the $Bottleneck$, $Y$ is the tensor from the previous layer, and $F_{conv}$ is the weight tensor used for the convolutional operations. After two convolutional operations, the output is fed into $BC$. Subsequently, $HMFB$ is applied to the output of the previous layer, and the final resultant tensor is fed into $BT$ to obtain the output (denoted as $Bottleneck$).

2.3.1. PERB

We placed the PERB before the MPL after each stage in the encoder part of the AHFF-Net. The PERB enhances feature extraction by capturing sophisticated details of crops and weeds, such as edges and textures. More detailed features can be extracted from crops and weeds using deeper convolutions while preserving the feature map size. The PERB consists of three $3 \times 3$ CLs. Here, the first and final CLs are followed by a BN, and the middle CL is followed by a BN with an activation function, ReLU. This is shown in Figure 6.

The mathematical expression for the PERB is given by Equation (6). The input feature vector for the PERB is $F_{PERB\left(inp\right)} \in {\mathbb{R}}^{H \times W \times C}$. The output is represented as $F_{PERB\left(out\right)}\in {\mathbb{R}}^{H \times W \times C}$. The intermediate layers of the PERB are expressed in three steps. In step 1, convolution and BN are applied to $F_{PERB\left(inp\right)}$, and the resultant output is denoted as $A$. In step 2, $A$ is considered as the input, and another CL (followed by BN with the ReLU activation function) is applied to produce output $B$. Finally, in step 3, $B$ is considered as the input, and a CL with BN is applied to produce output $C$. $C$ becomes the final output of the PERB and is denoted as $F_{PERB\left(out\right)}$.

$$\begin{array}{c}{F}_{PERB\left(out\right)} \in {\mathbb{R}}^{H\times W\times C}=C\left(B\left(A\right)\right), \\ where \left\{\begin{array}{c}A=BN\left(Conv\left(F_{PERB\left(inp\right)}\right)\right)\\ B=ReLU\left(BN\left(Conv\left(A\right)\right)\right)\\ C=BN\left(Conv\left(B\right)\right)\end{array}\right.\end{array}$$

(6)

2.3.2. HMFB

The HMFB is placed in the bottleneck layer and contains multiple stages. Here, each stage independently learns diverse and richer feature information of crops and weeds. The stages are combined hierarchically. After combining all the stages, a residual connection is applied to recover the previously degraded features. This process provides fine-grained patterns and detailed feature representations of crops and weeds. The structure of the HMFB is shown in Figure 7.

The HMFB comprises four stages. At each stage, 3 × 3 CLs (followed by the BN and ReLU) are applied to the input of HMFB. The output of the first two stages is concatenated channel-wise, and the resultant feature vector is concatenated with the output of the third stage. Finally, the concatenated result of the three stages is concatenated channel-wise with the outputs of the four stages. After all the stages of concatenation, element-wise addition is applied between the output after the fourth stage and the residual from the input. The resultant output after the element-wise addition is then convolved with a 1 × 1 CL to produce the final output layer of the HMFB. A mathematical representation of the HMFB is presented in Equations (7)–(13).

$$F_1 \in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}=ReLU\left(BN\left(Conv\left(F_{HMFB\left(inp\right)}\right)\right)\right)$$

(7)

$$F_2 \in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}=ReLU\left(BN\left(Conv\left(F_{HMFB\left(inp\right)}\right)\right)\right)$$

(8)

$$F_3 \in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}=ReLU\left(BN\left(Conv\left(F_{HMFB\left(inp\right)}\right)\right)\right)$$

(9)

$$F_4 \in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}=ReLU\left(BN\left(Conv\left(F_{HMFB\left(inp\right)}\right)\right)\right)$$

(10)

$$F \in {\mathbb{R}}^{H\times W\times C}=F_4 \copyright \left(F_3 \copyright \left(F_1 \copyright F_2\right)\right)$$

(11)

$$R \in {\mathbb{R}}^{H\times W\times C}=F_{HMFB\left(inp\right)}$$

(12)

$$F_{HMFB\left(out\right)} \in {\mathbb{R}}^{H\times W\times C}=Conv(F \oplus R)$$

(13)

The input feature vector for the HMFB is denoted as $F_{HMFB\left(inp\right)}\in {\mathbb{R}}^{H\times W\times C}$. The output feature vector is denoted as $F_{HMFB\left(out\right)}\in {\mathbb{R}}^{H\times W\times C}$. The output of the first stage, $F_1\in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}$, is computed using $F_{HMFB\left(inp\right)}$ by applying 3 × 3 CL, BN, and ReLU as the activation function. In the output of $F_1$, we obtain ${\displaystyle \frac{C}{4}}$ number of channels as the output, whereas C is the total number of channels considered as the input. Similarly, the output of the second stage, $F_2\in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}$, is also calculated by applying the operations to input $F_{HMFB\left(inp\right)}$ and obtaining an equal number $\left({\displaystyle \frac{C}{4}}\right)$ of channels. In a similar manner $F_3\in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}$ and $F_4\in {\mathbb{R}}^{H\times W\times {\displaystyle \frac{C}{4}}}$ are computed at the third and fourth stages, respectively. The concatenation process begins with channel-wise concatenation (denoted as $\copyright$) of $F_1$ and $F_2$ by obtaining ${\displaystyle \frac{C}{2}}$ number of channels. The result of this concatenation ($F_1 \copyright F_2$) is concatenated with $F_3$ and increases the number of channels by ${\displaystyle \frac{3C}{4}}$. Then, the subsequent results ($F_3 \copyright \left(F_1 \copyright F_2\right)$) are concatenated with $F_4$ and equal the numbers of channels with input C. The final output from all the stages is denoted as $F\in {\mathbb{R}}^{H\times W\times C}$. A residual connection is applied from $F_{HMFB\left(inp\right)}$, and element-wise addition (denoted as $\oplus$) is performed between $F$ and residual $R$ as shown in Equation (13). After this operation, a 1 × 1 CL is applied to produce the final output of the HMFB (denoted as $F_{HMFB\left(out\right)}$).

2.3.3. AFEB

The AFEB was placed before the final 1 × 1 CL in the AHFF-Net decoder. The AFEB is based on an attention mechanism that focuses on the critical regions of crops and weeds. In segmentation tasks, attention mechanisms are particularly advantageous because certain features are more important than others. In the AFEB, the input is passed through a 3 × 3 CL followed by a BN. The resultant is passed through a 1 × 1 CL and sigmoid function, whereas the other is used as a residual connection. The outputs of the sigmoid and residual operations are combined using an element-wise multiplication operation. This output functions as an attention mechanism within the AFEB. Furthermore, the output of the element-wise multiplication is passed through a 3 × 3 CL followed by a BN to produce the final output. The complete structure of the AFEB is shown in Figure 8. The mathematical details are provided in Equations (14)–(16).

$$R \in {\mathbb{R}}^{H\times W\times C}=BN\left(Conv\left(F_{AFEB\left(inp\right)}\right)\right)$$

(14)

$$D \in {\mathbb{R}}^{H\times W\times C}=Sigmoid\left(Conv\left(R\right)\right)$$

(15)

$$F_{AFEB\left(out\right)} \in {\mathbb{R}}^{H\times W\times C}=BN\left(Conv\left(R\odot D\right)\right)$$

(16)

In the mathematical equations of AFEB, $F_{AFEB\left(inp\right)} \in R^{H \times W \times C}$ is used as the input vector. Meanwhile, $F_{AFEB\left(out\right)} \in R^{H \times W \times C}$ is represented as the output feature vector of AFEB. In Equation (13), $F_{AFEB\left(inp\right)}$ is passed through 3 × 3 CL and BN to produce $R \in R^{H \times W \times C}$. $R \in R^{H \times W \times C}$ is also used as a residual connection. Concurrently, it is passed through 1 × 1 CL and a sigmoid function. The output of the sigmoid and residual skip connection is combined through an element-wise multiplication operation $\left(R\odot D\right)$. Finally, the result of this operation is passed through 3 × 3 CL and BN to calculate $F_{AFEB\left(out\right)} \in R^{H \times W \times C}$, which is the output feature vector of AFEB (Equation (16)).

2.4. Loss Function

The Dice loss function [57] was used as the loss function for training the AHFF-Net, as shown in Equation (17):

$$L_{dice} =1-{\displaystyle \frac{1}{classes}}\sum _{cl=1}^{classes}{\displaystyle \frac{2{\sum }_{i=1}^{H\times W}(M_{cl}\left(i\right) N_{cl}\left(i\right))}{{\sum }_{i=1}^{H\times W}\left(M_{cl}\left(i\right)+N_{cl}\left(i\right)\right)}}$$

(17)

where the number of classes ($classes$) used in all the experiments was three: weed, crop, and background. The ground truth label map and predicted segmentation output are denoted as ${N\in R}^{H\times W\times classes}$ and ${M\in R}^{H\times W\times classes}$, respectively. For a specific class $cl$, ${N_{cl}\in R}^{H\times W}$ and ${M_{cl}\in R}^{H\times W}$ represent the ground truth and predicted values, respectively.

2.5. Evaluation Metrices

We evaluated the semantic segmentation accuracy using the intersection over union ($IOU$) for all the three classes (background, weed, and crop), mean $IOU$ ($mIOU$), $Recall$, $Precision$, and $F1 score$ (harmonic mean of $Precision$ and $Recall$). The mathematical expressions for all the evaluation metrics are provided in Equations (18)–(22). These metrics were used to assess the semantic segmentation performance of the proposed AHFF-Net. The number of classes ($Cl$) was set to three. True positives (TPs) denote cases in which the true label matches the prediction, whereas true negatives (TNs) denote those in which a false label matches the prediction. False positives (FPs) occur when false labels are predicted incorrectly as true labels, and false negatives (FNs) occur when a true label is predicted incorrectly as a false label. The $IOU$ for class $j$ $\left(IO{U}_j\right)$ is defined in Equation (19). $Recall$ and $Precision$ were calculated for each class, and the average values were computed. The average values were then used to calculate the $F1 score$. This provided a comprehensive evaluation of the performance of the model.

$$IOU={\displaystyle \frac{TP}{TP+FN+FP}}$$

(18)

$$mIOU = {\displaystyle \frac{{\sum }_{j=1}^{Cl} IO{U}_j}{Cl}}$$

(19)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(20)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(21)

$$F1 score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(22)

2.6. FD Estimation

The FD is a mathematical measure to describe quantitative analysis of irregularities of different structures. It measures the complexity of a shape or pattern, especially in fractal shapes and geometric structures that possess self-similarity across the different scales. These repeating patterns at multiple scales can be effectively quantified using the FD. FD is a numerical value, usually varying between 1 and 2, showing its complexity level [58]. An increased FD value reflects a more complex structure. One of the most commonly used methods to calculate FD is the box-counting algorithm [59]. The pseudo-code to measure the FD using the box-counting algorithm is provided in Algorithm 1.

Algorithm 1. Pseudo-code to measure FD

Input: image (input image path)

Output: Fractal dimension (FD) value

1:         Read the input image and further convert it into grayscale 2:         Set the maximum box-size with the power of 2 and ensure the dimensions                   q = 2^(log(max(size(image))/log2)]             Add the padding if required to match the dimensions 3:         Initialize the number of boxes 4:         Calculate the number of boxes Y(q) to minimum pixels 5:         Decrease the box size by 2 and recalculate Y(q) iteratively                   while q > 1 6:         Calculate log(Y(q)) and log(1/q) for each q 7:         Draw a fitted line to the points (log(Y(q)) and log(1/q))

8:         FD value is the slope of the fitted line             Return Fractal Dimension Value

This algorithm enables the calculation of FD. Due to its flexibility, this technique is widely used in many fields, including agriculture [11,22] and many other systems. The formula for calculating FD using this method is given below:

$$\mathrm{FD} ={\lim }_{\mathrm{q}\to 0}{\displaystyle \frac{\log \left(\mathrm{Y}\right(\mathrm{q}\left)\right) }{\log (1/\mathrm{q})}}$$

(23)

Here $\mathrm{Y}\left(\mathrm{q}\right)$ represents the total number of boxes with size $\mathrm{q}$ needed to completely cover the curve. The variable $\mathrm{q}$ means the box size and FD means the fractal dimension that characterizes the complexity of the curve.

3. Experimental Results

3.1. Training Details

The proposed AHFF-Net utilized the BoniRob, CWFID, and Sunflower public datasets, with input images of size 512 × 512. Initially, Dataset-1 was trained using 70% of the data images with an Adam optimizer (AO) and an initial learning rate (LER) of ${1\times 10}^{-5}$ for 150 epochs. The learning rate was reduced gradually during training by a cosine annealing (CA) strategy [60]. The CA helps the model converge smoothly, improves generalization, and ensures better optimization without abrupt learning rate reductions. After training with Dataset-1, Dataset-2 was pre-processed and divided into two equal parts: one used for testing and the other for small training data. A single image from the small training set was augmented, and the two small training images were augmented using the patch-based techniques described in Section 2.2.2. These two small training images, in conjunction with the five patch-based augmented images, were used for training with the same AO, LER, and CA strategies. The test data remained unaltered. Subsequently, Dataset-2 was replaced with Dataset-3, and the segmentation results were calculated for crops and weeds in heterogeneous environments. Similarly, each dataset was first trained on one dataset, pre-processed, and tested on the other two datasets.

Table 2. Training, validation, and testing data split for all the experiments.

Dataset-1	Training	Validation	Dataset-2	Testing
BoniRob	400	30	CWFID	30
CWFID	45	5	BoniRob	246
Sunflower	120	17	BoniRob	246
BoniRob	400	30	Sunflower	86
CWFID	45	5	Sunflower	86
Sunflower	120	17	CWFID	30

shows the training, validation, and testing data split for all the experiments.

3.2. Testing of Proposed Method

3.2.1. Performance Evaluation with CWFID Following BoniRob Dataset Training

Ablation Study

We performed eight ablation studies with multiple potential scenarios while testing the CWFID, following training with the BoniRob dataset. We conducted a block-by-block ablation study that included individual combinations of two blocks (

Table 3. Block by block comparisons of the semantic segmentation result (B, W, and C denote background, weed, and crop, respectively).

PERB	HMFB	AFEB	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
			0.624	0.985	0.538	0.350	0.761	0.759	0.759
√			0.626	0.986	0.600	0.289	0.772	0.756	0.763
	√		0.625	0.986	0.598	0.288	0.770	0.752	0.759
		√	0.624	0.986	0.592	0.294	0.770	0.753	0.761
√	√		0.633	0.986	0.578	0.334	0.787	0.765	0.774
√		√	0.632	0.986	0.622	0.290	0.784	0.761	0.772
√	√	√	0.653	0.986	0.637	0.338	0.804	0.772	0.787

“√” indicates that the corresponding experiment includes the respective combination of blocks.

). It is noteworthy that the base model without any of the proposed blocks achieved the highest $IOU$ (C) for the crop class. This may be because it was challenging for the base model to separate visually similar weeds from crops. As a result, certain weed regions may have been labeled incorrectly as crops. This would have caused an increase in $IOU$ (C) at the cost of a reduction in $IOU$ (W). When the proposed blocks were introduced, the attention and feature representations of the model became distributed more uniformly across all the classes. Among the individual blocks, the overall segmentation performance (in terms of $F1 score$ and $mIOU$) achieved using only the PERB was better than that obtained using only the HMFB or AFEB. This is likely because the PERB enhances the initial feature representation and helps better distinguish between classes in coarse segmentation. However, the PERB alone is insufficient for capturing fine-grained boundaries, as reflected by the reduced $IOU$ (C) for the crop class. Compared with the use of a single block, the combination of two blocks resulted in better segmentation accuracy. This improvement can be attributed to the effect of the blocks: one block (e.g., PERB) provides robust base features, and the second block (e.g., HMFB or AFEB) complements it by enhancing contextual understanding or focusing on semantically important spatial regions. This complementary behavior yields a better class balance and more accurate segmentation results. We observed that the three proposed blocks yielded the best segmentation results for the semantic segmentation of crops and weeds.

Subsequently, we performed experiments by applying IMBA and patch-based augmentation using the proposed AHFF-Net.

Table 4. Result comparison of semantic segmentation with IMBA and patch-based augmentation (B, W, and C, denote background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
With IMBA	0.644	0.986	0.62	0.326	0.797	0.765	0.780
With patch-based augmentation (proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787

shows that the AHFF-Net trained with patch-based augmentation achieved superior performance in all the metrics compared with the same model trained using IMBA. Patch-based augmentation preserves the fine-grained spatial details and increases the diversity of localized training. In contrast, IMBA applies global transformations that may not preserve the small-scale structure and distribution of critical features, particularly in complex scenes with dense crop and weed mixtures.

We then investigated the effects of using different numbers of patches and compared the segmentation results. As shown in

Table 5. Results of semantic segmentation with different numbers of patches in patch-based augmentation (B, W, and C denote background, weed, and crop, respectively).

Number of Patches	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
4	0.623	0.986	0.548	0.335	0.772	0.751	0.760
5 (Proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787
6	0.646	0.986	0.583	0.369	0.792	0.764	0.778

, the use of five patches for augmentation yielded the best overall segmentation performance. It achieved the highest $mIOU$ (0.653) and $F1 score$ (0.787). When four patches were used, the performance reduced observably ($mIOU$ = 0.623 and $F1 score$ = 0.760). This reduction can be attributed to the limited diversity of the augmented training samples. However, when six patches were used, a marginal performance reduction (compared with the use of five patches) was observed. Although the use of more patches increases the number of training samples, it may introduce redundant or highly overlapping regions. This results in less informative or repetitive features. Moreover, although the overall performance was lower for six patches, the $IOU$ for the crop class was marginally higher than that for the proposed method using five patches. This may be because an increase in the number of patches results in an enhanced exposure of the model to small crop structures, thereby enabling more precise boundary learning for crops.

Additionally, we conducted separate ablation studies for the PERB, HMFB, and AEFB. For the PERB, we performed an ablation study based on the number of convolutions and convolutional manner (either parallel or sequential). This is explained in

Table 6. Ablation study results for different numbers of convolutions in PERB (B, W, and C denote background, weed, and crop, respectively).

Number of Convolutions	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
2	0.636	0.985	0.562	0.359	0.783	0.765	0.773
3 (Proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787
4	0.645	0.985	0.598	0.353	0.795	0.769	0.781

and

Table 7. Ablation study results comparing sequential and parallel convolutional manners in PERB (B, W, and C denote background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
With sequential convolutions (proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787
With parallel convolutions	0.637	0.985	0.554	0.371	0.785	0.764	0.774

. As shown in Table 6, using only two CLs resulted in the highest $IOU$ for the crop class. However, it yielded the lowest $IOU$ for weeds and overall performance ($mIOU$ = 0.636, $F1 score$ = 0.773). This may be because a shallower configuration has an inadequate capacity to extract sufficient semantic and contextual features, particularly for less distinct or more complex classes. Although the limited depth may enable the model to focus on clearer structures, it fails to capture the fine-grained variable patterns required for accurately segmenting weed regions. This results in imbalanced class performance. In contrast, using four convolutions marginally improved $IOU$ (C) to 0.353. However, it still underperformed in terms of $IOU$ (W) (=0.598), $mIOU$ (=0.645), and $F1 score$ (=0.781) compared with the proposed three-CL design. This indicates that adding more depth may cause redundancy, oversmoothing, or increased difficulty in optimizing feature representations, particularly in the early encoder stages. The proposed three-convolution configuration achieved the best overall results ($mIOU$ = 0.653, $F1 score$ = 0.787) by achieving an optimal balance between low-level detail preservation and mid-level semantic abstraction. This enabled robust performance across all the classes.

As shown in Table 7, the proposed sequential configuration achieved the best overall segmentation performance, with $mIOU$ = 0.653 and $F1 score$ = 0.787. In contrast, the parallel convolution variant yielded an $IOU$ for the crop class ($IOU$ (C) = 0.371) that was marginally higher than the 0.338 in the sequential case. However, it experienced significant degradation in weed segmentation ($IOU$ (W) = 0.554) and overall performance ($mIOU$ = 0.637, $F1 score$ = 0.774). This indicates that parallel convolutions enhance the spatial diversity and potentially capture more localized features that are advantageous for crops. However, sequential convolution provides a more structured and hierarchical flow of features. This helps in learning stable representations and improves interclass separation, thereby yielding a better generalization across all classes.

For the HMFB, we conducted an ablation study with and without residual connections, as listed in

Table 8. Ablation study results with and without residual connection in HMFB (B, W, and C denote background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
With residual connection (proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787
Without residual connection	0.631	0.986	0.564	0.341	0.792	0.754	0.772

. The inclusion of residual connections, similar to the proposed design (with residual connections), yielded the best performance across most key metrics: $mIOU$ = 0.653 and $F1 score$ = 0.787. In contrast, removing the residual connection caused an observable performance reduction, with $IOU$ (W) reducing to 0.564, $mIOU$ decreasing to 0.631, and $F1 score$ reducing to 0.772. It is noteworthy that $IOU$ (C) increased marginally without a residual connection. This was possibly because the model focused more narrowly on prominent and conveniently identifiable regions in the absence of competing fine-grained details preserved by residual connections. However, residual connections are critical in deep structures such as the HMFB. This is because these help mitigate gradient vanishing, maintain feature continuity, and enable multiscale fusion of low- and high-level features, which is particularly advantageous for challenging weed segmentation.

Table 9. Ablation study results for different numbers of convolutions in HMFB (B, W, and C denote background, weed, and crop, respectively).

Number of Convolutions	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
3	0.636	0.985	0.623	0.299	0.796	0.761	0.778
4 (Proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787
5	0.634	0.986	0.579	0.335	0.79	0.763	0.776

lists the ablation of the HMFB based on the number of convolutions. Using three CLs resulted in the lowest $IOU$ (C) of 0.299. This indicates that the depth was insufficient to model spatially diverse features and multiscale information, particularly for crops with varying shapes and sizes. However, an increase in the depth to five convolutions lowered the $mIOU$ and $F1 score$ to 0.634 and 0.776, respectively. This reduction is attributed to overfitting or excessive feature compression, which weakens the network’s capability to generalize, particularly in underrepresented crop and weed classes. The proposed four-convolution setup provides the best overall performance by effectively integrating multiscale contextual information without complicating the feature space. This balance configuration enhances both weed and crop segmentation and ensures stable generalization across all metrics.

For the AFEB, we performed ablation studies with and without an attention mechanism, as shown in

Table 10. Ablation study results with and without attention in AFEB (B, W, and C denote background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
With attention (Proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787
Without attention	0.64	0.986	0.562	0.371	0.778	0.781	0.779

. The proposed method (with attention) provided the most consistent and balanced results across most metrics. When attention was removed, the model showed a higher crop $IOU$ ($IOU$ (C) = 0.371). However, it was affected significantly in terms of weed segmentation ($IOU$ (W) = 0.562) and overall accuracy ($mIOU$ = 0.640, $F1 score$ = 0.779). This pattern indicates that without attention, the model tends to overfit the dominant crop class. It leverages its abundant and spatially consistent features while omitting less frequent and spatially dispersed weed regions. The attention mechanism effectively reallocates the focus of the model to class-specific and informative regions. This enhances the detection of small and complex objects such as weeds. This, in turn, results in an improved class balance, particularly under conditions of spatial complexity and class imbalance (common in agricultural segmentation tasks). Through these ablation studies, we assessed the impact of various blocks on the performance. The results emphasize the significance of specific model design selections (such as the number of convolutions and incorporation of attention mechanisms) in enhancing the segmentation accuracy.

Comparison of the Proposed and State-of-the-Art (SOTA) Methods

This subsection describes the experiments conducted using the CWFID as a testing dataset, following training with the BoniRob dataset. This study compared the SOTA methods with the proposed AHFF-Net, as shown in

Table 11. Comparison of semantic segmentation results for crops and weeds between the proposed AHFF-Net and various SOTA methods (B, W, and C denote background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
U-Net (RHT + small training data + IMBA) [22]	0.620	0.986	0.524	0.349	0.749	0.762	0.752
U-Net [61]	0.624	0.985	0.538	0.350	0.761	0.759	0.759
Modified U-Net [40]	0.606	0.985	0.546	0.288	0.765	0.738	0.751
Unet++ [46]	0.547	0.982	0.514	0.146	0.684	0.654	0.668
Reduced U-Net [62]	0.560	0.978	0.482	0.220	0.680	0.691	0.685
DWUNet [63]	0.535	0.979	0.447	0.178	0.646	0.655	0.650
MSFCA-Net [64]	0.538	0.982	0.465	0.166	0.647	0.658	0.652
DeepLAB V3 Plus [65]	0.597	0.974	0.556	0.262	0.712	0.713	0.712
SEG-Net [66]	0.558	0.961	0.516	0.197	0.631	0.719	0.672
CG-Net [67]	0.506	0.963	0.447	0.110	0.711	0.576	0.636
AHFF-Net (proposed)	0.653	0.986	0.637	0.338	0.804	0.772	0.787

. In all the metrics except for IOU (C), AHFF-Net surpassed the other SOTA methods in terms of the segmentation results for crops and weeds. The second-best model was U-Net [61]. It showed a 2.9% lower $mIOU$ and 2.8% lower $F1 score$ than the proposed AHFF-Net. U-Net [61] achieved a marginally higher $IOU$ (C) for the crop class (0.350) than AHFF-Net (0.338). This can be attributed to the structural simplicity of the U-Net, which tends to extract low-level features (such as shape and texture) more effectively and mixes the tighter boundaries of crops and weeds. U-Net is deficient in deeper refinement modules. Consequently, it may rely more significantly on early features and label mixed regions as crops. This, in turn, can result in a marginally higher $IOU$ (C). In contrast, AHFF-Net marginally forfeited $IOU$ (C) in favor of improved weed segmentation and class balance. Thereby, it achieved superior overall performance. The lower performance of the SOTA models can be attributed to the similar shapes of crops and weeds in the other datasets. However, AHFF-Net performed better at differentiating between crops and weeds in this heterogeneous dataset environment because of its use of patch-based augmentation, sequential convolutions, attention mechanisms, and residual connections. Patch-based augmentation provides the network with localized and balanced training samples. This is essential for learning the fine-grained details of small and irregular crop and weed structures that may be underrepresented or vary across fields in heterogeneous environments. The use of sequential convolutions facilitates deeper and more hierarchical feature extraction. This enables the model to capture semantic transitions and maintain spatial consistency across varying resolutions and object-scale conditions typical of heterogeneous agricultural scenes. The attention mechanism embedded in the AFEB module enables the model to focus selectively on class-discriminative regions. This effectively reduces misclassification in cases where crops and weeds have similar visual characteristics or overlap spatially. Furthermore, the HMFB is enhanced with residual connections. It combines multilevel contextual features while preserving low-level spatial indications. This combined mechanism ensures that both detailed textures and broader semantic relationships are retained. This, in turn, improves the model’s capability to accurately distinguish classes across diverse and unseen heterogeneous environmental data. These features enable AHFF-Net to achieve an accuracy higher than those of other SOTA models.

Figure 9 shows a visualization example of the semantic segmentation performance of the SOTA method and proposed AHFF-Net. In the visualization, crops are represented by red pixels, weeds by blue pixels, and the background by black pixels. We also visualized the error pixels, where crops were detected incorrectly as weeds or background (represented by yellow pixels). Similarly, the error pixels for weeds (where weeds were detected incorrectly as crops or backgrounds) are indicated by orange pixels. Finally, the error pixels for the background (where these were detected incorrectly as crops or weeds) are shown as gray pixels. From the visual analysis, it is evident that AHFF-Net (o) produced the most accurate and balanced segmentation with minimal misclassification and well-preserved class boundaries. It accurately captured both large crop regions and scattered weed structures, which are generally misclassified by other models. The boundary quality and object shape integrity were particularly significant in the AHFF-Net results. In contrast, earlier models such as Unet++ (h), reduced U-Net (i), and DWUNet (j) showed substantial misclassification, particularly in dense weed regions (as evidenced by the presence of orange and yellow artifacts). These errors reflect the inability to distinguish spatially similar crop and weed regions, which generally results in overlapping misclassifications. MSFCA-Net (k) and SEG-Net (m) performed moderately. Nonetheless, these displayed boundary degradation and loss of structural detail in fine plant branches. DeepLab V3 Plus (l) demonstrated improved crop segmentation. However, it failed to detect certain weed patches. Meanwhile, CG-Net (n) underperformed significantly. It displayed substantial background confusion and broken object continuity. Overall, we verified that AHFF-Net successfully outperformed the SOTA models in terms of the semantic segmentation performance of crops and weeds.

3.2.2. Performance Evaluation with BoniRob Following CWFID Training

This subsection describes the experiments conducted using the BoniRob as a testing dataset, following training with the CWFID. This study compared the SOTA methods with the proposed AHFF-Net, as shown in

Table 12. Comparison of semantic segmentation results for crops and weeds between the proposed AHFF-Net and various SOTA methods (B, W, and C mean background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
U-Net (RHT + small training data + IMBA) [22]	0.637	0.971	0.292	0.647	0.708	0.787	0.743
U-Net [61]	0.642	0.975	0.301	0.649	0.755	0.738	0.746
Modified U-Net [40]	0.584	0.966	0.277	0.509	0.644	0.778	0.704
Unet++ [46]	0.576	0.958	0.230	0.541	0.633	0.789	0.702
Reduced U-Net [62]	0.568	0.951	0.238	0.516	0.626	0.803	0.703
DWUNet [63]	0.533	0.967	0.077	0.555	0.667	0.578	0.619
MSFCA-Net [64]	0.570	0.972	0.225	0.513	0.652	0. 723	0.685
DeepLAB V3 Plus [65]	0.541	0.969	0.208	0.446	0.661	0.644	0.652
SEG-Net [66]	0.491	0.951	0.205	0.319	0.565	0.652	0.605
CG-Net [67]	0.475	0.930	0.211	0.285	0.553	0.735	0.631
AHFF-Net (proposed)	0.661	0.974	0.334	0.674	0.742	0.796	0.768

. In all the metrics except for $IOU$ (B), $Precision$, and $Recall,$ AHFF-Net surpassed the segmentation results for crops and weeds compared to the other SOTA methods. This showed that AHFF-Net significantly differentiated the crops and weeds in the heterogeneous environment. The second-best model was U-Net [61]. It showed a 1.9% lower $mIOU$ and 2.2% lower $F1 score$ than the proposed AHFF-Net. Notwithstanding its overall advantages, AHFF-Net reported slightly lower values in a few individual metrics. U-Net [61] achieved the highest $IOU$ for background ($IOU$ (B) = 0.975), as well as the highest $Precision$ (0.755). This can be attributed to U-Net’s tendency to focus on lower-level features. The encoder–decoder structure of U-Net with direct skip connections can preserve low-level information (such as large and uniform background areas) more effectively. Background regions generally have less texture and more consistent appearance, which enables the cleaner segmentation of these areas through the structural design of U-Net. It also tends to favor only high-confidence areas and avoid uncertain regions near object boundaries or transitions. This can inflate the values of both $IOU$ (B) and $Precision$. In contrast, AHFF-Net distributes focus more evenly across all classes because of its attention mechanism and patch-based learning, which may slightly reduce background accuracy while improving overall weed and crop detection. Similarly, reduced U-Net [62] reported the highest $Recall$ (0.803). This is likely because of its aggressive prediction strategy, favoring sensitivity over specificity. While this helped in identifying more correctly labeled crop and weed pixels, it also resulted in more false positives, leading to a trade-off where precision was marginally forfeited ($Precision$ = 0.626) in favor of $Recall$. By contrast, AHFF-Net maintained a better balance between $Precision$ (0.742) and $Recall$ (0.796), reflecting its strong generalization and class-wise discrimination. Although U-Net and reduced U-Net may have outperformed AHFF-Net in a few individual metrics, the comprehensive performance of AHFF-Net reflected in the highest $mIOU$ (0.661), $IOU$ (W) (0.334), $IOU$ (C) (0.674), and $F1 score$ (0.768) demonstrated its effectiveness in balanced and context-aware segmentation of crops and weeds.

Visual examples of semantic segmentation performance with the BoniRob evaluation dataset for proposed AHFF-Net and SOTA methods are shown in Figure 10. From the visualization, we can confirm that AHFF-Net outperformed the SOTA models in terms of crop and weed semantic segmentation performance with minimal misclassified pixels, particularly in distinguishing overlapping or tightly clustered weeds and crops. In contrast, the models such as Unet++ (h) and reduced U-Net (i) displayed significant misclassifications, particularly in background regions (gray) and between crops and weeds (orange/yellow areas), indicating confusion between spatially close or visually similar classes. Models like DWUNet (j), MSFCA-Net (k), and SEG-Net (m) also showed scattered misclassifications, particularly in boundary regions and small weed structures. DeepLab V3 Plus (l) showed improved performance over conventional U-Net-based models but still exhibited errors around crop edges and isolated weed patches. CG-Net (n) misclassified a large portion of the image, particularly mislabeling background and crops, which compromised its effectiveness in real-world applications.

3.2.3. Performance Evaluation with Sunflower Following BoniRob Dataset Training

In this subsection, the experiments were conducted using Sunflower as a testing dataset, following training with the BoniRob dataset. This study compared the SOTA methods with the proposed AHFF-Net, as shown in

Table 13. Comparison of semantic segmentation results for crops and weeds between the proposed AHFF-Net and various SOTA methods (B, W, and C mean background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
U-Net (RHT + small training data + IMBA) [22]	0.581	0.993	0.503	0.248	0.679	0.694	0.685
U-Net [61]	0.587	0.992	0.514	0.256	0.686	0.695	0.690
Modified U-Net [40]	0.575	0.990	0.577	0.158	0.651	0.700	0.674
Unet++ [46]	0.580	0.991	0.604	0.144	0.694	0.682	0.687
Reduced U-Net [62]	0.546	0.988	0.490	0.161	0.609	0.687	0.645
DWUNet [63]	0.427	0.980	0.000	0.300	0.503	0.486	0.494
MSFCA-Net [64]	0.583	0.992	0.520	0.236	0.685	0.693	0.688
DeepLAB V3 Plus [65]	0.566	0.99	0.586	0.121	0.657	0.636	0.646
SEG-Net [66]	0.505	0.934	0.544	0.037	0.571	0.699	0.628
CG-Net [67]	0.554	0.986	0.559	0.116	0.800	0.572	0.667
AHFF-Net (proposed)	0.601	0.993	0.512	0.300	0.724	0.691	0.707

. In all the metrics except for $IOU$ (W), $Precision$, and $Recall$, AHFF-Net surpassed the segmentation results for crops and weeds compared to the other SOTA methods. AHFF-Net performed better at differentiating between crops and weeds in this heterogeneous environment. The second-best model was U-Net [61]. It showed a 1.4% lower $mIOU$ and a 1.7% lower $F1 score$ than the proposed AHFF-Net. Although AHFF-Net achieved the highest overall performance ($mIOU$ = 0.601, $F1 score$ = 0.707), a few models marginally outperformed it in isolated metrics. For instance, Unet++ [46] reported the highest $IOU$ for weeds ($IOU$ (W) = 0.604), which can be attributed to its densely nested skip connections that enhance spatial detail and support fine-grained segmentation of thin weed structures. However, this results in the cost of poor performance in crop segmentation ($IOU$ (C) = 0.144) and overall balance, resulting in a lower $mIOU$ (0.580) and $F1 score$ (0.687) than AHFF-Net. Similarly, CG-Net [67] achieved the highest $Precision$ (0.800) because of its conservative prediction strategy, which likely reduced false positives by focusing on confident predictions. However, this strategy tends to suppress uncertain or ambiguous regions, such as overlapping or low-contrast crop or weed structures, resulting in a lower $Recall$ (0.572). The model’s limited exploration of harder to detect regions can result in many true positive pixels being missed, which negatively impacts and lowers $F1 score$ (0.667). Meanwhile, modified U-Net [40] recorded the highest $Recall$ (0.700) by assigning class labels more broadly, which helped it detect more relevant regions but also led to less accurate boundaries and a drop in $Precision$ (0.651) and $F1 score$ (0.674).

Visual examples of semantic segmentation performance with the Sunflower evaluation dataset for the proposed AHFF-Net and SOTA methods are shown in Figure 11. The visualization showed that AHFF-Net (o) achieved the most accurate and clean segmentation, with well-preserved boundaries and minimal misclassification, especially in complex regions where crops and weeds were densely overlapped. In contrast, models such as Unet++ (h) and reduced U-Net (i) exhibited significant confusion, particularly with prominent orange and yellow misclassified pixels, indicating difficulty in accurately distinguishing between crops and weeds. These models generally tended toward mislabeled crops as weeds and vice versa. Other models, including DWUNet (j), MSFCA-Net (k), and SEG-Net (m), showed scattered misclassifications, particularly around object boundaries or within dense weed areas, reflecting limitations in capturing fine-grained structural details of crops and weeds. Although DeepLab V3 Plus (l) performed moderately better, it still struggled to preserve class boundaries consistently, particularly in regions with overlapping class distribution. CG-Net (n) demonstrated significant misclassification, particularly with large gray and yellow regions, indicating poor differentiation between crops and background, as well as failure to detect small weed clusters. Overall, these results confirmed that AHFF-Net significantly outperformed other SOTA models in terms of semantic segmentation accuracy for crops and weeds, particularly in heterogeneous and challenging agricultural environments.

3.2.4. Performance Evaluation with BoniRob Following Sunflower Dataset Training

In this subsection, the experiments were conducted using the BoniRob as a testing dataset, following training with the Sunflower dataset. This study compared the SOTA methods with the proposed AHFF-Net, as shown in

Table 14. Comparison of semantic segmentation results for crops and weeds between the proposed AHFF-Net and various SOTA methods (B, W, and C mean background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
U-Net (RHT + small training data + IMBA) [22]	0.561	0.973	0.172	0.538	0.659	0.671	0.664
U-Net [61]	0.558	0.973	0.136	0.565	0.678	0.667	0.672
Modified U-Net [40]	0.579	0.973	0.217	0.548	0.671	0.701	0.685
Unet++ [46]	0.530	0.974	0.128	0.486	0.631	0.632	0.631
Reduced U-Net [62]	0.514	0.963	0.154	0.426	0.588	0.656	0.620
DWUNet [63]	0.424	0.966	0.077	0.230	0.521	0.501	0.510
MSFCA-Net [64]	0.570	0.970	0.214	0.525	0.663	0.685	0.673
DeepLAB V3 Plus [65]	0.471	0.971	0.124	0.319	0.580	0.558	0.568
SEG-Net [66]	0.479	0.955	0.159	0.324	0.556	0.630	0.590
CG-Net [67]	0.378	0.960	0.009	0.164	0.600	0.395	0.476
AHFF-Net (proposed)	0.595	0.972	0.225	0.587	0.699	0.711	0.704

. In all the metrics except for $IOU$ (B), AHFF-Net surpassed the segmentation results for crops and weeds compared to the other SOTA methods. AHFF-Net significantly differentiated between crops and weeds in the heterogeneous environment. The second-best model was the modified U-Net [40]. It showed a 1.6% lower $mIOU$ and a 1.9% lower $F1 score$ than the proposed AHFF-Net. Although AHFF-Net achieved the highest overall scores ($mIOU$ = 0.595, $F1 score$ = 0.704), Unet++ [46] marginally outperformed it in $IOU$ for background. This can be attributed to Unet++’s densely connected architecture, which reinforced the preservation of low-level features such as soil and background textures. As a result, Unet++ may have placed greater emphasis on detecting well-distributed and visually consistent background regions, leading to a small gain in $IOU$ (B). However, this focus generally came at the expense of segmenting more complex or overlapping structures such as weeds, which is evident in its lower $IOU$ (W) = 0.128, $IOU$ (C) = 0.486, and $F1 score$ = 0.631.

Visual examples of semantic segmentation performance with the BoniRob evaluation dataset for the proposed AHFF-Net and SOTA methods are shown in Figure 12. The visualization showed that AHFF-Net (o) achieved accurate semantic segmentation of crops and weeds, with minimum confusion among classes, particularly in regions with overlapping plants and background. The boundaries between classes were well-preserved, and misclassified pixels were minimal. In comparison, Unet++ (h), reduced U-Net (i), and DWUNet (j) exhibited noticeable confusion, particularly in the form of yellow and orange misclassifications, where crops and weeds were generally mislabeled. These models also showed irregular shapes and inconsistent coverage of crop regions. MSFCA-Net (k) and SEG-Net (m) displayed poor segmentation in weed regions and frequently confused background with plant structures, leading to scattered gray pixels. DeepLab V3 Plus (l) provided marginally better boundary awareness but still misclassified small weed clusters. Notably, CG-Net (n) performed the worst among all the models, generating large gray and yellow misclassifications and failing to properly detect crop regions. We can also confirm that AHFF-Net successfully outperformed the SOTA models in terms of the semantic segmentation performance of crops and weeds.

3.2.5. Performance Evaluation with Sunflower Following CWFID Training

In this subsection, the experiments were conducted using the Sunflower as a testing dataset, following training with CWFID. This study compared the SOTA methods with the proposed AHFF-Net, as shown in

Table 15. Comparison of semantic segmentation results for crops and weeds between the proposed AHFF-Net and various SOTA methods (B, W, and C mean background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
U-Net (RHT + small training data + IMBA) [22]	0.568	0.99	0.487	0.229	0.646	0.710	0.676
U-Net [61]	0.582	0.991	0.514	0.241	0.686	0.678	0.682
Modified U-Net [40]	0.569	0.986	0.509	0.211	0.617	0.759	0.680
Unet++ [46]	0.580	0.991	0.539	0.209	0.683	0.682	0.682
Reduced U-Net [62]	0.545	0.983	0.475	0.177	0.579	0.766	0.659
DWUNet [63]	0.498	0.988	0.416	0.090	0.621	0.550	0.583
MSFCA-Net [64]	0.578	0.992	0.498	0.243	0.678	0.673	0.675
DeepLAB V3 Plus [65]	0.569	0.987	0.622	0.098	0.746	0.598	0.663
SEG-Net [66]	0.437	0.930	0.344	0.035	0.468	0.675	0.552
CG-Net [67]	0.441	0.962	0.278	0.082	0.599	0.594	0.596
AHFF-Net (proposed)	0.592	0.992	0.513	0.272	0.694	0.710	0.701

. In all the metrics except for $IOU$ (W), $Precision$, and $Recall$, AHFF-Net surpassed the segmentation results for crops and weeds compared to the other SOTA methods. AHFF-Net significantly differentiated between crops and weeds in the heterogeneous environment. The second-best model was the U-Net [61]. It showed a 1% lower $mIOU$ and a 1.9% lower $F1 score$ than the proposed AHFF-Net. While AHFF-Net achieved the highest overall performance ($mIOU$ = 0.592, $F1 score$ = 0.701), a few models reported higher scores in individual metrics. DeepLab V3 Plus [65] achieved the highest $IOU$ for weeds ($IOU$ (W) = 0.622) and the highest $Precision$ (0.746). This can be attributed to its dilated convolution design, which is effective in capturing multi-scale contextual features and may enhance weed boundary detection in structured scenes. However, this generally comes with trade-offs in detecting finer structures like crops, which were reflected in its significantly lower $IOU$ (C) = 0.098 and $F1 score$ = 0.663. This suggested that while DeepLab V3 Plus exceled at isolating certain classes such as weeds, it was deficient in the balance needed for high overall performance. Similarly, reduced U-Net [62] recorded the highest $Recall$ (0.766), likely because of its more inclusive prediction behavior, which favors broader classes. This strategy may help identify more vegetation regions but generally forfeits accuracy in spatial detail, resulting in lower $Precision$ (0.579) and $F1 score$ (0.659). In contrast, AHFF-Net maintained a well-rounded segmentation strategy with high scores across all key metrics, while avoiding excessive focus on any single class.

Visual examples of semantic segmentation performance with the Sunflower evaluation dataset for the proposed AHFF-Net and SOTA methods are shown in Figure 13. From the visualization, we can determine that AHFF-Net (o) delivered the most accurate and refined segmentation, particularly in distinguishing scattered weed regions with minimal misclassified pixels. The boundaries between classes were clear and consistent, even in the areas with overlapping vegetation. In contrast, Unet++ (h) and reduced U-Net (i) showed substantial misclassifications, particularly in the form of orange and yellow pixels, indicating confusion between crops and weeds. These models also suffered from inconsistent boundary preservation and poor segmentation in background regions. DWUNet (j) and MSFCA-Net (k), while showing improved weed detection compared to U-Net-based models, still produced scattered errors near object edges and small plant patches. DeepLab V3 Plus (l) and SEG-Net (m) exhibited gray and yellow areas, particularly in boundary regions, signifying difficulty in separating background and crop features. CG-Net (n) failed to accurately segment the crop area, resulting in more noise and incorrect classification. Overall, AHFF-Net’s superior performance compared to those of other SOTA models was clearly visible in terms of the semantic segmentation performance of crops and weeds.

3.2.6. Performance Evaluation with CWFID Following Sunflower Dataset Training

In this subsection, the experiments were conducted using the CWFID as a testing dataset, following training with the Sunflower dataset. This study compared several SOTA methods with the proposed AHFF-Net, as shown in

Table 16. Comparison of semantic segmentation results for crops and weeds between the proposed AHFF-Net and various SOTA methods (B, W, and C mean background, weed, and crop, respectively).

Cases	$\mathit{m}\mathit{I}\mathit{O}\mathit{U}$	$\mathit{I}\mathit{O}\mathit{U}$ (B)	$\mathit{I}\mathit{O}\mathit{U}$ (W)	$\mathit{I}\mathit{O}\mathit{U}$ (C)	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{F}1 \mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
U-Net (RHT + small training data + IMBA) [22]	0.515	0.977	0.313	0.254	0.666	0.676	0.670
U-Net [61]	0.580	0.978	0.571	0.189	0.699	0.690	0.694
Modified U-Net [40]	0.581	0.975	0.496	0.273	0.675	0.727	0.700
Unet++ [46]	0.569	0.976	0.485	0.246	0.695	0.715	0.704
Reduced U-Net [62]	0.590	0.970	0.567	0.234	0.703	0.718	0.710
DWUNet [63]	0.506	0.966	0.492	0.060	0.647	0.617	0.631
MSFCA-Net [64]	0.539	0.982	0.560	0.076	0.640	0.644	0.641
DeepLAB V3 Plus [65]	0.568	0.963	0.514	0.226	0.697	0.693	0.694
SEG-Net [66]	0.483	0.857	0.523	0.067	0.554	0.731	0.627
CG-Net [67]	0.480	0.950	0.409	0.081	0.756	0.534	0.625
AHFF-Net (proposed)	0.595	0.982	0.485	0.319	0.714	0.750	0.731

. In all the metrics except for $IOU$ (W) and $Precision$, AHFF-Net surpassed the segmentation results for crops and weeds compared to the other SOTA methods. Moreover, AHFF-Net significantly differentiated between crops and weeds in the heterogeneous environment. The second-best model was the reduced U-Net [50]. It showed a 0.5% lower $mIOU$ and a 2.1% lower $F1 score$ than the proposed AHFF-Net, further confirming AHFF-Net’s superior performance. Although AHFF-Net achieved the best overall results ($mIOU$ = 0.595, $F1 score$ = 0.731), reduced U-Net [62] recorded the highest $IOU$ for weeds ($IOU$ (W) = 0.567). This may be attributed to its tendency to give stronger focus to broader weed regions, resulting in more complete labeling of larger weed areas. However, this generally comes at the cost of performance in other classes, as reflected in its lower $IOU$ (C) = 0.234 and $IOU$ (B) = 0.970, as well as a marginally lower $F1 score$ = 0.710 than the proposed AHFF-Net. In contrast, AHFF-Net maintained a more balanced segmentation across all classes, with $IOU$ (C) = 0.319, the highest among all models. Similarly, CG-Net [67] achieved the highest $Precision$ (0.756), which suggests a more conservative prediction approach. This approach may result in clear segmentation boundaries and fewer misclassified pixels in confident regions. However, it also results in a lack of coverage, particularly for less distinct classes such as weeds, as seen in CG-Net’s lower $IOU$ (W) = 0.409 and $IOU$ (C) = 0.081, and significantly lower $Recall$ (0.534) and $F1 score$ (0.625).

Visual examples of semantic segmentation performance with the CWFID evaluation dataset for the proposed AHFF-Net and SOTA methods are shown in Figure 14. From the visualization, we can confirm that AHFF-Net (o) again provided the most balanced segmentation, with clearly outlined weed and crop regions and significantly fewer misclassified pixels. In contrast, CG-Net (n) showed more errors, particularly around the middle section of the plant rows. For example, one can observe that the leaf structure visible in the ground truth (d) and correctly segmented by AHFF-Net (o) was poorly segmented in CG-Net (n), with parts either missing or misclassified (seen as orange or gray regions). This highlights AHFF-Net’s superior ability to preserve class boundaries and shape consistency even in cluttered scenes. Several other models, such as Unet++ (h), DWUNet (j), and SEG-Net (m), showed similar patterns of confusion, where large areas of weeds were misclassified as crops (orange), and vice versa. DeepLab V3 Plus (l) and MSFCA-Net (k) also exhibited performance degradation at the boundaries, with scattered background misclassifications and unclear plant structures. U-Net (f) and modified U-Net (g) provided marginally better performance compared to earlier variants but still struggled to resolve overlapping vegetation accurately. Importantly, while CG-Net (n) appeared visually closer to AHFF-Net at a glance, a closer inspection revealed clear segmentation gaps, particularly in small weed structures and finer crop boundaries. AHFF-Net (o), by contrast, achieved higher fidelity, accurately identifying even subtle vegetative elements. Overall, AHFF-Net successfully outperformed the SOTA models in terms of the semantic segmentation performance of crops and weeds.

3.3. Evaluation by FD Estimation

To estimate the approximate fractal dimensions of different shapes, we applied the box-counting technique and evaluated it using the CWFID. The results are summarized in

Table 17. FD values of images from CWFID. The 1st~4th row FD values are computed from the 1st~4th row images of Figure 16.

Weed FD	Crop FD
1.60	1.09
1.57	1.16
1.58	1.10
1.56	1.11

, which represents the FD value and the spatial distributions of crops and weeds across the field. Specifically, the FD values shown in the 1st–4th rows of Table 17 correspond to the 1st–4th row images displayed in Figure 15. Higher FD values observed for crops and weeds indicate greater complexity, suggesting that agricultural experts or autonomous vehicle systems should focus more on accurately distinguishing between them. This estimation technique also supports precision agriculture by enabling targeted weed removal through selective spraying in areas with dense or complex weed growth, ultimately contributing to increased crop yield. In segmentation tasks [8,22,68], it shows the real contributions of FD estimation. Fractal derivative analysis provides a useful approach for quantifying the complexity and irregularity of medical and agricultural imaging data and is also utilized in various other applications.

Figure 15. Visual examples of crops and weeds from the CWFID for estimating FD values: (a) crop and weed segmentation results; (b) weed segmentation result; (c) crop segmentation result.

Figure 15. Visual examples of crops and weeds from the CWFID for estimating FD values: (a) crop and weed segmentation results; (b) weed segmentation result; (c) crop segmentation result.

Figure 16. Sample images with Grad-CAM visualization examples at last 1 × 1 convolutional layer. (a) Input image, (b) ground truth mask (black, gray, and white pixels denote background, weeds, and crops, respectively), (c) Grad-CAM of weed (class 1), (d) Grad-CAM of crop (class 2), and (e) segmented results. Each row presents different examples.

Figure 16. Sample images with Grad-CAM visualization examples at last 1 × 1 convolutional layer. (a) Input image, (b) ground truth mask (black, gray, and white pixels denote background, weeds, and crops, respectively), (c) Grad-CAM of weed (class 1), (d) Grad-CAM of crop (class 2), and (e) segmented results. Each row presents different examples.

3.4. Comparisons of Algorithm Complexities

In this subsection, we evaluate the performance of AHFF-Net by measuring the number of parameters for the model, inference time, floating-point operations (FLOPs), and GPU memory requirements. The inference time was assessed on both desktop computer and the Jetson TX2 embedded system. On the desktop, the inference time was measured using two different GPUs: NVIDIA GeForce GTX 1070 (NVIDIA Corporation, Santa Clara, CA, USA) [69] and NVIDIA GeForce RTX 4080 SUPER (NVIDIA Corporation, Santa Clara, CA, USA) [70]. The Jetson TX2 system powered by an NVIDIA Pascal™-family GPU (NVIDIA Corporation, Santa Clara, CA, USA) with 256 CUDA cores was selected for testing [71]. It had a power consumption of less than 7.5 W and featured 8 GB of shared memory between the GPU and CPU.

The Jetson TX2 was selected for this evaluation because of its relevance in real-world agricultural applications. Crop and weed segmentation algorithms are typically executed onboard farming robots using embedded systems. The desktop computer specifications are outlined in Section 2.1. The inference times measured on the desktop computer and Jetson TX2 embedded system are listed in

Table 18. Inference time of proposed method on desktop and Jetson embedded system (unit: ms).

Methods	Desktop		Jetson Embedded System
	NVIDIA GeForce GTX 1070	NVIDIA GeForce RTX 4080 SUPER
U-Net [22,61]	412.05	41.50	629.12
Modified U-Net [40]	426.51	41.08	472.55
Unet++ [46]	443.83	40.61	502.86
Reduced U-Net [62]	411.98	33.73	453.18
DWUNet [63]	416.37	38.28	428.06
MSFCA-Net [64]	442.06	45.07	1319.61
DeepLAB V3 Plus [65]	456.16	42.43	977.56
SEG-Net [66]	419.40	27.72	446.93
CG-Net [67]	459.82	26.78	583.12
AHFF-Net (Proposed)	417.96	29.75	908.22

. On the desktop equipped with the NVIDIA GeForce GTX 1070 GPU, the proposed AHFF-Net achieved an inference time of 417.96 ms. This was slow compared with reduced U-Net (411.98 ms), U-Net (412.05 ms), and DWUNet (416.37 ms), but fast compared with the other SOTA methods. Using the more effective NVIDIA GeForce RTX 4080 SUPER, AHFF-Net achieved a significantly lower inference time of 29.75 ms. This was marginally slower than SEG-Net (27.72 ms) and CG-Net (26.78 ms), but faster than the other SOTA methods. In the Jetson embedded system, the inference time of AHFF-Net was 908.22 ms. Although it showed a higher inference time than other SOTA methods, it was still competitive compared to other SOTA methods such as DeepLab V3 Plus and MSFCA-Net. This was larger than those of the other SOTA methods. However, it outperformed DeepLab V3 Plus and MSFCA-Net.

Table 19. Comparisons of the number of parameters, GPU memory, and FLOPs of AHFF-Net and the SOTA methods.

Methods	Number of Parameters (Unit: Mega)	GPU Memory Requirements (Unit: MB)	FLOPs (Unit: G)
U-Net [22,61]	31.04	3352	378.10
Modified U-Net [40]	31.03	3971	437.79
Unet++ [46]	26.08	3426	147.75
Reduced U-Net [62]	7.76	1906	96.25
DWUNet [63]	3.26	1571	63.12
MSFCA-Net [64]	30.88	6617	523.96
DeepLAB V3 Plus [65]	54.6	8137	201.6
SEG-Net [66]	29.44	6647	339.6
CG-Net [67]	0.503	962	30.055
AHFF-Net (Proposed)	47.88	5271	287.71

compares the GPU memory requirements, numbers of parameters, and FLOPs of the proposed AHFF-Net with those of the other SOTA methods. Although AHFF-Net had a larger number of parameters than most SOTA methods, it had fewer than DeepLab V3 Plus. Similarly, its GPU memory requirements were lower than those of MSFCA-Net, DeepLab V3 Plus, and SEG-Net, but higher than those of the other SOTA methods. U-Net, modified U-Net, MSFCA-Net, and SEG-Net had higher FLOPs than AHFF-Net, whereas the other SOTA methods had lower FLOPs. Although the algorithm complexities of the proposed method were not the lowest (Table 18 and Table 19), the segmentation accuracies of our AHFF-Net were higher than those of all the SOTA methods (Table 11, Table 12, Table 13, Table 14, Table 15 and Table 16). Moreover, there is always a tradeoff between performance and computational requirements, so we managed it by prioritizing the performance.

4. Discussion

We performed Student’s t-test [72] for statistical analysis and calculated both p-value and Cohen’s d-value [73]. For analysis, we calculated the mean and standard deviation of the F1 score. The F1 scores of the proposed AHFF-Net and second-best U-Net [61] were obtained from Table 11 and Table 13, respectively, to compute the mean and standard deviation. We achieved a p-value of 0.038. This verified that our method outperformed the second-best U-Net at the 95% confidence level. In general, Cohen’s d-value indicates the effect size. Herein, 0.2 indicates a small effect size, 0.5 shows a medium effect, and ≥0.8 denotes a large effect. The measured Cohen’s d-value was 2.17. It reflected a large effect size. Based on these results, we verified that the proposed AHFF-Net statistically outperformed the other SOTA methods in terms of semantic segmentation accuracy.

We applied gradient-weighted class activation mapping (Grad-CAM) [74] for a more detailed analysis. Grad-CAM visualizes important features in yellow and red (indicating high confidence), whereas unimportant features in blue reflect a lower confidence in the predictions. Figure 16 presents the Grad-CAM analysis of the proposed AHFF-Net in the final 1 × 1 CL. This demonstrates a high confidence level in the extraction of important features for crops and weeds in the semantic segmentation task. Grad-CAM images also verified that the proposed AHFF-Net accurately distinguished between crops and weeds in a heterogeneous dataset environment.

Examples of incorrect semantic segmentation results predicted by the proposed AHFF-Net are shown in Figure 17. The incorrect segmentation can be attributed to the similarity in shapes and colors between crops and weeds, which hinders their differentiation, particularly when the objects have thin regions. In addition, the presence of other objects with shapes similar to those of crops and weeds further contributes to segmentation errors. In the first row, although the central crop structure was well identified, the surrounding weeds with similar leaf shapes and orientations were partially misidentified as crops (orange regions), and a few finer weed structures faded into the background (black). This indicates that even with attention mechanisms, it may be challenging for the model to maintain clear boundaries in complex arrangements. Besides this similarity in the shapes and colors of crops and weeds, there was also a similarity in their appearance, as shown in Figure 17. Our proposed method still provided satisfactory segmentation performance. The second row shows a case where the low contrast between the plant and soil (upper right part of the image) resulted in inferior segmentation. The model successfully captured the main crop in the lower part of the image. However, many small weed patches were missing. This was likely owing to the weak feature information and insufficient contextual clues from the surroundings. In the third row, the non-vegetative elements shown in cyan introduced additional confusion. Although the crop was mostly detected correctly, parts of the background were misclassified (gray regions), and several peripheral weeds were mislabeled as crops or left unsegmented. This indicates that unobserved or irregular objects in the field can disturb the learned class boundaries of a model. Moreover, when crops and weed plants overlap, it causes lower performance and also leads to misclassification. This is the common limitation among all the SOTA methods.

In our study, we utilized LLaMA [75] to recommend pesticides for early-stage weed management and introduced a data-driven approach to improve decision-making (

Table 20. Pesticide recommendations for early-stage weeds by LLaMA.

Input Image	Segmentation Mask	Weed Morphology (Early Stage)	Weed Name	Pesticides
		Narrow and pointed leaves that grow in clusters from the base, spreading outward in a pattern	Barnyard Grass (Echinochloa crus-galli)	Atrazine Pendimethalin Sulfometuron Glyphosate *
		Thin, branching stems with narrow, deeply lobed leaves forming a spreading growth habits	Knotweed (Polygonum aviculare)	Imazapyr Triclopyr * Dicamba * Glyphosate *
		Small, oval-shaped green leaves with smooth edges, arranged in opposite pairs on delicate stems	Chickweed (Stellaria media)	2,4-D Dicamba * Sulfometuro Triclopyr *

* Pesticides with * sign are generally used for multiple weeds.

). Unlike conventional methods that depend substantially on manual observations and predefined guidelines, LLaMA enables the analysis of weed species profiles, environmental conditions, and pesticide efficacy records. Our observations showed that the LLaMA effectively identified suitable pesticides with enhanced accuracy. This would help farmers make informed decisions, minimize environmental risks, and promote sustainable agricultural practices. These systems can be utilized by non-agricultural experts to determine the appropriate pesticide using the LLaMA model, input images, and segmentation mask by our method in real applications. However, if we feed only input images to LLaMA, it perhaps fails to differentiate crops and weeds owing to the same attributes, such as shape, color, and overlapping portions. Ultimately, it leads to the wrong pesticide recommendations. The segmentation output of the proposed method helps LLaMA in distinguishing and identifying pixels belonging to weeds and crops, resulting in accurate pesticide recommendations by LLaMA. Our proposed system also benefits the practitioners with in-field weed control and optimizes the pesticide usage with enhanced accuracy. In this study, the proposed method shows satisfactory performance using three different datasets with extensive variability. The proposed method can be integrated into a vision-based robot-assisted system to discriminate between crops and weeds. Although our framework is tested with limited data, it is scalable for large datasets and other applications. Therefore, the proposed method is expected to better cope with field variability and scalability.

5. Conclusions

This study introduced the AHFF-Net. It is a novel method that incorporates small amounts of training data and patch-based augmentation to enhance the accuracy of semantic segmentation for crops and weeds. Three open databases of crops and weeds were used to evaluate the performance of AHFF-Net. Thereby, it was verified that our method outperformed the SOTA methods. The architecture of AHFF-Net featured the PERB, HMFB, and AFEB. When combined, these improved feature extraction by capturing fine details such as edges and textures. These blocks also enabled the model to learn richer and more diverse feature information, capture fine-grained patterns, and apply an attention mechanism to critical regions within crops and weeds in heterogeneous database environments. In addition, we integrated an FD estimation approach into our system to provide valuable insights into the distributional characteristics of crops and weeds. This also confirmed the AHFF-Net’s ability to effectively extract essential features for distinguishing between them. We verified through a t-test and Cohen’s d-value that our proposed AHFF-Net outperformed previous methods. Additionally, we validated the method’s capability to extract key features essential for the accurate segmentation of crops and weeds using Grad-CAM. We also verified that our method can be applied to the Jetson TX2 embedded system for farming robots. Furthermore, we integrated LLaMA to generate pesticide recommendations specific to the types of weeds identified in the segmented images. Notwithstanding the presence of a few segmentation inaccuracies (mainly owing to the visual similarities between crops and weeds, overlapping thin structures, and background clutter), the LLaMA demonstrated the capability to provide context-aware and effective recommendations. The segmentation results showed key challenges in which crops and weeds shared similar shapes, colors, and thin structures. This resulted in boundary ambiguity and inaccurate predictions. Even with attention mechanisms, it was challenging for the model to address fine weed details, low-contrast regions, and irregular non-vegetative elements that disrupted class boundaries. These issues emphasized the limitations of the model in capturing complex spatial and contextual variations under real field conditions.

In future studies, we plan to focus on pre-processing methods to address these issues and consider more diverse datasets. We will also evaluate generative adversarial networks and diffusion model-based techniques to improve the limitation of small and thin crops and weeds with similar shapes and colors in heterogeneous data environments for better segmentation performance. Additionally, we will study methods for creating more lightweight models with low computational requirements and parameters while maintaining accuracy based on the knowledge distillation method.