FFEDet: Fine-Grained Feature Enhancement for Small Object Detection

Abstract

Small object detection poses significant challenges in the realm of general object detection, primarily due to complex backgrounds and other instances interfering with the expression of features. This research introduces an uncomplicated and efficient algorithm that addresses the limitations of small object detection. Firstly, we propose an efficient cross-scale feature fusion attention module called ECFA, which effectively utilizes attention mechanisms to emphasize relevant features across adjacent scales and suppress irrelevant noise, tackling issues of feature redundancy and insufficient representation of small objects. Secondly, we design a highly efficient convolutional module named SEConv, which reduces computational redundancy while providing a multi-scale receptive field to improve feature learning. Additionally, we develop a novel dynamic focus sample weighting function called DFSLoss, which allows the model to focus on learning from both normal and challenging samples, effectively addressing the problem of imbalanced difficulty levels among samples. Moreover, we introduce Wise-IoU to address the impact of poor-quality examples on model convergence. We extensively conduct experiments on four publicly available datasets to showcase the exceptional performance of our method in comparison to state-of-the-art object detectors.

1. Introduction

The objective of object detection is to classify and precisely locate objects, with small object detection (SOD) representing a distinct subdomain that emphasizes the identification of small-sized objects [1]. SOD finds extensive applications in various domains, including natural resource surveys, environmental monitoring, disaster management, agriculture, autonomous driving, and military applications. In recent years, there has been significant advancement in the domain of small object detection, thanks to the utilization of convolutional neural networks (CNN) in deep learning approaches. However, despite these exciting advancements, SOD continues to present challenges in object detection research. For example, the state-of-the-art detector YOLOv7 [2] achieved an average precision (mAP) of 56.0 and 66.7 on medium and large instances in the COCO test set. Still, its performance dropped significantly to 35.2 when dealing with small objects. This performance gap can be attributed not only to the typical obstacles encountered in general object detection, including occlusions and lighting variations, but also to specific issues inherent to SOD. Specifically, existing feature extractors [3,4,5,6,7,8] have limitations in representing features for small objects, as conventional downsampling operations result in the deprivation of details associated with small objects, and small object features are susceptible to contamination from backgrounds and other instances, making it challenging for networks to capture the discriminative information required for subsequent tasks. Furthermore, the problem of sample imbalance between easy and difficult examples causes models to overlook the learning of regular samples and challenging ones during the optimization process, further affecting SOD performance. Small objects also exhibit lower tolerance to bounding box perturbations compared to larger objects, which can impact the model’s convergence. Overcoming the challenges of small object detection requires overcoming these limitations, enhancing feature representation quality, and reducing interference from other factors.

Currently, in two-stage detection algorithms (e.g., ref. [9,10,11]), the process of object detection is executed through iterative refinement steps. Initially, these algorithms utilize a specially designed region proposals network (RPN) to generate high-quality candidate boxes. Subsequently, these candidate regions undergo classification and position regression. In contrast, one-stage detection algorithms conduct detection in a solitary step, showcasing enhanced computational efficiency by eliminating the requirement for an RPN. One-stage object detection algorithms are mainly divided into SSD ([12,13]) and YOLO ([5,14,15,16,17,18]) as the mainstream. The YOLO series is widely recognized in industrial applications for its impressive balance of detection accuracy and speed. For instance, ref. [14] introduced DarkNet-19 as a backbone network and integrated batch normalization preprocessing, higher-resolution classifiers, multi-scale training mechanisms, and binary cross-entropy loss functions, resulting in significant improvements in recall and accuracy. Nonetheless, there are still possibilities for further improving the detection capability of small objects. The deeper DarkNet-53 residual network was adopted in their work [5], along with an FPN (feature pyramid network), more scales of feature maps, and additional anchor points to enhance small object detection performance. Nevertheless, this led to an increase in model complexity. Ref. [19] introduced a cascade query strategy aimed at eliminating redundant computations on low-level features. Ref. [16] incorporated CSPDarkNet53, SPP+PAN fused features, data augmentation, and DropBlock regularization, resulting in significantly improved detection accuracy while maintaining speed. Ref. [20] proposed an efficient rep-style Gaussian–Wasserstein network (ERGW-net), which effectively addressed the challenges of small object sizes and low contrast in infrared aerial imagery. Ref. [17] introduced mosaic data augmentation, focus, CSP structure, and GIoU loss functions, offering fast inference speed and powerful detection performance. Incorporating attention mechanisms at the pixel and channel levels [21], this design aimed to enhance the feature information of small objects, simultaneously suppressing background noise. Furthermore, ref. [22] proposed a novel metric, dot distance, to mitigate the sensitivity of IoU (intersection over union) boundary box losses. Additionally, ref. [23] proposed NWD (normalized Wasserstein distance) and ref. [24] proposed a novel loss function that does not require additional computation or time cost during inference, thereby enhancing the localization accuracy of small objects, which improves the detection capability for small objects.

To tackle the challenges posed by complex background interference, diverse object scales, and performance bottlenecks of existing detectors in handling small objects, we introduce fine-grained feature enhancement for small object detection (FFEDet) within a comprehensive architectural framework depicted in Figure 1. Small objects often become obscured within intricate backgrounds, making it challenging for single-scale feature extraction to fully capture these targets. To efficiently fuse features across different scales, we propose the effective cross-scale feature fusion attention (ECFA) module. Leveraging attention mechanisms, this module adeptly integrates features from various scales, better capturing the details and contextual information of small objects while emphasizing relevant features and suppressing irrelevant noise. This cross-scale fusion enriches feature representation and enhances the model’s robustness to complex backgrounds. Another critical challenge in small object detection is maintaining efficient computation while boosting feature extraction capability. Thus, we introduce the simple yet efficient convolution (SEConv) module, which reduces computational redundancy and provides multiscale receptive fields, thereby enhancing feature learning capacity. During training, due to uneven sample distribution, the contributions of different samples to the model should dynamically vary. To address this, we propose the dynamic focal sample weighting function (DFSLoss), which dynamically adjusts the focus on optimizing normal and challenging samples during model training. By reducing the impact of simple samples on loss, this function enhances algorithm effectiveness. Furthermore, in the detection process, low-quality samples can adversely affect model performance. Hence, we incorporate Wise-IoU as the localization loss function to alleviate the sensitivity of detection boxes to IoU. Implementing this strategy helps mitigate the negative impact of low-quality samples, facilitating better model convergence.

This paper offers the following key contributions:

• We present the efficient cross-scale information fusion attention (ECFA) module, which efficiently fuses information across different scales through attention mechanisms. This module effectively reduces feature redundancy while improving the representation of small objects;
• We develop SEConv, a simple and highly efficient convolutional module that effectively reduces computational redundancy and provides multi-scale receptive fields, resulting in enhanced feature learning capabilities;
• We design DFSLoss, a dynamic focal sample weighting function, to overcome the issue of imbalanced hard and easy samples and improve network model optimization. Moreover, we introduce Wise-IoU to alleviate the negative effects of poor-quality examples on model convergence.

2. Related Work

2.1. Scale-Aware Methods

In image processing applications, objects typically exist at various scales, especially in fields such as traffic monitoring and remote sensing, posing significant challenges for single detectors. Traditional handcrafted feature-based methods often perform poorly in detecting small objects due to their limited feature representation capability. Early deep learning detection methods, primarily relying on high-level features, similarly struggle to effectively capture the details of small objects. To address this issue, multiscale detection strategies have been developed to enhance the recognition of small objects. For example, ref. [25] assumed that information within regions of interest (RoI) can be distributed across different layers of the backbone network, necessitating an effective organization strategy. By combining and integrating features from larger to smaller scales, this method obtains hyper-features that retain the capability to identify small objects. However, this approach may perform poorly in complex backgrounds, as the fusion of features from different layers can introduce extraneous background information. Ref. [26] employed scale-dependent pooling (SDP) to determine the optimal feature layers for aggregation. Although this method can adaptively select the feature layers best suited for small objects, it may lead to information loss during the multiscale feature aggregation process. This issue is particularly pronounced in scenes with objects of varying scales, potentially resulting in reduced detection performance. Ref. [27] generates detection proposals at various intermediate layers, with each layer specializing in objects within a specific scale range. While this method optimizes sensitivity to small objects, the primary issue is the lack of feature sharing between layers, resulting in insufficient overall feature representation. When objects in a scene exhibit significant scale variation, insufficient coordination between layers may affect detection performance. Ref. [12] proposed identifying small targets on high-resolution feature maps, leveraging the rich detail contained within these maps. The main problem with this approach is that high-resolution feature maps are susceptible to noise and background information during processing, leading to increased false detection rates. Additionally, the generation and processing of high-resolution feature maps demand high data quality, and low-quality images can severely impact detection performance. Ref. [15] adopted parallel branches for multiscale prediction, utilizing high-resolution feature maps to handle small objects. The main issue with this method is inadequate feature fusion and information sharing between branches, potentially resulting in poor coordination of the overall detection system. Moreover, the design of parallel branches requires careful consideration of each branch’s feature extraction capability and detection accuracy to avoid performance imbalances between branches. Inspired by [28,29], a connection was established between RoIs and a fusion of pooled features at various scales along with global features. This approach effectively combines local and global information, enhancing the robustness of small object detection. However, the fusion of features at different scales requires highly precise design and tuning, with extensive experimentation needed to determine appropriate weight distribution and fusion strategies. Additionally, the selection of RoIs and the precise matching of pooled features are critical issues; any misalignment may lead to suboptimal detection performance. In summary, although various multiscale detection strategies have significantly improved small object detection performance, challenges remain in feature fusion, information sharing, noise filtering, and scene adaptability.

2.2. Feature Fusion Methods

Deep convolutional neural networks (deep CNNs) generate hierarchical feature maps with varying spatial resolutions. These networks leverage low-level features to capture fine details, thereby providing accurate localization cues, while high-level features extract semantic information, enhancing representation. However, downsampling layers in deep feature maps can diminish their responsiveness to small objects. To address this issue, recent research has adopted feature fusion strategies that integrate multi-level features to obtain higher-quality representations for small objects. For instance, ref. [30] employed a hierarchical approach with lateral connections to combine features at different scales, capturing fine-grained details and semantic information to enhance small object detection. While this method effectively integrates features from different levels, it may struggle in complex scenes where lateral connections fail to adequately fuse low and high-level features, resulting in decreased detection accuracy. Additionally, this approach requires the precise alignment of feature maps, and misalignment can lead to significant performance issues. Building on FPN, ref. [28] introduced a bottom-up pathway to address gradient inconsistency issues found in FPN-based methods, thereby enhancing the representational capacity of lower-level features. Although this improvement enhances gradient propagation and strengthens low-level feature representation, it may introduce noise and irrelevant information in highly dense scenes, negatively impacting overall detection performance. Moreover, ref. [9] aggregated features at multiple spatial scales to accommodate objects of different sizes. While this method improves detection performance across various scales, it can encounter feature confusion when dealing with objects that have significant scale variation, especially when objects fall near the boundary of the fused features’ scales. In summary, feature fusion methods have made significant strides in mitigating the spatial and semantic discrepancies between low and high-level pyramid layers. However, challenges remain in capturing the fine-grained details crucial for the accurate localization and classification of small objects. Issues such as background noise, feature misalignment, and feature confusion in complex scenes continue to pose problems. Future research needs to optimize feature fusion strategies further to ensure stable and efficient small object detection across diverse and complex scenarios.

2.3. Context Modeling Methods

Context modeling methods aim to enhance the performance of salient object detection (SOD) by integrating contextual information and leveraging the correlation between objects and their surrounding environment to boost neural networks’ discriminative ability. For example, ref. [31] employed dilated convolution operations to expand the receptive field, acquiring richer contextual information, albeit susceptible to background interference in dense scenes. Additionally, ref. [32] proposed a method utilizing spatial context to compute inter-class and intra-class distances among different object instances. While successful in SOD, it may struggle to adapt to complex scene dynamics. Attention mechanisms capture contextual details but may have limitations in handling positional information, as discussed in [33]. On the other hand, ref. [34] combined channel and spatial attention mechanisms, making them easily incorporable into various convolutional neural networks. Its structural depiction is presented in Figure 2. Traditional convolutional operations may not capture local relationships effectively, especially in tasks requiring long-range dependencies. Ref. [35] addressed this by embedding spatial coordinate information into channel attention mechanisms, enhancing focus on informative regions while suppressing background interference. However, this method risks information loss when modeling long-range dependencies for small objects. In summary, context modeling methods hold promise for improving small object detection performance by leveraging global or local context cues. However, careful consideration of their strengths, limitations, and applicability to specific scenarios is essential during selection and optimization.

2.4. Loss Functions

In object detection networks, object localization accuracy is typically assessed using IoU. However, in cases where there is no overlap between the two bounding boxes, IoU loss can lead to gradient vanishing, affecting the model’s effective updates. To address this issue, a series of improved loss functions have been introduced. Ref. [36] proposed ratio-balanced angle loss, enabling better balance between low and high-aspect ratio objects, avoiding redundant designs, and enhancing small object detection quality. Despite its effectiveness in handling small objects with varying aspect ratios, this method may be susceptible to background noise in complex environments, potentially increasing false detections. Ref. [37] introduced the concept of the minimum bounding box to alleviate the issue of gradient vanishing associated with IoU loss, enhancing training stability by providing a more accurate representation of object bounding boxes. Additionally, ref. [38] proposed incorporating a distance factor to accelerate model convergence, though it may have limitations in handling objects of varying scales. Furthermore, ref. [39] considered the aspect ratio associated with anchor boxes, leading to enhanced detection performance, particularly for objects with diverse aspect ratios. However, the presence of low-quality instances in the dataset unavoidably introduces significant discrepancies in both the distance and aspect ratios. These factors may excessively penalize poor-quality examples, diminishing the algorithm’s generalization capability. To overcome this challenge, we integrate the bounding box loss function proposed by [40] into our model. This approach effectively balances the treatment of instances with varying qualities, thereby improving the model’s robustness and generalization ability.

3. Methods

This section presents a detailed overview of our proposed method, FFEDet. We use YOLOv7, a well-known anchor-based one-stage detector, as the foundation for presenting our approach. It is important to emphasize that our method is not only applicable to YOLOv7 but also effectively utilized in both single-stage and two-stage detectors. In Section 3.1, we will revisit YOLOv7 to establish a foundation. Subsequently, Section 3.2 and Section 3.3 will delve into the structures of ECFA and SEConv. Section 3.4 will introduce the dynamic focus sample weighting function (DFSLoss), while the similarity measurement methods will be described in Section 3.5.

3.1. Revisiting YOLOv7

YOLOv7, which represents a progression in the YOLO series, has exhibited remarkable efficiency and precision in comparison to various object detection methods. The model mainly comprises three parts: a backbone to extract features, a neck structure to fuse these features, and a head structure to output the detection results. YOLOv7 incorporates several innovative technologies to enhance its performance, including an SPPCSPC module, reparameterized convolutions, efficient layer aggregation networks, and dynamic label assignment. Figure 3 illustrates the structure of the SPPCSPC module. First, the input feature map undergoes a 1 × 1 convolution operation (Conv1 × 1), followed by a 3 × 3 convolution operation (Conv3 × 3), and then another 1 × 1 convolution operation. These convolution operations generate intermediate feature maps. Next, these feature maps are processed through three max-pooling layers of different sizes (MP5 × 5, MP9 × 9, and MP13 × 13) for multi-scale pooling. The pooled feature maps are concatenated with the original feature map, creating a higher-dimensional feature map. This high-dimensional feature map then undergoes further processing with another series of 1 × 1, 3 × 3, and 1 × 1 convolution operations. Finally, the processed feature maps are concatenated again, producing the enhanced output feature map. In the realm of YOLOv7, including variants like YOLOv7-tiny and YOLOv7-X, our primary focus is on standard YOLOv7 as the reference model. We place special emphasis on the effectiveness and capability to operate in real time.

3.2. Efficient Cross-Scale Information Fusion Attention

In object detection tasks, objects exhibit variations in size and shape across different scales. Consequently, cross-scale feature fusion plays a pivotal role in enhancing the understanding of objects within images and refining detection accuracy. By amalgamating features across different scales, the representational capacity of features is fortified, thereby fostering improved model generalization and robustness. Particularly noteworthy is the escalating depth of the model, exacerbating the challenge of fine-grained detail loss, especially pronounced in detecting small objects, thereby complicating the detection process. The efficient cross-scale information fusion attention (ECFA) module employs an efficient strategy for cross-scale feature fusion, effectively integrating multi-scale information through upsampling, downsampling, and concatenation of feature maps from different scales. This enables the model to dynamically adapt to the diverse scales inherent in object detection tasks. Attention mechanisms, initially successful in natural language processing, have now found broad application across various domains, including computer vision. These mechanisms aim to enhance model performance by guiding the model’s attention towards salient information, mitigating noise, and facilitating contextual modeling and information interaction across different scales of feature maps. Consequently, the ECFA module utilizes attention mechanisms for cross-scale feature fusion and interaction, in conjunction with SAM and CAM, to capture crucial spatial and channel information, thereby amplifying the model’s discriminative capability and accuracy. Figure 4 illustrates the structure of ECFA. Firstly, we independently apply convolutional layers to feature maps $P_{i-1}$ ($W_{i-1}$, $H_{i-1}$, $C_{i-1}$), $P_i$ ($W_i$, $H_i$, $C_i$), and $P_{i+1}$ ($W_{i+1}$, $H_{i+1}$, $C_{i+1}$) obtained from three adjacent scale levels. Subsequently, we perform downsampling and upsampling operations on $P_{i-1}$ and $P_{i+1}$, resulting in three feature maps of equal shape: $P_{i-1}$ ($W_i$, $H_i$, C), $P_i$ ($W_i$, $H_i$, C), and $P_{i+1}$ ($W_i$, $H_i$, C). Next, we perform element-wise addition and channel-wise concatenation on these feature maps, obtaining feature maps of sizes ($W_i$, $H_i$, C) and ($W_i$, $H_i$, $3C$). Then, we apply spatial attention and channel attention operations separately. Finally, we combine the results using weighted fusion to extract more comprehensive object information. This module enhances the representation capability for small objects by effectively focusing on different feature maps. The spatial and channel attention operations can be mathematically represented as follows:

$$\begin{array}{cc}\hfill M_s\left(F\right)& =\sigma \left(f^{7\ast 7}\left([AvgPool\left(F\right),MaxPool\left(F\right)]\right)\right)\hfill \\ \hfill & =\sigma \left(f^{7\ast 7}\left([F_{avg}^S;F_{max}^S]\right)\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill M_c\left(F\right)& =\sigma \left(MLP\right(AvgPool\left(F\right))+MLP(MaxPool\left(F\right)\left)\right)\hfill \\ \hfill & =\sigma (W_1\left(W_0\left(F_{avg}^C\right)\right)+W_1\left(W_0\left(F_{max}^C\right)\right)),\hfill \end{array}$$

(2)

where F represents the input feature. In the spatial attention operation, F undergoes channel-wise average and max pooling, then concatenation along the channel dimensions, and passes sequentially through the convolutional layer followed by the Sigmoid activation function, generating weight $M_s$. Finally, the original feature F is multiplied by these weights to acquire the updated feature. Next, we apply the spatial average and max pooling to F during the channel attention operation. Then, the outcomes are individually passed through a two-layer neural network using ReLU activation. Parameters in this neural network are shared among its layers. Then, the two resulting features are summed, and the Sigmoid activation function is used to obtain the weights $M_c$. Ultimately, the updated feature emerges from the multiplication of the weight with the initial feature F.

3.3. Simple and Efficient Convolution Module

To enhance the feature learning capability of the model, ref. [41] introduced the CSP module, which can be integrated with the residual modules in ResNet to become a core structure in many mainstream detectors. Figure 5a provides a detailed illustration of the CSP module. The input undergoes two branches, with one employing a recurrent, residual structure for multiple iterations. Subsequently, the outputs from both branches are concatenated along the channel dimension. In ref. [2], ELAN and E-ELAN abandoned the dual-branch structure, as depicted in Figure 5b. In the ELAN module, feature fusion is achieved by integrating the output of each stacked module layer, while in the E-ELAN model, feature fusion is accomplished by incorporating the output of each convolutional layer within the stacked module. This allows the feature maps to gather information from multiple receptive fields, enhancing the model’s capability to extract features at various scales. However, standard 3 × 3 convolutional layers extract a significant portion of redundant features and introduce a considerable number of parameters and FLOPs. Therefore, we propose the simple and efficient convolution (SEConv) module, which reduces redundant computations, promotes the learning of representative features, and provides a multi-scale receptive field. The structure of SEConv is illustrated in Figure 6a. Experimental results in

Table 1. Performance comparison between Conv and SEConv.

Method	Params(M)	FLOPs	Input	Output
Conv(3 × 3)	0.59	24.2	(1,256,640,640)	(1,256,640,640)
SEConv	0.21	8.6	(1,256,640,640)	(1,256,640,640)

demonstrate that SEConv exhibits notable advantages in terms of parameter count and computational complexity compared to standard 3 × 3 convolutional layers. We extend SEConv into the ELAN framework, proposing the S-ELAN module. As shown in Figure 6b, S-ELAN represents an enhanced version derived from the ELAN and E-ELAN structures. S-ELAN partitions the input feature map along the channel dimension into two equally shaped segments, each directed into separate branches. Subsequently, S-ELAN replaces the stacked standard convolution with the SEConv module proposed in this study. Finally, the resulting feature maps from each layer are concatenated along the channel dimension, and channel dependencies are extracted through a standard convolution with a 1 × 1 kernel size. The results in Table 5 illustrate that the S-ELAN module maintains excellent detection performance while simultaneously reducing the model’s computational complexity.

3.4. Dynamic Focal Sample Weighting Function

Regarding the IoU based on the prediction box to ground truth box, we categorize samples into two classes: easy and hard. Typically, easy samples have a more pronounced impact on the loss function, which can make it challenging for the model to effectively handle hard samples. To tackle this challenge, we devise an innovative, dynamic focal sample weighting function, referred to as DFSLoss.

$$f\left(x\right)=\left\{\begin{array}{cc}1\hfill & x⩽\mu -0.1\hfill \\ e^{1-\mu }\hfill & \mu -0.1<x<\mu +0.1\hfill \\ e^{1-x}\hfill & x⩾\mu +0.1,\hfill \end{array}\right.$$

(3)

which empowers the model to prioritize its attention on samples that are closer to the threshold boundaries, both normal and hard, by assigning higher weights to them during training. Consequently, the model has the capability to minimize the influence of simple samples in the model optimization process and more efficiently learn from normal and hard samples. To enhance the training stability of the model, we employ an exponential moving average (EMA) method to optimize IoU values between all bounding boxes, using their average as the threshold $\mu$. Based on this threshold, samples are classified as negative (an IoU less than $\mu$) or positive (an IoU greater than $\mu$). However, due to the errors introduced by classification, samples near the threshold often suffer more substantial losses. Therefore, we aim for the model to more comprehensively learn from and optimize these samples near the threshold, leading to improved network training.

3.5. Similarity Measurement

In object detection, bounding box regression is the key to accurately locating objects. The IoU loss function ensures that prediction boxes closely align with the actual objects, ultimately leading to improved accuracy in object localization.

$$IoU=\frac{\mathrm{Intersection}}{\mathrm{Union}}=\frac{A\cap B}{A\cup B},$$

(4)

where A represents the prediction boxes, while B represents the ground truth boxes. YOLOv7 employs the CIoU loss function.

$$L_{CIoU}=1-IoU+\frac{{(x-x_{gt})}^2+{(y-y_{gt})}^2}{({W_g}^2+{H_g}^2)}+\alpha \nu ,$$

(5)

where $\alpha$ is used to balance the different terms within the loss function, and $\upsilon$ quantifies the aspect ratio consistency with the anchor box.

$$\upsilon =\frac{4}{\pi }{\left(arctan\frac{w_{gt}}{h_{gt}}-arctan\frac{w}{h}\right)}^2,$$

(6)

$$\alpha =\frac{\upsilon }{(1-IoU)+\upsilon },$$

(7)

where x, y, w, h, $x_{gt}$, $y_{gt}$, $w_{gt}$, and $h_{gt}$, respectively, denote the center coordinates and dimensions of the predicted box and the corresponding ground truth box. Moreover, $W_g$ and $H_g$ indicate the dimensions of the minimum anchor box that encompasses both the predicted box and the ground truth box. The CIoU loss function considers various factors, such as the center point distance or the width-to-height proportion of bounding boxes. However, training data may contain low-quality examples with issues like background noise, inconsistent aspect ratios, and other geometric factors. These issues can adversely affect the model’s training, especially when it comes to addressing geometric factors. To mitigate these problems, we introduce Wise-IoU [40], which helps to reduce the penalty for geometric factors when anchor boxes and object boxes have good overlap, thus reducing interference from training data.

$${\mathrm{L}}_{WIoUv1}={\mathrm{R}}_{WIoU}(1-IoU),$$

(8)

$${\mathrm{R}}_{WIoU}=\exp (\frac{{(x-x_{gt})}^2+{(y-y_{gt})}^2}{{({W_g}^2+{H_g}^2)}^{*}}),$$

(9)

where $R_{WIoU}$ has the capability to magnify the IoU loss for common quality anchor boxes while reducing high-quality anchor boxes. When there is a lot of overlap between the bounding box and the object box, $R_{WIoU}$ pays more attention to the distance between their centers. To prevent potential gradient issues introduced by $R_{WIoU}$, we separate $W_g$ and $H_g$ (indicated by ${ }^{*}$). This separation effectively eliminates factors that might hinder convergence without the need to introduce new metrics like aspect ratio. WIoUv2 constructs a monotonic focusing factor.

$$L_{WI\mathrm{oUv}2}={\left(\frac{L_{I\mathrm{oU}}^{*}}{\overline{L_{I\mathrm{oU}}}}\right)}^{\gamma }{L}_{WI\mathrm{oUv}1},\gamma >0,$$

(10)

where $\overline{L_{IoU}}$ represents an exponentially moving average with momentum $m,$ utilized to address the problem of diminished convergence rate in the latter phases of model. We reflect anchor box quality by defining outliers, with high-quality anchor boxes having smaller outliers. By matching them with smaller gradient gains, it can effectively mitigate larger harmful gradients caused by low-quality samples. WIoUv3 introduces a non-linear focus factor, $\beta$, to dynamically adjust the gradient amplification allocation strategy.

$$L_{WI\mathrm{oUv}3}=\frac{\beta }{\delta {\alpha }^{\beta -\delta }}{L}_{WI\mathrm{oUv}1},$$

(11)

$$\beta =\frac{L_{I\mathrm{oU}}^{*}}{\overline{L_{I\mathrm{oU}}}}\in [0,+\infty ).$$

(12)

By allowing $L_{IoU}$ to be dynamic and accommodating different standards for evaluating the quality of anchor boxes, WIoUv3 is able to adapt to various anchor box quality assessment standards. This fine-tuning helps to reduce the detrimental impacts of varying anchor box quality during training.

4. Experiments

4.1. Implementation Details

In this study, we employed an NVIDIA RTX A6000 GPU for model training and implemented the algorithms using the PyTorch 1.8 framework. Training was conducted for 150 iterations on each dataset, with each iteration taking approximately 50 s, resulting in a total training time of roughly 2 h. For specific details regarding the experimental setup, please refer to

Table 2. Experimental environment.

Configuration	Parameter
Operating System	Ubuntu 18.04
GPU	NVIDIA RTX A6000
CUDA	12.2
Frame	PyTorch 2.0.1
Programming Language	Python 3.8

.

We adopted YOLOv7 as our baseline model, configuring the input image size to 640 × 640, and employed SGD optimization with momentum. During the initial three iterations of training, the momentum value was set to 0.8 and subsequently adjusted to 0.937. The model was initialized with a learning rate of 0.01, which was decreased by a factor of 0.1 during the 50th and 100th iterations. To mitigate overfitting, we applied data augmentation techniques, including mosaic augmentation and random cropping, and incorporated a weight decay factor of 0.0005. Speed and accuracy tests were conducted with a batch size of 1. The IoU threshold for non-maximum suppression (NMS) was set to 0.5.

4.2. Datastes

To evaluate the superiority of our suggested approach, four demanding public datasets were chosen for this paper: VisDrone-DET2021 [42], TGRS-HRRSD [43], DOTAv1.0 [44], and PASCAL VOC [45]. As shown in Figure 7, we showcase a selection of examples taken from these four public datasets. The choice of these datasets aimed to cover various application scenarios across different domains, providing a comprehensive evaluation of our method’s performance.

VisDrone-DET2021 dataset: This dataset consists of 10,209 high-quality images with high resolution. It contains 10 different classes of objects with 6471 training, 548 validation, and 3190 testing images. The use of this dataset for evaluating the performance of small object detection models is challenging due to the noticeable imbalances in class distribution and object sizes.

TGRS-HRRSD dataset: This dataset comprises a comprehensive collection of21,761 images encompassing 13 classes of objects, totaling 55,740 object instances. Furthermore, the dataset is partitioned into 5401 training, 5417 validation, and 10,943 testing images. It was created by the Chinese Academy of Sciences in 2019. Image resolutions vary from 150 × 150 to 1200 × 1200, with average aspect ratios ranging from 41.96 to 276.50 pixels per category.

DOTAv1.0 dataset: The DOTAv1.0 dataset comprises aerial images that have been annotated using horizontal bounding boxes. It comprises 2806 images and encompasses 15 object categories. Due to the large original image resolutions, handling them directly can be challenging. A common practice is to crop each image into 640 × 640 patches with a 320-pixel stride, generating corresponding annotations for each cropped image. The processed dataset totals 21,046 images and 188,282 instances, and covers 15 object categories.

PASCAL VOC dataset: Recognized as a extensively utilized benchmark, the PASCAL VOC dataset holds significant importance for object detection tasks within the domain. For this study, we combined Pascal VOC 2007 and 2012 and obtained a consolidated dataset comprising 16,551 training images and 40,058 object objects, as well as 4952 validation images with 12,032 objects.

4.3. Evaluation Metrics

Average precision (AP) is a comprehensive performance measure frequently employed in the field of object detection, utilized to assess the detection capability for different classes. It relies on both precision and recall. Precision (P) can be formulated as follows:

$$\mathrm{P}=\frac{TP}{TP+FP}.$$

(13)

Recall (R) can be defined as follows:

$$\mathrm{R}=\frac{TP}{TP+FN},$$

(14)

where TP, FP, and FN represent the true positive, false positive, and false negative, respectively. The formula for calculating the average precision (AP) can be written as follows:

$$AP={\int }_0^{1}P\left(R\right)dR,$$

(15)

where the mean average precision (mAP) at an IOU threshold of 0.5 is denoted as $mA{P}_{0.5}$. The h within the IOU threshold range from 0.5 to 0.95, with an increment of 0.05, is referred to as $mA{P}_{0.5:0.95}$. The mean average precision is defined as follows:

$$\mathrm{m}AP=\frac{{\sum }_{i=1}^{N}A{P}_i}{\mathrm{N}},$$

(16)

where $A{P}_i$ and N denote the average precision and number of categories for category i. In addition, $A{P}_s$ corresponds to the average precision for objects with an area smaller than ${32}^2$. $A{P}_m$ pertains to objects with an area ranging from ${32}^2$ to ${96}^2$, while $A{P}_l$ pertains to objects with an area larger than ${96}^2$. FPS (frames per second) measures the detection speed of a model in terms of the number of recognizable frames per second and is widely used to evaluate the real-time capability of a model.

4.4. Result and Analysis

In this segment, a full analysis of our experimental findings will be presented. We selected mAP as our primary evaluation metric. The quantitative results are summarized in

Table 3. Experimental results of different detection algorithms on the PASCAL VOC, VisDrone-DET2021, TGRS-HRRSD, and DOTAv1.0 datasets.

Datasets	Method	Params(M)	${\mathbf{mAP}}_{0.5}$	AP	${\mathbf{mAP}}_{0.75}$	${\mathbf{mAP}}_{\mathit{s}}$	${\mathbf{mAP}}_{\mathit{m}}$	${\mathbf{mAP}}_{\mathit{l}}$	FPS(f/s)
PASCAL VOC	Baseline	37.62	84.4 ± 0.2	62.5 ± 0.1	67.8 ± 0.3	25.9 ± 0.2	50.9 ± 0.1	71.4 ± 0.2	68.6
	YOLOv5l	46.20	80.5 ± 0.3	58.8 ± 0.2	62.9 ± 0.4	21.6 ± 0.3	44.0 ± 0.2	66.7 ± 0.3	73.2
	YOLOv8l	43.60	82.7 ± 0.4	62.3 ± 0.3	64.0 ± 0.2	24.9 ± 0.3	48.7 ± 0.2	68.2 ± 0.4	70.5
	Ours	39.52	85.5 ± 0.2	63.8 ± 0.1	69.3 ± 0.3	27.0 ± 0.2	51.7 ± 0.1	73.0 ± 0.2	69.2
VisDrone-DET2021	Baseline	37.62	49.1 ± 0.3	28.0 ± 0.2	27.6 ± 0.2	18.7 ± 0.1	39.1 ± 0.3	49.3 ± 0.2	71.6
	YOLOv5l	46.20	41.4 ± 0.2	24.4 ± 0.1	24.8 ± 0.2	16.1 ± 0.1	34.0 ± 0.2	41.7 ± 0.3	77.4
	YOLOv8l	43.60	42.9 ± 0.3	25.9 ± 0.2	26.4 ± 0.2	17.4 ± 0.1	33.9 ± 0.2	43.1 ± 0.3	60.8
	Ours	39.52	53.9 ± 0.2	32.1 ± 0.1	32.7 ± 0.2	23.2 ± 0.1	42.9 ± 0.2	49.7 ± 0.2	72.7
TGRS-HRRSD	Baseline	37.62	89.8 ± 0.2	68.9 ± 0.1	81.5 ± 0.3	28.7 ± 0.2	58.1 ± 0.1	60.3 ± 0.2	74.5
	YOLOv5l	46.20	89.1 ± 0.3	63.9 ± 0.2	75.0 ± 0.4	30.2 ± 0.3	54.0 ± 0.2	58.7 ± 0.3	74.1
	YOLOv8l	43.60	89.9 ± 0.2	69.4 ± 0.1	80.9 ± 0.3	28.6 ± 0.2	60.4 ± 0.1	62.3 ± 0.2	69.8
	Ours	39.52	91.4 ± 0.2	70.1 ± 0.1	83.1 ± 0.3	31.0 ± 0.2	61.0 ± 0.1	63.4 ± 0.2	69.9
DOTAv1.0	Baseline	37.62	76.7 ± 0.2	51.6 ± 0.1	54.1 ± 0.2	25.4 ± 0.1	51.3 ± 0.2	60.5 ± 0.1	73.4
	YOLOv5l	46.20	73.0 ± 0.3	49.0 ± 0.2	50.9 ± 0.3	20.5 ± 0.2	45.3 ± 0.1	58.6 ± 0.2	68.3
	YOLOv8l	43.60	74.5 ± 0.2	52.9 ± 0.1	56.1 ± 0.2	24.9 ± 0.1	50.4 ± 0.2	57.3 ± 0.1	74.4
	Ours	39.52	78.2 ± 0.2	53.1 ± 0.1	55.0 ± 0.2	26.8 ± 0.1	53.7 ± 0.2	60.9 ± 0.1	70.6

, and visual outcomes are showcased in Figure 8, Figure 9, Figure 10 and Figure 11.

The experimental results outlined in Table 3 delineate the performance metrics obtained by various detection algorithms across four distinct datasets (PASCAL VOC, VisDrone-DET2021, TGRS-HRRSD, and DOTAv1.0). For each dataset, performance parameters of the baseline method (Baseline), YOLOv5l, YOLOv8l, and our proposed method (Ours) are itemized, encompassing the model parameter count (Params(M)), average precision at varying IoU thresholds ($mA{P}_{0.5}$, $mA{P}_{0.75}$), average precision across small, medium, and large objects ($A{P}_s$, $A{P}_m$, $A{P}_l$), and frames per second processing speed (FPS).

Overall, our method (Ours) eclipsed other algorithms across all datasets. Notably, a substantial improvement was observed in the VisDrone-DET2021 dataset. Relative to the YOLOv7 baseline model, our method exhibited enhancements of 4.8% and 4.1% in $mA{P}_{0.5}$ and AP, respectively. Moreover, $A{P}_s$, $A{P}_m$, and $A{P}_l$ experienced boosts of 4.5%, 3.8%, and 0.4%, correspondingly. The significant augmentation in APs underscores the efficacy of our method in detecting and classifying small objects. On the PASCAL VOC dataset, our method achieved improvements of 1.1%, 1.3%, and 1.1% in $mA{P}_{0.5}$, AP, and $A{P}_s$, respectively; on the TGRS-HRRSD dataset, enhancements of 1.6%, 1.2%, and 2.3% were achieved; on the DOTAv1.0 dataset, enhancements of 1.5%, 1.5%, and 1.4% were observed. Our method consistently maintained optimal performance.

Despite not emerging as the top performer in FPS, our method still approached the optimal level. Although our model’s inference speed slightly lagged behind that of competing methods, it still satisfied the real-time detection requirement (i.e., processing over 60 frames per second). Hence, our method strikes a balance between performance and speed, furnishing a viable solution for practical applications. Furthermore, the standard deviation following each performance metric did not exceed 0.3, indicating the stability and reliability of the experimental results. These findings underscore the superiority of our method in object detection tasks and its promising prospects for application.

Furthermore, we present visualizations of the detection results from different algorithms applied to various datasets, as illustrated in Figure 8, Figure 9, Figure 10 and Figure 11. From top to bottom, examples of detections (a), (b), and (c) are presented. Each row displays the detection results from left to right, including the ground truth bounding boxes, the YOLO series algorithms, and our proposed algorithm. From these figures, it can be observed that YOLOv5, YOLOv7, and YOLOv8 are prone to missing and misclassifying small objects in scenes with complex backgrounds and dense occlusions. In contrast, our approach excels at accurately identifying small objects in complex situations, including occlusion, complex backgrounds, and densely populated object environments.

While the FFEDet model excels in object detection tasks, it may encounter limitations when dealing with highly cluttered scenes, necessitating improvements in its robustness. Additionally, enhancements in computational efficiency and generalization capabilities are required to accommodate diverse scenarios and datasets. Future research endeavors should focus on enhancing the model’s performance and applicability, thereby facilitating its widespread adoption in practical applications.

4.5. Ablation Study

To assess the contribution of each module to the model performance, we conducted ablation studies on the VisDrone-DET2021 dataset, with the baseline model being YOLOv7. The experimental results are shown in

Table 4. Ablation study on the VisDrone-DET2021 dataset.

ECFA	SEConv	DFSLoss	WIoUv3	${\mathbf{mAP}}_{0.5}$	AP
				49.1	28.0
√				52.0	30.6
	√			49.7	28.2
		√		50.3	28.6
			√	50.5	29.2
√	√			52.4	30.8
		√	√	51.1	29.7
√	√	√		53.1	29.8
√	√		√	53.5	31.9
√	√	√	√	53.9	32.1

. The baseline (YOLOv7) achieved a $mA{P}_{0.5}$ of 49.1% and an AP of 28.0%. Each module (ECFA, SEConv, DFSLoss, and WIOUv3) exhibited improvements in performance compared to the baseline. Notably, the ECFA module significantly increased $mA{P}_{0.5}$ and AP by 2.9% and 2.6%, respectively. Subsequently, the SEConv module further enhanced $mA{P}_{0.5}$ and AP, with respective increments of 0.4% and 0.2%. Building upon this, the combination of the DFSLoss and WIOUv3 modules raised $mA{P}_{0.5}$ and AP to 53.9% and 32.1%, respectively, representing respective improvements of 4.8% and 4.1% over the baseline. These experimental results emphasize how each proposed module contributes to improving model performance, especially when considering the cumulative effects of ECFA, SEConv, DFSLoss, and WIOUv3. This further validates the efficacy of our approach in SOD tasks.

To verify the superiority of SEConv, we conducted comparative experiments employing multiple widely used convolutional modules [46,47,48,49]. As shown in Table 5, PConv, DWConv, SCConv, and GSConv exhibited a decrease in parameters and computational complexity in comparison to the standard Conv layer, but their performance deteriorated. In contrast, our proposed SEConv achieved lower parameters and computational complexity than the standard Conv layer while demonstrating a noteworthy 0.6% improvement in $mA{P}_{0.5}$.

Table 5. A comparison of the performance of various convolution methods on the VisDrone-DET2021 dataset.

Method	Params(M)	FLOPs	${\mathbf{mAP}}_{0.5}$	AP
Conv(Baseline)	37.62	106.5	49.1	28.0
DWConv	34.12	98.3	48.9	27.4
SCConv	35.36	99.9	48.4	26.8
PConv	30.52	80.3	48.5	27.3
GSConv	29.54	77.1	48.1	26.9
SEConv(Ours)	35.84	102.8	49.7	28.2

Table 5. A comparison of the performance of various convolution methods on the VisDrone-DET2021 dataset.

Method	Params(M)	FLOPs	${\mathbf{mAP}}_{0.5}$	AP
Conv(Baseline)	37.62	106.5	49.1	28.0
DWConv	34.12	98.3	48.9	27.4
SCConv	35.36	99.9	48.4	26.8
PConv	30.52	80.3	48.5	27.3
GSConv	29.54	77.1	48.1	26.9
SEConv(Ours)	35.84	102.8	49.7	28.2

To demonstrate the effectiveness of combining DFSLoss with Wise-IoU (WIoU), we selected the popular bounding box regression (BBR) loss for comparative experiments. As shown in

Table 6. Performance comparison of DFSLoss combined with each bounding box loss on VisDrone-DET2021.

Method	${\mathbf{mAP}}_{0.5}$	AP	${\mathbf{mAP}}_{0.75}$
CIoU [39] + DFSLoss	50.3	28.6	28.1
GIoU [37] + DFSLoss	50.2	28.7	28.3
DIoU [38] + DFSLoss	50.1	28.3	27.9
SIoU [50] + DFSLoss	49.7	27.9	27.4
EIoU [51] + DFSLoss	49.3	27.4	27.1
MPDIoU [52] + DFSLoss	50.0	28.1	27.8
WIoUv1 + DFSLoss	50.4	28.9	28.3
WIoUv2 + DFSLoss	50.9	29.3	28.8
WIoUv3 + DFSLoss	51.1	29.7	29.3

, it can be observed that among various BBR loss functions, the performance achieved by DFSLoss and WIoUv3 consistently outperformed other models.

4.6. Comparative Experiments

We conducted a comparative analysis of our method with other state-of-the-art detectors on the VisDrone-DET2021 dataset. According to the information provided in

Table 7. Comparison results with other state-of-the-art methods on the VisDrone-DET2021 dataset.

Method	Backbone	Params(M)	${\mathbf{mAP}}_{0.5}$	AP	${\mathbf{mAP}}_{0.75}$	FPS(f/s)
YOLOv3 [5]	Darknet53	61.53	40.0	22.2	22.4	54.6
YOLOv4 [16]	Darknet53	52.50	39.2	23.5	23.4	55.0
YOLOv5l [17]	Darknet53	46.20	41.4	24.4	24.8	77.4
YOLOX [15]	Darknet53	54.20	39.1	22.4	22.7	68.9
YOLOv6l [18]	EfficientRep	58.50	41.8	25.4	25.8	116
YOLOv8l [55]	Darknet	43.60	42.9	25.9	26.4	60.8
CascadeNet [56]	ResNet101	184.00	47.1	28.8	29.3	-
RetinaNet [57]	ResNet50	59.20	44.9	26.2	27.1	54.1
HRDNet [53]	ResNet18 + 101	63.60	49.3	28.3	28.2	-
GFLV2 [58] (CVPR 2021)	ResNet50	72.50	50.7	28.7	28.4	19.4
RFLA [59] (ECCV 2022)	ResNet50	57.30	45.3	27.4	-	-
QueryDet [19] (CVPR 2022)	ResNet50	-	48.1	28.3	28.8	14.9
ScaleKD [54] (CVPR 2023)	ResNet50	43.57	49.3	29.5	30.0	20.1
Ours	DarkeNet53	39.52	53.9	32.1	32.7	72.7

, our approach demonstrated significant advantages in performance. Firstly, our model achieved the highest levels of $mA{P}_{0.5}$, AP, and $mA{P}_{0.75}$, reaching 53.9, 32.1, and 32.7, respectively. Compared to other methods, our approach showed a remarkable improvement in object detection accuracy. It is worth noting that HRDNet [53], queryDet [19], and ScaleKD [54] are advanced detectors specifically designed for small object detection. In contrast, our method achieved the best detection performance with relatively fewer parameters (39.52 M). This indicates that our method maintained model simplicity and efficiency while enhancing detection performance. Additionally, our method also demonstrated outstanding performance in inference speed, achieving a speed of 72.7 frames per second. This suggests that our model excels not only in detection accuracy but also in real-time applicability and efficiency. In summary, our method achieved significant performance improvement in small-object detection tasks, exhibiting higher detection accuracy, fewer parameters, and faster inference speed, demonstrating clear superiority.

5. Conclusions

In this study, we proposed FFEDet. Particularly, we proposed ECFA, an effective attention module for cross-scale feature aggregation, which effectively integrates feature information from various scales, emphasizes relevant features, suppresses irrelevant noise, and enhances feature representation. Additionally, we proposed SEConv, a simple and efficient convolutional module that reduces computational complexity and expands the receptive field across multiple scales. Furthermore, we proposed DFSLoss, which is a dynamic focus sample weighting function that assigns greater weights to normal and challenging samples, thus addressing the imbalance in the distribution of easy and hard samples. Moreover, we introduced Wise-IoU to alleviate the detrimental impact of low-quality examples on model convergence. The outcomes of the experiments conducted on four public datasets evidently demonstrate the outstanding performance of FFEDet. For future research, we aim to explore more lightweight model design options to enhance the model’s inference speed while preserving high accuracy. This may entail utilizing more compact network architectures, introducing more effective parameter-sharing strategies, or adopting more flexible attention mechanisms. Additionally, we will further investigate data augmentation techniques and explore more diverse and refined sample selection methods to bolster the model’s generalization ability and robustness. Moreover, we intend to explore the potential applications of the FFEDet model in various domains and tasks. For instance, we plan to deploy it in fields such as medical image analysis and intelligent surveillance systems to validate its adaptability and versatility across diverse scenarios. This exploration will offer us additional opportunities to refine the model and assess its performance in real-world applications.