SAFF-SSD: Self-Attention Combined Feature Fusion-Based SSD for Small Object Detection in Remote Sensing

Abstract

SSD is a classical single-stage object detection algorithm, which predicts by generating different scales of feature maps on different convolutional layers. However, due to the problems of its insufficient non-linearity and the lack of semantic information in the shallow feature maps, as well as the fact that small objects contain few pixels, the detection accuracy of small objects is significantly worse than that of large- and medium-scale objects. Considering the above problems, we propose a novel object detector, self-attention combined feature fusion-based SSD for small object detection (SAFF-SSD), to boost the precision of small object detection. In this work, a novel self-attention module called the Local Lighted Transformer block (2L-Transformer) is proposed and is coupled with EfficientNetV2-S as our backbone for improved feature extraction. CSP-PAN topology is adopted as the detection neck to equip feature maps with both low-level object detail features and high-level semantic features, improving the accuracy of object detection and having a clear, noticeable and definitive effect on the detection of small targets. Simultaneously, we substitute the normalized Wasserstein distance (NWD) for the commonly used Intersection over Union (IoU), which alleviates the problem wherein the extensions of IoU-based metrics are very sensitive to the positional deviation of the small objects. The experiments illustrate the promising performance of our detector on many datasets, such as Pascal VOC 2007, TGRS-HRRSD and AI-TOD.

1. Introduction

The widespread use of object detection technology has increased with the advancement of deep learning technology in the military, national defense, transportation, industrial automation and other fields [1]. Small objects are poorly detected compared with objects of normal scale due to their low resolution, small coverage in the image, and inadequate feature expression. Therefore, the question of how to accurately recognize and locate small objects is a significant topic in the current computer vision field. The criteria for defining small objects in different scenarios vary, but there are two basic types of current defining methodologies: (1) definition based on absolute size, where objects of a size less than 32 pixels × 32 pixels are defined as small objects in the MS COCO dataset; and (2) definition based on relative size, where an object’s imaging points are less than 80 pixels in a 256 pixels × 256 pixels image (i.e., a small object is defined as an object whose ratio of the number of pixels occupied by the object to the total number of pixels in the original image is less than 0.12%).

Existing convolutional neural network (CNN)-based [2] object detection methods can be divided into two categories, namely, two-stage object detection methods and one-stage object detection methods. With regard to the two-stage method, a region proposal network (RPN) and region of interest pooling (RoI pooling) are used first to obtain candidate regions. Then, classification and regression are performed on the candidate regions to obtain the final detection results, representing algorithms such as Fast R-CNN [3] and Faster R-CNN [4].

However, compared with the two-stage method, use of the one-stage object detection methods offers a more end-to-end strategy. The YOLO series is widely recognized as one of the most popular detection frameworks in industrial applications. This success is largely due to its exceptional balance between speed and accuracy. The series’ pioneering works, YOLOv1–3 [5,6,7], have paved the way for one-stage detectors and been substantially improved over time. YOLOv4 [8] has been a game-changer in this regard, reorganizing the detection framework into separate parts (backbone, neck, and head) and introducing the bag-of-freebies and bag-of-specials to design a single-GPU training framework. Today, efficient detectors such as YOLOv5 [9], YOLOX [10], PPYOLOE [11], and YOLOv7 [12] are credible alternatives for deployment. Different-sized models are often obtained through scaling techniques. Aiming at the problem of large object scale gap, SSD [13] improves detection accuracy by setting multiple scales of feature maps for detection without losing real-time performance. Single-stage object detection models employ VGG-16 as their backbone network for feature extraction. To improve detection accuracy, several convolutional layers are added on top of the backbone network to produce a diverse set of feature maps. SSDs generate six of these, making them ideal for detecting smaller objects, while the deep feature maps contain more semantic information, making them suitable for detecting larger objects. For each input image, SSD utilizes the anchor concept introduced in Faster R-CNN by setting prior boxes with different scales and aspect ratios. These prior boxes serve as a reference for predicting the bounding boxes, thus reducing the complexity of training. Finally, SSD applies non-maximum suppression to generate the ultimate detection results. However, the problem of small object detection still arises due to the lack of semantic information derived from the shallow features and spatial information of deep feature maps. Hence, several techniques have been proposed by researchers to enhance the semantic information of the feature maps. ESSD [14] uses transposed convolution to upsample the feature map once more and then skips to connect feature maps of the same size. This improves situations where there is a difficulty in classifying due to shallow feature maps. FFESSD [15] presents a lightweight feature fusion module that integrates several outputs of the backbone and passes them through the feature pyramid again. This yields new features with stronger semantic information. ZHAO applied transposed convolution, dilated convolution and deformable convolution to improve SSD for multi-scale feature fusion in terms of receptive field and adaptive object shape [16]. Similar to the above, RSSD [17] also uses transposed convolution. However, it fuses the outputs of the various stages of the backbone. In addition, sharing parameters between feature maps becomes feasible when the channels of the fused feature maps exhibit consistency. Thereafter, in MDSSD [18], the deeper feature maps are considered to possess richer semantic information, so the improved model retains more of the backbone. This allows it to make jump connections over a larger span and finally use elaborate feature fusion modules.

This paper presents SAFF-SSD, an enhanced small object detection algorithm that is based on SSD and incorporates a transformer model and a multi-scale feature fusion strategy. As shown in Figure 1, comparisons with another SSD-based object detectors show that our method achieves the best accuracy on Pascal VOC2007. The structure is illustrated in Figure 2. To begin, more development is required for the VGG-16 backbone of the original SSD network’s feature extraction efficiency and capability. Taking into account the fact that Transformer [19] has good global self-focus modeling capabilities, in our work we seek to combine Transformer and convolution in the network’s backbone. To this end, we propose the Local Lighted Transformer block (2L-Transformer) and couple it with the lightweight neural network EfficientNetV2-S as the new backbone network, in order to achieve a balance between accuracy and efficiency. Second, SSD only predicts on a single-scale, shallow feature map, which cannot effectively integrate positional information with semantic features. Hence, SSD often results in significant false and missed detection rates, particularly for small items and objects of varying sizes. In order to resolve this issue, CSP-PAN is incorporated into the SSD. This is a feature fusion approach that improves the accuracy and stability of detection by integrating feature maps of different sizes and resolutions to offer additional contextual information. With CSP-PAN, the model may leverage input from several layers concurrently to capture spatial and semantic information about the object more accurately. This can aid the model in detecting objects with varying sizes and resolutions and prevent it from being affected by resolution and scale discrepancies, resulting in a decrease in false and missing detection rates. Finally, the NWD metric [20] is utilized to overcome the sensitivity of the position variation of the microscopic objects and dramatically improve the detection performance.

The contributions are summarized as follows:

• We propose Local Lighted Transformer block (2L-Transformer), a new transformer module, and integrate it with EfficientNetV2-S as our backbone for improved feature extraction. On the AI-TOD dataset, an increase of 1.3% in AP is observed when we replace the original backbone with EfficientNetV2-S. When further embedding the proposed Transformer block in EfficientNetV2-S, we achieve an additional 1.8% increase in AP.
• By optimizing the original bottleneck network on the PAN topology utilizing CSP as an internal building block, our experiments demonstrate that the optimized model is more effective in integrating multi-scale features.
• We also evaluate the possibility of using the normalized Wasserstein distance (NWD) as an alternative similarity metric for small targets. Specifically, on the AI-TOD dataset, the proposed model with NWD has a 14.1% increase in AP compared with the standard AP baseline. Moreover, on the VOC 2007 dataset, our proposed multi-level NWD shows better results than the NWD metric.

The rest of the article is arranged as follows: some related works are reviewed in Section 2. Materials and methods are introduced in Section 3. Section 4 presents the results of our experiments and discusses their implications. Finally, Section 5 offers a concise summary of our findings and suggests directions for future research.

2. Related Work

2.1. Feature Pyramid

Pyramidal feature representation is a fundamental building block for solving multi-scale problems. SSD first tried to use pyramidal features for object detection. FPN [21] use lateral connections and top-down pathfinding to construct a feature pyramid with strong semantics. Further, PAN [22] adds bottom-up augmentation based on FPN to improve information flow. A bidirectional route is repeated several times by EfficientDet [23] to facilitate higher level feature fusion. In addition to designing the network manually, NAS-FPN [24] tries to find a stronger feature pyramid structure with the aid of a neural architectural search.

2.2. Transformers/Self-Attention Mechanism

Transformer is a self-attention-based neural network architecture proposed by Vaswani et al. [25] in 2017, where the self-attention enables Transformer to weigh the importance of various input sequence segments when creating the output. Nowadays, several versions of Transformer models are available, such as the BERT model for pre-trained language representations [26] and the GPT model for language production [27,28,29]. These have emerged as highly effective models in the field of natural language processing, outperforming previous models in various tasks, including machine translation, language modeling, and text classification. This has further brought Transformer into the focus of research. The Transformer has become one of the fundamental components of contemporary deep learning models. Self-attention layers have been utilized to supplement backbones [30,31,32,33,34] and head networks [35,36], enabling them to capture distant dependencies or diverse interactions. Moreover, the Transformer’s encoder–decoder architecture has recently been employed for object detection and instance segmentation tasks [37,38,39,40].

2.3. Similarity Metrics

The Intersection over Union metric is commonly employed for measuring the similarity between bounding boxes. However, it has limitations in cases where the boxes do not overlap. To address this limitation Generalized IoU (GIoU) [41], which penalizes the smallest box converting bounding boxes, is proposed as a solution to this issue. The degradation from GIoU to IoU occurs in cases where one bounding box completely encloses another.

Consequently, it is suggested that DIoU [42] and CIoU [42] be used in order to overcome the constraints of IoU and GIoU. Yang et al. [43] also proposed a Gaussian Wasserstein distance (GWD) loss for oriented object detection by measuring the positional relationship of oriented bounding boxes. The goal of GWD is to address the oriented object identification border discontinuities and square-like difficulty. Moreover, Wang et al. have introduced the normalized Wasserstein distance (NWD) [20] as a solution to the sensitivity issue of Intersection over Union (IoU) in measuring the accuracy of object detection algorithms for small objects.

3. Materials and Methods

This section will introduce the proposed self-attention combined feature fusion-based SSD (SAFF-SSD) in detail. Figure 2 displays the SAFF-SSD architectural plan. Section 3.1 and Section 3.2 introduce, respectively, the structures of the two essential constituents. The similarity metric is described in Section 3.3.

3.1. Self-Attention Mechanism

Transformer was initially introduced to address natural language processing (NLP) problems, but it has also been applied to the computer vision (CV) domain with good results on many tasks. In order to use Transformer with CV tasks, the self-attention mechanism of Transformer must first allow for interactions between different locations of the image, enabling the model to capture dependencies between different regions. Secondly, the multi-headed self-attention mechanism in Transformer can also process features from multiple channels simultaneously and combine them into a global representation. However, spatial structure information cannot be handled by the traditional Transformer. Even more seriously, image input will raise computational efficiency issues compared with text input. Therefore, applying Transformer models in the field of object detection remains a challenge.

In view of this, we aim to address this challenge by proposing a novel approach that combines the Transformer and convolution in the backbone network. This approach can improve computational efficiency while also leveraging the global self-attention modeling ability of Transformer. We use EfficientNetV2-S [44] as the base framework for the backbone. This is a lightweight and high performance convolutional neural network designed to solve the conflict between efficiency and accuracy and uses a module called “Squeeze-and-Excitation (SE)” [45]. Although it may improve detection accuracy, it is only an abbreviated enhancement or suppression of channels, lacks attention to spatial position, and is not stable across tasks or datasets. Therefore, SE is not real self-attention. Our proposed 2L-Transformer module truly combines the self-attention of the transformer with the spatial invariance of the convolution, and it has worked well in many subsequent experiments.

Let us first review the method of ViT [19]. This divides the image into 16 × 16 patches and each part is embedded into a vector. It then extracts features by standard transformer, which is the stacking of several multiheaded self-attention (MSA) modules, a multilayer perceptron module (MLP) and a layer norm (LN). We consider the module stacked L times in ViT as a block, named Transformer. The process can be computed as

$$\begin{gathered} {\mathrm{z}}_0=\left[{\mathrm{x}}_{\mathrm{class}}{; \mathrm{x}}_{\mathrm{p}}^1{\mathrm{E}; \mathrm{x}}_{\mathrm{p}}^1{\mathrm{E}; \dots ; \mathrm{x}}_{\mathrm{p}}^1\mathrm{E}\right]{+\mathrm{E}}_{\mathrm{pos}},\mathrm{E}\in {\mathrm{R}}^{\left({\mathrm{P}}^2·\mathrm{C}\right)\times \mathrm{D}}{,\mathrm{E}}_{\mathrm{pos}}\in {\mathrm{R}}^{(\mathrm{N}+1)\times \mathrm{D}} \\ {\mathrm{z}\prime }_{\mathrm{l}}{=\mathrm{MSA}\left(\mathrm{LN}\right(\mathrm{z}}_{\mathrm{l}-1}{\left)\right)+\mathrm{z}}_{\mathrm{l}-1},\mathrm{l}=1\dots \mathrm{L} \\ {\mathrm{z}}_{\mathrm{l}}{=\mathrm{MLP}\left(\mathrm{LN}\right(\mathrm{z}\prime }_{\mathrm{l}}{\left)\right)+\mathrm{z}\prime }_{\mathrm{l}},\mathrm{l}=1\dots \mathrm{L} \\ {\mathrm{Y}=\mathrm{LN}(\mathrm{z}}_{\mathrm{L}}^0) \end{gathered}$$

(1)

However, this method produces a quadratic multiplication of computational expenses as the image resolution increases, and the constrained image resolution can have a negative effect on small object detection. Therefore, we present an improved module.

The Local Lighted Transformer block (2L-Transformer) is shown in Figure 3. To start with, we split the feature map into two parts according to the channels. Subsequently, one part does window local self-attention [46], which is to extract features within the window. For the other part, embedding is first implemented using dilation convolution, followed by global self-attention to interoperate information between windows. Next, we reshape the output of both branches to the original size, where the globally attended branch is downsampled and requires additional upsampling. Finally, we concatenate the output of the two branches according to the channel. At the same time, to preserve spatial location information and ensure valid gradients, we also design a shortcut from the input to the output [47]. After that, smoothing is performed using standard convolutional layers (Conv + BN + SiLu). Our module can be computed as

$$\begin{gathered} {\mathrm{x}}_{\mathrm{loc}}{, \mathrm{x}}_{\mathrm{glo}}=\mathrm{split}\left(\mathrm{x}\right), \mathrm{W} \times \mathrm{H} \times 2\mathrm{C}\to \mathrm{W} \times \mathrm{H} \times \mathrm{C}, \mathrm{W} \times \mathrm{H} \times \mathrm{C} \\ {\mathrm{z}}_{\mathrm{glo}}=\left[{\mathrm{x}}_{\mathrm{glo},\mathrm{p}}^1{\mathrm{E}; \dots ; \mathrm{x}}_{\mathrm{glo},\mathrm{p}}^1\mathrm{E}\right]{+\mathrm{E}}_{\mathrm{glo},\mathrm{pos}},\mathrm{E}\in {\mathrm{R}}^{\left({\mathrm{P}}^2·\mathrm{C}\right)\times \mathrm{D}}{,\mathrm{E}}_{\mathrm{pos}}\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{D}} \\ {\mathrm{Y}}_{\mathrm{glo}}=\mathrm{unreshape}\left(\mathrm{LN}\left({\mathrm{Transformer}(\mathrm{z}}_{\mathrm{glo}})\right)\right), \\ {\mathrm{z}}_{\mathrm{loc}}{=\mathrm{reshapeToWindow}\left(\mathrm{LN}\right(\mathrm{x}}_{\mathrm{loc}}\left)\right),\mathrm{W} \times \mathrm{H} \times \mathrm{D}\to {(\mathrm{p}}_{\mathrm{w}}·{\mathrm{p}}_{\mathrm{h}}{) \times (\mathrm{win}}_{\mathrm{w}}·{\mathrm{win}}_{\mathrm{h}}) \times \mathrm{D} \\ {\mathrm{Y}}_{\mathrm{loc}}=\mathrm{unreshape}\left(\mathrm{LN}\left({\mathrm{Transformer}(\mathrm{z}}_{\mathrm{glo}})\right)\right), \\ \mathrm{Y}=\mathrm{Conv}\left(\mathrm{concat}\left[{\mathrm{Y}}_{\mathrm{loc}}{, \mathrm{Y}}_{\mathrm{glo}}{, \mathrm{x}}_{\mathrm{loc}}{, \mathrm{x}}_{\mathrm{glo}}\right]\right) \end{gathered}$$

(2)

3.2. Feature Fusion Strategy

3.2.1. Spatial Pyramid Pooling-Fast

After the forward process consisting of standard convolution, each point on the feature map corresponds to a certain fixed small region of the original image. On scenes as complex as we can foresee—where small objects appear simultaneously with large ones—this fixed correspondence limits the model’s capacity for detection. Therefore, in our work, inspired by the Spatial Pyramid Pooling (SPP) module proposed in Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition by K. He [48], we add the Spatial Pyramid Pooling-Fast block after the backbone. As illustrated in Figure 4, we integrate local features by concatenating the maximum pooling outputs of different kernel sizes. Furthermore, doing so will enrich the expressiveness of the feature map. It is effective when there are large differences in object sizes, and it does not require additional parameters that need training. It is worth mentioning that stacking pooling layers with small kernel sizes gives exactly the same results and significantly reduces the computational cost when compared with the direct use of pooling layers with large kernel sizes.

3.2.2. Path Aggregation Network

Figure 5a shows the detailed structure of the PANet proposed by Liu et al. To improve performance, it conducts path augmentation and aggregation. Specifically, the position features present in the shallow layers are gradually blurred or even lost as the feature extraction goes deeper. Considering this, bottom-up path augmentation is added, which will allow the combination of positional features at the shallow layers with semantic features at the deeper layers. PAN is an extension of FPN, which defines layers producing feature maps with the same spatial size as belonging to the same network stage, with each stage corresponding to a feature level. And PAN uses a set notation to denote the feature levels generated by FPN, specifically {${\mathrm{P}}_1$, ${\mathrm{P}}_2$, ${\mathrm{P}}_3$}. The enhancement path starts from the lowest level ${\mathrm{P}}_1$ and progresses towards ${\mathrm{P}}_3$, as illustrated in Figure 5a. From ${\mathrm{P}}_1$ to ${\mathrm{P}}_3$, the spatial size is progressively downsampled by a factor of 2. We use {${\mathrm{N}}_1$, ${\mathrm{N}}_2$, ${\mathrm{N}}_3$} to denote the newly generated feature maps that correspond to {${\mathrm{P}}_1$, ${\mathrm{P}}_2$, ${\mathrm{P}}_3$}. Note that ${\mathrm{N}}_1$ is just ${\mathrm{P}}_1$, without any modification. Each building block acquires a higher resolution feature map Ni and a coarser feature map ${ \mathrm{P}}_{\mathrm{i}+1}$ through lateral concatenation and generates a new feature map ${\mathrm{N}}_{\mathrm{i}+1}$. Each feature map ${\mathrm{N}}_{\mathrm{i}}$ is first passed through a 3 × 3 convolutional layer with step size 2 to reduce the space size. Then, each element of the feature map ${\mathrm{P}}_{\mathrm{i}+1}$ and the downsampled map are joined laterally by summation. After obtaining the fused feature maps, a 3 × 3 convolutional layer is applied to generate the next feature map, $N_{\mathrm{i}+1}$. This is an iterative process that terminates after approaching ${\mathrm{P}}_3.$ A rectified linear unit (ReLU) activation function follows each convolutional layer. Subsequently, the object is then detected on {${\mathrm{N}}_1$, ${\mathrm{N}}_2$, ${\mathrm{N}}_3$}.

3.2.3. Cross Stage Partial-PAN

While adding the PAN module as the neck of the model (which is very lightweight compared with the backbone) does work, doing so can cause problems when we consider improving it. Generally, to improve the performance, increasing the depth and width of the network is considered the most straightforward approach. However, if we do this, issues such as parameter bloat, overfitting, reduced gradient stability, and difficulty in convergence will follow. One paper, which goes deeper with convolutions [49], reports the assembly of multiple convolution or pooling operations into modules and the building of a network. The networks designed in this way reached a much higher level at that time. Inspired by this, we expanded the standard convolutional layer into multiple branches in the neck part of the network. However, unlike inception [50], we did not use a 1 × 1 convolution to reduce the number of channels. Instead, we directly split the input into two parts by channel. We have found that this has a minimal effect on the results while saving a considerable number of parameters.

The architecture of CSP-PAN is shown in Figure 5b. The modified CSP-PAN embeds the CSP-block [50], which has the structure shown in Figure 6a. In one CSP-block, the feature map in a stage is split into two parts through channel $X_0$ = [$X_0^{\prime }$, $X_0^{″}$]. Between $X_0^{\prime }$ and $X_0^{″}$, the former goes through one ConvBNSiLu layer, and is then directly connected to the end of the stage, and the latter goes through one ConvBNSiLu layer and some bottlenecks. The output of these parts will be concatenated and passed through another ConvBNSiLu layer, until it finally generates output $X_U$. Unlike the original CSP, we have found that concatenating each bottleneck layer to the output results in better performance.

3.3. Similarity Metric

IoU is actually the Jaccard similarity coefficient for computing similarity of two limited sample sets. According to this fact, in [20], Wang proposed a novel metric for small objects using Wasserstein distance, which can better reflect the distance between distributions even if they have no overlap. Compared with IoU, this metric has better properties for measuring the similarity between small objects. Normalized Wasserstein distance is a distance metric that utilizes the Wasserstein distance, derived from Optimal Transport theory, to measure the distance between probability distributions. When comparing two 2D Gaussian distributions denoted by ${\mu }_1=\mathcal{N} \left(m_1,{\mathsf{\Sigma }}_1\right)$ and ${\mu }_2=\mathcal{N}\left(m_2,{\mathsf{\Sigma }}_2\right)$, the second order Wasserstein distance can be used to quantify their distance from one another as follows:

$${\mathrm{W}}_2^2\left({\mu }_1{,\mu }_2\right)=\Vert m_1-m_2{\Vert }_2^2{+\mathrm{Tr}(\mathsf{\Sigma }}_1{+\mathsf{\Sigma }}_2-{2(\mathsf{\Sigma }}_2^{1/2}{\mathsf{\Sigma }}_1{\mathsf{\Sigma }}_2^{1/2}{)}^{1/2})$$

(3)

and it can be simplified as:

$${\mathrm{W}}_2^2\left({\mu }_1,{\mu }_2\right)=\Vert m_1-m_2{\Vert }_2^2+\Vert {\mathsf{\Sigma }}_1^{\frac{1}{2}}-{\mathsf{\Sigma }}_2^{\frac{1}{2}}{\Vert }_{\mathrm{F}}^2,$$

(4)

where $\Vert ·{\Vert }_{\mathrm{F}}$ is the Frobenius norm.

Furthermore, for Gaussian distributions ${\mathcal{N}}_{\mathrm{a}}$ and ${\mathcal{N}}_{\mathrm{b}}$, which are modeled from bounding boxes $\mathrm{A}=\left({\mathrm{cx}}_{\mathrm{a}},{\mathrm{cy}}_{\mathrm{a}},{\mathrm{w}}_{\mathrm{a}},{\mathrm{h}}_{\mathrm{a}}\right) \mathrm{and},\mathrm{B}=\left({\mathrm{cx}}_{\mathrm{b}},{\mathrm{cy}}_{\mathrm{b}},{\mathrm{w}}_{\mathrm{b}},{\mathrm{h}}_{\mathrm{b}}\right)$, respectively, Equation (4) can be simplified as:

$${\mathrm{W}}_2^2\left({\mu }_1,{\mu }_2\right)=\Vert ({\left[{\mathrm{cx}}_{\mathrm{a}},{\mathrm{cy}}_{\mathrm{a}},\frac{{\mathrm{w}}_{\mathrm{a}}}{2},\frac{{\mathrm{h}}_{\mathrm{a}}}{2}\right]}^{\mathrm{T}}),{\left[{\mathrm{cx}}_{\mathrm{b}},{\mathrm{cy}}_{\mathrm{b}},\frac{{\mathrm{w}}_{\mathrm{b}}}{2},\frac{{\mathrm{h}}_{\mathrm{b}}}{2}\right]}^{\mathrm{T}}{\Vert }_2^2$$

(5)

An exponential form normalization is applied to Wasserstein distance, resulting in a new metric known as normalized Wasserstein distance (NWD):

$$\mathrm{NWD}\left({\mathcal{N}}_{\mathrm{a}},{\mathcal{N}}_{\mathrm{b}}\right)=\exp (-\frac{\sqrt{{\mathrm{W}}_2^2\left({\mathcal{N}}_{\mathrm{a}},{\mathcal{N}}_{\mathrm{b}}\right)}}{\mathrm{C}}),$$

(6)

where C is a constant that has a strong connection to the dataset.

In addition, the NWD metric has a single hyperparameter C that is strongly dependent on the dataset, making it difficult for model training in multi-scale object detection tasks. To address this issue, we expanded the normalized Wasserstein distance (NWD) to multi-level NWD. The process of setting the hyperparameters for the multi-level NWD is shown in Figure 7. First, we categorized objects in the training set into three groups based on their length, corresponding to the three detection heads in object detection. Next, we computed the NWD metric for various scales based on the origin of the predicted bounding boxes. This direct approach has been effective in bringing the NWD performance in multi-scale object detection tasks closer to that of IoU.

4. Results

4.1. DataSet

Three publicly available object-detection datasets are used to evaluate the proposed methods in the experiments. Some examples, the Pascal VOC 2007 [51], TGRS-HRRSD [52], and AI-DOT [53], are shown in Figure 8.

The first dataset is Pascal VOC 2007 which is a benchmark dataset for object detection and has been extensively used for evaluating object detection models. It was introduced as part of the PASCAL Visual Object Classes Challenge in 2007 and has since become a standard benchmark for evaluating the performance of computer vision algorithms. The dataset consists of over 5000 images, each of which is annotated with object class labels and bounding box coordinates for up to 20 object categories, including people, cars, and animals.

The second dataset is TGRS-HRRSD, which was published by the University of Chinese Academy of Sciences in 2019. The spatial resolution of the 21,761 image samples in the TGRS-HRRSD, which ranges from 0.15 m to 1.2 m, were collected from Google Earth and Baidu Map. There are 55,740 object instances total, divided across 13 object categories. The TGRS-HRRSD dataset contains occlusion levels and truncation rates, which are key factors in evaluating the performance of small object detection models and makes it a more accurate measure of small object detection model performance in challenging real-world scenarios.

The third dataset is AI-TOD. This is a recent addition to the family of object detection datasets, with a focus on small objects. The dataset comprises 4280 images with an average absolute size of instances of only 12.8 pixels, which is much smaller than other datasets such as PASCAL VOC (156.6 pixels), MS COCO (99.5 pixels) [54], or DOTA (55.3 pixels) [55], and which makes it one of the smallest object detection datasets. The dataset presents a challenging task for object detection models due to the extremely small object size and limited number of pixels available for each object instance. Therefore, it serves as a valuable benchmark for evaluating the performance of object detection models when detecting small objects.

4.2. Experimental Setup

To compare the performance of the proposed method, we evaluate it against several algorithms. Among these, SSD [13], YOLOv3 [7], YOLOv4 [8], RetinaNet [56], YOLOv6-M [57] are one-stage methods. In addition, the FoveaBox [58] and YOLOv8n [59] are anchor-free methods. Some algorithms of remote sensing are EFNet [60] and HRCNN-regression [52]. Due to the unavailability of public codes, the main sources for various experimental results of domain algorithms are already published references.

In the object detection experiments, we use the following default hyper-parameters: the optimizer is SGD and a step decay learning rate scheduling strategy is used with an initial learning rate of 0.01, multiplied by a factor of 0.1 at 50 and 100 epochs. The architecture uses a single GPU to perform training with a batch size of 8 due to GPU memory limitations. The momentum is set to 0.95 with the weight decay set to 0.0005. All experiments are run with an NVIDIA RTX 3080ti GPU.

4.3. Evaluation Metric

The average precision (AP) and its derivative metrics are adopted to quantitatively evaluate the proposed method. The AP is a comprehensive metric in the task of object detection and is based on the precision and recall of Equations (7) and (8).

$$precision=\frac{TP}{TP+FP},$$

(7)

$$recall=\frac{TP}{TP+FN},$$

(8)

where the terms TP, FP, and FN are true positives, false positives, and false negatives, respectively. The terms TP, FP, and FN are calculated from the Intersection over Union (IoU) between the bounding boxes of ground-truth and the bounding boxes of prediction as follows:

$$IoU=\frac{B_{pred}\cap B_{gt}}{B_{pred}\cup B_{gt}},$$

(9)

where $B_{pred}$ denotes the bounding box of prediction, and $B_{gt}$ is the bounding box of ground-truth. The standard HRRSD metrics include AP, AP_0.5, AP_0.75, AP_s, AP_m, and AP_l. AP denote AP at IoU = 0.50:0.05:0.95 (average over IoU thresholds), AP_0.5 denotes AP at IoU = 0.50 (equally to PASCAL VOC metric). AP_0.75 denotes AP at IoU = 0.75 (very strict metric). AP_s denotes AP for small objects whose areas are smaller than ${32}^2.$ AP_m denotes AP for small objects whose areas are between ${32}^2$ and ${96}^2.$ AP_l denotes AP for large objects whose areas are bigger than ${96}^2$. In addition, AP_vt, AP_t, AP_s, AP_m are APs for very tiny, tiny, and small medium scales, respectively. For the detection of multi-category objects, the AP usually denotes mean average precision (mAP) which is obtained by the average of different category APs.

4.4. Results and Analysis

The details of the experimental findings are presented in this section. The quantitative results of the comparative methods are shown in

Table 1. Results of comparative approaches for object detection on Pascal VOC 2007. In natural scenes, our algorithm yields performances that are almost on par with the state-of-the-art one-stage algorithm yolov6. Nevertheless, our approach still boasts highly satisfactory results in this context.

Method	Backbone	Input Size	mAP
SSD300	VGG-16	300 × 300	77.2
SSD512	VGG-16	512 × 512	79.8
ESSD [14]	VGG-16	300 × 300	78.7
ESSD [14]	VGG-16	512 × 512	81.7
RSSD [17]	VGG-16	300 × 300	78.5
RSSD [17]	VGG-16	512 × 512	80.8
FFESSD [15]	VGG-16	300 × 300	79.1
FFESSD [15]	VGG-16	512 × 512	80.9
MDSSD [18]	VGG-16	300 × 300	78.6
MDSSD [18]	VGG-16	512 × 512	81.0
YOLOv6-M [57]	EfficientRep	640 × 640	86.0
YOLOv8n [59]	Darknet	640 × 640	80.2
DETR [39]	Transformer Encoder	640 × 640	82.6
Ours300	VGG-16	300 × 300	79.9
Ours512	VGG-16	512 × 512	82.4
Ours300	2L-Transformer+EfficientNetV2-S	300 × 300	82.2
Ours512	2L-Transformer+EfficientNetV2-S	512 × 512	84.9

,

Table 2. Results of comparative approaches for object detection on TGRS-HRRSD. Although many algorithms can detect most of the targets in TGRS-HRRSD based on AP_0.5, our approach stands out by achieving superior overall performance and demonstrating particular strengths in detecting small targets (AP_S).

Method	Backbone	AP	AP0.5	AP0.75	APs	APm	APl
HRCNN-regression [52]	AlexNet	/	51.4	/	/	/	/
YOLO-v4	CSPDarknet-53	59.9	89.7	68.3	11.2	50.1	58.2
YOLOv3-608	Darknet-53	59.4	89.0	66.8	10.6	48.3	55.6
RetinaNet	ResNet-50	59.6	89.3	67.7	10.9	49.2	57.4
EFNet [60]	ResNet-50	62.2	90.7	71.7	12.4	53.5	60.2
SSD512	VGG-16	52.7	87.3	57.4	9.5	42.1	51.7
FoveaBox	Resnet-50	61.8	90.4	70.6	18.4	51.8	58.7
SSD512	EfficientNetV2-S	53.4	88.0	62.1	9.6	43.5	53.8
YOLOv6-M [57]	EfficientRep	62.9	91.4	71.2	18.3	54.0	61.2
YOLOv8n [59]	Darknet	61.3	90.4	70.7	18.0	53.1	59.3
DETR [39]	Transformer Encoder	58.8	88.5	69.5	14.5	53.5	61.8
Ours512	2L-Transformer+ EfficientNetV2-S	63.2	91.1	72.2	20.4	53.8	60.6

and

Table 3. Results of comparative approaches for object detection on AI-TOD. Unlike object detection in daily life, detecting objects in aerial images is more challenging and so fully demonstrates the advantages of our approach.

Method	Backbone	AP	AP0.5	AP0.75	APvt	APt	APs	APm
SSD512 [13]	ResNet-50	7.0	21.7	2.8	1.0	4.7	11.5	13.5
TridentNet [61]	ResNet-50	7.5	20.9	3.6	1.0	5.8	12.6	14.0
FoveaBox [58]	ResNet-50	8.1	19.8	5.1	0.9	5.8	13.4	15.9
RepPoints [62]	ResNet-50	9.2	23.6	5.3	2.5	9.2	12.9	14.4
FCOS [63]	ResNet-50	9.8	24.1	5.9	1.4	8.0	15.1	17.4
M-CenterNet [53]	DLA-34	14.5	40.7	6.4	6.1	15.0	19.4	20.4
RetinaNet	ResNet-50	4.7	13.6	2.1	2.0	5.4	6.3	7.6
ATSS [64]	ResNet-50	12.8	30.6	8.5	1.9	11.6	19.5	29.2
Faster R-CNN	ResNet-50	11.1	26.3	7.6	0.0	7.2	23.3	33.6
Cascade R-CNN [65]	ResNet-50	13.8	30.8	10.5	0.0	10.6	25.5	36.6
DetectorRS [66]	ResNet-50	14.8	32.8	11.4	0.0	10.8	28.3	38.0
SSD512	EfficientNetV2-S	6.9	21.4	2.7	0.8	4.7	11.0	13.5
YOLOv6-M [57]	EfficientRep	16.3	37.9	12.1	2.6	10.2	30.5	40.2
YOLOv8n [59]	Darknet	14.9	32.8	11.6	2.0	9.8	26.4	33.6
DETR [39]	Transformer Encoder	10.6	26.4	7.4	0.0	6.7	20.5	35.2
Ours	2L-Transformer+ EfficientNetV2-S	21.1	49.9	14.9	7.0	20.8	30.1	38.8

, and qualitative results are illustrated in Figure 9, Figure 10 and Figure 11. We apply mAP to be evaluation metric, which is a widely used standard in the field of object detection. Therefore, it allows direct comparison of the performance of different models or mechanisms.

For the Pascal VOC 2007, based on Table 1, it can be observed that when the input size is 512 × 512, our method outperforms the classical SSD algorithm (baseline) with 5.1% improvement in mAP, and also outperforms other improved SSD methods such as FSSD, RSSD, DSSD, and MDSSD. Although we design SAFF-SSD for small object detection, it still achieves good results in VOC, outperforming YOLOv8n [59] and second only to YOLOv6-M [57]. This proves that the proposed SAFF-SSD is a promising method.

For the TGRS-HRRSD, it can be observed, based on Table 2, that our method achieves the best results on the TGRS-HRRSD dataset, outperforming methods such as RetinaNet [56] and YOLOv4 [8], and, with regard to APs, beating the second-best, FoveaBox [58], by 2.0%, showing the advantage for small object detection.

For the AI-TOD, and based on Table 3, it becomes apparent that AP_vt of current state-of-the-art detectors tends towards zero, which indicates that they are not suitable for producing satisfactory results on tiny objects. In addition, our proposed detectors surpass the AP_t metric of SSD [13], TridentNet [61], FoveaBox [58], RepPoints [62], FCOS [63] and M-CenterNet [53] by 16. 1%, 15.0%, 15. 0%, 11.6% and 12.8%, respectively. Notably, our proposed method achieves state-of-the-art performance (21.1% AP) on the AI-TOD dataset among one-stage detectors, particularly in cases where objects are extremely small. This indicates a significant improvement in performance compared with existing models.

Figure 9, Figure 10 and Figure 11 show visual comparison of the detection performance of the SSD algorithm and the proposed algorithm. Each row, from top to bottom, represents the image that needs to be detected with labels, the SSD algorithm’s detection result, and our proposed algorithm’s detection result, respectively. From these figures, it can be seen that SSD is very prone to miss detection in images with complex scenes and dense objects. The proposed algorithm uses a transformer block to enhance the features and focus more on the spatial location information of small objects, and combines feature fusion to strengthen the semantic information of the feature map, thus effectively locating and detecting dense small object objects and reducing the phenomenon of missed detection, thus proving the superiority of the proposed algorithm in detecting dense small objects in a high-resolution image.

4.5. Ablation Study

4.5.1. Influence of Local Lighted Transformer Module

To demonstrate the efficacy of the proposed algorithm, we have chosen to perform ablation experiments on several datasets, such as AI-TOD. This also helps us to further analyze the degree of impact of each module on the detection performance. Among these datasets, AI-TOD is designed for evaluating the performance of the challenging task of small object detection. The average absolute size of the instances in this dataset is only 12.8 pixels, which makes it appropriate for performance testing of small object detectors.

We propose an efficient, lightweight, local self-attention 2L-Transformer module and have replaced some of the modules in the backbone network EfficientNetV2-S. To provide a more intuitive and clear analysis of the effect of the proposed attention module, a heat map is generated and visualized for a scene with small objects, when the input image size is set to 512 × 512. The 2L-Transformer module’s detecting impact is shown in Figure 12. It is evident that the area of interest is more accurately objected to the small object, which effectively avoids the missed detection and improves the detection accuracy of small objects.

Table 4. Effectiveness of Local Lighted Transformer block visualization results. Several backbone feature extractors were compared, including VGG-16, which was featured in the original SSD, ResNet-50 which is now widely utilized, the native version of EfficientNet, and the final mixed architecture version.

	Backbone	AP	AP0.5
AI-TOD	VGG-16	7.0	21.7
	ResNet-50	9.1	26.2
	EfficientNetV2-S	8.3	24.4
	2L-Transformer + EfficientNetV2-S	10.1	27.5
VOC-07	VGG-16	60.7	79.8
	ResNet-50	62.1	81.6
	EfficientNetV2-S	61.5	80.3
	2L-Transformer + EfficientNetV2-S	62.5	82.8
TGRS-HRRSD	VGG-16	52.7	87.3
	ResNet-50	58.8	88.9
	EfficientNetV2-S	58.7	88.4
	2L-Transformer + EfficientNetV2-S	59.6	89.3

presents the results of the ablation experiment, indicating a significant improvement in the feature extraction ability of EfficientNetV2-S with the proposed attention module. First, we use EfficientNetV2-S as the backbone of SSD, and its detection accuracy gains 1.3% mAP improvement. It can be seen that using an improved design for the backbone does work. We then proceed to replace some of the modules in the EfficientNetV2-S with 2L-Transformer and end up with a nearly 3.1% improvement in detection accuracy compared with the original SSD. This is due to the good performance of the 2L-Transformer module in global modeling and context capture. To a certain extent, it addresses the limitations of the receptive field and compensates for the lack of features specific to small objects. However, EfficientNet is considered a lightweight backbone, and increasing the number of modules does not guarantee their superiority. Therefore, in the end, we replaced the backbone with the most commonly used ResNet50, and the experiment proved that the backbone with 2L-Transformer had better feature extraction ability than ResNet50, resulting in an improvement of nearly 1% in the final results. Moreover, the parameter counts of the backbone networks are comparable and all within 30 M.

4.5.2. Influence of Different Feature Fusion Modules

To further investigate the impact of the feature fusion module on small object detection, we completely trained three variants of the model on two datasets (AI-TOD and VOC-07). By comparing the effect of the models on the test set, we can analyze the role and applicability scenarios of CSP-PAN and SPP-Fast. As expected, the experiments prove that the modules we used are effective.

• Spatial Pyramid Pooling-Fast

SPP-Fast uses maximum pooling for the inputs with different kernel sizes, then concatenates by channel. There are two main starting points for this: firstly, this facilitates the processing of objects of different sizes; and secondly, we aim to improve the robustness and generalization of the model. Nevertheless, adding the module only after the backbone may lead to weak prediction accuracy. The reasons are clear. Firstly, the output of this module is a small-sized feature map for predicting large objects, and the module has a limited range of influence; meanwhile, pooling may result in the loss of certain crucial information, for example, leading in a reduction of spatial resolution or a lack of location information. Therefore, some improvements are still needed.

• Cross Stage Partial-PAN

To verify the superiority of the CSP-PAN feature fusion module, we ensured that other components were the same and selected different necks for feature fusion. The baseline was the native neck of the SSD model which extracted feature maps from different stages, passed them through several convolutional layers, and maintained their sizes. Some details can be found in

Table 5. Effectiveness of different feature fusion modules. Through comparison, CSP-PAN structure is not overly complex yet still manages to achieve an almost 0.5-point improvement.

Dataset/Component	SSD_Neck (AP0.5)	FPN (AP0.5)	PAN (AP0.5)	CSP-PAN (AP0.5)
AI-TOD	27.5	29.8	31.8	32.4
VOC-07	82.8	83.5	84.3	84.9
TGRS-HRRSD	89.3	90.2	90.5	90.6

. The experimental results showed that CSP-PAN consistently outperformed FPN. Even for the structurally similar PFPN, CSP-PAN still achieved the best performance on all three datasets, with an improvement of nearly 0.5 points.

CSP-PAN combines the location information in the shallow layer with the semantic information in the deep layer, thus achieving feature fusion and ultimately improving the capability of feature representation. As is to be expected, with this cross-layer feature fusion, the model is more likely to make a correct classification and localization of small objects. Specifically,

Table 6. The contribution of SPPF in feature fusion. Although it is a plug-and-play multi-scale sampling component that can be used for feature fusion, it is not suitable for standalone usage.

Dataset/Component	SPP-Fast	CSP-PAN	AP0.5
AI-TOD			27.5
	√		27.3
		√	32.2
	√	√	32.4
VOC-07			82.8
	√		82.4
		√	84.1
	√	√	84.9
TGRS-HRRSD			89.3
	√		89.4
		√	90.4
	√	√	90.6

shows that the proposed method achieves a boost of 1.3% mAP on the VOC test set and nearly 4.7% mAP on the AI-TOD test set.

As analyzed above, adding SPP-Fast after the backbone without consideration will have negative effects, contrary to our expectations, because SPP-Fast has a limited range of influence and may cause information loss. If we couple CSP-PAN with SPP-Fast, the effect will be completely different, not only will the impact range be extended, but the lost information may also be filled from the cross-layer connection. Experiments show that the detection effect is optimal in the model wherein CSP-PAN is combined with SPP-Fast: 2.1% mAP improvement on the VOC test set and 4.9% mAP improvement on the AI-TOD test set.

4.5.3. Influence of Normalized Wasserstein Distance

Table 7. Effectiveness of normalized Wasserstein distance. We modified the existing models as previously described and conducted comprehensive tests on multiple datasets to explore NWD’s efficacy in terms of matching positive and negative samples, model training, and validation.

Dataset	Method	Assigning	Loss	AP	AP0.5
AI-TOD	Baseline			12.8	32.4
	NMD	√		18.9	46.2
			√	13.4	36.4
		√	√	21.1	49.9
VOC-07	Baseline			64.8	84.9
	NMD	√		62.2	80.2
			√	60.7	78.8
		√	√	60.5	78.6
TGRS-HRRSD	Baseline			62.5	90.6
	NMD	√		62.4	90.4
			√	63.0	91.1
		√	√	62.4	90.5

displays the effectiveness of NWD metric, 7.1% AP improvement is achieved with the NWD-based positive and negative sample assignment compared with the baseline method. This indicates that the small object sensitivity to IoU severely affects the assignment of labels, and this confounding error has a negative impact on the training. However, the NWD-based assignment strategy we adopt greatly improves the assignment quality and when we apply NWD to both assigning and loss modules, the best performance of 49.9% AP_0.5 is achieved.

As observed from the aforementioned table, the NWD values for objects of larger proportions undergo a significant reduction in the VOC-07 dataset, leading to a decline of almost 6% in mAP. In order to maintain the admirable descriptive features of NWD for small-sized objects, while simultaneously confronting its limitations in recognizing objects of diverse scales, we employed the multi-level NWD methodology, as delineated in the ensuing

Table 8. Comparison of NWD and multi-level NWD methods in VOC-07. The multi-level NWD effectively addresses the challenge of multi-scale target detection in NWD, yielding results comparable to those achieved with IoU.

Method	AP	AP0.5
IoU	64.8	84.9
NWD	60.7	78.8
Multi-level NWD	62.6	84.1

. In instances where the prevalent size of the objects in the dataset was small, NWD yielded better outcomes in comparison with the traditional IoU technique. Nonetheless, for entities exhibiting a varied range of proportions, normalizing NWD may impede the regression of bounding boxes, leading to distortion and reduced precision that may ultimately impair the model’s training. The said predicament can be circumvented successfully by employing the multi-level NWD technique, which facilitates closer alignment of the NWD yield to that of IoU in diverse scale contexts, minimizing the disparity to a mere 20% of its original value.

5. Conclusions

In this paper, we have proposed SAFF-SSD: an improved small object detection algorithm, which aims to address the issue of low small object detection precision in SSD and which has an obvious advantage over the previous work and the currently presentative methods. One of the main contributions is that we propose Local Lighted Transformer block (2L-Transformer) as a novel self-attention module, which is embedded in the lightweight EfficientNetV2-S as our backbone. Experimentally, we have shown that the proposed 2L-Transformer increases the effectiveness of EfficientNetV2-S for superior feature extraction. In addition, our detector can leverage feature maps of different scales by using CSP-PAN to improve the accuracy and stability of detecting small objects. This structure can achieve better performance than even the most advanced neck networks. Finally, we used a large number of reliable experiments to verify the superiority of normalized Wasserstein distance (NWD) in small object detection and slightly extend it to improve the problem of handling multi-scale targets. Moving forward, our future research efforts will be directed towards enhancing the real-time performance of small object detection.