A Novel Adaptive Edge Aggregation and Multiscale Feature Interaction Detector for Object Detection in Remote Sensing Images

Abstract

Object detection (OD) in remote sensing (RS) images is an important task in the field of computer vision. OD techniques have achieved impressive advances in recent years. However, complex background interference, large-scale variations, and dense instances pose significant challenges for OD. These challenges may lead to misalignment between features extracted by OD models and the features of real objects. To address these challenges, we explore a novel single-stage detection framework for the adaptive fusion of multiscale features and propose a novel adaptive edge aggregation and multiscale feature interaction detector (AEAMFI-Det) for OD in RS images. AEAMFI-Det consists of an adaptive edge aggregation (AEA) module, a feature enhancement module (FEM) embedded in a context-aware cross-attention feature pyramid network (2CA-FPN), and a pyramid squeeze attention (PSA) module. The AEA module employs an edge enhancement mechanism to guide the network to learn spatial multiscale nonlocal dependencies and solve the problem of feature misalignment between the network’s focus and the real object. The 2CA-FPN employs level-by-level feature fusion to enhance multiscale feature interactions and effectively mitigate the misalignment between the scales of the extracted features and the scales of real objects. The FEM is designed to capture the local and nonlocal contexts as auxiliary information to enhance the feature representation of information interaction between multiscale features in a cross-attention manner. We introduce the PSA module to establish long-term dependencies between multiscale spaces and channels for better interdependency refinement. Experimental results obtained using the NWPU VHR-10 and DIOR datasets demonstrate the superior performance of AEAMFI-Det in object classification and localization.

1. Introduction

Object detection (OD) is a widely explored area within computer vision and digital image processing, with applications in robot navigation, intelligent video surveillance, industrial inspection, and various other domains. With the refinement of satellite launching technology and the continuous development of spatial sensors, remote sensing (RS) technology has made marvelous progress. OD also plays an important role in processing large-scale optical RS images, and its influence extends to the field of spatial observation, which finds extensive application in environmental monitoring, ocean monitoring, precision agriculture, and urban planning, among others [1,2]. OD has received great attention because of its important scientific research value and wide application prospects [3]. In general, the purpose of remote sensing object detection (RSOD) is to automatically identify and localize objects of interest, such as buildings, vehicles, etc. [4]. The rapid advancement of convolutional neural networks (CNNs) has greatly enhanced the performance of OD in processing natural images. This progress has further stimulated the advancement of RSOD [5,6].

The development of deep learning has driven the rapid advancement of algorithmic models in various fields, such as image classification [7,8,9,10,11], behavioral recognition [12], and object detection. Prior to the widespread adoption of deep learning (DL) in computer vision, traditional OD algorithms [13,14,15,16,17], such as HOG [18], SIFT [19], and Adaboost [20], were considered relatively mature. However, they were often insensitive to variations in object scale, angle, and shape, which limited their effectiveness in handling diverse object instances. DL has emerged as a fundamental pillar in the field of OD owing to its exceptional feature extraction and representation capabilities [21,22,23]. This rapid development has led to the proliferation of OD frameworks based on DL, which are broadly classified into two-stage algorithms and one-stage algorithms. In two-stage algorithms [24,25,26], candidate boxes or regions are initially generated, and then object classification and positional regression are performed for each candidate. This type of algorithm has high computational complexity but high detection accuracy. One-stage algorithms [27,28,29] perform object detection and localization directly in the image without an explicit candidate box generation step. This approach has the advantages of good real-time performance, simplicity, and efficiency. However, the accuracy of these algorithms decreases in some complex scenes. Through continuous development, generalized OD methods have achieved remarkable success in processing natural images. However, when applied directly to RS images, the results have been less than satisfactory [30].

Many large open-sourced datasets, including DOTA [31], DIOR [32], NWPU VHR-10 [33], HRRSD [34], and HRSC2016 [35], have been used to delve further into RSOD algorithms based on DL. Normal cameras take natural images from a horizontal perspective, whereas RS images are taken by satellite or high-altitude airborne cameras from an overhead perspective. Because of the unique imaging perspective, the RSOD task is more challenging. This is because RS images are featured with complex backgrounds, dense instances, and significant scale variations, resulting in the features captured by the network not being aligned with the real objects.

RS images tend to exhibit higher levels of complexity than regular images. They often contain intricate backgrounds and densely cluttered instances of objects. Such complexity can result in a network’s focus that does not correspond to the actual objects of interest within the images. To address this problem, SCRDet [36] introduces a joint supervised pixel attention and channel attention network. This method aims to enhance detection performance by effectively addressing the challenges posed by cluttered scenes and small objects in RS images. ABNet [37] proposes an enhanced effective channel attention mechanism to enhance feature discriminative power by focusing on interactions between channels, which captures local cross-channel correlations and mitigates the obstruction of foreground objects by complex backgrounds. Liu et al. [38] propose a center-boundary dual attention mechanism for RSOD that captures features from specific regions. However, since convolutional kernels use a fixed receptive field for feature sampling, the features captured by the above attention methods are localized [39,40,41]. Most of the mainstream network models for RSOD nowadays ignore the important feature of the object’s edge features. Although objects within RS images will have various sizes of scales, the same class of objects shares almost the same geometric features and structural patterns when viewed from a top-down perspective, exhibiting similar edge characteristics. Huang et al. [42] proposed nonlocal-aware pyramid attention (NP-Attention) to capture multiscale long-distance dependencies of features. NP-Attention directs the neural network model to focus more on effective features and suppresses background noise, but it cannot fit the object’s shape well because of the limitations of the edge features it captures using traditional operators. Therefore, for OD tasks, the improved attention mechanism should be better able to learn multiscale nonlocal dependencies and acquire effective edge features of the object.

Second, RS images exhibit a notable characteristic where objects possess a considerable scale variation that surpasses the scale variation found in natural images. Consequently, this causes the extracted features to be inconsistent with the scale of the actual objects in the image. Feature pyramid networks (FPN) [43] are commonly employed to solve the multiscale problem in OD tasks. However, FPN follows a top-down pyramid structure and cannot propagate shallower features containing position and texture information to deeper layers. SME-Net [44] introduces the feature split-and-merge module. This module aims to eliminate the prominent features of large objects, allowing for better feature capture of small objects. It also transfers the feature information of large objects to deeper feature maps, thereby alleviating feature confusion. FGUM [17] uses a novel stream warping to align features in an upsampling operation, thus improving cross-scale feature fusion. However, these detectors do not consider long-range modeling and focus only on acquiring local features of the object. In addition, in RSOD, there are still challenges related to missing semantic information and correlation noise during the multiscale feature aggregation process. This occurs because the coarse-grained features extracted by the backbone network can be affected by noise arising from intra-class diversity and inter-class similarity, leading to potential interference. The contextual information contains important information that depicts the detected object. Especially for the detection of small-scale objects, context information is particularly important. Huang et al. [45] used a pyramid pool to summarize context information from various regions. This module constructs a global scene representation by incorporating contextual cues from different areas of the images. GCWNet [46] proposed a global context-aggregation module that combines contexts into a feature representation, thereby improving feature discrimination and accuracy. Similarly, SRAF-Net [47] is a scene-enhanced multiscale feature fusion network that enhances the discriminative power of the model by incorporating scene context information into the feature extraction process. Although these obtain richer feature information by obtaining contextual information, they also inevitably bring noise interference. Consequently, a better feature pyramid should use self-attentive modulation and effective context to learn global information about an object, capture the correlation between the object and the global scene, and fuse multiscale features more effectively [48,49,50].

In summary, the misalignment challenges associated with RS images may limit detector performance. To tackle the mentioned challenges, we developed a novel one-stage detection framework for the adaptive fusion of multiscale features, and we propose a novel adaptive edge aggregation and multiscale feature interaction detector (AEAMFI-Det) for RSOD. First, considering the effective assisting role of edge information, we design an adaptive edge aggregation (AEA) module that includes an edge enhancement mechanism. This mechanism facilitates adaptive focusing on multidirectional edge features, allowing the module to learn the spatial multiscale nonlocal dependencies effectively. Next, we developed a new context-aware cross-attention feature pyramid network (2CA-FPN) to enhance multiscale feature interactions. 2CA-FPN adopts a level-by-level feature fusion approach that fully fuses the features of various scales through multiscale feature interaction, effectively alleviating the problem of scale inconsistency between the extracted features and the actual objects. 2CA-FPN includes a feature enhancement module (FEM) designed to transform the self-attention of individual features into the cross-attention of pairs of features. Local and nonlocal context information [45] is first mined as auxiliary information to enhance the feature representation by interacting information between multiscale features in a cross-attention manner. Then, we introduce the pyramid squeeze attention (PSA) module [51] to process the spatial information of multiscale features further and establish long-term dependencies among multiscale channel attentions effectively for better interdependency refinement. Extensive experiments with challenging DIOR [32] and NWPU VHR-10 [33] datasets provide compelling evidence for the superiority of AEAMFI-Det. The paper’s key contributions are highlighted as follows:

• We propose a novel end-to-end one-stage framework for RSOD called AEAMFI-Det, whose most significant feature is multiscale feature adaptive fusion. Because of the framework’s characteristic of feature adaptive fusion, AEAMFI-Det can achieve high detection accuracy and robustness.
• A new 2CA-FPN is proposed to achieve multiscale perceptual aggregation with level-by-level feature fusion to mitigate the scale misalignment between extracted features and real objects effectively. The context-aggregated FEM in 2CA-FPN was designed to mine contextual information as auxiliary information to enhance feature representation by relating the information among the multiscale features in a cross-attention manner.
• We propose an AEA module with an edge enhancement mechanism to direct attention to focus adaptively on multidirectional edge features. The AEA module can learn spatial multiscale nonlocal dependencies and reduce the misalignment between the network’s focus and real objects.

The remainder of this paper is organized as follows. Section 2 describes the two RSOD modules. Section 3 describes the proposed AEAMFI-Det. The details and results of the experiments are presented in Section 4. Section 5 discusses the experimental results. The study effort, results, and conclusions drawn are presented in Section 6.

2. Related Work

2.1. Contextual Transformer (CoT) Block

Relying solely on an object’s features often falls short of achieving satisfactory results in RSOD. The contextual information surrounding an object contains important information depicting the object [15] and using contextual information can improve the accuracy and robustness of OD. Therefore, contextual enhancement in FEM is used to facilitate the contextual information interaction of multiscale features. We use a contextual transformer (CoT) block [52] to capture valid local and nonlocal contextual information.

As Figure 1 shows, the CoT block employs a $3\times 3$ convolutional operation to encode the input keys contextually to obtain the static contextual feature $K^1$. This encoding process enhances the input representation by incorporating contextual information, thus capturing the contextual relationships within the encoded features. We connect $K^1$ and query ${\mathrm{Q}}_c$ to learn the dynamic multi-headed attention matrix via two successive $1\times 1$ convolutions:

$$A=\mathrm{Cat}\left(K^1,Q_c\right)W_{\theta }{W}_{\delta }$$

(1)

where $W_{\theta }$ and $W_{\delta }$ are convolutions with and without ReLU activation functions, respectively, $\mathrm{Cat}(\cdot )$ represents the concatenation of operations along the channel dimension. We guide self-attention learning through $K^1$. We impose contextual attention to value $V_c$ to compute feature $K^2$:

$$K^2=V_c\times A$$

(2)

where $K^2$ is a dynamic context feature that represents the adaptive feature interaction across the inputs containing the context. The final output is the fusion of static context features and dynamic context features.

2.2. Pyramid Squeeze Attention (PSA) Module

A pyramid squeeze attention (PSA) module [51] is introduced at the end of AEAMFI-Det to fuse the obtained multiple features and perform information interaction between the channel attentions for better interdependence refinement. As Figure 2, the PSA module has two main parts: multiscale feature extraction and channel attention weight acquisition. First, features of different spatial resolutions containing different levels of information can be obtained using grouped convolutions of different sizes in the pyramid structure to extract key information from feature maps at different scales. Through the compression of channel dimensions in each branch and the execution of local cross-channel interactions, the approach effectively extracts multiscale spatial features from the channel-level feature maps. For computational convenience, we set the number of grouped convolutions to four.

$$F_i=Con{v}_{(2\times (i+1)+1)}(X),i=0,1,2,3$$

(3)

$$F_{\mathrm{O}}=\mathrm{Cat}([F_0,F_1,F_2,F_3])$$

(4)

where $Con{v}_{(2\times (i+1)+1)}(\cdot )$ represents the dimensions of the convolution kernel used in group convolution, $F_i$ represents the feature map with channel $\raisebox{1ex}{$C$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$ in each branch, and $F_O$ represents the overall multiscale feature map output in the four branches. $F_i$ is passed through the SE weight acquisition module along the channel dimension. This module calculates attention weights for feature maps of various scales. The weights are then normalized using the $softmax$ function to assign importance to each feature map. Finally, the weights of each channel are concatenated along the channel dimension and multiplied by $F_O$ to obtain the final output. The whole process can be written as follows:

$$Out=F_O\odot \mathrm{Cat}(softmax(SE(F_i)))$$

(5)

where $F_i$ represents the overall multiscale feature of each channel, $SE(\cdot )$ represents the weight acquisition operation in SENet, and $\odot$ represents the multiplication between channels.

3. Proposed Method

Substantial scale variations, dense distribution, and complex backgrounds in RS images introduce challenges in accurately detecting objects. These factors often result in misalignment between features extracted by the network and the actual objects in the images. This misalignment is manifested as a misalignment between the focus of the network and the actual objects in RS images and a scale misalignment between the extracted features and the actual objects. Previously proposed object detectors, especially one-stage frameworks, cannot produce satisfactory detection performance in this case. Several approaches have been proposed to address these problems, but they use coarse-grained features for learning and prediction, which may lead to incorrect classification and regression due to the lack of rich semantic features and necessary feature interactions.

This paper proposed a novel end-to-end one-stage framework for RSOD called AEAMFI-Det that detects objects in RS images. This framework combines context-awareness and cross-attention to establish dependencies between multiscale features. It employs an edge enhancement mechanism to adaptively focus on multidirectional edge features, which enables the object detector to capture more discriminative and informative aspects of the objects and thus improves the accuracy of the object detector. This section presents an outline within the suggested framework, following which two crucial components are mainly described: AEA and 2CA-FPN with embedded FEM. These components help to solve the problems of feature misalignment and coarse-grained feature noise interference extracted by the OD model and thus improve the accuracy and robustness of RSOD.

3.1. Framework Overview

AEAMFI-Det is a one-stage framework that combines CNN and cross-attention for multiscale RSOD in realistic scenarios. The framework enables long-range modeling while suitably retaining the inductive bias of CNNs, which allows for superior accuracy while keeping the computational effort relatively low. As Figure 3 shows, the RS image $I\in {ℛ}^{H\times W\times 3}$ is provided as input. Four feature layers, from low levels $\left({\mathrm{E}}_1{\text{\&}\mathrm{E}}_2\right)$ to high levels $\left({\mathrm{E}}_3{\text{\&}\mathrm{E}}_4\right)$, are first extracted using the CSPDarknet53 [53] backbone, containing coarse-grained features. Next, a new aggregation strategy, AEA, is used to fuse low-level edge features $\left({\mathrm{E}}_1{\text{\&}\mathrm{E}}_2\right)$ adaptively. This aggregation strategy uses an edge enhancement mechanism to attend adaptively to multidirectional edge features and enables the model to learn and incorporate spatial dependencies across multiple scales. Then, in an attempt to enhance the multiscale feature interactions, a new fusion architecture, 2CA-FPN, is designed. The 2CA-FPN fully fuses the feature information of different scales by adopting a level-by-level feature fusion approach. In 2CA-FPN, we design FEM, which first mines effective local and nonlocal contextual information as auxiliary information to acquire semantic similarities and differences across multiscale features through information interaction at the spatial level in a cross-attention manner. After effectively extracting the multiscale spatial feature information at a finer granularity level, the PSA module is presented to process spatial information with multiscale features further to establish the long-term dependency between multiscale channel attentions effectively to achieve better refinement of the spatial and channel dimensional interdependencies. Finally, the detection head of the framework consists of two compact and parallel sub-networks: one for classification and the other for regression. These sub-networks work in tandem to generate the final detection results.

3.2. AEA Module

We include edge information as auxiliary clues in the model to enhance the network’s ability to capture fine-grained features. We employ the DCN v3 [54] method to integrate the extraction of edge features seamlessly into the CNN. This method uses convolutional layers to effectively extract edge features, allowing for a cohesive integration within the overall network architecture. In the traditional convolution operation, a fixed convolutional kernel is used for sampling and cannot adapt to different object shape and size variations. In contrast, deformable convolution allows the convolution kernel to deform within a certain range, thus effectively capturing object’s shape and contour information [55] and enabling the study of more accurate feature representations. By incorporating the DCN v3 operator, the model can capture long-range dependencies more effectively, and it can dynamically adjust the aggregation of features based on the specific spatial characteristics of the objects being detected.

In general, low-level features encompass a mix of visual clues and noise, including edges, corners, and lines, as well as uncorrelated high-frequency signals [56]. However, this amalgamation can lead to a misalignment of the network’s attention with the actual objects in RS images. To address this issue, we introduce the AEA module, specifically designed for the low-level feature map $\left({\mathrm{E}}_1{\text{\&}\mathrm{E}}_2\right)$. The module uses an edge enhancement mechanism to capture edge information between features at different scales. We aim to selectively emphasize the relevant visual cues while suppressing noise and irrelevant information.

As Figure 4 shows, AEA consists of two branches that receive input $\left(E_1\in {ℛ}^{\frac{H}{8}\times \frac{W}{8}\times 128},E_2\in {ℛ}^{\frac{H}{8}\times \frac{W}{8}\times 256}\right)$ from the encoder and then adjust the number of channels by convolving block ${ℱ}_i^e(\cdot )$ by $3\times 3$, which can be expressed as follows:

$${\stackrel{-}{E}}_1={ℱ}_1^e\left(E_1\right),{\stackrel{-}{E}}_2={ℱ}_2^e\left(E_2\right).$$

(6)

${\stackrel{-}{E}}_1$ and ${\stackrel{-}{E}}_2$ effectively capture the nonlocal edge features of the object by DCN v3:

$${\tilde{E}}_1={ℱ}_1^{dcn}\left({\stackrel{-}{E}}_1\right),{\tilde{E}}_2={ℱ}_2^{dcn}\left({\stackrel{-}{E}}_2\right)$$

(7)

where ${ℱ}_i^{dcn}(\cdot )$ denotes the convolution operation of the edge extraction operator. We designed a selectively weighted switcher ${\theta }^s$ to learn adaptively how to weigh different attentions at different levels $\left(i=1,2\right)$:

$${\theta }^s=ℱ\left({\tilde{E}}_1\odot {\tilde{E}}_2\right).$$

(8)

The primary function of this switcher is to suppress background noise and emphasize discriminative edge features by mutually weighting them, thereby preserving and incorporating the valuable cues. However, this process carries the risk of disregarding valuable clues. To address this concern, we developed a residual aggregation process that combines features as follows:

$$F^e=ℱ\left(Cat\left(ℱ\left(\stackrel{-}{E_1}\oplus {\theta }^s\right),ℱ\left(\stackrel{-}{E_2}\oplus {\theta }^s\right)\right)\right)$$

(9)

where $ℱ(\cdot )$ and $Cat(\cdot )$ denote the $3\times 3$ convolution operation and the cascade operation, respectively. This process addresses the above issue well using residual learning [57]. Finally, we obtain the output feature map $F^e$, which contains rich edge information.

3.3. 2CA-FPN Module

The variation in object size can result in scale misalignment between the extracted features and the actual object. To address this misalignment, FPN introduces a hierarchical structure that combines high-resolution features from lower levels with semantically rich features from higher levels, but FPN only fuses with a simple convolution operation on lateral connections. Furthermore, FPN does not propagate low-level features containing more location and edge information from shallow to deep layers, and shallow features are not fully fused with deep features. In addition, the current approach in the multiscale feature fusion network focuses primarily on merging features at different scales without fully considering the contextual relationships within the object. As a result, the network may struggle to capture the holistic understanding of the object and exploit the contextual cues that can aid in accurate detection.

To address the above problems, firstly, we design a FEM that mines local and nonlocal valid contextual information as auxiliary information to enhance the feature representation by interacting information between multiscale features in a cross-attention manner. Meanwhile, we propose a new 2CA-FPN and embed FEM into it to enhance multiscale perception. Unlike FPN, the 2CA-FPN incorporates a level-by-level feature fusion strategy. This strategy aims to capture semantic similarities and differences across multiple scales.

3.3.1. 2CA-FPN

To enhance the representation of multiscale features, 2CA-FPN employs a level-by-level feature fusion method in a top-down path. Specifically, in the FEM, the input features ${\mathrm{P}}_4$ and ${\mathrm{P}}_3$ undergo a process of capturing semantic similarities and differences. This is achieved through the interaction of information at the level of space, where contextual auxiliary information is incorporated to enhance the feature representation of the output feature maps $T_4$ and $\stackrel{-}{T_3}$. ${\mathrm{P}}_4$ is obtained from ${\mathrm{E}}_4$ after $3\times 3$ convolutions, and ${\mathrm{P}}_4$ is upsampled and concatenated with ${\mathrm{E}}_3$ after convolution to obtain ${\mathrm{P}}_3$.

$$T_4,\stackrel{-}{T_3}=\mathrm{Fem}\left({\mathrm{P}}_4, {\mathrm{P}}_3\right)$$

(10)

$${\mathrm{P}}_3=ℱ\left(\mathrm{Cat}\left(\mathcal{U}\left(P_4\right), {\mathrm{E}}_3\right)\right), {\mathrm{P}}_4=ℱ\left({\mathrm{E}}_4\right)$$

(11)

where $\mathrm{Fem}(\cdot )$ and $\mathcal{U}(\cdot )$ denote the fusion of features through the FEM module and upsampling operations, respectively. Similarly, $T_2$ and $T_3$ are input into the FEM module for information interaction to enhance cross-image semantic similarity and mitigate the effect of correlated noise to obtain the enhanced output feature maps $F_2$ and $\stackrel{-}{F_3}$. $T_4$ is upsampled and concatenated with $\stackrel{-}{T_3}$ and then convolved to obtain $T_3$; $T_3$ is upsampled and concatenated with the AEA output of $F^e$, and then convolved to obtain $T_2$.

$$F_2,\stackrel{-}{F_3}=\mathrm{Fem}\left(T_2, T_3\right)$$

(12)

$$T_3=ℱ\left(\mathrm{Cat}\left(\mathcal{U}\left(T_4\right),\stackrel{-}{T_3}\right)\right), T_2=ℱ\left(\mathrm{Cat}\left(\mathcal{U}\left(T_3\right), F^e\right)\right).$$

(13)

Then, the bottom-up path is further fused to obtain $F_3$ and $F_4$.

$$F_3=ℱ\left(\mathrm{Cat}\left(\mathcal{D}\left({\mathrm{F}}_2\right),\stackrel{-}{F_3}\right)\right),F_4=ℱ\left(\mathrm{Cat}\left(\mathcal{D}\left(F_3\right), {\mathrm{T}}_3\right)\right)$$

(14)

where $\mathcal{D}(\cdot )$ denotes a downsampling operation.

3.3.2. FEM Module

Inspired by the concept of nonlocal blocks that are recognized for their ability to capture a broader context of contextual information [58], we propose a novel FEM module. This FEM module adopts a similar principle by prioritizing contextual enhancement to obtain rich features through the exchange of information between features at various scales. Unlike the self-attention mechanism, the proposed FEM facilitates information exchange between features at various scales through spatial-level information communication, which enables the module to understand and use the semantic relations among features better.

Since the inputs to the FEM are paired feature maps, we uniformly denote the input feature pairs of different scales as $\left(F^1,F^2\right)$, where $F^1,F^2\in {ℛ}^{C\times H\times W}$. As Figure 5 shows, the more representative features $Q$ and $K$ are first obtained using CoT blocks to capture both local and nonlocal context information of the input for $F^1$ and $F^2$, respectively, where $Q,K\in {ℛ}^{C_k\times H\times W}$, $C_k$ is the channel count of the low-dimensional feature map. We then reshape $Q$ and $K$ into the $C_k\times N$, where $N=H\times W$. The $Q$ and $K$ communicate across image information through feature interaction. Therefore, we designed two branches to augment the feature representation for each of the two inputs. We multiply query $Q$ and the transpose of key $K$ to obtain the first feature map and transpose that feature map to obtain the second feature map. Then, we compute the attention maps $A^1$ and $A^2$ using $softmax$ layers.

$$A_{ji}^1=\frac{\exp \left(Q_i\cdot K_j\right)}{{\displaystyle \sum _{i=1}^N\exp \left(Q_i\cdot K_j\right)}},A^2=transpose\left(A^1\right)$$

(15)

where $A_{ji}^1$ is the correlation degree between the $i^{th}$ pixel position of $F^1$ and the $j^{th}$ pixel position of $F^2$. The degree of similarity between these positions determines the strength of the observed correlation. Meanwhile, $F^1$ and $F^2$ are passed through the same convolution layer to generate two new features, $V^1$ and $V^2$, respectively, where $V^1,V^2\in {ℛ}^{C_k\times H\times W}$. Next, we reshape them into $C_k\times N$, where $N=H\times W$. We then do a matrix multiplication of $V^1$ and $A^1$ and reshape it into the shape of $C_k\times H\times W$. We do the same for $V^2$ and $A^2$. We feed them into convolutional block layers $\mathcal{C}{\mathrm{onvs}}_1$ and $\mathcal{C}{\mathrm{onvs}}_2$, respectively. Finally, we use residual connections to prevent valuable clues from being lost, resulting in the final results in Figures $E^1$ and $E^2$. They are calculated as follows:

$$E^1=\mathcal{C}{\mathrm{onvs}}_1\left({\displaystyle \sum _{i=1}^N\left(A_{ji}^1{V}_i^1\right)}\right)+F^1$$

(16)

$$E^2=\mathcal{C}{\mathrm{onvs}}_2\left({\displaystyle \sum _{i=1}^N\left(A_{ji}^2{V}_i^2\right)}\right)+F^2.$$

(17)

Incorporating global contextual information into enhanced feature graphs allows for selective aggregation of contextual information using attention graphs. This selective integration is based on the attention weights assigned to different regions. As a result, similar semantic features gain mutual benefit, enhancing intra-class relevance and promoting semantic consistency.

4. Experiments and Analysis

In this section, we present an overview of the experimental dataset, evaluation metrics, and practical implementation details. We then evaluate the effects of AEA and 2CA-FPN with FEM on AEAMFI-Det via ablation experiments. Finally, we present the results of a comparison of the performance of AEAMFI-Det with the performance of state-of-the-art methods, illustrating the superiority of AEAMFI-Det.

4.1. Dataset and Evaluation Metrics

We validated our proposed method using two datasets: NWPU VHR-10 [33] and DIOR [32]. NWPU VHR-10 is an open dataset that has been widely used in RSOD research. It consists of 715 RGB color images from Google Earth and 85 infrared images after pan-sharpening. It has ten object categories: airplane (AL), baseball diamond (BB), basketball court (BC), bridge (BD), ground track field (GTF), harbor (HB), ship (SP), storage tank (ST), tennis court (TC), and vehicle (VH). The NWPU VHR-10 dataset is challenging, with similarities between object categories, small visual differences in object backgrounds, and varying object sizes. Of all the samples in the database, 150 unlabeled negative samples were excluded from consideration in the experiment. We randomly selected 80% of the 650 positive sample images as the training set and used the rest for testing.

The DIOR dataset, released by Northwestern Polytechnical University in 2019, is currently the largest large-scale benchmark dataset for object detection of optical remote sensing images. DIOR contains 23,463 images and 192,472 instances. It has 20 object categories: airplane (AL), airport (AT), baseball field (BF), basketball court (BC), bridge (BD), chimney (CH), dam (DM), expressway service area (ESA), expressway toll station (ETS), golf course (GC), ground track field (GTF), harbor (HB), overpass (OP), ship (SP), stadium (SD), storage tank (ST), tennis court (TC), train station (TS), vehicle (VH), and windmill (WD). The DIOR dataset has more object categories than the NWPU VHR-10 dataset, more complex backgrounds, and larger scale variations. We randomly selected 11,731 images as the training set and 3910 images as the validation set and used the remaining 7822 images for testing.

The mean average precision (mAP) is a commonly used evaluation metric in OD tasks that can be used to evaluate the detection performance of a model comprehensively. The average precision (AP) for the category is obtained by calculating the area under the precision–recall curve. AP is an averaging of the area under the curve that takes into account the variation in precision at different recall rates. To account for the model’s performance on different categories, the APs of all the object categories were averaged to obtain the final mAP. The mAP is calculated as follows:

$$\mathrm{AP}={\displaystyle {\int }_0^1{P}_i\left(R_i\right)d{R}_i}$$

(18)

$$\mathrm{mAP}=\frac{1}{C}{\displaystyle \sum _{i=1}^C{\mathrm{AP}}_i}$$

(19)

where $C$ represents the number of categories included and $P_i$ and $R_i$ represent the accuracy rate and recall rate belonging to category $i$, respectively.

4.2. Implementation Details

We executed the algorithms using the widely used PyTorch1.7.1 framework and a 16-GB RAM Nvidia GeForce RTX 4080 GPU. First, we randomly flipped 50% of the images in the dataset horizontally to increase the amount of data. The input image size of this calculation model was $768\times 768$. SGD was used as the optimization algorithm, and the momentum, weight decay, and batch size were 0.9, 0.0001, and 4, respectively. The NWPU VHR-10 dataset was trained for 100 iterations. The learning rates were set to $1\times {10}^{-3}$ for the first 30 iterations that constituted the warm-up phase, $1\times {10}^{-4}$ for iterations 31–70, and $1\times {10}^{-5}$ for iterations 71–100. The DIOR dataset was trained for 150 iterations. The learning rates were set to $1\times {10}^{-3}$ for the first 30 iterations that constituted the warm-up phase, $1\times {10}^{-4}$ for iterations 31–90, and $1\times {10}^{-5}$ for iterations 91–150.

4.3. Ablation Evaluation

To assess the effects of the two proposed modules, i.e., AEA and 2CA-FPN with FEM, on the performance improvement of the AEAMFI-Det model, we performed ablation experiments on the NWPU VHR-10 dataset. These experiments aimed to investigate various network models, primarily focusing on the following architectures.

(1)

Baseline: We used the CSPDarknet53 backbone network, classical FPN, and universal OD head as the baseline network.

(2)

Baseline + AEA: baseline network with the addition of AEA.

(3)

Baseline + 2CA-FPN with FEM: We replaced the original FPN with 2CA-FPN with FEM on top of baseline.

(4)

AEAMFI-Det: The proposed multiscale featured adaptive fusion network, assembling AEA, 2CA-FPN with FEM and PSA on baseline network.

Table 1. Ablation studies of model components. The highest values in each row are indicated by bold numbers (%).

Methods	AL	GTF	ST	BB	BC	TC	SP	VH	HB	BD	mAP
Baseline	99.06	95.08	90.12	98.41	67.92	96.46	85.56	80.08	71.11	70.52	85.43
+AEA	99.69	96.61	99.63	98.43	87.73	97.81	89.94	90.38	77.24	74.64	91.21
+2CA-FPN with FEM	99.91	97.32	95.45	98.82	80.51	97.72	93.96	88.95	93.53	81.63	92.78
AEAMFI-Det	99.95	97.38	99.36	99.16	97.23	98.85	95.93	92.55	92.88	80.28	95.36

shows the results of each category in the ablation experiments for various models. From Table 1, it can be concluded that the baseline model has a mAP of only 85.43%. In contrast, both our proposed AEA and 2CA-FPN modules with FEM improve the model performance, thus improving the detection precision compared to the baseline model. With the addition of AEA, the mAP obtains an improvement of 5.78%, and the APs of each category are well improved, indicating that the use of edge enhancement mechanism to guide the network to learn the spatial multiscale nonlocal dependence adaptively significantly improves the detection accuracy. Baseline +2CA-FPN with FEM improves the mAP by 7.35%, proving that information interaction demonstrates the effectiveness of information interaction through cross-attention with the help of contextual information in RSOD. Finally, compared with the baseline, AEAMFI-Det increases mAP from 85.43% to 95.36%, a greater improvement than is achieved by adding either module alone. These results suggest that the combined use of AEA and 2CA-FPN with the FEM module in AEAMFI-Det synergistically enhances the model’s detection capabilities.

From an individual category perspective, the proposed model composed of a single module and AEAMFI-Det shows enhancements compared to the baseline. For example, the mAP of vehicle (VH) and storage tank (ST) are improved by 12.47% and 9.24%, respectively, which suggests that AEAMFI-Det is effective in accurately detecting and localizing small objects and effectively solves the misalignment between the focus of the network and the real objects in RS images. In addition, the features of harbor (HB) and bridge (BD) have some similarities. By adding context enhancement, the mAP of the two are improved by 21.77% and 9.76%, which proves that the proposed AEAMFI-Det effectively suppresses the noise caused by interclass similarity and enhances the intraclass correlation. Therefore, the proposed AEAMFI-Det can effectively solve the scale misalignment problem the extracted features have with the actual objects.

Figure 6 demonstrates the visualization effect of 2CA-FPN with FEM and AEA module. The baseline model accurately detects the storage tank but fails to detect the ship. By incorporating the AEA module into the baseline, the network successfully detects two ships of similar scales by leveraging the edge enhancement mechanism. It captures the edge information of the objects and learns the spatial multiscale nonlocal dependency. Furthermore, with the addition of 2CA-FPN with FEM on the baseline, the network effectively explores both global and local contextual information of the objects. It employs cross-attentive interactions to fuse information, enabling the capture of cross-image semantic similarity between multiscale features. This approach effectively suppresses matching noise caused by local similarity between classes and enhances intra-class correlation. As a result, the network successfully detects the small-scale ship. However, due to the limited number of samples in the NWPU VHR-10 dataset, the transformer-based improved FEM could not be adequately trained, resulting in the unsuccessful detection of another ship. In contrast, our AEAMFI-Det model combines the AEA module and 2CA-FPN with FEM, benefiting from their synergistic effects and achieving the best detection results.

4.4. Comparison Experiments

To evaluate the superiority of AEAMFI-Det, AEAMFI-Det was evaluated using the proposed excellent RSOD algorithms, and comparative experiments were performed on representative RSOD datasets. The selected control groups for comparative experiments include not only popular methods in computer vision such as Faster R-CNN [59], Faster R-CNN with FPN [43], Mask R-CNN, SSD [60], YOLOv4, YOLOv5, and YOLOX [61], but also RSOD-based methods such as NPMMR-Det [42] and SME-NET [44]. These OD algorithms encompass both one-stage and two-stage methods, providing a comprehensive representation of different approaches in OD.

(1) Comparative experiments with the NWPU VHR-10 dataset: More specific experimental values of the excellent detectors with respect to the NWPU VHR-10 dataset are provided in

Table 2. Comparison of object detection with other methods on NWPU VHR-10 (%), the highest values in each row are indicated by bold numbers.

Methods	AL	GTF	ST	BB	BC	TC	SP	VH	HB	BD	mAP	FPS
Faster R-CNN	98.65	95.49	90.63	89.55	66.33	87.33	79.91	69.17	80.67	67.05	82.48	45.44
Faster R-CNN with FPN	96.05	97.10	88.92	93.57	85.65	90.21	84.20	75.96	84.30	71.38	86.73	31.25
Mask R-CNN	85.28	97.69	88.10	90.05	75.20	90.13	82.31	73.68	89.65	67.65	83.97	—
SSD	84.61	99.45	85.31	88.41	71.23	89.33	71.52	62.67	86.41	64.95	80.39	133.13
YOLOv4	98.74	91.80	95.64	93.95	80.22	86.56	94.38	91.83	83.40	76.21	89.27	68.80
YOLOv5	99.62	99.29	98.01	95.22	81.11	89.86	95.55	91.40	90.53	79.76	92.04	67.37
YOLOX	99.82	97.03	99.74	98.03	91.96	98.08	94.69	91.19	91.40	76.85	93.88	63.55
NPMMR-Det	99.35	98.64	95.90	92.14	93.51	95.63	93.16	87.21	89.50	76.08	92.11	67.50
SME-NET	99.69	100.00	91.93	97.15	97.15	90.31	94.65	85.28	92.82	81.06	93.00	56.64
Proposed	99.95	97.38	99.36	99.16	97.23	98.85	95.93	92.55	92.88	80.28	95.36	48.74

. The best values are shown in bold. Since some papers report only the detection precision (mAP), not the detection speed (FPS), we re-implemented them and tested the algorithms in a fair environment.

The experimental results suggest that AEAMFI-Det achieved a mAP of 95.36%, which was better than the mAPs achieved by the other object detectors considered in the study. The mAP of the proposed approach was 2.36% better than that of SME-NET. The FPS of the proposed method is moderate because our approach adds AEA, 2CA-FPN with FEM, and PSA modules to the baseline, which significantly enhances the detection ability of the network, although this increases the amount of computation necessary and slightly affects the efficiency of the model. AEAMFI-Det achieved the top AP performance in 7 of 10 categories, and its mAP was the best among the methods compared. In conclusion, comparative experiments with the NWPU VHR-10 dataset confirm that AEAMFI-Det achieves the best performance.

Figure 7 displays the visualization results of the proposed AEAMFI-Det for the NWPU VHR-10 dataset. As the figure shows, vehicles (VH) obscured by surrounding objects and shadows are correctly detected, thanks to the inclusion of contextual information and enhanced intra-class correlation through cross-attention. For tennis court (TC), ship (SP), etc., which are affected by complex and variable backgrounds, we achieve correct detection by the action of the edge enhancement mechanism.

(2) Comparative experiments with the DIOR dataset: To validate the detection capability and generality of AEAMFI-Det for RSOD tasks more, comparative experiments were conducted using the DIOR dataset.

As

Table 3. Comparison of object detection with other methods on DIOR (%), the highest values in each row are indicated by bold numbers.

Methods	AL	AT	BF	BC	BD	CH	DM	ESA	ETS	GC	GTF	HB	OP	SP	SD	ST	TC	TS	VH	WD	mAP	FPS
Faster R-CNN	53.60	53.25	78.83	66.28	50.11	70.90	62.33	69.02	55.20	68.15	56.91	50.27	28.31	27.76	72.05	39.81	75.23	38.66	23.62	45.40	54.28	43.21
Faster R-CNN with FPN	54.32	70.62	66.50	80.10	55.00	71.52	57.96	66.54	62.10	72.81	76.12	41.85	42.56	72.30	56.88	52.63	79.92	55.20	44.58	80.60	63.01	28.59
Mask R-CNN	53.84	72.65	63.20	78.43	50.20	66.28	55.92	66.15	55.30	70.69	69.21	44.31	30.65	68.03	55.35	50.66	81.14	56.23	45.60	78.30	60.61	—
SSD	59.53	72.72	72.41	75.70	48.13	65.85	56.64	63.32	53.02	65.37	68.62	49.40	28.23	59.22	61.26	46.63	76.34	55.10	27.46	64.71	58.48	122.52
YOLOv4	80.30	86.63	75.52	82.15	60.40	70.10	66.31	75.92	65.82	84.20	78.03	60.21	47.74	89.15	58.05	72.85	90.98	63.37	58.43	81.28	72.37	65.36
YOLOv5	86.96	85.21	82.12	88.32	68.73	70.23	71.86	80.88	70.51	80.60	81.65	62.34	48.92	86.50	70.95	72.56	92.10	67.58	56.30	82.26	75.33	63.48
YOLOX	88.83	85.43	80.40	91.21	70.20	70.05	72.59	83.21	68.54	81.96	84.60	63.21	51.30	88.95	68.43	73.85	88.20	69.40	61.29	81.87	76.18	62.25
NPMMR-Det	88.61	86.49	83.21	90.94	65.02	71.43	74.86	82.55	71.49	83.20	83.54	60.32	50.66	87.58	72.16	72.00	90.94	70.23	55.68	83.52	76.22	66.43
SME-NET	87.63	88.21	82.31	92.14	71.64	72.11	73.41	83.70	70.51	84.30	84.49	61.65	48.52	90.50	73.10	72.12	93.41	69.85	58.62	81.52	76.99	55.57
Proposed	89.52	87.13	82.62	92.60	64.52	71.54	73.58	85.21	71.75	85.21	85.61	65.90	52.65	91.05	74.30	74.91	92.71	73.15	60.21	82.22	77.82	45.10

shows, AEAMFI-Det achieves the highest mAP of 77.82% of all one-stage detectors (e.g., YOLOX, NPMMR-Det, and SME-NET) when applied to the DIOR dataset and also outperforms some two-level detectors, such as Faster R-CNN, Mask R-CNN and Faster R-CNN with FPN. There are 16 categories with an AP of more than 70% and 10 categories with an AP of more than 75%. However, the performance of the proposed method in detecting objects in the overpass (OP) category in the DIOR dataset was unsatisfactory. This is because the training sample size for overpasses was small, which greatly increased the possibility of misidentifying overpasses as bridges.

To illustrate the detection capability of AEAMFI-Det, a visualized view of the DIOR dataset is shown in Figure 8. As the figure shows, AEAMFI-Det performs well in detecting not only small and dense objects, such as ship (SP), windmill (WD), storage tank (ST), and airplane (AL), but also large objects, such as ground track field (GTF), chimney (CH), and dam (DM). For classes with large variations in scale and shape, such as airplane (AL) and ship (SP), the problem of scale inconsistency between extracted features and actual objects is effectively solved by filtering noise interference caused by intra-class diversity and enhancing intra-class correlation through a cross-attention mechanism.

5. Discussion

Based on the analysis of the results of the ablation experiments in Section 4.3 and the results of the comparison experiments in Section 4.4, we conduct a discussion of the experimental results in this section.

First, ablation experiments are conducted on the NWPU VHR-10 dataset to investigate different network models, and based on the experimental results of different models, it can be seen that all of our proposed modules are able to improve detection accuracy to different degrees. The addition of the AEA module gives a good improvement in the AP of each category, indicating that the use of the edge enhancement mechanism to guide the network to adaptively learn the spatial multiscale nonlocal dependencies significantly improves the detection accuracy. Moreover, 2CA-FPN with FEM enhances the feature representation with the help of contextual information through cross-attention information interaction. The PSA module further processes the spatial information of multiscale features and effectively establishes the long-term dependency relationship between the attention of multiscale channels for better interdependency refinement. AEA, 2CA-FPN with FEM and PSA gain from each other, and AEAMFI-Det shows absolute advantages over adding either module alone.

AEAMFI-Det employs a level-by-level feature fusion strategy to capture semantic similarities and differences between multiscales, and effectively solves the scale misalignment between network-extracted features and real objects by mining global and local valid contextual information to interact information between multiscale features in a cross-attentive manner, suppressing the noise caused by interclass similarity. Meanwhile, the edge enhancement mechanism is used to learn the spatial multiscale nonlocal dependency and enhance the detection accuracy of small targets. The combination of the two effectively solves the misalignment between the focus of the network and the real objects in the image. The results of the ablation experiments illustrate our motivation well.

Comparing the detection performance of AEAMFI-Det and the state-of-the-art methods on the NWPU VHR-10 dataset and the DIOR dataset, it can be concluded that AEAMFI-Det is optimal based on the detection results of each detection algorithm. On the NWPU VHR-10 dataset, AEAMFI-Det obtained the best AP performance in 7 out of 10 categories, and the AP difference with other detectors was within 1% in 2 out of the other three categories. We added the cross-attention module to AEAMFI-Det, which is an improvement based on self-attention due to the large amount of image data required to train the transformer, resulting in slightly poorer detection results for ground track field categories with insufficient samples. From some of the visualization results, it can be concluded that vehicles that are occluded by surrounding objects and shadows are correctly detected, thanks to the inclusion of contextual information and the enhancement of intraclass correlation through cross-attention. For tennis courts, ships, and other cases affected by complex and variable backgrounds, we achieve correct detection through the action of the edge enhancement mechanism.

On the DIOR dataset, AEAMFI-Det had 16 categories with APs above 70% and 10 categories above 75%. Thirteen of the 20 categories had the highest AP performance, and six of the other seven categories had APs within 1.5% of the APs of the other detectors. The small number of training samples for bridges and the interclass similarity between the samples for bridges and overpasses greatly increases the likelihood of confusing bridge and overpasses detections. As can be concluded from some of the visualization results, AEAMFI-Det has excellent detection results for small and large objects. For objects with large-scale variations, such as airplanes and ships, satisfactory detection performance is obtained by filtering the noise interference caused by intraclass diversity and enhancing intraclass correlation through the cross-attention mechanism.

Overall, the proposed AEAMFI-Det effectively captures spatial and channel dependencies from both global and local perspectives, achieves feature capture through edge enhancement, context-assisted, and cross-attention, and employs a level-by-level feature fusion strategy to achieve multiscale perceptual aggregation, which effectively solves the misalignments between features extracted from the target model and the real object.

6. Conclusions

This paper proposes a novel one-stage detection mechanism with adaptive fusion in multiscale features called AEAMFI-Det for RSOD to address the misalignment between the model features extracted and the actual object. The mechanism can be used to locate and identify instances in RS images accurately. The AEA module uses an edge enhancement mechanism to bootstrap multiscale nonlocal dependencies in the network learning space and improve the misalignment of the network’s attention with actual objects. The 2CA-FPN enhances multiscale perception with level-by-level feature fusion, embedded FEM captures the effective context, and enhances the feature representation by interacting information between multiscale features in a cross-attention manner to effectively improve the scale misalignment between the features and the actual object. Long-term dependencies between channels are established by introducing PSA for better interdependent refinement of spatial and channel dimensions.

The structural design of the proposed method is complicated, resulting in a relatively high computational complexity. In the next few years, we will adjust the network with guaranteed precision and design a lighter, more general, and more effective modular structure to calibrate objects for RSOD better.