A Review: Remote Sensing Image Object Detection Algorithm Based on Deep Learning

Abstract

Target detection in optical remote sensing images using deep-learning technologies has a wide range of applications in urban building detection, road extraction, crop monitoring, and forest fire monitoring, which provides strong support for environmental monitoring, urban planning, and agricultural management. This paper reviews the research progress of the YOLO series, SSD series, candidate region series, and Transformer algorithm. It summarizes the object detection algorithms based on standard improvement methods such as supervision, attention mechanism, and multi-scale. The performance of different algorithms is also compared and analyzed with the common remote sensing image data sets. Finally, future research challenges, improvement directions, and issues of concern are prospected, which provides valuable ideas for subsequent related research.

1. Introduction

Optical remote sensing image object detection has always been one of the research hotspots in the field of computer vision, and its main task is to locate and classify the objects of interest in remote sensing images [1]. As illustrated in Figure 1, the trained deep-learning algorithm is utilized to identify and predict the object of remote sensing images. The task of remote sensing image object detection is highly challenging and has broader application prospects, such as automatic driving [2], face recognition [3], pedestrian detection [4], medical detection [5], and so on. At the same time, object detection can also be used for image segmentation [6], image description [7], object tracking [8], action recognition [9], and other more complex computer vision tasks. Preprocessing, feature extraction, and object classification are typically the three processes of traditional remote sensing picture object detection techniques. First, the entire image is scanned using the sliding window approach, which may be configured with various aspect ratios to improve the capacity to recognize objects of various forms. This allows for the division of the image into multiple candidate regions.

Then, each sliding window is subjected to feature extraction. The matching technique scale-invariant feature transform [10] (SIFT), the [11] histogram of oriented gradient (HOG), and Haar features are examples of traditional features. These characteristics mainly depend on the image’s texture, color, and scale. By extracting features, the object can be more accurately characterized. Finally, the sliding window is classified using conventional machine-learning classifiers like support vector machines (SVMs) and Adaboost to perform object detection. Using the retrieved features, the classifier will decide if the object item is present in the sliding window. Accurate object recognition can be achieved by training the classifier and learning the feature distribution of the object. In 2001, the birth of V-J detector was mainly used for face detection. The HOG+SVM approach first surfaced in 2006 and was primarily used to detect pedestrians. In 2008, Felzenszwalb presented the deformable part model (DPM) methodology [12]. For a long time before the deep-learning method matured, the DPM algorithm has been playing a role in the field of object detection. These are conventional image processing and computer vision-based item detection techniques. Traditional object detection algorithms for optical remote sensing images mainly rely on hand-designed feature extraction and traditional machine-learning classifiers. These methods have a strong interpretability of features such as texture, color, and scale of the object. However, traditional methods have some limitations in dealing with complex scenes, object scale changes, occlusion, and so on. Traditional approaches also have certain restrictions regarding computational correctness and efficiency due to the high resolution and broad coverage area of remote sensing images.

Deep-learning-based object recognition techniques have steadily emerged in recent years as computer vision and deep learning have advanced, which can better solve the limitations of traditional methods and have made significant breakthroughs in remote sensing image object detection.

When compared to the conventional method based on manual features, the object detection method based on deep learning has significant advantages. Traditional methods usually require manual designs of feature extractors, and the effect is limited under complex scenes and highly variable conditions. However, object detection methods based on deep learning can learn feature representations directly from the original image via end-to-end training without relying on hand-designed feature extractors. The Convolutional Neural Network (CNN) deep-learning model can extract rich feature information from images and capture semantic information at different levels, thus improving the accuracy and generalization ability of object detection.

Deep-learning-based object identification algorithms like Region-based Convolutional Neural Networks (Faster RCNN) [13], You Only Look Once (YOLO) [14], Single Shot MultiBox Detector (SSD) [15], and others have recently been popular research topics. These algorithms effectively address the balance between accuracy and efficiency in object detection by introducing different network structures and optimization methods. For example, Faster RCNN [13] used the Region Proposal Network (RPN) to generate candidate object frames before classifying and locating objects utilizing a series of convolutional and fully connected layers. However, YOLO [14] turned the object identification challenge into a regression issue and used a single CNN model to directly forecast the category and location of the object.

This paper provides a detailed summary of the research progress based on the YOLO series, SSD series, candidate region series, and Transformer algorithm. This study also explores the practical applications of these algorithms in the domain of optical remote sensing image object detection. This study involves a comparative investigation of various algorithms’ performance, and the commonly used remote sensing image data sets are used. Additionally, the influence of common enhancement techniques, including supervised learning, attention mechanism, and multi-scale on the object detection algorithm is summarized. By conducting these comparisons and analyses, we can enhance our comprehension of the benefits and constraints associated with different algorithms. This will enable us to offer valuable insights and recommendations for future research endeavors, thereby fostering the advancement and utilization of this particular field.

2. Object Detection Algorithms for Optical Remote Sensing Images

2.1. Remote Sensing Object Detection Algorithm Based on Anchor Frame

2.1.1. YOLO Series Object Detection Algorithms

Due to their quick and precise capabilities, YOLO series object recognition algorithms have drawn a lot of attention. The first YOLO algorithm to be proposed was YOLOv1 [14]. Using an end-to-end single-stage network, it classified and located objects simultaneously, greatly increasing detection speed while retaining detection accuracy. YOLOv2 [16] improved YOLOv1 by introducing Darknet-19 as the backbone network and adopting the Anchor mechanism and multi-scale training strategy while using BN (Batch Normalization) to accelerate convergence and avoid overfitting. It had better recall and positioning accuracy and could adapt to multi-size image inputs. Based on YOLOv2, YOLOv3 [17] employed a more powerful darknet-53 backbone network and introduced the FPN structure to achieve multi-scale detection. It improved detection accuracy and speed by removing the pooling and fully connected layers and switching to logistic regression from softmax for classification. In addition, YOLOv3 provided a flexible version of tiny-darknet. YOLOv4 [18] introduced CSPDarknet53 as the backbone network with mosaic data enhancement and SPP+PAN structure to improve feature representation and cross-scale information fusion capabilities. It used GIOU_Loss instead of the Smooth L1 Loss function to improve the detection accuracy. The YOLOv5 [19] adopted CSPDarknet53 and Focus network based on YOLOv4 with Mosaic data enhancement. It used FPN (Feature Pyramid Network) and PAN (Pyramid Attention Network) structures for feature fusion and improved loss function and prediction frame filtering methods. Its lightweight model and high-accuracy object detection made it an excellent choice. The YOLOX [20] algorithm further improved its accuracy and speed via clever integration schemes such as data augmentation, anchor-free detection, and label classification. YOLOv6 [21] used EfficientRep Backbone and Rep-PAN Neck structure with an anchor-free paradigm and SIoU (Scale-Invariant Intersection over Union) bounding box regression loss optimization model. At the same time, it also utilized post-training quantization and quantitative perception training to improve the reasoning speed. YOLOv7 [22] was an improved object detection algorithm based on YOLOv5 and YOLOR, which introduced ELAN (Efficient and Lightweight Aggregation Network, ELAN), Rep (Reparameterization Convolution, Rep), and Auxiliary Head detection and other new techniques, and used YOLOX’s SimOTA (Sim Optimal Transport Algorithm, SimOTA) strategy and YOLOV5’s cross-grid search for label assignment. YOLOv7 was a worthwhile model to learn. YOLOv8 [23] was an improved version of YOLOv5 based on YOLOv5, which was fast, accurate, and easy to use for various object detection and tracking, instance segmentation, image classification, and pose estimation tasks. It adopted the overall structure of the CSPDarknet backbone network, the PAN-FPN neck, and the decoupled head. It improved the details such as using Context Cascaded Fully Convolutional Network C2f (Context Cascaded Fully Convolutional Network, C2f) module instead of the C3 (Context Cascaded Convolutional Network, C3) module and removing some convolutional structures in PAN-FPN. Compared with YOLOv5, YOLOv8 was fundamentally different, which was a model based on the idea of anchor-free. The object detection algorithm has undergone several iterations and upgrades of YOLOv1, YOLOv2, YOLOv3, YOLOv4, YOLOv5, YOLOX, YOLOv6, YOLOv7, and YOLOv8, which mainly includes the use of different backbone networks, the introduction of the Anchor mechanism, multi-scale training strategy, BN, logistic regression instead of softmax, FPN structure, CSP (CSP, Cross Stage Partial) Darknet, Mosaic data enhancement, SPP (Spatial Pyramid Pooling, SPP) and PAN structure, GIoU_Loss (Generalized Intersection over Union Loss, GIoU_Loss), Focus network, anchor-free detection, and other new technologies and methods, which significantly improve the speed and accuracy of object detection, and gradually move towards lightweight and efficient. Among them, YOLOv5 and YOLOX are relatively novel and excellent models, while YOLOv8 is an improved model based on an anchor-free idea.

Table 1. Comparison of YOLOv1-v8 versions.

Version	Time	Characteristics	Weakness	Advantage
YOLOv1 [14]	2015	Fully convolutional networks, high real-time, simple, and effective.	Poor positioning accuracy, and poor detection of small objects.	Fast, easy to implement and deploy.
YOLOv2 [16]	2016	Anchor box, multi-scale prediction, and Darknet-19 are used as the basic network with significant accuracy improvement.	Detection of small objects remains difficult.	Fast speed, high overall accuracy, and good robustness.
YOLOv3 [17]	2018	Multi-scale prediction, FPN, better objecting	Relatively slow, higher computing resources are needed.	High detection accuracy, the effect of small object detection is improved, and strong robustness.
YOLOv4 [18]	2020	Powerful detection performance, faster speeds, higher accuracy, CSPDarknet53 network.	Requires higher computational resources and higher complexity.	Excellent detection accuracy, good detection effect on small objects and occluded objects, and strong robustness.
YOLOv5 [19]	2020	Lightweight network, fast speed, high precision, easy to train and deploy.	Relatively new, there may be some instability and room for improvement.	High speed and high accuracy, efficient performance and training deployment efficiency, multi-language support, and smaller model volume.
YOLOX [20]	2021	Decoupled head, anchor-free, and SimOTA.	Currently, there are only 640 × 640 pre-trained weights.	Better performance at lower image resolutions.
YOLOv6 [21]	2022	Support model training, reasoning, and multi-platform deployment, improvement, and optimization of network structure and training strategy.	High false detection rate, version maintenance update speed is too slow, not suitable for industrial fields.	Full-chain industrial application requirements, improved and optimized network structure, and algorithm level.
YOLOv7 [22]	2023	Better precision and rate, with high accuracy and fast detection speed.	Newer knowledge may be difficult for some users to learn.	It has better accuracy and speed while ensuring accuracy and can process video and images in real time.
YOLOv8 [23]	2023	Fast and efficient, high accuracy, supports multi-category object detection, suitable for real-time applications.	Poor detection of small objects, A large amount of training data, and training time are required.	Fast speed, high accuracy, fitting real-time scenarios, supporting multi-category object detection.

provides a summary of the features, benefits, and drawbacks of YOLOv1-v8:

2.1.2. Remote Sensing Object Detection Algorithm of the YOLO Series

The architecture of the optical remote sensing object recognition algorithm based on regression analysis is shown in Figure 2. Due to its benefits of lightweight deployment, the YOLO series of object identification algorithms has the potential for a wide range of applications in the field of remote sensing. To increase the functionality of YOLO in the area of remote sensing photos, researchers have experimented with a variety of improvement and optimization techniques. For example, Tang et al. [24] proposed a moving object detection method based on dual-beam SAR that utilized two independent YOLO networks to fuse the sub-image results and improve detection accuracy. In tasks like aircraft detection in remote sensing photos [25], multi-source underwater object wake [26], and remote sensing building detection, the technique based on YOLOv5 proved accurate and effective [27]. FRN (Feature Refinement Network, FRN)-YOLO [28] was a feature refusion network for remote sensing object detection with high accuracy and robustness. In addition, there are some studies devoted to improving the performance of small object detection. Wei et al. [29] improved YOLOX and introduced a bilateral attention mechanism to effectively detect small-sized objects and reduce the false detection rate. A lightweight YOLOv4 algorithm with excellent efficiency and accuracy in remote sensing image recognition was proposed by Ma et al. [30]. It extracted feature information using the residual module and FPN and further improved performance via data enhancement and transfer learning. In addition, some researchers have considered the characteristics and application requirements of remote-sensing images. Li et al.’s [31] accurate and effective identification was made possible by tailoring the network structure and loss function to the properties of remote sensing images. The performance was then further enhanced using methods such as data augmentation and a priori box optimization. In order to improve performance, Hong et al. [32] accomplished multi-scale ship detection in synthetic aperture radar (SAR) and optical pictures and introduced multi-scale feature fusion. Yang et al. [33] improved the accuracy and robustness by redesigning the network structure and adjusting the loss function. In addition to the above methods, there are many improvements and applications based on the YOLO algorithm [34,35,36,37,38,39,40,41,42]. The related ideas of the above algorithms are shown in

Table 2. Comparison of object detection algorithms for remote sensing images based on YOLO series.

Literature	Paper Highlights	Applicability
Tang et al. [24]	Moving object detection method based on the YOLO, processing of dual-beam SAR images	Applying for moving object detection in dual-beam SAR images.
Jindal et al. [25]	Using the YOLOV5 architecture to realize aircraft detection in remote sensing images.	Applying for aircraft inspection tasks.
Shi et al. [26]	The improved YOLOv5 realizes the wake detection of underwater objects based on multi-source images.	Applying for the field of underwater object wake detection and having certain practical value.
Ding et al. [27]	Remote sensing image building detection with high accuracy and robustness.	Applying for remote sensing image building detection and using in urban planning, resource management, and other fields.
Sun et al. [28]	Feature re-fusion extracts details, reducing false detections.	Fast and accurate detection of the object, providing important data support.
Wei et al. [29]	Utilizing bilateral attention, detailed information about small objects in remotely sensed images is effectively captured.	The network focuses on improving small object detection.
Ma et al. [30]	Optimizing the network structure and parameters and achieving a lightweight.	Providing accurate object detection results.
Li et al. [31]	Data enhancement techniques are employed.	Real-time monitoring of ships, improving collection efficiency and safety.
Hong et al. [32]	Multi-scale detection, improved network structure and loss function, etc.	Maritime traffic monitoring, maritime patrol and island defense, marine resource management, and other areas.
Yang et al. [33]	The GIoU evaluation metric is introduced in the loss function.	Applying for aviation monitoring, military reconnaissance, disaster monitoring, and other fields.
Xin et al. [34]	Optimizing the characteristics of remote sensing images, such as high resolution and complex backgrounds.	Large-scale and complex remote sensing image data, with the ability to detect multiple objects simultaneously.
Wang et al. [35]	A saliency adjustment mechanism to weight the input image for saliency.	For efficient and accurate detection of ship objects in optical satellite images.
Zhang et al. [36]	Optimizing the network architecture, designing the loss functions, adjusting the prior frames, etc.	Processing large-scale image data for real-time ship inspection.
Zhang et al. [37]	Hybrid attention mechanism of similarity mask.	Efficient and accurate detection of small ship objects in optical remote sensing images.
Zhu et al. [38]	Methods such as introducing attention mechanisms, adapting feature fusion strategies, and optimizing feature propagation paths.	Handling the task of object detection in remote sensing images.
Zhou et al. [39]	A multi-scale feature fusion mechanism is introduced.	Handling vehicle detection tasks and combining multi-scale information to improve detection performance.
Liu et al. [40]	Specific training strategies and optimization methods for aircraft.	Handling the tasks of aircraft inspection and optimizing aircraft-specific problems.
Sharma et al. [41]	Designing a lightweight object detection model.	Handling real-time object detection tasks can be deployed for use on devices with restricted resources or limited computing power.
Wang et al. [42]	Adjusting the network structure, mining, and fusion of ship characteristics.	Handling the task of ship detection in remote sensing images and high detection accuracy can be obtained.

. Although the YOLO series algorithms have achieved significant advancements in the field of remote sensing image object detection, they encounter challenges due to the limited accuracy in detecting small objects, distant objects, and objects with typical contrast. Furthermore, these algorithms struggle to adapt to variations in data across different areas, seasons, and weather conditions. This paper discusses the various challenges that arise in ensuring the robustness and generalization capabilities of algorithms. In addition, in some application scenarios, it may be imperative to further improve and optimize the algorithm to improve the detection accuracy and reduce the false detection rate. Therefore, further research is needed to solve these problems and improve the performance of the algorithm.

2.1.3. Application of SSD Framework in Remote Sensing Detection

SSD [15] is an efficient single-stage object detection algorithm for object detection tasks in real-time scenarios. It realized object detection at different scales and categories by designing multi-scale feature maps and priori boxes and achieved good performance. In addition to remote sensing images, the SSD algorithm has also been applied to object detection in SAR images [43]. The SSD-based SAR object detection technique improved accuracy and resilience by integrating data augmentation and migration learning, and it also offered fresh suggestions for enhancing the SAR object detection algorithm. In addition, some other methods have achieved specific results in remote sensing image object detection, such as anchor-free methods [44], sparse anchor-guided capsule network method [45], etc. These approaches introduced many technologies, such as adaptive feature selection, channel filter fusion, and attention mechanism [46], to increase the precision and dependability of remote sensing picture object detection. In remote sensing image change detection, Yang et al. [47] proposed an end-to-end deep-learning framework that realized automatic, high-precision, and high-efficiency change detection and had better generalization performance. Additionally, certain enhanced techniques for object detection in remote sensing images were used. For example, the algorithm improved with the attention mechanism and feature fusion achieved excellent performance [48]. When employed to detect small objects in remote sensing photos, the SSD network with dilated convolution and feature fusion [49] produced great results. Meanwhile, Suidong et al. [50] proposed a multi-scale feature fusion and adaptive positive and negative sample filtering mechanism to improve the detection accuracy and performance of small objects in remote sensing images [51]. In conclusion, the SSD algorithm and its upgraded approaches for object detection in remote sensing photos have obtained good performance in many circumstances [52,53,54,55], which is of considerable significance and wide application value. However, further research is still needed to solve the specific challenges in remote sensing images, such as the low detection accuracy of small objects, remote objects, and low-contrast objects, in order to improve the performance and robustness of the algorithm.

2.2. Remote Sensing Object Detection Algorithm Based on Candidate Box and Regional Convolutional Neural Network

In the field of object detection, the RCNN series [56,57,58] algorithms dominated before the advent of YOLOv1. The architecture of the optical remote sensing object recognition algorithm based on the candidate region is shown in Figure 3. The spatially oriented object detection framework that Yu et al. [59] developed had the highest accuracy and resilience in remote sensing images and showed significant application potential in remote sensing image analysis. While this was going on, the upgraded Mask-RCNN model [60] was used to recognize objects in remote sensing images, and the segmentation accuracy and precision were enhanced using new techniques for multi-scale feature fusion, region-generating networks, and segmentation precision optimization. In addition, the aircraft detection method based on multi-scale Faster-RCNN successfully dealt with complex scenes and images of different scales by introducing multi-scale feature pyramids and RoI (region of interest, RoI) complete utilization strategy [61] and achieved excellent detection performance. Sha et al. [62] proposed a multi-scale aircraft object detection method for remote sensing images that improved object detection performance and had significant applications in the fields of aviation monitoring, military investigation, and other areas. In addition, using a multi-scale feature pyramid network and an adaptive label mapping strategy [63], In low-altitude UAV images, Singh et al. [64] proposed an efficient instance segmentation method for railroad sleepers, which was based on the Mask RCNN model and achieved accurate localization and segmentation with high accuracy and efficiency via pixel-level segmentation and optimization strategies. Further, He et al. [65] proposed a seismic wave P-wave arrival pickup method based on the Faster-RCNN model, which improved the picking accuracy by detecting the local window via the object orientation. This way, it can accurately detect the P-wave arrival time and has high computational efficiency. Regarding small object detection in optical remote sensing video, Feng et al. [66] proposed a motion-guided RCNN-based method that achieved high accuracy and robustness by combining motion and appearance features, providing an effective solution for small object detection. In summary, the RCNN series algorithms had a dominant position until the emergence of YOLOv1 and have been widely used in remote sensing image object detection. These methods include candidate region-based algorithms, spatial directional object detection frameworks, and improved Mask-RCNN and Faster-RCNN models. They improve the accuracy, robustness, and segmentation accuracy of object detection by introducing spatial context, direction information, multi-scale feature fusion, and region of interest strategy. However, these methods may face limitations such as high computational complexity and limited detection performance for small objects and complex scenes and need to be further improved and optimized to meet specific application requirements.

2.3. End-to-End Remote Sensing Object Detection Algorithm Based on Transformer Network

The end-to-end remote sensing object detection algorithm based on the Transformer network is an emerging method in the field of object detection and shows potential advantages in remote sensing images. These algorithms extract global context information by introducing a Transformer encoder and using a self-attention mechanism for object detection and object relationship modeling. Some specific methods include a multi-scale visual Transformer based on angle segmentation in Reference [67] and Efficient-Matching-Oriented Object Detection Transformer (EMO2-DETR) in Reference [68]. These methods can handle objects of different scales and directions and achieve high accuracy and robustness in remote sensing images. Compared with the traditional two-stage method, the end-to-end algorithm based on Transformer has the advantages of reducing computational complexity, processing scale changes and complex backgrounds, and providing more global context information, which is conducive to accurate positioning and classification of objects. However, these algorithms still face some challenges. Firstly, transformer-based algorithms usually have high computational complexity and require more computing resources. Secondly, due to the need for a large amount of training data to train the Transformer model, the data demand is enormous, especially in the field of remote sensing images.

Therefore, future research needs to explore further and improve the end-to-end remote sensing object detection algorithm based on Transformer to improve its performance and meet the practical application requirements. This includes research on reducing computational complexity, optimizing the training process, and introducing more domain-specific technologies. In general, the end-to-end remote sensing object detection algorithm based on the Transformer network has broad development prospects and application potential, but it still needs further improvement and research. As shown in Figure 4, the process of optical remote sensing object detection algorithm is based on a Vision Transformer (ViT).

2.4. Remote Sensing Object Detection Method for Specific Scenes

2.4.1. Object Detection in Remote Sensing Images Based on Supervision

Deep learning and other technical methods can be used to detect objects in remote sensing photos accurately, and this technique has potential applications in many different fields. For example, Li et al. [69] used deep-learning techniques to detect airports in a real remote sensing environment, extracted features via convolutional neural networks, and used multi-layer perceptrons for classification. With the help of transfer learning and data enhancement technology, satisfactory performance has been achieved. Wu et al. [70] proposed an end-to-end multi-side output fusion deep supervised network for change detection of remote sensing images, which achieved automated and accurate detection with high accuracy and consistency. In addition, there were several studies that used optimized data processing and feature extraction methods [71] to efficiently and accurately extract information from large-scale remote sensing data, which had potential value for remote sensing applications. Meanwhile, some studies based on weakly supervised learning and hierarchical fusion techniques [72] can effectively detect various types of objects and achieve good performance in complex contexts and backgrounds. In addition, the methods based on semi-supervised learning and object-first mixup techniques [73] have also been applied to remote sensing image object detection, which can effectively detect and locate objects with high accuracy and robustness. These studies also explored the application of semi-supervised learning in specific domains, such as power transmission tower object detection and SAR image object detection [74]. These techniques have a wide range of applications and can enhance the effectiveness of object detection by combining both a small amount of labeled data and a large amount of unlabeled data. Among them, the method of introducing semi-supervised migration learning and multi-layer feature fusion mechanisms [75] showed better performance in infrared object detection tasks. In contrast, the SAR object detection network method based on semi-supervised learning [76] improved the detection performance using unlabeled SAR image data to assist training. Finally, the semi-supervised SAR ship object detection method based on a graph attention network had better performance and robustness in ship object detection by constructing a graph structure and applying an attention mechanism [77]. Important application solutions for the fields of marine monitoring and safety prevention were supplied by these studies. The research shows that the method based on deep learning has achieved satisfactory performance in the fields of airport detection, change detection, and information extraction and shows the characteristics of high accuracy, robustness, and automation. However, remote sensing image object detection still faces some challenges, such as high computational complexity, processing requirements for large-scale data, and performance in complex backgrounds. Therefore, future research needs to improve further and optimize the algorithm to improve efficiency, accuracy, and adaptability to meet the needs of practical applications.

2.4.2. Remote Sensing Image Object Detection Method Based on Attention Mechanism

Performance can be increased using the attention mechanism in remote sensing image object detection. It improves the accuracy and robustness of object detection by assigning different attention weights to different regions in remote sensing images so that the network can pay more attention to the object region. Numerous fields, including remote sensing picture processing, geological research, agricultural monitoring, and others can use this technique. Wang et al. [78] proposed an improved model of Double U-Net for remote sensing image change detection. The model adopted a double-head structure and utilized multi-scale feature fusion and separable convolution techniques to efficiently process the images of the current and previous moments to achieve efficient change detection. The experiment proves that the Double U-Net model is superior to the traditional U-Net model with better performance and anti-interference ability, which is expected to be widely promoted in practical applications. A cross-modal attention feature fusion method for object detection in multispectral remote sensing images was proposed by Qingyun et al. [79]. In order to increase the reliability and accuracy of object identification, the technique combined a feature fusion network with a cross-modal attention mechanism. Experiments demonstrated that the method performed well in multispectral remote sensing images and could accurately detect and locate objects, which was of potential application. A quick self-attentive cascade network method for object detection in vast scene remote sensing photos was proposed by Hua et al. [80]. The method utilized the self-attention mechanism to extract features by combining cascade modules and performed object prediction via a classification regression module. Experiments proved that the method had good performance and high detection speed in large scene remote sensing image object detection. A dual-branch network approach for remote sensing image change detection was proposed by Ma et al. [81]. By considering the information and self-attention mechanism of the two moments at the same time, it can accurately detect the changing area and have application value. Zhang et al. [82] proposed a global context detection model using an attention mechanism and multi-scale feature fusion for optical remote sensing image object detection. By concentrating on region formation and feature fusion, our technique enhanced the detection efficiency and accuracy and completed the mission successfully. Nong et al. [83] proposed a method that utilized graph convolutional neural networks and attention mechanisms to achieve spatial relationship detection of remote sensing objects. The method’s ability to efficiently gather object connection information and enhance the object detection process’s robustness and accuracy was crucial in this field. An attention-guided twin network for change detection in high-resolution remote sensing images was proposed by Yin et al. [84]

The application of attention mechanisms in remote sensing image object detection has shown the potential to improve performance and superior performance in many application fields. In particular, by introducing attention mechanisms, such as cross-modal attention, self-attention, and global context attention, the detection accuracy and robustness of the object can be enhanced. However, the application of attention mechanisms in remote sensing image object detection still faces challenges, such as high computational complexity, processing requirements for large-scale data, and performance optimization in complex scenes. Therefore, future research needs to improve further and optimize the design of attention mechanisms to enhance efficiency, stability, and adaptability to meet practical applications’ needs better.

2.4.3. Remote Sensing Image Object Detection Method Based on Multi-Scale Processing

By processing remote sensing images at various scales, the multi-scale remote sensing image object detection approach can increase the detection accuracy and rate. In the specific study, Han et al. [85] proposed a contextual scale-aware detector that enhanced the perception of small and weak objects by introducing context information and scale-aware modules. Dong et al. [86] proposed a multi-scale building detection method based on boundary protection, which effectively solved the problems of missed and false detection in traditional methods. A quick and accurate approach to object detection in remote sensing images was proposed by Zhang et al. [87] and is based on aerial optical sensors. Song et al. [88] proposed an ERMF (edge refinement multi-feature, ERMF) method to improve the accuracy of dual-temporal remote sensing image change detection by fusing multiple features and adopting a refined segmentation strategy. Gao et al. [89] proposed a global-to-local (GL) network, which achieved accurate and robust remote sensing object detection via global and local feature fusion. Chen et al. [90] proposed the Info-FPN network improved the remote sensing image object detection algorithm via an information transfer mechanism and an adaptive weighted loss function. Su et al. [91] proposed the multi-scale context-aware RCNN method, which showed high accuracy and stability in detecting small sample objects in remote sensing images. Dong et al. [92] proposed the multi-scale deformable attention and multilevel feature aggregation method, which exhibited high accuracy and robustness in remote sensing object detection. Zhang et al. [93] proposed the multi-scale structural conditional feature transformation network showed high accuracy and robustness in object detection of remote sensing images. Dong et al. [94] proposed the convolutional neural network method based on appropriate object scale features to solve the detection problem caused by object scale differences in high-resolution remote sensing images using multi-scale features and adaptive pooling operations. Meng et al. [95] proposed a multi-scale convolutional neural network remote sensing object detection method that demonstrated higher accuracy and robustness on real datasets. Yao et al. [96] proposed the remote sensing multi-scale object detection method that utilized multivariate feature extraction and characterization optimization to achieve efficient and accurate detection of different scale objects in remote sensing images. Zhang et al. [97] proposed an all-around accurate detection algorithm for dense small objects and achieved efficient and accurate detection by introducing RPN and multi-scale feature fusion. Zhang et al. [98] proposed an image segmentation method that utilized an adaptive pyramid network and a parallel spatial channel to enhance performance. Zhou et al. [99] proposed the APS-Net (adaptive point set network, APS-Net), which achieved an accurate detection of complex scenes and small objects via adaptive mechanisms and multi-scale feature fusion. As shown in

Table 3. Comparison of object detection algorithms for remote sensing images based on multi-scale.

Literature	Paper Highlights	Applicability
Han et al. [85]	Context scale-aware detector, providing a new benchmark data set.	Remote sensing small weak object detection task in UAV images.
Dong et al. [86]	Accurate maintenance of building boundaries is emphasized, and the accuracy and robustness of detection are improved using a multi-scale strategy.	Building detection tasks in earthquake disasters and other remote sensing images.
Zhang et al. [87]	An efficient object detection method based on multi-scale aerial optical sensor.	Remote sensing image, aerial photography, UAV image analysis, and other fields.
Song et al. [88]	Innovative design with edge refinement and multi-feature fusion.	Two-phase remote sensing image change detection, surface environmental change monitoring, and other tasks.
Gao et al. [89]	Innovative design with scale perception and global-to-local strategy.	Remote sensing object detection and multi-scale object detection tasks.
Chen et al. [90]	An innovative design with an information feature pyramid and feature selection based on information gain.
Su et al. [91]	The small sample object detection of multi-scale context-aware.	Few-shot object detection task in remote sensing images
Dong et al. [92]	Multi-scale deformable attention mechanism and multi-level feature aggregation method are used to improve the accuracy and robustness of object detection.	Remote sensing image object detection tasks.
Zhang et al. [93]	The multi-scale structural condition feature transformation and attention module are introduced.
Dong et al. [94]	The method of applying object scale feature extraction and structural optimization.	High-resolution remote sensing image object detection task.
Meng et al. [95]	The method of multivariate feature extraction and characterization optimization.	Remote sensing multi-scale object detection tasks.
Yao [96]	The method of multi-scale fusion feature and convolutional neural network	Remote sensing imagery aircraft object detection Mission.
Zhang [97]	Using the specific object detection network structure and algorithm optimization technology to achieve accurate detection of dense small objects.	Detection task of dense small objects in remote sensing images.
Zhang [98]	An adaptive point set network and a point set modeling and matching method are introduced.	Optical remote sensing image target detection task.
Zhou [99]	Autonomous structure pyramid network and parallel space-channel attention mechanism.	Change detection task of high-resolution remote sensing images.

, the remote sensing image object detection method based on multi-scale processing has many applications in improving detection accuracy and detection rate. These methods effectively solve the problems of object size change and background complexity in remote sensing images by introducing multi-scale information, context awareness, feature fusion, and other technologies, and achieve good performance. However, these methods may face the challenges of high computational complexity and extensive memory usage when dealing with large-scale data and may have poor detection results for objects with large-scale changes. Therefore, in practical applications, it is necessary to optimize the algorithm’s efficiency and robustness to meet the needs of large-scale and multi-scale remote sensing image object detection.

2.4.4. Based on Deep Learning and Traditional Manual Feature Extraction Methods

Deep-learning technology has made significant progress in the field of computer vision and pattern recognition. However, traditional manual feature extraction methods are still significant in some fields. A deep-learning network structure was proposed by Wang et al. [100]. It is based on two-dimensional discrete wavelet transform and adaptive feature weighted fusion. Xu et al. [101] proposed a new neural network structure that realized pixel-by-pixel classification of high-resolution remote sensing images by introducing control gates and feedback attention mechanisms. Liu et al. [102] proposed a new network architecture called contourlet CNN (C-CNN). It combines traditional spectral analysis with convolutional neural networks to make feature representation for texture classification tasks sparse and effective. Hu et al. [103] proposed an object detection method based on a small number of samples called Gabor-CNN. By constructing a feature extraction convolution kernel library, improving the region proposal process, and using a deep utilization feature pyramid network, the accuracy and recall rate on a small sample data set are better than the existing comparison models, which have a strong application prospect. Chen et al. [104] proposed a hyperspectral image classification method that combines the Gabor filter and convolution filter. This would help alleviate the overfitting problem of deep CNN when there are not enough training samples. Zheng et al. [105] proposed a new frequency domain direction learning module that encodes the direction information of directional object detection via a frequency domain feature extraction network and a direction-enhanced self-attention layer. El-Khamy et al. [106] proposed a new convolutional neural network model using a discrete wavelet transform pool (DWTPL) to extract spectral information. They obtained a better classification accuracy and f1 score than other models on the AID dataset. This method can effectively process high-resolution images and improve scene classification performance. EL GAYAR et al. [107] proposed a new image decoding model, combining a wavelet-driven convolutional neural network with a two-stage discrete wavelet transform to extract salient features in images. Using the deep visual prediction model and long-term and short-term memory as the decoder, the model can automatically generate semantic titles according to the image’s semantic context information and spatial features. He et al. [108] proposed a method combining the adequate channel attention (ECA) module with the ResNet model to improve the accuracy of remote sensing image classification. The ECA module shows high accuracy on the AID dataset by avoiding dimensionality reduction and using a deep residual structure. Tsourounis et al. [109] proposed a new SIFT-CNN combination method, which implicitly obtains local rotation invariance by converting SIFT descriptors into multi-channel images and training CNN to use these images as input. Li et al. [110] proposed a color texture convolutional neural network, which combines the characteristics of traditional CNN and Gabor CNN. The image classification task is completed by inputting the original image and the Gabor processed image into two stream networks and combining the output at the end.

In summary, based on some research results of deep learning and traditional methods, these methods improved the accuracy and efficiency of remote sensing image processing. However, these methods may require a lot of data support and complex computing resources, and there are still limitations in specific scenarios, which need further practical application and improvement.

2.4.5. Fast Image Processing Method Based on VHR

With the wide application of VHR images, it is necessary to develop fast processing methods to improve processing efficiency. Huo et al. [111] proposed a new high-resolution image change detection method that uses fast object-level feature extraction and progressive change feature classification. By improving the distinguishability between changed classes and unchanged classes, dynamically adjusting training samples, and gradually adjusting the separation hyperplane, this method achieves higher classification accuracy and automation, and experiments prove its effectiveness. Mboga et al. [112] developed a land cover classification method based on a fully convolutional network, using aerial RGB images for end-to-end training and introducing skip connections to restore high spatial details. This method has a lower computational cost and performs better in edge depiction. Zhang et al. [113] proposed a coarse-to-fine large-size ultra-high resolution image registration method, which accelerates the acquisition of control points and image correction by calculating a unified device architecture. Saha et al. [114] proposed an unsupervised context-sensitive framework, called depth change vector analysis, for change detection of multi-temporal high spatial resolution images. This method uses convolutional neural network features to model the spatial relationship between pixels. It uses a hierarchical automatic feature selection strategy to select features emphasizing the change information of high and low prior probabilities. The depth change vector is obtained by comparing the depth feature supervector, and then the change pixels are identified. The experimental results show that the proposed method achieves effective change detection results on different data sets.

In summary, some fast for high-resolution remote sensing image processing methods are introduced. These methods have achieved specific results in improving processing efficiency and accuracy. However, in practical applications, it is still necessary to pay attention to the robustness and generalization ability of these methods in different scenarios, and further verify their applicability in large-scale data and complex environments.

3. Performance Evaluation and Comparison of Optical Remote Sensing Image Object Detection

3.1. Optical Remote Sensing Image Data Sets

Optical remote sensing image data sets play a vital role in remote sensing object detection tasks. These datasets provide valuable standard remote sensing data for model training and an objective and unified benchmark for comparing different networks and algorithms. In recent years, with the development of satellite remote sensing technology, some high-quality optical remote sensing image object detection data sets have become increasingly popular. In this paper, 15 representative data sets are selected for introduction. Their sample statistical information is shown in

Table 4. Overview of commonly used optical remote sensing image object detection datasets [115].

Data Set	Publisher and Content Description	Number of Object Categories	Number of Images
TAS [116]	Vehicle objecting dataset published by Stanford University.	1	30
OIRDS [117]	Vehicle objecting datasets published by Raytheon Corporation.	5	900
SZTAKI [118]	Rotating building object dataset published by Mta Sztaki.	1	9
UCAS-AOD [119]	Vehicle and aircraft object datasets published by CAS, and background negative samples.	2	976
NWPU VHR-10 [120]	The data set of aircraft, ships, oil tanks, baseball courts, tennis courts, basketball courts, and other objects released by Northwestern Polytechnical University.	10	1510
VEDAI [121]	Vehicle objecting dataset published by Caen University.	9	1210
HRSC2016 [122]	Ship objecting dataset released by Northwestern Polytechnical University.	1	1061
DLR3k [123]	Vehicle object dataset published by German Aerospace Center.	7	20
RSOD [124]	Aircraft, oil tank, stadium, and overpass object datasets released by Wuhan University.	4	976
TGRS-HRRSD [125]	Object datasets for ships, bridges, athletic fields, oil tanks, basketball courts, tennis courts, and other object data sets released by the Chinese Academy of Sciences.	13	21,761
LEVIR [126]	The object data set of aircraft, ships, and oil tanks released by Beijing University of Aeronautics and Astronautics.	3	22,000
ITCVD [127]	Vehicle objecting dataset published by Twente University.	1	135
DIOR [128]	Aircraft, airports, basketball courts, bridges, chimneys, dams, and other object data sets published by Northwestern Polytechnical University.	20	23,463
DOTA [129]	object data sets of ships, swimming pools, track and field fields, ports, helicopters, football fields, and other object data sets released by Wuhan University.	16	2806
FAIR1M [130]	The data set of 5 large categories and 37 fine-grained categories such as aircraft, ships, vehicles, stadiums, and roads published by the Chinese Academy of Sciences is the world’s largest fine-grained object detection and recognition data set for optical remote sensing images.	37	15,000

, including publisher, and content description, number of object categories, and number of images contained in the data set. These open optical remote sensing image data sets promote the rapid development of remote sensing object detection technology based on deep learning.

3.2. Algorithm Performance Evaluation and Comparison

Currently, the commonly used performance indicators for evaluating optical remote sensing image object detection algorithms are precision, recall, mean average precision (mAP), and frame per second (FPS). The accuracy reflects the proportion of actual positive samples in the test results. The recall rate reflects the proportion of positive samples that are correctly detected in all positive samples to be noticed, and there is a trade-off combination between accuracy and recall rate. The precision–recall curve (PR) can be obtained by plotting the precision as the ordinate and the recall as the abscissa. The area under the curve represents the AP of a specific category of objects. The AP mean of multiple categories is the average precision mean mAP, which represents the algorithm’s overall performance on the data set.

Table 5. Performance comparison of typical optical remote sensing image object detection algorithms.

Data Set	Algorithm	Backbone Network	Literature	Release Time	mAP/%
RSOD [124]	FPN-YOLO	DarkNet53	Sun et al. [28]	2021	87.40
	DAM-YOLOX	CSPDarkNet	Wei et al. [29]	2023	93.90
	YOLOv5-DNA		Xin et al. [34]	2022	77.51
	DF-SSD	ResNet-50	Qu et al. [45]	2020	51.78
	SSOD-RS		Zhang et al. [73]	2021	90.70
	RCNN-FCD		Su et al. [91]	2022	96.60
	MLFAM		Dong et al. [92]	2022	92.50
	I-SSD	VGG-16	Liu et al. [47]	2022	80.53
	AFF-SSD		Yin et al. [55]	2022	75.19
DOTA [129]	GSC-YOLO	CSPDarkNet	Ma et al. [30]	2022	93.44
	YOLOv4-CD		Zhu et al. [38]	2023	90.88
	SAHR-CapsNet		Yu et al. [50]	2021	93.04
	AF-SSD	ResNet-50	Lu et al. [44]	2021	52.60
	FFC-SSD		Xue et al. [52]	2022	74.90
	SOSA-FCN		Hua et al. [80]	2020	95.25
	ATMTransformer	DETR	Zhang et al. [67]	2022	77.30
	EMO2-DETR		Hu et al. [68]	2023	70.91
	Info-FPN	FPN	Chen et al. [90]	2023	75.84
FAIR1M [130]	YOLM	CSPDarkNet	Liu et al. [40]	2022	88.70
NWPU VHR-10 [120]	AF-SSD	ResNet-50	Lu et al. [44]	2021	69.80
	DF-SSD		Qu et al. [45]	2020	65.35
	FESSD		Shi et al. [54]	2020	79.36
	MSCNN		Yao et al. [96]	2019	96.00
	CenterNet	DLA-34	Liu et al. [49]	2020	95.70
	GCDN	ResNet-18	Zhang et al. [82]	2020	97.60
DIOR [128]	RSADet	DLA-34	Yu et al. [59]	2021	72.20
	CenterNet		Yu et al. [59]	2021	69.40
DIOR [128]	MLFAM	ResNet-50	Dong et al. [92]	2022	73.90
	MFC	MFE	Meng et al. [95]	2023	70.90
VEDAI [121]	YOLOFusion	CSPDarkNet	Qingyun [79]	2022	78.60
TGRS-HRRSD [125]	SOSA-FCN	ResNet-50	Hua et al. [80]	2020	97.25
	MSFT	SCFT	Zhang et al. [93]	2021	86.33
	MFC	MFE	Meng et al. [95]	2023	90.20

 displays a performance comparison of popular optical remote sensing picture object detection techniques. The following conclusions may be drawn from the previous analysis of the critical properties of various algorithms and the performance comparison of typical algorithms on the same data set in Table 5.

4. Challenge and Improvement Direction

For remote sensing object detection tasks, selecting data sets is crucial. Large-scale, diversified, and high-resolution data sets are needed, and the balance of different categories of samples is maintained. In terms of annotation, the bounding box is usually used to mark the position and size of the object, and the partial annotation or splitting of overlapping objects can be considered. For multi-class object detection, it can be transformed into a multi-label classification problem, which combines prior knowledge and deep-learning models to improve accuracy. For minor sample problems, data augmentation techniques and transfer learning methods can be used. In order to improve robustness, data enhancement, abnormal sample detection and repair, multi-scale information fusion, and other measures can be taken. To improve the interpretability of the model, many methods can be employed, including visual attention, model anatomy, introduction of rules, and interpretable network architecture.

When employing deep-learning techniques for remote sensing object detection, it is imperative to consider various challenges, such as computational cost, availability of datasets, and real-time processing capabilities. Many techniques can be employed to reduce the computational cost, including adopting lightweight models, utilizing hardware accelerators, implementing distributed computing, incorporating hardware acceleration libraries, and applying optimization algorithms. When the dataset is limited and expensive, it is possible to find a suitable dataset to cooperate or consider using synthetic data for training and reduce the amount of calculation via data preprocessing and enhancement. In terms of real-time processing capabilities, selecting real-time object detection algorithms and using parallel computing based on GPU clusters can improve processing speed.

5. Conclusions and Prospect

This paper summarizes the research progress of object detection in optical remote sensing images based on the YOLO series, SSD series, candidate region series, and Transformer algorithm. Among them, the YOLO series algorithm has fast detection speed and high accuracy, which is suitable for real-time scenes, but has limitations in small object detection; SSD series algorithms use multi-scale feature maps for object detection, which can better solve the problem of small object detection, but the detection accuracy is relatively low. The candidate region series algorithm performs object detection in two steps: candidate region extraction and classification, which has high accuracy but high computational complexity. The Transformer algorithm has made a significant breakthrough in the field of natural language processing. In recent years, it has also been applied to object detection tasks with good detection performance and interpretability.

Future research needs to focus on the following directions: small object detection, multi-scale information fusion, diversity and scale of data sets, and real-time and efficiency of algorithms. These directions will promote the development and application of optical remote sensing image object detection. In addition, there are some potential areas of concern:

Applying attention to new remote sensing tasks: In addition to object detection, the attention mechanism can be applied to other remote sensing tasks, such as instance segmentation, scene classification, etc. Researchers can explore how to use attention mechanisms to improve the performance and efficiency of these tasks.

Developing interpretable attention mechanisms: In order to improve the model interpretability, researchers can explore the development of interpretable attention mechanisms to explain model decisions. This can enhance the user’s trust in model prediction and help explain the model’s attention to different objects in remote sensing images.

Attention for multi-modal and multi-source remote sensing data fusion: With the progress of remote sensing technology, multi-modal and multi-source remote sensing data have become more abundant. Researchers can study how to effectively integrate information from multi-modal data (such as optics, radar, hyperspectral, etc.) and multi-source data (such as satellites, aircraft, ground sensors, etc.) to improve the performance and robustness of object detection.

Attention mechanisms for few-shot and semi-supervised learning with limited labeled data: In optical remote sensing image object detection, obtaining a large amount of labeled data may be difficult and expensive. Therefore, researchers can explore the semi-supervised learning method with few samples and combine the attention mechanism to make full use of limited labeled data to improve the model’s performance.

Lightweight attention modules to reduce computational complexity: In order to achieve real-time and efficiency in a resource-constrained environment, researchers can develop lightweight attention modules to reduce computational complexity. This will make it possible to perform optical remote sensing image object detection in an environment with limited embedded devices or computing resources.

Self-supervised pre-training with attention for improved generalization: Self-supervised learning has shown potential in optical remote sensing image object detection. Further research can explore how to combine the attention mechanism for self-supervised pre-training to improve the model’s generalization performance in different datasets and scenarios.

Extending to video analysis and change detection over time: In addition to static images, researchers can extend the attention mechanism to the field of remote sensing video analysis, such as object tracking, behavior recognition, and time-varying change detection. This will help to better understand the dynamic scene in remote sensing images and provide more comprehensive information.