1. Introduction
SAR (Synthetic Aperture Radar) is a high-resolution active microwave remote sensing imaging technology, which can carry out remote sensing observations at any time and under any weather conditions, is not affected by clouds and atmospheric barriers, and has an all-weather observation capability that enables it to obtain richer information about the target [
1]. It has been widely used in the fields of earth observation and military reconnaissance, etc. SAR ship classification is a crucial step in Automatic Target Recognition (ATR) [
2], enabling precise identification of ship types or categories [
3]. This detailed maritime information is invaluable for enhancing marine surveillance, streamlining trade management, monitoring transportation activities, and supporting sustainable fisheries management [
4]. The classification of ship identification is frequently determined by the specific purpose or function of the vessel, with bulk carriers transporting large quantities of industrial and commercial goods, container ships handling vital international trade cargo, oil tankers transporting industrial oils, and fishing vessels harvesting marine fish [
5].
In the early stages of classification research, manual feature engineering was the dominant paradigm [
6,
7]. These features include geometric, texture, scattering intensity, and orientation gradient histogram features. The extracted feature set is then fed into a traditional machine learning, such as Support Vector Machines (SVMs), Decision Trees, Random Forests, Perceptrons, and Naive Bayes for ship category identification [
8]. For instance, Zhao et al. [
9] employed a hierarchical analysis method, utilizing a feature set comprising geometric, transformational, and local invariant features, to identify the most effective feature subset. They then leveraged a K-Nearest Neighbor (KNN) classifier to achieve successful classification of cargo ships, oil tankers, and container ships from TerraSAR-X imagery. Wu et al. [
10] delved into the joint optimization of feature selection and classifier design, proposing a BDA-KELM algorithm that automatically performs feature selection and parameter optimization to determine the optimal feature–classifier combination. Zhou et al. [
11] introduced MIMR, an optimization framework designed to maximize information content while minimizing redundancy. This framework allows for the extraction of a non-redundant subset of complementary features, enhancing feature representativeness and ultimately improving the classification performance of ships with limited pixel information in medium-resolution SAR images. The reliance on manual expertise, while effective in specific contexts, can become a bottleneck in multi-sensor fusion and diverse scene understanding, limiting the scalability and adaptability of these approaches.
Deep-learning-based ship classification methods for SAR imagery have been garnering significant interest from scholars in recent years. Researchers have leveraged deep convolutional neural network models to achieve SAR ship classification. These models have demonstrated superior accuracy and speed compared to traditional manual feature recognition models [
12]. Convolutional Neural Networks (CNNs) have garnered significant attention in SAR ship classification due to their ability to automatically learn discriminative features from labeled data, eliminating the need for laborious manual feature extraction. These learned features represent hierarchical representations of the target objects, and the end-to-end training paradigm simplifies the workflow and enhances efficiency. For instance, Li et al. [
13] developed a dense residual network (DRNet) incorporating upsampling data augmentation and in-batch balanced sampling to address the challenge of class imbalance in ship classification. They validated the efficacy of their approach on the publicly available OpenSARShip dataset. However, the large size of DRNet can impact training and testing efficiency. Bentes et al. [
14] proposed a multi-resolution input CNN to enhance multi-scale feature representation. They integrated four distinct CNN classifiers to distinguish between cargo ships, oil tankers, offshore platforms, and port facilities. Their experimental results demonstrated superior recognition accuracy compared to traditional machine learning classifiers like Support Vector Machines (SVMs). Dong et al. [
15] designed a residual learning network for fine-grained ship type recognition in Gaofen-3 SAR imagery, achieving high-accuracy classification of cargo ships, container ships, and oil tankers. Huang et al. [
16] explored the application of transfer learning in SAR ship classification. They transferred knowledge learned from the MSTAR and ImageNet datasets to the OpenSARShip dataset for ship classification. Furthermore, they proposed the Deep SAR-Net model to facilitate knowledge transfer from optical to SAR imagery, ultimately improving recognition accuracy for bulk carriers, container ships, and oil tankers. Despite the strong feature extraction capabilities of CNNs, their inherent inductive bias limits their ability to capture global context, potentially hindering further performance improvements.
Discriminative features for ship classification often manifest as subtle variations within localized image regions. Accurately identifying and harnessing these non-obvious cues is paramount to achieving robust classification performance. The deep self-attention network, Transformer, originally introduced in the seminal paper “Attention Is All You Need” [
17], revolutionized the field of natural language processing. It quickly became the dominant model in the field and formed the foundation for large-scale language models like GPT-3. Since 2020, Transformer’s influence has extended to computer vision, marking a significant advancement in the field. Following the pioneering work of Google’s Vision Transformer (ViT) [
18], several noteworthy vision Transformers have emerged, including Facebook’s DeiT [
19] and Microsoft Research Asia’s Swin Transformer [
20]. These models have demonstrated remarkable success in image classification tasks, highlighting the potential of Transformer to match or even surpass the performance of CNNs in computer vision. In 2022, Li et al. [
21] introduced the Transformer model to SAR ship detection, aiming to enhance ship feature representation. By leveraging a non-local neural network, the Transformer model establishes long-range spatial dependencies, enabling a broader global receptive field and allowing the network to focus on global contextual information. Their results demonstrated that the Transformer achieved superior detection performance compared to CNNs. The Transformer architecture excels at extracting target features by considering both global and local information. However, despite its high recognition accuracy, the model’s performance in detecting small objects remains a challenge, primarily due to the increased computational complexity associated with high-resolution features. To address these limitations, this paper introduces Deformable Convolution Network and Multi-scale Feature Fusion with Transformer architecture (DCN-MSFF-TR), a novel ship classification model specifically tailored for SAR imagery. Inspired by the Transformer encoder–decoder paradigm, our model incorporates the DCN deformable convolution module into the backbone network. This strategic integration enables the model to focus on sparse spatial information, thereby enhancing small target recognition in SAR images. Furthermore, to meet the demanding feature extraction requirements of ship classification, the model employs multi-scale attention processing of feature layers within the Transformer framework. This process is followed by feature fusion at appropriate locations within the model, guided by a feature pyramid. Consequently, each layer benefits not only from its own information but also from a comprehensive integration of features from other layers. Extensive evaluations conducted on the publicly available OpenSARShip-3-Complex (three-class) and OpenSARShip-6-Complex (six-class) datasets demonstrate the effectiveness of our proposed method. The experimental results confirm that DCN-MSFF-TR achieves superior performance in accurately identifying ship categories within SAR images.
3. Results and Analysis
3.1. Dataset Collections
The OpenSARShip dataset, which includes 11,364 SAR ship images integrated with AIS messages, possesses five key characteristics: specificity, large scale, diversity, reliability, and public accessibility [
24]. We constructed the OpenSARShip-3-Complex dataset by extracting a total of 3990 samples, comprising Cargo, Fishing, and Tanker ship categories, from the OpenSARShip dataset for training and testing. To mitigate the potential adverse effects of class imbalance on training, the number of training and testing samples for each ship category was balanced. The OpenSARShip-3-Complex is a dataset for categorizing SAR ship images into triples, with various categories of ship image samples. An example of the triple categorized data are displayed in
Figure 6. To enhance the data quality, data augmentation techniques such as rotation, mirror inversion, and brightness adjustment were applied, resulting in almost 4000 ship image samples. Details are provided in
Table 1.
To further validate the proposed method’s efficacy, we applied it to the publicly available OpenSARShip-6-Complex dataset. This dataset, comprising 6000 samples for six-class classification, is detailed in
Table 2, and a sample of the data are shown in
Figure 7.
3.2. Experimental Procedure
The experimental platform utilized Ubuntu 16.0 with an NVIDIA Tesla V100 GPU (NVIDIA, Santa Clara, CA, USA). Ablation experiments were conducted separately on the OpenSARShip-3-Complex (three-category) and OpenSARShip-6-Complex (six-category) datasets.
Figure 8 illustrates the convergence of the model loss function. The primary performance metrics for ship recognition are precision (P) and recall (R), where P = TP/ (TP + FP), R = TP/(TP + FN). Precision (P) is defined as the ratio of correctly predicted ships (True Positives, TP) to the total number of ships predicted as belonging to that category (TP + False Positives, FP). Recall (R) is the ratio of correctly predicted ships (TP) to the total number of ships actually belonging to that category (TP + False Negatives, FN). In addition to precision and recall, mean Average Precision (mAP) and mean Average Recall (mAR) are also used to evaluate overall performance. mAP represents the average precision across all ship categories, while mAR represents the average recall across all categories.
The experimental results, presented in
Table 3 and
Table 4, demonstrate the effectiveness of our proposed model enhancements. Specifically, our approach achieves a notable improvement in mean Average Precision (mAP) for both three-classification (+5.8%) and six-classification (+4.3%) recognition tasks.
This improvement can be attributed to addressing the limitations of the original Model A’s Transformer architecture. While advantageous for large target recognition, the Transformer’s self-attention mechanism struggles with the small target sizes characteristic of SAR imagery. Our enhancements target this issue.
Model B: By incorporating deformable convolution, Model B demonstrates a superior ability to capture small target features. Compared to Model A, this translates to mAP improvements of 0.3% and 0.9% for three- and six-classification tasks, respectively.
Model C: Multi-scale feature fusion (MSFF) enables Model C to leverage information across different scales. This results in even more pronounced mAP gains compared to Model A: +2.3% for three-classification and +1.3% for six-classification.
Model D: Our final model, Model D, integrates both deformable convolution and multi-scale feature fusion. This combination synergistically leverages the strengths of both approaches, achieving the most significant improvement in deep learning-based target recognition performance.
These findings highlight the importance of tailoring model architectures to the specific challenges posed by SAR imagery. Our proposed enhancements effectively address the limitations of standard Transformer-based approaches, leading to substantial improvements in small target recognition accuracy.
Figure 9 showcases the enhanced recognition capabilities of the improved model compared to its predecessor. The baseline model exhibits misclassifications for Tanker, Tug, and Dredging vessel types, and fails to detect Passenger vessels altogether. In contrast, the improved model accurately identifies all vessel types, demonstrating its superior performance.
Table 5 and
Table 6 present the classification accuracy achieved on the OpenSAR Ship three-category and six-category datasets, respectively. The results clearly indicate that DCN-MSFF-TR outperforms the original TR architecture, achieving higher ship recognition accuracy across all categories. This improvement underscores the efficacy of integrating the DCN and MSFF modules for enhanced ship recognition.
Specifically, the “Cargo” category consistently achieves the highest accuracy in both datasets, likely due to its greater representation in the training data. While “Fishing” vessels are accurately classified in the three-category scenario, their accuracy noticeably declines in the six-category dataset. This suggests potential difficulties in distinguishing between finer-grained ship types.
This observation highlights the challenge of class imbalance. Despite employing data balancing techniques to ensure uniform data distribution, categories with limited training instances still exhibit lower classification accuracy. Nonetheless, the strong performance of DCN-MSFF-TR, particularly with well-represented categories, demonstrates its capacity to effectively extract discriminative features from sufficient data.
Table 7 and
Table 8 present the categorized single-category recall rates on the SARShip-3-Complex and SARShip-6-Complex datasets, respectively. A comparison of the experimental results demonstrates that the DCN-MSFF-TR architecture effectively mitigates the issue of missed detections for single-category ships compared to the original TR architecture. The tables reveal an improvement in recall rates across all categories in both datasets, with an average recall improvement of 9.23%. Notably, the Cargo ship type exhibits the highest recall rate at 85.5%, performing well in both the three-category and six-category classifications. This superior performance is likely attributed to the relatively stable image features inherent to the Cargo ship type. Furthermore, the Fishing ship type demonstrates the most significant improvement in recall rate within the six-category classification. These results underscore the effectiveness of the DCN-MSFF-TR model in reducing the miss detection rate across all categories within the given datasets.
Figure 10 and
Figure 11 illustrate the classification results obtained on the Sentinel-1 OpenSARShip-3-Complex (three-class) and OpenSARShip-6-Complex (six-class) datasets, respectively.
Table 9 and
Table 10 present the confusion matrices for DCN-MSFF-TR on the OpenSARShip-3-Complex (three-class) and OpenSARShip-6-Complex (six-class) datasets, respectively. The dominant values along the diagonals of both matrices indicate that DCN-MSFF-TR correctly classifies the majority of ships. This observation highlights the model’s effectiveness in accurately recognizing different ship types.
3.3. Benchmarking against Other Models
Table 11 clearly demonstrates that DCN-MSFF-TR achieves superior recognition accuracy compared to other state-of-the-art methods. Specifically, on the OpenSARShip-3-Complex dataset, DCN-MSFF-TR achieves an accuracy of 78.1%, significantly surpassing the second-best performing model, SqueezeNet, at 72.2%. On the OpenSARShip-6-Complex dataset, DCN-MSFF-TR achieves a precision of 66.7%, substantially outperforming the second-best model, DenseNet, by 13.2%. Furthermore, DCN-MSFF-TR achieves recall rates of 78.0% and 75.0% on the OpenSARShip-3-Complex and OpenSARShip-6-Complex datasets, respectively, again exceeding the performance of the second-best model, DenseNet.
This enhanced performance is attributed to the synergistic integration of key components within the DCN-MSFF-TR architecture. Specifically, the deformable convolution network (DCN) allows for more effective feature extraction by adapting to the varying shapes and orientations of ships in SAR images. Simultaneously, the multi-scale feature fusion (MSFF) module within the Transformer architecture facilitates the aggregation of information from different scales, enriching the ship feature representation. This combination enables DCN-MSFF-TR to achieve higher accuracy in SAR ship recognition.