Adaptive Adversarial Self-Training for Semi-Supervised Object Detection in Complex Maritime Scenes

Abstract

Semi-supervised object detection helps to monitor and manage maritime transportation effectively, saving labeling costs. Currently, many semi-supervised object detection methods use a combination of data augmentation and pseudo-label to improve model performance. However, these methods may get into trouble in complex maritime scenes, including occlusion, scale variations and lighting variations, leading to distribution bias between labeled data and unlabeled data and pseudo-label bias. To address these problems, we propose a semi-supervised object detection method in complex maritime scenes based on adaptive adversarial self-training, which provides a teacher–student detection framework to use a robust pseudo-label with data augmentation. The proposed method contains two modules called adversarial distribution discriminator and label adaptive assigner. The adversarial distribution discriminator is proposed to match the distribution between augmented data generated from different data augmentations, while the label adaptive assigner is proposed to reduce the labeling bias for unlabeled data so that the pseudo-label of unlabeled data contributes to the detection performance effectively. Experimental results show that the proposed method achieves a better mean average precision of 91.4%, with only 5% of the labeled samples compared with other semi-supervised object detection methods, and its detection speed is 11.1 frames per second. Experiments also demonstrate that the proposed method improves the detection performance compared with fully supervised detectors.

1. Introduction

Maritime object detection plays an important role in waterway management, and its effective application greatly improves intelligent maritime transportation systems. Currently, vision-based maritime object detection methods [1,2,3,4,5,6] are becoming more and more popular because of their rich visual information and the popularity of marine surveillance. Additionally, multiple research studies on general object detection, which help to develop the field of maritime object detection, are proposed [7,8].

It is well known that object detection should estimate the locations and the categories of objects of interest at the same time. Although deep-learning-based detection methods [9] are good at learning rich and high-level feature representation directly from data, the large number of trainable parameters need a lot of labeled data to train a robust model. Labeling such large datasets is a very costly and time-consuming task. The complex scenes in real-world maritime conditions present many challenges, including occlusion, scale variations and lighting variations, which increase labeling costs. Vast unlabeled maritime data hold the potential for enhanced detection performance, making reduced reliance on labeled datasets a feasible and vital goal.

To improve robustness and alleviate overfitting with limited labeled data, some semi-supervised detection methods such as unbiased teacher (UT) [10] combined data augmentation and pseudo-labeling to achieve excellent detection performance under the teacher–student framework [11,12,13]. However, these methods used different pipelines of data augmentation, such as strong–weak augmentations, for labeled data and unlabeled data, which may produce distribution bias between labeled data and unlabeled data [14]. The generation of pseudo-labels on unlabeled data often employs a method involving a fixed threshold. As a result, these methods are likely to generate pseudo-label bias on unlabeled data whose distribution is different from the training data. This problem has occurred during object detection in complex maritime scenes where the semi-supervised setting degrades the detection performance.

To address the problems mentioned above, we propose a semi-supervised object detection method based on adaptive adversarial self-training, which consists of two modules called adversarial distribution discriminator (ADD) and label adaptive assigner (LAA). The proposed method aims to dynamically evaluate the learning status of the detection model and narrow the difference in data distribution in a teacher–student detection framework. Thus, the distribution bias and pseudo-label bias are reduced in complex maritime scenes during the semi-supervised training. The main contributions of this paper are summarized as follows:

(1) ADD is proposed to reduce distribution bias between different augmented views of training data, which encourages the detection model to generate unbiased and robust feature distribution with adversarial learning. This module also helps to alleviate the pseudo-label bias caused by distribution bias.

(2) LAA is proposed to reduce the pseudo-label bias, which uses augmentation with N times to obtain the average output. It adaptively estimates the current confidence thresholds of different classes to generate the accuracy pseudo-labels for unlabeled data.

(3) A new semi-supervised objective function is proposed to train in an end-to-end semi-supervised detection framework. This objective function consists of three items, including supervised loss, unsupervised loss and adaptive adversarial loss, which use class-balance loss to further alleviate the class imbalance problem.

This manuscript is organized as follows: In Section 2, literature that is relevant to our approach is reviewed. In Section 3, we give a detailed description of the proposed method. In Section 4, several comparative experiments are shown. Discussion is presented in Section 5. Finally, conclusions are provided in Section 6.

2. Related Works

Studies relevant to the proposed method are reviewed from two aspects: object detection based on deep learning and object detection based on semi-supervised learning.

2.1. Object Detection Based on Deep Learning

In recent years, deep learning has made great breakthroughs in the field of object detection [7,9]. Without designing feature representations by hand, they learn robust and high-level feature representation directly from data. Many detection models based on deep learning have been proposed and are becoming milestones [15,16,17,18]. Spraul et al. [19] reviewed a marine object detection study that amalgamated multiple datasets and incorporated various training techniques such as multi-scale training, which significantly bolsters detection accuracy. Kim et al. [2] fine-tuned Faster region-based convolutional neural network (R-CNN) [15] to detect targets and improve detection accuracy by performing Bayesian fusion on the detection results between video frames. It improved the detection performance of nearby objects in the videos, but scale variations in complex maritime would lead to frequent detection failures if Faster R-CNN was not trained on a large-scale dataset. Moosbauer et al. [3] trained the Mask R-CNN [20] to detect maritime objects with instance segmentation labels generated by local background subtraction. The Mask R-CNN [20] has more trainable parameters than Faster R-CNN [15], which is easier to fall into overfitting without enough labeled data. Shao et al. [21] proposed a saliency-aware convolutional neural network for maritime object detection using coastline segmentation to reduce the inspection range, improving the accuracy and efficiency of object detection. This method used the detection framework called you only look once (YOLO) [16], which was trained on a large-scale dataset to produce the initial detection results, and then performed the saliency detection to refine the detection results in complex scenes such as occlusion.

The methods mentioned above use pre-processing or post-processing to assist deep neural networks and improve detection results in complex maritime scenes such as occlusion, scale variations and lighting variations [22]. Many technologies are proposed to alleviate overfitting such as data augmentation [23,24,25] and dropout [26,27] when lacking enough training data. But they may produce the distribution bias for the limited labeled data and therefore damage the detection performance when the test data sampled from other maritime scene. We propose ADD to reduce the distribution bias of augmented data and match the labeled data and unlabeled data, so that the detection model can learn the unbiased and robust feature.

2.2. Object Detection Based on Semi-Supervised Learning

It is known that semi-supervised learning is a power technique to discover useful information from unlabeled data and limited labeled data [28]. Many classic semi-supervised methods are proposed including mean teacher [29], virtual adversarial training (VAT) [30], Pseudo-Label [31] and FlexMatch [32,33], spawning many excellent solutions for the semi-supervised object detection such as the teacher–student framework [34].

Chen et al. [35] proposed a domain adaptive detection framework to perform semi-supervised detection at maritime. It performed well for small objects, which is designed with cross-domain co-attention feature correlation and multi-level feature alignment. Jeong et al. [36] proposed a consistency-based semi-supervised learning method for object detection. It applied the consistency regularization to the divergence between the original image and its flipped version, which tends to encourage the learning of flip-invariant features. Tang et al. [37] transferred both visual and semantic knowledge to a target task that shares common attributes with the source task, which found that visually similar objects contributed semi-supervised learning. However, it still requires a more effective way to solve the problems in complex maritime scenes. Sohn et al. [38] proposed a semi-supervised detection framework with self-training, augmentation and consistency regularization, referred to as STAC. It pretrained a detector on the labeled data and then generate the pseudo labels for the unlabeled data. Strong data augmentations and consistency regularization were than applied to train a final detector. But this method relied on a well-trained model to generate the high-quality pseudo labels and could not support the end-to-end semi-supervised training. Liu et al. [10] proposed the semi-supervised detection framework based on mean teacher [29]. It utilized focal loss [17] to alleviate class imbalance and trained student network and teacher network in a mutually beneficial manner. To detect targets in complex maritime scenes well, its data augmentations need to be adapted to reflect the real maritime environment.

These detection methods above got the benefits from semi-supervised learning, which enabled the models to use the potential information of unlabeled data during training phase. Further, it is important to learn the robust features to improve the object detection in complex maritime scenes. Therefore, we propose a semi-supervised object detection method based on adaptive adversarial self-training, which provides a teacher–student detection framework to achieve semi-supervised object detection in complex maritime scenes. This method combines suitable data augmentation with accuracy online pseudo-labeling.

3. The Proposed Method

The proposed method aims to detect objects in complex maritime scenes within a semi-supervised setting, where the training set includes a labeled dataset $D_L={\{d_L^i,y_L^i\}}_{i=1}^{N_L}$ and an unlabeled dataset $D_U={\left\{d_U^i\right\}}_{i=1}^{N_U}$. $N_L$ and $N_U$ are the number of the labeled data and the unlabeled data, respectively, and $y_L^i$ is the annotation for the i-th labeled sample. It is noted that both the labeled and unlabeled data are sampled from the same maritime scenes, denoted as $D_M$. So their relationships can be formulated as $D_L\subset D_M$ and $D_U\subset D_M$. In addition, the data distribution of labeled and unlabeled data is constrained as $\underset{N_L\to \infty }{\lim }P\left(d_L\right)=\underset{N_U\to \infty }{\lim }P\left(d_U\right)$. In fact, $N_L$ and $N_U$ are limited with $N_L\ll N_U$, which leads to the statistically distribution bias between $P\left(d_L\right)$ and $P\left(d_U\right)$. Part of pseudo-label bias comes from this distribution bias as well as the class imbalance. Therefore, our work focuses on narrowing the distribution bias and applying the adaptive technologies to reduce pseudo-label bias, thereby contributing to the semi-supervised object detection in complex maritime scenes.

3.1. Objective Function Based on Adaptive Adversarial Self-Training

The training illustration of the proposed method based on adaptive adversarial self-training is shown in Figure 1. This figure presents the teacher–student framework, which primarily consists of six components: the data augmentation module (depicted in blue), the teacher network (in orange), the student network (in green), the ADD (in yellow), the LAA (in purple), and the objective function (in gray). It is noted that both the teacher network and student network use the same detection framework based on Faster R-CNN [15], where the backbone combines Resnet50 [39] with feature pyramid network (FPN) [40] to extract the multi-scale features. The proposed ADD and LAA are employed to alleviate distribution bias and pseudo-label bias, thereby enhancing semi-supervised detection performance in complex maritime scenes.

To start the semi-supervised training, the teacher network is pretrained with labeled data and then predict the outputs for the unlabeled data. In pseudo-labeling, the unlabeled data is fed into the data augmentation module to generate N weakly augmented samples. After the teacher network makes predictions on these samples, the LAA is used to generate the robust pseudo-label through ensemble judgement on different augmented version of the unlabeled sample. With the pseudo-labels, the student network can be trained in a “fully supervised” manner, where supervised loss and unsupervised loss are the important items in objective function. In the student network, a mix of labeled data and weakly augmented version of the unlabeled data is put into strong augmentation first. Subsequently, the student network generates a feature map from the weakly augmented set of the unlabeled data. Since different data augmentations will bring different data distribution that easily causes the pseudo-label bias in teacher network, the ADD is proposed to encourage the student network to improve the robustness of features derived from strong–weak augmented samples. This process of ADD is achieved through adversarial learning, which produces the adaptive adversarial loss. Combing the supervised loss, unsupervised loss and adaptive adversarial loss, the objective function can be formulated as follow:

$$\begin{array}{c}\mathcal{L}\left(D_L,D_U,{\theta }_s\right)={\mathcal{L}}_L\left(D_L,{\theta }_s\right)+{\lambda }_1{\mathcal{L}}_U\left(D_U,{\theta }_s\right)\\ +{\lambda }_2{\mathcal{R}}_A\left(D_L,D_U,{\theta }_s\right)\end{array}$$

(1)

where ${\theta }_s$ is the parameter of the student network. ${\mathcal{L}}_L\left(·\right)$ is the supervised loss, ${\mathcal{L}}_U\left(·\right)$ is the unsupervised loss and ${\mathcal{R}}_A\left(·\right)$ is the adaptive adversarial loss. ${\lambda }_1$ and ${\lambda }_2$ are the coefficient for corresponding items. For convenience, we omit the ${\theta }_s$.

In each training step, the objective function is optimized with stochastic gradient descent (SGD) [41], formulated as ${\theta }_s^i=\underset{{\theta }_s^i}{\arg \min }\mathcal{L}\left({\theta }_s^{i-1}\right)$, where $i$ denotes the updating step. After that, the parameters of teacher network are gradually updated to predicted stable pseudo-label via exponential moving average (EMA), formulated as:

$${\theta }_t^i=\epsilon {\theta }_t^{i-1}+\left(1-\epsilon \right){\theta }_s^i$$

(2)

where ${\theta }_t$ is the parameters of teacher network and $\epsilon$ is the smoothing coefficient.

At the end of the training, the teacher network aggregates the knowledge from the student network, which reduces the overfitting risk and the pseudo-label bias. We use the final teacher network as detection model to improve object detection in complex maritime scenes.

With the Faster R-CNN [15] as our detection framework, the supervised loss ${\mathcal{L}}_L\left(·\right)$ consists of four items: the classification loss and regression loss of region proposal network (RPN), the classification loss and regression loss of the task head for region of interest (ROI). They can be formulated as:

$$\begin{array}{ll}{\mathcal{L}}_L\left(D_L\right)=& {\displaystyle \frac{1}{\left|D_L\right|}}{\displaystyle \sum _{d\in D_L}{\mathcal{L}}_c^{RPN}\left(d\right)+{\mathcal{L}}_r^{RPN}\left(d\right)}\\ & +{\displaystyle \frac{1}{\left|D_L\right|}}{\displaystyle \sum _{d\in D_L}{\mathcal{L}}_c^{ROI}\left(d\right)+{\mathcal{L}}_r^{ROI}\left(d\right)}\end{array}$$

(3)

where $\left|D_L\right|$ denotes the number of the labeled samples. The classification losses ${\mathcal{L}}_c^{RPN}\left(d\right)$ and ${\mathcal{L}}_c^{ROI}\left(d\right)$ use the focal loss function [17] to address the class-imbalance about foreground-background and foreground classes. While the regression losses ${\mathcal{L}}_r^{RPN}\left(d\right)$ and ${\mathcal{L}}_r^{ROI}\left(d\right)$ use the smooth-L₁ function [42] to generate robust bounding box.

Similarly, the unsupervised loss ${\mathcal{L}}_U\left(·\right)$ for unlabeled data has the same mathematical form as the supervised loss, which leverages supervised information from the teacher network. It can be formulated as:

$${\mathcal{L}}_U\left(D_U\right)={\mathcal{L}}_L\left(D_U\right)\cdot f_1\left(\widehat{y}\ne 0\right)$$

(4)

where $f_1\left(·\right)$ is the indicator function and only outputs 1 when the pseudo-label $\widehat{y}$ is not equal to 0.

More details of the adaptive adversarial loss ${\mathcal{R}}_A\left(·\right)$ related to ADD and the robust pseudo-label $\widehat{y}$ related to LAA will be introduced in Section 3.2 and Section 3.3, respectively.

3.2. Adversarial Distribution Discriminator

In teacher–student framework, data augmentation and pseudo-labeling are two important technologies to improve the semi-supervised object detection. However, we found that the strong–weak augmentation would produce the distribution bias for the training data, which has the negative influence in the complex maritime scenes. To address this problem, we propose ADD that produces the adaptive adversarial loss ${\mathcal{R}}_A\left(·\right)$ to reduce the distribution bias, which is shown in Figure 2.

The proposed ADD uses three convolutional layers as the domain classifier with gradient reversal layer (GRL) [43], and makes a domain discrimination on the input feature maps, formulated as:

$$\left\{\begin{array}{c}g=f_D\left(f_{GRL}\left(F\right)\right)\\ F=F_w\oplus F_s\end{array}\right.$$

(5)

where $f_D\left(·\right)$ and $f_{GRL}\left(·\right)$ are the mapping functions of the domain classifier and GRL, respectively. $\widehat{g}\in {ℝ}^{b\times w\times h}$ is the augmentation prediction of domain map, where $b$, $w$ and $h$ are, respectively, the batch size, width and height of feature map. $F\in {ℝ}^{b\times w\times h}$ is a batch of feature maps concatenating feature maps $F_w\in {ℝ}^{w\times h}$ and $F_s\in {ℝ}^{w\times h}$ from weakly and strongly augmented version, respectively. ⊕ is the concatenation operation.

The value of target label $g\in {ℝ}^{b\times w\times h}$ of the predicted domain map is 0 if the feature maps from weakly augmented version otherwise 1. It is noted that sign of the gradient is reversed when passing through the GRL in back-propagation, which performs the adversarial training to reduce the feature distribution bias between labeled data and unlabeled data. So the adaptive adversarial loss in Equation (1) can be formulated as:

$${\mathcal{R}}_A\left(D_L,D_U\right)={\displaystyle \sum _{F\in D_L,D_U}{f}_c\left(g,f_{ADD}\left(F\right)\right)}$$

(6)

where $f_{ADD}=f_D·f_{GRL}$ is the mapping function of ADD. $f_c$ is the focal loss function [17] that address the hard domain samples, it is formulated as:

$$f_c\left(v,u\right)=\left\{\begin{array}{cc}-{\displaystyle \sum _{u,v\in u,v}{\left(1-u\right)}^{\gamma }\log u}& \mathrm{if} v=1\\ -{\displaystyle \sum _{u,v\in u,v}{u}^{\gamma }\log \left(1-u\right)}& \mathrm{otherwise}\end{array}\right.$$

(7)

where $u$ and $v$ are the inputs, $\gamma$ is the focusing coefficient usually set to 2.

We further define the feature distribution bias as $\left|P\left(F_s\right)-P\left(F_w\right)\right|$, where the feature map is obtained as:

$$\left\{\begin{array}{cc}{F}_w=f_E\left(\alpha \left(d\right)\right)& d\in D_U\\ F_s=f_E\left(\mathcal{A}\left(d\right)\right)& d\in D_L\cup D_U\end{array}\right.$$

(8)

where $f_E$ is feature backbone in the student network. $\alpha \left(·\right)$ and $\mathcal{A}\left(·\right)$ are the weak and strong augmentation, respectively. The weak augmentation includes techniques such as the gaussian blur, horizontal flip and identity mapping. The strong augmentation employs Cutout, color jitter, random fog, random rain, random sun flare and KeepAugment [44].

During the training in ADD, minimize the objective function is going to reduce the feature distribution bias by adversarial manner, formulated as:

$$\underset{{\theta }_s}{\min }\mathcal{L}\left(D_L,D_U,{\theta }_s\right)\to \underset{f_D}{\min }\underset{f_{GRL}}{\max }{\mathcal{R}}_A\left(D_L,D_U;F\right)$$

(9)

As the feature distribution bias is reduced, the labeled and unlabeled data come to share a more similar feature space, which helps the teacher network to extract the unbiased features and generate the accuracy pseudo-label for the unlabeled data.

3.3. Label Adaptive Assigner

In self-training, it is important to generate accuracy pseudo-label for the unlabeled data, which helps to improve the semi-supervised detection performance. Unlike pseudo-labeling with fixed threshold, we use an adaptive threshold filter shown in Figure 3 to determine the pseudo-label of an unlabeled sample based on its multiple augmented weakly versions.

As the sample augmented weakly with N_A times fed into the teacher network, the average output in LAA helps to achieve the consistency regularization, which reduces the pseudo-label bias, formulated as:

$$\overline{y}={\displaystyle \frac{1}{N_A}}{\displaystyle \sum _{k=1}^{N_A}{f}_{{\theta }_t}\left(d_k\right)}$$

(10)

where $\overline{y}=\left[{\overline{y}}_{\left(c\right)},{\overline{y}}_{\left(r\right)}\right]$ is the average output of teacher network, which includes the classification output ${\overline{y}}_{\left(c\right)}$ and regression output ${\overline{y}}_{\left(r\right)}$. $f_{{\theta }_t}\left(·\right)$ is the mapping function of teacher network with parameters ${\theta }_t$, $d_k$ is the input sample at k-th augmentation. N_A is the number of augmentations.

To consider the learning difficulties across all classes, we adjust the thresholds adaptively for each class using the adaptive threshold filter. Motivated from curriculum learning [33], the adaptive threshold filter determines the flexible thresholds for each class based on the current learning status of the detection model and saves the pseudo-labels whose confidence scores exceed these thresholds. The pseudo-label $\widehat{y}$ is obtained as follows:

$$\widehat{y}=\left\{\begin{array}{cc}{f}_{sharp}\left(\overline{y}\right)& \mathrm{if} \max \left({\overline{y}}_{\left(c\right)}\right)>{\mathcal{T}}_i\left(\arg \max \left({\overline{y}}_{\left(c\right)}\right)\right)\\ 0& \mathrm{otherwise}\end{array}\right.$$

(11)

where $f_{sharp}\left(·\right)$ is a sharpening function to reduce the entropy of the class distribution, ${\mathcal{T}}_i(c)$ is the flexible threshold for class $c=\arg \max \left({\overline{y}}_{\left(c\right)}\right)$ at training step i.

In more detail, the sharpening function is formulated as:

$$f_{sharp}\left(y\right)=\left[{\displaystyle \frac{y_{\left(c\right)}^{\delta }}{{\displaystyle \sum y_{\left(c\right)}^{\delta }}}},y_{\left(r\right)}\right]$$

(12)

where $\delta$ is the hyperparameter. As $\delta \to \infty$, the class distribution of $y$ becomes a Dirac distribution.

The flexible threshold ${\mathcal{T}}_i\left(c\right)$ is formulated as:

$${\mathcal{T}}_i\left(c\right)={\kappa }_i\left(c\right)\tau$$

(13)

where ${\kappa }_i\left(c\right)$ is the current learning status of the detection model with a range between 0 and 1, allowing ${\mathcal{T}}_i\left(c\right)$ to be adjusted within a certain range $\left(0,\tau \right]$. $\tau$ is the predefined threshold.

As the high threshold may produce accuracy pseudo-label but result in a low recall rate, a low threshold can easily produce false positives. We use the F1 score of each class to evaluate the current learning status of the detection model, which balances the precision and recall rate, formulated as:

$${\kappa }_i\left(c\right)=f_{F1}\left(d\right) \mathrm{if} f_1\left(c\in d\right)=1$$

(14)

where ${\kappa }_i\left(c\right)$ is updated only if $f_1\left(c\in d\right)$ that is the indicator function, and only outputs 1 when the class $c$ appears in the sample $d\in D_L$. $f_{F1}\left(·\right)\in \left[0,1\right]$ is the F1 score function to measure the learning status of detection model.

3.4. Semi-Supervised Training for the Proposed Method

We propose a semi-supervised object detection in complex maritime scenes based on adaptive adversarial self-training. The key components are the ADD and LAA, which are used to reduce the distribution bias and pseudo-label bias, respectively. There are two same network structures named teacher network and student network, which are parameterized by ${\theta }_t$ and ${\theta }_s$, respectively. As the student network pretrained with the labeled data and share its parameters ${\theta }_s$ to the teacher network, the teacher network acquires a certain level of discrimination to generate pseudo-labels for the unlabeled data. With the proposed ADD, the distribution bias of augmented training data from strong–weak augmentations is reduce through optimizing the adaptive adversarial loss ${\mathcal{R}}_A\left(·\right)$, which helps to improve the accuracy the pseudo-label. Also, the strong–weak augmentations applied in teacher network and student network, performs implicitly the consistency regularization for the training data. In each training step with the ground-true and the pseudo-label, the student network is trained in a “fully supervised” manner and its parameters ${\theta }_s$ is optimized by SGD method with momentum [41], formulated as:

$${\theta }_s^{\ast }=\underset{{\theta }_s}{\arg \min }\mathcal{L}\left(D_L,D_U,{\theta }_s\right)$$

(15)

After that, the parameters ${\theta }_t$ of the teacher network is updated via EMA, as shown in Equation (2), which helps to generate the robust pseudo-labels. Further, the proposed LAA leads to average output and an adaptive threshold filter to adjust the threshold for each class according to the learning status of detection model. This further reduces the pseudo-label bias and improve the accuracy of pseudo-labels. Finally, at the end of training, the teacher network is used as the final detection model.

4. Experiments and Results

4.1. The Dataset and Evaluating the Protocol

We have made a large marine object dataset combining two public datasets [45,46] to evaluate the proposed method. This marine object dataset consists of 16,980 images (1920 × 1080 pixels) and covers 14 categories. We split the marine object dataset into a training set and a test set at a 4:1 ratio. The training set therefore contains 13,584 samples and the test set contains 3396 samples. Among these training samples, up to 10% are labeled samples while the remaining samples are unlabeled samples whose annotations are artificially discarded. Specially, the test set typically contains the samples in complex maritime scenes including occlusion, scale variations and lighting variations, with 1661, 1144 and 940 samples, respectively. It is noted that some samples in the test set cover multiple scenes. Detailed information of the marine object dataset is shown in

Table 1. Detailed information of the marine object dataset to evaluate the proposed method.

Class Name	Label	Number	Percentage
Passenger ship	1	1536	2.41%
Ore carrier	2	3516	5.52%
General cargo ship	3	2426	3.81%
Fishing boat	4	3458	5.43%
Sail boat	5	2409	3.78%
Kayak	6	3532	5.55%
Vessel	7	22597	35.48%
Buoy	8	2474	3.89%
Ferry	9	5463	8.58%
Container ship	10	1773	2.78%
Other	11	4673	7.34%
Boat	12	1270	1.99%
Speed boat	13	5100	8.01%
Bulk cargo carrier	14	3454	5.42%
Total	-	63,681	100.0%

and

Table 2. Category information of the test set of the marine object dataset.

Category	Occlusion	Scale Variations	Lighting Variations
Passenger ship	2.8%	2.3%	2.5%
Ore carrier	6.3%	5.2%	6.0%
General cargo ship	4.3%	6.3%	3.7%
Fishing boat	6.4%	6.4%	5.4%
Sail boat	3.0%	3.8%	3.6%
Kayak	4.5%	5.6%	5.5%
Vessel	33.2%	31.4%	36.0%
Buoy	4.4%	4.1%	3.8%
Ferry	8.6%	8.6%	8.5%
Container ship	3.1%	2.8%	2.7%
Other	7.0%	7.4%	7.3%
Boat	2.2%	1.7%	1.8%
Speed boat	8.2%	9.2%	7.9%
Bulk cargo carrier	5.9%	5.2%	5.3%
Total	1161	1144	940

.

Following the standard protocol to evaluate detection task, we adopt (i) precision rate, (ii) recall rate, (iii) mean average precision (mAP) [47], (iv) frame per second (FPS) and (v) average center error (ACE) [4] as the evaluation metrics. Among them, mAP is the average of the average precisions (AP) for each class, where AP is calculated as the area under the precision-recall curve for that class. This metric is the tradeoff between precision and recall at different classification thresholds, providing a comprehensive measure of how well a detector can identify objects across multiple classes. ACE is taken as the metric for evaluating the location accuracy of object detection. It calculates the mean distance between the center points of the predicted bounding boxes and their corresponding ground truth. A lower ACE value indicates higher positional precision, indicating that the predicted locations of object centers are closer to the actual locations. The FPS evaluates the real-time performance.

4.2. Implementation Details

We propose a semi-supervised object detection method in complex maritime scenes based on adaptive adversarial self-training, which can be trained end-to-end in a semi-supervised manner. All training data are fed into the detection model in batches to minimize the objective function using SGD method with momentum. Due to the limitation of memory of GPU, the batch size is set to 2, which consists of labeled data and unlabeled data. The number N_A of weak augmentations is also set to 2. The input images are resized so that the shorter side is 800 pixels. Training epoch is set to 30 to achieve convergence, and first 5 epochs are dedicated to full supervision training using only the labeled data for the stability of model training. The settings of hyperparameters of the objective function in Equation (1) are as follows: ${\lambda }_1=0.1$ for the unsupervised loss ${\mathcal{L}}_U\left(·\right)$ to alleviate the pseudo-label bias in the early training stage; ${\lambda }_2=0.01$ for the adaptive adversarial loss ${\mathcal{R}}_A\left(·\right)$ as a regularization item to reduce the distribution bias. The hyperparameter $\epsilon$ of EMA in Equation (2) is set to 0.999. The hyperparameter $\delta$ of the sharpening function in Equation (12) is set to 0.5. The upper threshold $\tau$ in Equation (13) is experimentally set to 0.9 which is a high enough confidence. It is noted that the proposed method and the compared methods use FPN [40] and same data augmentations including Cutout, random fog, random rain, random sun flare and KeepAugment [44] to deal with the complex maritime scenes.

4.3. Experimental Results

To evaluate the proposed method in complex maritime scenes, three experiments are conducted as follows: (i) ablation studies on ADD and LAA; (ii) performance of various data amounts; (iii) performance of different methods.

(i) Ablation studies on ADD and LAA: In our way of semi-supervised training, ADD and LAA are proposed to reduce the distribution bias and pseudo-label bias. To analyze the influence of the ADD and LAA, we conduct the experiments by setting, respectively, ${\lambda }_1$ and ${\lambda }_2$ in Equation (1) to zero. We use 10% of the training samples as the labeled data (1358) and the remaining samples (12,226) as the unlabeled data. Experimental results are shown in

Table 3. Performance of the proposed ADD and LAA. The mAPs (%) are obtained combing different modules of the proposed method. Best Results are displayed in bold.

ADD	LAA	mAP (%)
		All	Occlusion	Scale Variations	Lighting Variations
		90.5	86.9	90.3	91.2
	√	90.2	86.4	90.1	90.9
√		90.9	87.5	90.3	92.0
√	√	92.9	88.8	92.7	93.8

, the proposed method with ADD and LAA obtains the best mAPs overall that are 92.9%, 88.7%, 92.7% and 93.8%, respectively, in corresponding maritime scenes. In all maritime scenes, the fully supervised baseline method [15], the method with only LAA and the method with ADD obtain mAP values of 90.5%, 90.2% and 90.9%, respectively. This demonstrates that the strong–weak augmentations for labeled and unlabeled data can produce distribution bias, and the proposed ADD helps to reduce this bias.

To confirm that the distribution bias is generated from the strong–weak augmentation, we perform a feature reduction between the original data and their augmented versions. The features are extracted from the last layer of the backbone. As shown in Figure 4a, the distribution of the augmented data exhibits an offset from that of the original data, and there are some feature points of augmented data that form an isolated cluster at the bottom. In Figure 4b, the distribution of augmented data matches the distribution of the original data, laying a solid foundation for the effectiveness of data augmentation.

To highlight the advantages of the proposed adaptive threshold, the compared experiment about the different threshold strategies is shown in

Table 4. Performance of different threshold strategy in all maritime scenes. Best results are displayed in bold.

Threshold Strategy	LAA	0.5	0.6	0.7	0.8	0.9
mAP (%)	92.9	85.6	88.1	87.9	89.9	89.6

. The adaptive threshold strategy with LAA obtains the highest mAP of 92.9% in Table 4, while the fixed threshold strategy requires manual search for the optimal threshold which may not be suitable across different datasets. The trend of adaptive threshold during training is shown in Figure 5. As seen in Figure 5a, the thresholds of all classes gradually converge around 0.8, which explains why the fixed threshold of 0.8 achieves the second-highest mAP of 89.9% to some extent. And the proposed LAA provides more appropriate threshold according to the learning status of the detection model, which reduces the pseudo-label bias and improves the performance semi-supervised object detection.

(ii) Performance of various data amounts: To evaluate the sensitivity of varying data amounts, we conduct this experiment by changing the proportions of labeled and unlabeled samples. The total numbers of all the labeled and unlabeled samples are 1,358 and 12,226, respectively. We set 4 types of ratios (1%, 2%, 5% and 10%) for labeled samples and 3 types of ratios (0%, 50%, 100%) for unlabeled samples. It is noted that the 0% ratios of unlabeled samples denote the baseline in fully supervised manner. As shown in

Table 5. The mAPs (%) of different amounts of labeled and unlabeled samples. Best results are displayed in bold.

	Labeled	1%	2%	5%	10%
Unlabeled
0%		63.8	71.2	79.8	90.5
50%		64.1	78.2	88.7	92.0
100%		64.1	80.1	91.4	92.9

, the more labeled samples, the higher the mAP, which aligns with the widely held intuition in training detection model. Similarly, more unlabeled samples also help improve the mAP in Table 5. However, the following two cases attract our attention. (1) Little improvement is achieved while training with unlabeled samples and 1% labeled samples, which suggests that the number of labeled samples is too small to effectively leverage the unlabeled samples. (2) The mAP obtained by combing 2% of labeled samples with all unlabeled samples is higher than that achieved with 5% of labeled sample in a fully supervised manner. The same phenomenon occurs when combing 5% of labeled samples with all unlabeled samples, whose mAP is higher than that with 10% of labeled sample in a fully supervised manner. This demonstrates that a large number of unlabeled samples can bring greater performance gains than a limited number of labeled samples. More discussion will be presented in Section 5.

(iii) Performance of different methods: To highlight the advantage of semi-supervised learning, we compare the proposed method with other existing detection method [10,15,38,48], using 5% of labeled samples and all unlabeled data. All these methods are trained with the same hyperparameters and tested on the same test set with the complex maritime scenes of occlusion, scale variations and lighting variations. Faster R-CNN [15] is considered as the baseline method that only trained with labeled samples.

Experimental results are listed in

Table 6. Detection results of different methods in complex maritime scenes. Best Results are displayed in bold.

Method	mAP (%)				ACE	FPS
	All	Occlusion	Scale Variations	Lighting Variations
FRCNN [15]	79.8	76.9	79.7	80.4	0.110	-
YOLOv8 [48]	85.8	82.5	85.9	87.3	0.093	25.8
STAC [38]	85.6	82.4	85.1	86.9	0.102	-
UT [10]	87.4	85.5	86.9	89.5	0.100	-
Proposed	91.4	89.3	91.0	92.6	0.093	11.1

. The baseline method [15] gets the mAPs of 79.8%, 76.9%, 79.7% and 80.4 in corresponding maritime scenes, respectively. YOLOv8 [48] improves specifically for small object detection and has slightly higher mAP than STAC [38], which gets the mAPs of 85.8%, 83.3%, 85.1% and 87.6 in corresponding scenes, respectively. All the compared semi-supervised detection methods have the higher mAPs than the baseline method, demonstrating the powerful strength of the semi-supervised detection and the potential to reduce labeling costs. Specifically, the semi-supervised detection method named STAC [38] uses pseudo-label to improve the detection performance and gets the mAPs of 85.6%, 85.5%, 86.9% and 89.5% in corresponding maritime scenes. The other semi-supervised detection method named UT [10], which combines the mean teacher framework [29] with pseudo-labeling, achieves the second-best mAPs of 87.4%, 85.5%, 86.9% and 89.5% in corresponding maritime scenes. The proposed method uses the ADD and LAA to obtain the best mAPs of 91.4%, 89.3%, 91.0% and 92.6% in corresponding maritime scenes. Our method aligns the feature distribution of strong–weak augmentation and reduces the pseudo-label bias, thereby helping to improve the detection performance in complex maritime scenes. Regrading ACE, the proposed method and YOLOv8 show best value, indicative of minimal center deviation and thus superior location accuracy. Since different methods in this part are performed in the same two-stage detection framework, their detection speeds are considered roughly to be the same. The FPS of the proposed method is 11.1, which meets the real-time requirements.

The visualization results in complex maritime scenes for the proposed method and UT [10] are shown in Figure 6. The results with different colors indicate different target classes. Different columns show the detection results from different methods and the ground truths (GT) are showed in the first column. Complex maritime scenes, including occlusion, scales variations and lighting variations, are presented in different rows. In Figure 6(1) with scene of occlusion, the proposed method detects two targets with high confidence score while UT [10] detects a false positive of bulk cargo carrier. In Figure 6(2) with scene of scale variations, the proposed method detects all the targets while UT [10] misses some small targets in a distant. In Figure 6(3) with scene of lighting variations, the proposed method misses a hard target while UT [10] generates the false positives.

5. Discussion

To reduce the distribution bias generated from strong–weak data augmentation and reduce the pseudo-label bias for different classes, we propose a semi-supervised object detection method in complex maritime scenes based on adaptive adversarial self-training. ADD and LAA are the two important components that provide solutions to these challenges. We make a further discussion according to three experiments implemented in this work.

(i) Ablation studies on ADD and LAA: The results of the first experiment are shown in Table 3. This experiment conducts the ablation studies of ADD and LAA which are proposed to reduce the distribution bias and pseudo-label bias. Although the strong–weak augmentation [32] has become the popular setting, we found that this augmentation strategy produces the distribution bias for the training data in this experiment, which has the negative influence on pseudo-labeling in the complex maritime scenes. As shown in Figure 4a. There are some feature points of augmented data become an isolated cluster, which will produce the unpredictable output through the nonlinear mapping of neural network. With the proposed ADD, the distribution bias can be reduced and therefore the consistency regularization is applied implicitly. Further, LAA is proposed to adaptive adjust the pseudo-label threshold for different classes according to the learning status of the detection model. Compared with the fixed threshold strategy, the adaptative threshold achieves the best mAP of 92.9% shown in Table 4, which indicates that more accuracy pseudo-labels are generated and the costs of manual search for the optimal threshold can be reduced.

(ii) Performance of various data amounts: The results of the second experiment are shown in Table 5, which shows the performance with various data amounts. This experiment demonstrates intuitively that more labeled samples help improve detection performance. It is crucial to improve the accuracy of the pseudo-labels for unlabeled samples. But the accuracy pseudo-labels also depends on the number of labeled samples. On one hand, as shown in Table 5, it is difficult to make full use of the unlabeled sample with just 1% of labeled samples. On the other hand, the accuracy of these pseudo-labels is inherently contingent upon the availability of labeled samples, which lay a foundation to discover the useful information of the unlabeled samples.

(iii) Performance of different methods: The results of the third experiment are shown in Table 6 for complex maritime scenes including occlusion, scale variations and lighting variations. It can be observed that the semi-supervised detection methods achieve better mAPs than fully supervised detection method in the same detection framework, and the proposed method obtains the best mAP and ACE in each maritime scene. Also, thanks to the FPN [40] and data augmentations, each method in this experiment does not generate large gap in mAPs for different maritime scenes. The FPN is used to extract the multi-scale features to deal with scale variations. The data augmentations of Cutout and KeepAugment are used to deal with occlusion. And the data augmentations including color jitter and random fog are used to deal with the lighting variations.

Overall, the proposed method yields commendable outcomes in terms of localization accuracy and detection performance. However, there remains a likelihood of encountering persistent pseudo-label errors during the semi-supervised training, which can undermine the performance of the detection model. Particularly the risk of mislabeling escalates in scenarios of extreme ambiguity and noise label, where the model is confronted with conditions that are inadequately represented within the labeled sample set. This deficiency in coverage exacerbates the model’s susceptibility to errors, potentially compromising its robustness and reliability in real-world applications.

6. Conclusions

In this work, we propose adaptive adversarial self-training for semi-supervised object detection in complex maritime scenes, leveraging the unlabeled data with limited labeled data. To address the problem of distribution bias and the pseudo-label bias, we employ technologies including adversarial learning, data augmentation, and entropy minimization, and therefore proposed two important modules called ADD and LAA, respectively. Also, an end-to-end semi-supervised detection framework with objective function is proposed to alleviate the class imbalance problem. They are designed to enhance the robustness of feature extraction from different objects and broaden applicability across various scenarios such as different weather and locations. Experimental results show that the proposed method achieves the best mAP and ACE compared with the other methods and meets the real-time requirement, saving the labeling costs and aiding object detection in maritime intelligence surveillance.

However, it is important to pay additional attention to the propagation of learning bias that stem from the use of inexact pseudo-labels. Understanding and mitigating these inaccuracies are crucial for enhancing the overall performance and reliability of detection models, particularly in semi-supervised learning scenarios, where pseudo-labels play a pivotal role. In the future, we will focus on extending our current framework to incorporate video-based semi-supervised object detection, aiming to delve into the complexities of learning under conditions of label noise and ensuring they perform optimally within an open-world context.