Cross-Domain Object Detection through Consistent and Contrastive Teacher with Fourier Transform

Abstract

The teacher–student framework has been employed in unsupervised domain adaptation, which transfers knowledge learned from a labeled source domain to an unlabeled target domain. However, this framework suffers from two serious challenges: the domain gap, causing performance degradation, and noisy teacher pseudo-labels, which tend to mislead students. In this paper, we propose a Consistent and Contrastive Teacher with Fourier Transform (CCTF) method to address these challenges for high-performance cross-domain object detection. To mitigate the negative impact of domain shifts, we use the Fourier transform to exchange the low-frequency components of the source and target domain images, replacing the source domain inputs with the transformed image, thereby reducing domain gaps. In addition, we encourage the localization and classification branches of the teacher to make consistent predictions to minimize the noise in the generated pseudo-labels. Finally, contrastive learning is employed to resist the impact of residual noise in pseudo-labels. After extensive experiments, we show that our method achieves the best performance. For example, our model outperforms previous methods by 3.0% on FoggyCityscapes.

1. Introduction

In recent years, deep learning algorithms have achieved excellent performance in various tasks such as object detection, and these algorithms typically assume that the training and inference data are independent and identically distributed (i.i.d.). However, real-world data distributions rarely match this assumption. The distributional differences between domains can lead to severe performance degradation when pre-trained models are deployed to new domains. In addition, manually labeling annotations for all samples is highly time-consuming and labor-intensive, so inference is often performed with new domains that lack labels for supervision. As a result, unsupervised domain adaptation (UDA) has received widespread attention from the research community. As a semi-supervised method that effectively utilizes abundant unlabeled data, the teacher–student framework is widely employed for UDA, e.g., in the Unbiased Mean Teacher (UMT) [1] and the Adaptive Teacher (AT) [2]. Within the teacher–student framework, the teacher model generates pseudo-labels for the target domain, allowing the student model to be supervised without additional annotations.

Nevertheless, since the teacher–student framework was not initially introduced to solve UDA problems, it suffers from two serious problems: domain gaps and noisy pseudo-labels. Due to the domain gap, it is difficult for the teacher trained in the source domain to provide adequate guidance to the student in the target domain. In addition, the pseudo-labels generated by the teacher in the target domain usually contain noise, as shown in Figure 1, which includes many false positives and errors. Noisy pseudo-labels will mislead the student model and cause further performance degradation. In addition, although the quality of the pseudo-label can be improved, it is difficult to eliminate noise, which inherently limits the student’s performance.

To address these problems, we propose a Consistent and Contrastive Teacher with Fourier Transform (CCTF) for cross-domain object detection. First, we point out that style mismatches caused by low-frequency spectral variations reduce the perception of high-frequency semantics, leading to large domain gaps. For this reason, we introduce the Fourier transform to exchange the low-frequency components between the spectra of the source and target domain images to allow the style of the target image to be transferred to the source image, minimize the style mismatch, and reduce the domain gap. As shown in Figure 2, the discrepancy between the source distributions and target distributions is narrowed after the Fourier transform. Second, we maximize the consistency of the localization scores and the classification scores in both supervised and unsupervised ways. Due to discrepancies in domain distributions, the pseudo-labels are characterized by significant noise. Noisy pseudo-labels suggest that the IoU between predictions and ground truth may be low. Intuitively, a low IoU indicates the erroneous detection of an instance as an object, necessitating that the classification score attributed to this bounding box be correspondingly low. Conversely, a high IoU implies the accurate detection of an object, which encourages a correspondingly high classification score for the bounding box. In conclusion, the consistency between classification scores and IoU scores for a predicted instance directly correlates with the probability of accurately identifying an object. Thus, the quality of pseudo-labels is improved with original classification scores evolving to be consistent predictions that represent both classification and localization qualities. Third, we employ cross-domain contrastive learning on the object level. Despite enhancements to the quality of pseudo-labels, it remains inevitable that they incorporate a certain degree of noise. Inspired by the fact that contrastive learning can extract powerful representations without relying on accurate labels, we introduce contrastive learning to mitigate the influence of noise in pseudo-labeling and to facilitate consistency between student and teacher representations.

The contributions of our work can be summarized as follows:

• We propose to swap low-frequency image spectra between the source and target domains to narrow the domain gaps, thereby facilitating the cross-domain performance of the model.
• We propose to improve the consistency between the classification and localization branches of the teacher model in both supervised and unsupervised ways to optimize the quality of its generated pseudo-labels and reduce noise.
• We use contrast learning to reduce the distance between samples belonging to the same category in the source and target domains to improve knowledge transfer between teacher and student and to counteract the negative effects of noise in pseudo-labels.
• We carried out extensive experiments to demonstrate the state-of-the-art performance of our method. For instance, our method achieves 54.9% mAP on FoggyCityscapes, which is 3.0% higher than previous methods.

2. Related Work

• Unsupervised domain adaptation. UDA is proposed to transfer labeled source domain knowledge to another unlabeled target domain and various methods have been designed. The discrepancy-based methods try to minimize the distribution difference between domains to align them. Maximum mean discrepancy (MMD) [3] is an effective loss that many methods are proposed based on. For example, to balance source data distributions, WDAN [4] assigns an auxiliary weight for each class. JAN [5] measures the HilbertSchmidt norm between the kernel mean embedding of empirical joint distributions of source data and target data. Inspired by generative adversarial networks [6], adversarial-based methods introduce an adversarial objective for domain confusion through a domain discriminator. Ref. [7] excludes globally dissimilar images and only aligns globally similar images. Ref. [8] utilize gradient reverse layers (GRL) [9] to employ the adversarial objective and align two domains on both image level and instance level. Ref. [10] aligns discriminative regions that are mined with clustering methods. Lately, researchers have discovered the potential of the teacher–student framework in UDA. Ref. [2] introduces adversarial-based methods to the teacher–student framework to reduce domain gaps to alleviate the noise in pseudo-labels. Ref. [1] effectively alleviates model biases toward the source domain by healing the teacher framework with distillation and healing the student with style transfer.
• Cross-domain object detection. Cross-domain object detection (CDOD) is the extension of UDA from classification to object detection tasks. Some have proposed curriculum learning methods [11], annotation-level methods [12], and graph matching methods [13] for it. The adversarial learning is also employed to align two domains in [14,15,16,17]. As an effective semi-supervised method, the researchers also introduce the teacher–student framework to this field to help exploit target domain samples [1,2,18,19]. With the mutual learning between the teacher and the student, the framework can gain stronger robustness against data variance [1]. However, when facing large domain gaps and distribution discrepancies, most of these methods still suffer from large performance drops.
• Contrastive learning. Contrast learning pushes the model to learn generalized features that are insensitive to transformations, improving the representation of the model [20]. Recent studies have shown that pre-trained models based on contrastive proxy tasks can replace their supervised learning counterparts [21,22,23,24] with even better performance [25]. Naturally, contrastive learning can be employed for UDA. For example, [26] utilizes contrastive learning to align the text and vision encoders to disentangle the domain information. Therefore, in this paper, we use contrast learning to reduce the distance between samples belonging to the same class in the source and target domains to counteract the negative effects of residual noise in pseudo-labels.

3. Method

3.1. Problem Formulation

Given labeled source images $D_s=\left\{(X_s,B_s,C_s)\right\}$, where $X_s$, $B_s$, and $C_s$ represent the source image set, the bounding boxes and class labels, respectively, and unlabeled target images $D_t=\left\{\left(X_t\right)\right\}$, where $X_t$ represent the target image set, UDA aims at transferring knowledge learned from labeled source domain to unlabeled target domain and obtaining an effective model for the target domain.

3.2. Fourier Transform

The teacher–student framework was originally proposed for semi-supervised problems, whereas in UDA, the performance of the vanilla mean teacher is limited by the domain gap. To unleash the full potential of the teacher–student framework in UDA, we introduce the Fourier transform to mitigate the negative impact of the domain gap on the teacher model.

Referring to [27], given an RGB source image $x_s$ and an RGB target image $x_t$, we denote the amplitude and phase components of the Fourier transform as ${\mathcal{F}}_A,{\mathcal{F}}_{\mathcal{P}}$. After the Fourier transform, the image x in RGB space is transformed to $\mathcal{F}\left(x\right)(a,b)$ in frequency domain:

$$\mathcal{F}\left(x\right)(a,b)=\sum _{h,w}x(h,w)e^{-j2\pi \left({\displaystyle \frac{h}{H}}a+{\displaystyle \frac{w}{W}}b\right)},j^2=-1.$$

(1)

We then swapped the low-frequency components of the source and target inputs in the frequency domain and transformed them back into RGB space using the inverse Fourier transform ${\mathcal{F}}^{-1}$:

$$x_{s\to t}={\mathcal{F}}^{-1}\left([M_{\beta }\circ {\mathcal{F}}^A\left(x_t\right)+(1-M_{\beta })\circ {\mathcal{F}}^A\left(x_s\right),{\mathcal{F}}^P\left(x_s\right)]\right),$$

(2)

where the Fourier transform $\mathcal{F}\left(x\right)$ is implemented following [28] in a simple way using the Fast Fourier Transform (FFT) algorithm, and $M_{\beta }$ denotes a mask, which is defined as follows:

$$M_{\beta }(h,w)={\mathbf{1}}_{(h,w)\in [-\beta H:\beta H,-\beta W:\beta W]},$$

(3)

whose value is zero unless $\beta \in (0,1)$. Note that $\beta$ is not decided by image size nor resolution since $\beta$ is not pixel-wise, and we assume the center of a sample is $(0,0)$. By using the binary mask in the frequency domain space, the low-frequency portions of the source and target image amplitudes are exchanged, and the distribution difference between the hybridized image and the target image is reduced, facilitating domain adaptation. After that, without changing the phase component of $x_s$, the modified spectral representation of it is mapped back to the image $x_{s\to t}$. $x_{s\to t}$ contains the same content as $x_s$, but resembles the appearance of $x_t$.

With the Fourier transform exchanging of the low-frequency components between the image spectra of two domains, the style mismatches and domain gaps between the two domains are reduced. As a result, the performance of the vanilla mean teacher can be further unleashed, thus facilitating the performance of the student model.

3.3. Consistent Teacher

In the vanilla mean teacher, the teacher model is updated with the exponential moving average (EMA) of the student model to output robust target domain pseudo-labels. The student model is updated with both source domain ground truth and target domain pseudo-labels. Therefore, we can define the following optimization objective for the vanilla mean teacher:

$${\mathcal{L}}_{MT}={\mathcal{L}}_{source}+{\mathcal{L}}_{target},$$

(4)

where ${\mathcal{L}}_{source}$ and ${\mathcal{L}}_{target}$ denote the loss from source ground truth and the loss from target pseudo-labels.

With the employment of the Fourier transform, we initially improved the performance of the vanilla mean teacher. However, the vanilla mean teacher will produce low-quality pseudo-labels, which are harmful to the training process. Therefore, we propose the consistent teacher to generate high-quality pseudo-labels to help the student learn stable representations in both supervised (source domain samples) and unsupervised (target domain samples) ways.

Normally, the teacher–student framework optimizes the classification branch and the localization branch of the object detection loss separately. We argue that the inconsistency between the predicted outputs of these two branches is the source of the noise in the teachers’ pseudo-labels. Thus, we facilitate the consistency of these two branches by making localization quality the learning goal of the classification branch. To be specific, in the source domain, we set y to the IoU for the ground truth class in the localization branch and keep other classes at 0, which is formulated as follows:

$$\mathcal{L}(y,pred)=\left\{\begin{array}{cc}(y^2-y)log(1-pred)-y^{2}log\left(pred\right)\hfill & y>0\hfill \\ -{\left(pred\right)}^{\gamma }log(1-pred)\hfill & y=0\hfill \end{array}\right.,$$

(5)

where $pred$ is the classification prediction. The supervised loss can also be modified as follows:

$${\mathcal{L}}_{cls}=\sum _{ls}\sum _{cs}\mathcal{L}(y_{ls,cs},pre{d}_{ls,cs}),$$

(6)

where $ls$ and $cs$ represent the lth object of class c in the source domain. By changing the learning target from one-hot vectors to localization IoU, the classification score now represents both classification quality and localization quality. In addition, this softened learning objective also acts as a label-smoothing effect, further contributing to classification accuracy.

In the target domain, since we cannot access the annotation, we alternatively calculate the IoU between a box and all other box predictions from the student and choose the largest IoU as $\hat{y}$. Similarly to the supervised loss, the unsupervised loss can be remolded as follows:

$${\hat{\mathcal{L}}}_{cls}^t=\sum _{lt}\sum _{ct}\mathcal{L}({\hat{y}}_{lt,ct},pre{d}_{lt,ct}).$$

(7)

where $lt$ and $ct$ represent the lth object of class c in the target domain, and ${\hat{y}}_{lt,ct}$ and $pre{d}_{lt,ct}$ represent the pseudo-IoU and the classification prediction for the lth object of class c in the target domain.

With the modification in both the supervised way and the unsupervised way, the teacher model can output pseudo-labels with less noise, and the teacher–student framework will be more robust in the target domain.

3.4. Cross-Domain Contrastive Learning

Due to the lack of supervision, the Fourier transform and the consistent teacher cannot eliminate the noise in pseudo-labels. To this end, we introduce cross-domain contrastive learning to optimize feature representations and maximize beneficial learning signals.

To conduct contrastive learning, we first feed the same target mini-batch $MB$ to both the teacher’s RoIAlign [29] and the student’s RoIAlign to extract object-level features $OB{F}^T$ and $OB{F}^{S\to T}$. Meanwhile, by feeding $MB$ to the teacher, we also obtain corresponding pseudo-classes $C_t=\{C_1,...,C_N\}$ (N is the number of objects in $MB$). With object-level features and pseudo-classes, the contrastive loss can be formulated as follows:

$${\mathcal{L}}_{\mathrm{cont}}={\displaystyle \frac{{\lambda }_{\mathrm{cont}}}{N}}\sum _{i=1}^N{\displaystyle \frac{-1}{\left|\mathcal{P}\right(i\left)\right|}}\sum _{p\in \mathcal{P}\left(i\right)}log{\displaystyle \frac{exp(OB{F}_i^{S\to T}·OB{F}_p^T/\tau )}{{\sum }_{j=1}^{N}exp(OB{F}_i^{S\to T}·OB{F}_j^T/\tau )}},$$

(8)

where the positive pair $\mathcal{P}\left(i\right)=\left\{p\right|C_p=C_i,p\in \{1,...,N\}\}$ consists of all objects that have same pseudo-class as object i in mini-batch $MB$.

With the employment of contrastive learning, our student model can maximize the similarity of related samples and minimize that of unrelated samples to learn target sample representations.

3.5. Learning Objective

The overview of our framework is presented in Figure 3, and it is optimized as follows:

$$\mathcal{L}={\hat{\mathcal{L}}}_{cls}+{\mathcal{L}}_{reg}+{\lambda }_1({\hat{\mathcal{L}}}_{cls}^t+{\mathcal{L}}_{reg}^t)+{\lambda }_2{\mathcal{L}}_{cont},$$

(9)

where ${\hat{\mathcal{L}}}_{cls},{\mathcal{L}}_{reg}$ are the supervised classification and localization losses, and ${\hat{\mathcal{L}}}_{cls}^t,{\mathcal{L}}_{reg}^t$ are the unsupervised classification and localization losses. ${\mathcal{L}}_{con}$ is the object-level cross-domain contrastive loss. ${\lambda }_1$ and ${\lambda }_2$ are hyper-parameters used to control the weights of losses.

4. Experiments

4.1. Datasets

• Cityscapes. Cityscapes contains 2975 training images and 500 validation images collected from 50 different scenes of 8 categories for object detection tasks. Annotations are converted from segmentation masks.
• FoggyCityscapes. Foggy Cityscapes shares the same city scenes, categories, and annotations with Cityscapes, and Foggy Cityscapes adds fog to Cityscapes with depth information. Foggy Cityscapes simulates three fog levels (0.005, 0.01, 0.02) corresponding to different visibility ranges.
• PASCALVOC. PASCAL VOC [30] is composed of a combination of PASCAL VOC 2007 and 2012, which contains 17125 real-world images of 20 categories and corresponding annotations.
• Comic2k. Comic2k [31] contains 2000 images of 6 categories that are in comic styles.
• Clipart1k. Clipart1k [31] consists of 1000 clipart images that share the same categories with PASCAL VOC.
• Watercolor2k. Watercolor2k [31] contains watercolor-style images and shares the same categories with the Comic2k dataset.

For Comic2k, Clipart1k, and Watercolor2k, we follow [7,32] and equally divide three datasets into the training set and the testing set. Visualizations of samples of VOC, Clipart, Watercolor, and Comic are shown in Figure 4, and large domain gaps can be observed.

Experiments are conducted on two domain adaptation settings: from normal images to foggy images (Cityscapes → Foggy Cityscapes) and from realistic images to article images (Pascal VOC → Watercolor, Pascal VOC → Clipart1k, and Pascal → Comic2k). We use the training sets of both the Fourier-transformed source samples and the target samples for training and the validation sets of the target domain samples for evaluation. For comparison, we use the mean average precision (mAP) as our evaluation metric.

4.2. Implementation Details

In our experiments, we employ Faster R-CNN as the base object detection model in our method. The backbones of the Faster R-CNN are VGG16 [33] and ResNet-101 [34] pre-trained on ImageNet. The input images are scaled by resizing their shorter sides to 600 and keeping their original ratios. For the loss hyper-parameters, we set ${\lambda }_1=1.0$ and ${\lambda }_2=0.001$. For the hyper-parameters in Equation (4), we set $\alpha =0.75$ and $\gamma =2.0$. For the hyper-parameters in Equation (8), we set $\tau =0.07$ following [35,36]. We first train the Student model for 60k iterations. Then, the Teacher model is updated with EMA every iteration, and the Teacher and the Student are trained together for another 40k iterations. We set the learning rate as 0.016 and optimize our model using Stochastic Gradient Descent (SGD). Several data augmentation methods are applied, including random horizontal flip, random color jittering, gray-scaling, Gaussian blurring, and Erasing, to improve the generalization of our model. The weight smooth coefficient parameter of the EMA is set to 0.9996. We conduct our experiments on 2 NVIDIA TITAN RTX. The experiments are all implemented in PyTorch.

• Hyper-parameter discussions. We believe that performance drops in the teacher–student framework are mainly caused by noise in pseudo-labels. The consistent teacher is introduced to effectively reduce noise in pseudo-labels, hence we set ${\lambda }_1$ to be large as 1.0. We set ${\lambda }_2$ to be as small as 0.001 since we introduce the contrastive learning to enhance the consistent teacher.
• Avoiding strong augmentation issues. We observe that some objects are completely cut off from the strong augmented image while they are kept in the weak augmented image. Under such circumstances, object features extracted by the teacher and the student cannot be matched. To enforce the consistency between the teacher and the student, for each bounding box, we count the number of pixels where the RGB difference is larger than 40 between the teacher’s prediction and the student’s prediction [37]. If the difference ratio of pixels is higher than 50%, this object should be considered cut off and should be excluded from the contrastive learning.

4.3. Results and Comparisons

• Normal weather to severe weather. In this experiment setting, we evaluate our model on the benchmark normal weather → foggy weather (Cityscapes → FoggyCityscapes). In this setting, various complex scenes will bring challenges to the adaptation.

The results are shown in Table 1. For a complete demonstration of our model, we perform experiments on both the foggy level “0.02” split (the foggiest split) and the foggy level “ALL” split. We can observe from Table 1 that our model outperforms the previous best performance by 2.1% on the foggy level “0.02” split and by 3.0% on the foggy level “ALL” split, and this can demonstrate our model’s ability to extract robust features from both labeled data and unlabeled data. Besides, by achieving the best mAP in six categories, our model shows its effectiveness. Furthermore, these results can demonstrate the practical significance of our model. Sufficient unlabeled data can be easily collected for real-world applications. With more unlabeled data, our model is capable of extracting more target domain learning signals and the robustness of our model can further increase, leading to better performance in the target domain.

• Real to art. In this experiment setting, we evaluate our model on real-world images to artistic images. Compared with the experiment setting “Severe Weather”, domain shifts of this setting are much larger, bringing more challenges to our model. We use Pascal VOC (real-world images) as the source dataset and Watercolor, Clipart1k, and Comic2k (artistic images) as the target datasets. The results are shown in Table 2, Table 3 and Table 4. Specifically, our method achieves the highest 48.6% mAP among all compared methods and achieves the highest mAP in 10 categories. Similar results for Pascal VOC to Comic2k and Watercolor2k can be observed from Table 3 and Table 4: our model achieves 39.7% and 55.9% mAP, which surpasses the previous method by 2.0% and 0.7% respectively on Comic2k and Watercolor2k.

We also provide some visualization results for our model. In Figure 5, our model demonstrates the ability to accurately detect objects within images.

Table 1. The results for Cityscapes → Foggy Cityscapes. Our proposed method (in red) outperforms other methods on both foggy level “0.02” split and foggy level “ALL” split. SIGMA and MTOR (in blue) are based on ResNet-50 while other methods are based on VGG-16. The highest mAP of each class is bold.

Method	Split	Person	Rider	Car	Truck	Bus	Train	Motor	Bike	mAP
DM [38]	0.02	30.8	40.5	40.5	27.2	38.4	34.5	28.4	32.3	34.6
HTCN [14]	0.02	33.2	47.5	47.9	31.6	47.4	40.9	32.3	37.1	39.8
MeGA-CDA [39]	0.02	37.7	49.0	52.4	25.4	49.2	46.9	34.5	39.0	41.8
TIA [40]	0.02	34.8	46.3	49.7	31.1	52.1	48.6	37.7	38.1	42.3
SIGMA [41]	0.02	46.9	48.4	63.7	27.1	50.7	35.9	34.7	41.4	43.5
MTOR [19]	0.02	30.6	41.4	44.0	21.9	43.4	40.2	31.7	33.2	35.1
UMT [1]	0.02	33.0	46.7	48.6	34.1	56.5	46.8	30.4	37.3	41.7
PT+CMT [37]	0.02	42.3	51.7	64.0	26.0	42.7	37.1	42.5	44.0	43.8
AT+CMT [37]	0.02	45.9	55.7	63.7	39.6	66.0	38.8	41.4	51.2	50.3
Ours	0.02	49.5	60.3	65.6	40.2	65.8	41.0	40.3	56.5	52.4
Source	0.02	22.4	26.6	28.5	9.0	16.0	4.3	15.2	25.3	18.4
Oracle	0.02	39.5	47.3	59.1	33.1	47.3	42.9	38.1	40.8	43.5
PDA [42]	ALL	36.0	45.5	54.4	24.3	44.1	25.8	29.1	35.9	36.9
ICR-CCR [17]	ALL	32.9	43.8	49.2	27.2	36.4	36.4	30.3	34.6	37.4
PT+CMT [37]	ALL	45.6	55.1	66.5	34.0	59.4	42.4	43.9	47.4	49.3
AT+CMT [37]	ALL	47.0	55.7	64.5	39.4	63.2	51.9	40.3	53.1	51.9
Ours	ALL	50.9	54.8	68.1	45.3	65.1	55.8	42.9	56.3	54.9
Source	ALL	27.9	33.4	40.4	12.1	23.2	10.1	20.7	30.9	24.8
Oracle	ALL	41.2	49.1	61.6	32.6	56.6	49.0	37.9	42.4	46.3

Table 1. The results for Cityscapes → Foggy Cityscapes. Our proposed method (in red) outperforms other methods on both foggy level “0.02” split and foggy level “ALL” split. SIGMA and MTOR (in blue) are based on ResNet-50 while other methods are based on VGG-16. The highest mAP of each class is bold.

Method	Split	Person	Rider	Car	Truck	Bus	Train	Motor	Bike	mAP
DM [38]	0.02	30.8	40.5	40.5	27.2	38.4	34.5	28.4	32.3	34.6
HTCN [14]	0.02	33.2	47.5	47.9	31.6	47.4	40.9	32.3	37.1	39.8
MeGA-CDA [39]	0.02	37.7	49.0	52.4	25.4	49.2	46.9	34.5	39.0	41.8
TIA [40]	0.02	34.8	46.3	49.7	31.1	52.1	48.6	37.7	38.1	42.3
SIGMA [41]	0.02	46.9	48.4	63.7	27.1	50.7	35.9	34.7	41.4	43.5
MTOR [19]	0.02	30.6	41.4	44.0	21.9	43.4	40.2	31.7	33.2	35.1
UMT [1]	0.02	33.0	46.7	48.6	34.1	56.5	46.8	30.4	37.3	41.7
PT+CMT [37]	0.02	42.3	51.7	64.0	26.0	42.7	37.1	42.5	44.0	43.8
AT+CMT [37]	0.02	45.9	55.7	63.7	39.6	66.0	38.8	41.4	51.2	50.3
Ours	0.02	49.5	60.3	65.6	40.2	65.8	41.0	40.3	56.5	52.4
Source	0.02	22.4	26.6	28.5	9.0	16.0	4.3	15.2	25.3	18.4
Oracle	0.02	39.5	47.3	59.1	33.1	47.3	42.9	38.1	40.8	43.5
PDA [42]	ALL	36.0	45.5	54.4	24.3	44.1	25.8	29.1	35.9	36.9
ICR-CCR [17]	ALL	32.9	43.8	49.2	27.2	36.4	36.4	30.3	34.6	37.4
PT+CMT [37]	ALL	45.6	55.1	66.5	34.0	59.4	42.4	43.9	47.4	49.3
AT+CMT [37]	ALL	47.0	55.7	64.5	39.4	63.2	51.9	40.3	53.1	51.9
Ours	ALL	50.9	54.8	68.1	45.3	65.1	55.8	42.9	56.3	54.9
Source	ALL	27.9	33.4	40.4	12.1	23.2	10.1	20.7	30.9	24.8
Oracle	ALL	41.2	49.1	61.6	32.6	56.6	49.0	37.9	42.4	46.3

Table 2. The results for real images to article images on Pascal VOC → Clipart1k. All methods are based on ResNet-101 (ours in red). The highest mAP of each class is bold.

Method	Aero	Bcycle	Bird	Boat	Bottle	Bus	Car	Cat	Chair	Cow	Table	Dog	Horse	m-cyc	Person	Plant	Sheep	Sofa	Train	tv	mAP
Source	23.0	39.6	20.1	23.6	25.7	42.6	25.2	0.9	41.2	25.6	23.7	11.2	28.2	49.5	45.2	46.9	9.1	22.3	38.9	31.5	28.8
SCL [32]	44.7	50.0	33.6	27.4	42.2	55.6	38.3	19.2	37.9	69.0	30.1	26.3	34.4	67.3	61.0	47.9	21.4	26.3	50.1	47.3	41.5
UMT [1]	39.6	59.1	32.4	35.0	45.1	61.9	48.4	7.5	46.0	67.6	21.4	29.5	48.2	75.9	70.5	56.7	25.9	28.9	39.4	43.6	44.1
HTCN [14]	33.6	58.9	34.0	23.4	45.6	57.0	39.8	12.0	39.7	51.3	21.1	20.1	39.1	72.8	63.0	43.1	19.3	30.1	50.2	51.8	40.3
DM [38]	25.8	63.2	24.5	42.4	47.9	43.1	37.5	9.1	47.0	46.7	26.8	24.9	48.1	78.7	63.0	45.0	21.3	36.1	52.3	53.4	41.8
CMT [37]	39.8	56.3	38.7	39.7	60.4	35.0	56.0	7.1	60.1	60.4	35.8	28.1	67.8	84.5	80.1	55.5	20.3	32.8	42.3	38.2	47.0
Ours	41.6	55.9	39.8	43.1	61.8	37.2	59.4	8.1	62.6	62.9	37.0	27.7	71.2	87.9	81.4	55.3	21.7	36.2	44.9	37.6	48.6
Oracle	33.3	47.6	43.1	38.0	24.5	82.0	57.4	22.9	48.4	49.2	37.9	46.4	41.1	54.0	73.7	39.5	36.7	19.1	53.2	52.9	45.0

Table 2. The results for real images to article images on Pascal VOC → Clipart1k. All methods are based on ResNet-101 (ours in red). The highest mAP of each class is bold.

Method	Aero	Bcycle	Bird	Boat	Bottle	Bus	Car	Cat	Chair	Cow	Table	Dog	Horse	m-cyc	Person	Plant	Sheep	Sofa	Train	tv	mAP
Source	23.0	39.6	20.1	23.6	25.7	42.6	25.2	0.9	41.2	25.6	23.7	11.2	28.2	49.5	45.2	46.9	9.1	22.3	38.9	31.5	28.8
SCL [32]	44.7	50.0	33.6	27.4	42.2	55.6	38.3	19.2	37.9	69.0	30.1	26.3	34.4	67.3	61.0	47.9	21.4	26.3	50.1	47.3	41.5
UMT [1]	39.6	59.1	32.4	35.0	45.1	61.9	48.4	7.5	46.0	67.6	21.4	29.5	48.2	75.9	70.5	56.7	25.9	28.9	39.4	43.6	44.1
HTCN [14]	33.6	58.9	34.0	23.4	45.6	57.0	39.8	12.0	39.7	51.3	21.1	20.1	39.1	72.8	63.0	43.1	19.3	30.1	50.2	51.8	40.3
DM [38]	25.8	63.2	24.5	42.4	47.9	43.1	37.5	9.1	47.0	46.7	26.8	24.9	48.1	78.7	63.0	45.0	21.3	36.1	52.3	53.4	41.8
CMT [37]	39.8	56.3	38.7	39.7	60.4	35.0	56.0	7.1	60.1	60.4	35.8	28.1	67.8	84.5	80.1	55.5	20.3	32.8	42.3	38.2	47.0
Ours	41.6	55.9	39.8	43.1	61.8	37.2	59.4	8.1	62.6	62.9	37.0	27.7	71.2	87.9	81.4	55.3	21.7	36.2	44.9	37.6	48.6
Oracle	33.3	47.6	43.1	38.0	24.5	82.0	57.4	22.9	48.4	49.2	37.9	46.4	41.1	54.0	73.7	39.5	36.7	19.1	53.2	52.9	45.0

Table 3. The results for real images to article images on PASCAL VOC → Comic2k. All methods are based on ResNet-101 (ours in red). The highest mAP of each class is bold.

Method	Bike	Bird	Car	Cat	Dog	Person	mAP
Source	32.5	12.0	21.1	10.4	12.4	29.9	19.7
SWDA [7]	36.4	21.8	29.8	15.1	23.5	49.6	29.4
HTCN [14]	50.3	15.0	27.1	9.4	18.9	46.2	27.8
DT+PL [31]	53.0	23.7	34.4	27.4	27.2	44.0	35.0
AT [2]	34.0	9.1	56.2	24.6	38.8	63.6	37.7
Ours	34.0	10.7	61.1	24.9	40.2	67.4	39.7
Oracle	61.9	38.9	50.8	48.9	45.2	76.6	53.7

Table 3. The results for real images to article images on PASCAL VOC → Comic2k. All methods are based on ResNet-101 (ours in red). The highest mAP of each class is bold.

Method	Bike	Bird	Car	Cat	Dog	Person	mAP
Source	32.5	12.0	21.1	10.4	12.4	29.9	19.7
SWDA [7]	36.4	21.8	29.8	15.1	23.5	49.6	29.4
HTCN [14]	50.3	15.0	27.1	9.4	18.9	46.2	27.8
DT+PL [31]	53.0	23.7	34.4	27.4	27.2	44.0	35.0
AT [2]	34.0	9.1	56.2	24.6	38.8	63.6	37.7
Ours	34.0	10.7	61.1	24.9	40.2	67.4	39.7
Oracle	61.9	38.9	50.8	48.9	45.2	76.6	53.7

Table 4. The results for real images to article images on PASCAL VOC → Watercolor2k. All methods are based on ResNet-101 (ours in red). The highest mAP of each class is bold.

Method	bike	bird	car	cat	dog	person	mAP
Source	68.8	46.8	37.2	32.7	21.3	60.7	44.6
BDC-Faster [7]	68.6	48.3	47.2	26.5	21.7	60.5	45.5
DA-Faster [8]	75.2	40.6	48.0	31.5	20.6	60.0	46.0
WST-BSR [43]	75.6	45.8	49.3	34.1	30.3	64.1	49.9
HTCN [14]	78.6	47.5	45.6	35.4	31.0	62.2	50.1
AT [2]	83.8	34.3	54.2	36.1	31.6	71.3	51.9
SCL [32]	82.2	55.1	51.8	39.6	38.4	64.0	55.2
Ours	82.0	56.5	52.5	40.0	39.1	65.0	55.9
Oracle	61.9	38.9	50.8	48.9	45.2	76.6	50.6

Table 4. The results for real images to article images on PASCAL VOC → Watercolor2k. All methods are based on ResNet-101 (ours in red). The highest mAP of each class is bold.

Method	bike	bird	car	cat	dog	person	mAP
Source	68.8	46.8	37.2	32.7	21.3	60.7	44.6
BDC-Faster [7]	68.6	48.3	47.2	26.5	21.7	60.5	45.5
DA-Faster [8]	75.2	40.6	48.0	31.5	20.6	60.0	46.0
WST-BSR [43]	75.6	45.8	49.3	34.1	30.3	64.1	49.9
HTCN [14]	78.6	47.5	45.6	35.4	31.0	62.2	50.1
AT [2]	83.8	34.3	54.2	36.1	31.6	71.3	51.9
SCL [32]	82.2	55.1	51.8	39.6	38.4	64.0	55.2
Ours	82.0	56.5	52.5	40.0	39.1	65.0	55.9
Oracle	61.9	38.9	50.8	48.9	45.2	76.6	50.6

While assessing Clipart1k, Comic2k, and Watercolor2k within the same setting, it should be noted that the domain distances from Pascal VOC to Watercolor2k are considerably smaller than those observed between Pascal VOC and Clipart1k or Comic2k. To verify our statement, we calculate domain distances between Pascal VOC (source domain) and Clipart1k, Comic2k, and Watercolor2k (target domains). Strictly speaking, domain divergence should be measured with the $\mathcal{H}\Delta \mathcal{H}$-divergence. Since calculating the $\mathcal{H}\Delta \mathcal{H}$-divergence is complex, we employ the utilization of $\mathcal{A}$-distance, which is written as follows:

$$\mathcal{A}(D_s,D_t)=2(1-2\theta ),$$

(10)

where $\theta$ denotes the domain classification error of a binary classifier which classifies the source domain from the target domain. A reduced $\mathcal{A}$-distance indicates a smaller domain distance.

Table 5. The results of the $\mathcal{A}$-distance between domains of real images to article images. A smaller $\mathcal{A}$-distance value indicates a smaller domain distance.

Domains	Clipart1k	Comic2k	Watercolor2k
$\mathcal{A}$-distance	1.15	1.27	0.73

shows the results of domain distances. We can observe that domain distances between Pascal VOC and Watercolor2k are the smallest, corresponding to the best performance. Accordingly, domain distances between Pascal VOC and Comic2k are the largest, corresponding to the worst performance. Intuitively, watercolor paintings are more similar to real-world images than clipart or comics. Hence, the performances on Watercolor2k of all mentioned methods, including ours, is higher than those on Clipart1k and Comic2k.

4.4. Ablation Studies

Experiments conducted in Section 4.4 and Section 4.5 demonstrate that our model is robust under various circumstances. In this section, we further conduct ablation studies on each of our important modules.

• Fourier transform. The effectiveness of our Fourier transform is benchmarked in

  Table 6. Results of the ablation studies on our Consistent and Contrastive Teacher. Mean average precision (mAP, %) is reported on each module (performance drops in red).

  Model	Foggy “0.02”	Foggy “ALL”	Clipart1k	Comic2k	WaterColor2k
  Ours	52.4	54.9	48.6	39.7	55.9
  w/o Fourier transform	50.9 (−1.5)	52.9 (−2.0)	47.5 (−1.1)	38.7 (−1.0)	54.6 (−1.3)
  w/o consistent teacher	51.0 (−1.4)	53.3 (−1.6)	47.3 (−1.3)	38.5 (−1.2)	54.3 (−1.6)
  w/o contrastive learning	51.0 (−1.4)	53.4 (−1.5)	48.2 (−1.4)	38.4 (−1.3)	53.9 (−2.0)

  . From the results, we can observe large performance drops up to 2.0% for the “Severe Weather” setting. In addition, the performance drop of split “ALL” is larger than that of split “0.02”. Meanwhile, the performance drops for the “Real to Art” setting are only up to 1.3%, which is much smaller. Our Fourier transform is proposed to exchange the low-frequency parts of the amplitude of images from two domains. As shown in Figure 6, it works more naturally in the “Severe Weather” setting than in the “Real to Art” setting. In the “Severe Weather” setting, the Fourier transform can effectively convert an original image to its foggy version. In the “Real to Art” setting, however, the Fourier transform can only convert the original car to match the color of the target car. Based on these observations, we infer that the Fourier transform works more effectively when the low-level statistics of the distributions of two domains are more similar.
• The consistent teacher–student framework. We further analyze the importance of our consistent teacher–student framework. Results are shown in Table 6. We replace our consistent teacher–student framework with the MT and evaluate it on all datasets (up to 1.6%). Varying degrees of performance drops are observed on all datasets. The results show that our model is more robust after enforcing the consistency between the classification and the localization branches in both supervised and unsupervised ways.
• The contrastive learning. We also analyze the effectiveness of our cross-domain contrastive learning from two aspects. For one thing, in each iteration, we perturb pseudo-labels by randomly re-assigning random labels to a fraction of pseudo-label objects. By introducing noise to pseudo-labels, the domain adaptation pipeline may be compromised. The results are shown in Figure 7. As we perturb the pseudo-labels, the performance of the model without contrastive learning drops considerably (4.6% performance drop). In contrast, our model can partially resist the noise and recover the accuracy from the noise (2.8% performance drop). Therefore, contrastive learning can help our model improve stability and avoid collapsing. On the other hand, when we directly remove contrastive learning from our model, we can observe performance drops from 1.3% to 2.0% in Table 6. The results demonstrate that contrastive learning can resist the noise in labels and acquire robust visual representations to improve performance.

4.5. Discussion

We propose a novel method for cross-domain object detection tasks which seamlessly integrates Fourier transform, contrastive learning, and the consistent teacher. First, the Fourier transform alleviates domain gaps, thereby facilitating the effectiveness of subsequent modules. Then, enhanced representations derived from contrastive learning, in conjunction with the higher quality of pseudo-labels generated by a consistent teacher, mutually reinforce each other. Large performance improvements in two settings can demonstrate the effectiveness and adaptability of our model.

• Training time. We also compare the training time of our method with some state-of-the-art methods. For fair comparison without severe performance drops, we warm up each teacher–student-framework-based method for 20k iterations and then train them for another 10k iterations. For other methods, we train them for 30k iterations.

  Table 7. Results (in red) of the training time of our method and some SOTA methods. For fair comparison, the validation time within the training process (blue) is also reported.

  	City → Foggy	Voc → Clipart
  AT	4.5 h (−1.5 h)	2.5 h (−10 min)
  CMT+AT	4.5 h (−1.0 h)	2.5 h (−7.5 min)
  CMT+PT	5.0 h (−1.5 h)	−
  SIGMA	5.0 h (−1.0 h)	−
  Ours	4.75 h (−1.0 h)	2.75 h (−7.5 min)

  shows the training time (in red) of each method. In practice, we notice that the time consumed by different methods during the validation process within the training process is different, so we also present the validation time (in blue) for fair comparison. It can be seen that our method requires only marginally more time compared with other methods. The results in Table 7 and Section 4.3 demonstrate that our method can enhance performance while minimizing the increase in complexity.
• Application prospects. Given our model’s superior performance, it exhibits broad applicability across various perspectives, especially within autonomous driving. In practice, the weather will change from sunny to severe weather, e.g., foggy, requiring the model to be adaptive across different domains. Large performance improvements over previous DA methods on Cityscapes and FoggyCityscapes demonstrate that our model is adaptive to foggy weather. Therefore, our model may suit the field of autonomous driving.

5. Conclusions

In this paper, we introduce a novel method for cross-domain object detection. First, we introduce the Fourier transform to align two domains. The Fourier transform exchanges the low-frequency spectrum of the source and target domains to transfer target styles to source samples. Therefore, the initial alleviation of the domain shifts between two domains is achieved. Then, to improve the quality of pseudo-labels, we enforce the consistency between the classification and the localization branches in both supervised and unsupervised ways. Finally, to further resist the noise in pseudo-labels and improve the accuracy, we introduce contrastive learning to extract powerful visual representations. Extensive experiments and ablation studies demonstrate our method’s effectiveness and broad applicability within autonomous driving.