Complex Background Reconstruction for Novelty Detection

Abstract

Novelty detection aims to detect samples from classes different from the training samples (i.e., the normal class). Existing approaches predominantly make the target reconstruction better and choose the appropriate reconstruction error measurement method but ignore the influence of background information on this process. This paper proposes a novel reconstruction network and mutual information Siamese network. The reconstructed network aims to make the distribution of reconstructed samples consistent with that of original samples, intending to reduce background interference in the reconstruction process. After this, we measure the distance between the original and generated images based on a mutual information Siamese network, which extracts more discriminative features to calculate the similarity between the original images and their reconstructed ones. This part of the network uses global context information to improve the detection accuracy. We conduct extreme experiments to evaluate the proposed solution on two challenging public datasets. The experimental results show that the proposed method significantly outperforms the state-of-the-art methods.

1. Introduction

Novelty detection tackles an important unsupervised learning problem where novelty samples are not known a priori and the majority of the training dataset consists of “normal” data [1]. This problem has been widely applied in many areas, including abnormality detection [2,3], intruder detection [4], biomedical data processing [5], imbalance learning [6], vehicle tracking [7], and specific sign detection [8]. Different from other machine learning tasks, methods for one-class novelty detection are trained on only one class (i.e., the normal class) and aim to determine whether the given sample is a novelty during the inference stage. The challenges associated with anomaly detection tasks encompass several key aspects. (1) Lack of manual supervision: Anomaly detection often operates in an unsupervised or semi-supervised manner, meaning that there is limited availability of labeled data specifically indicating anomalies. This scarcity of labeled data hampers the ability to train models effectively. (2) Limited novelty samples: Novelty or anomaly samples are typically rare and occur infrequently in real-world data. The small number of available anomaly samples makes it challenging for models to learn and generalize effectively. (3) Imbalanced training data: In most cases, the training dataset consists primarily of instances from the normal class, making it imbalanced. This imbalance can lead to model bias towards the majority class and the poor detection of anomalies. (4) Unknown anomaly patterns: Anomalies can take various forms, and their characteristics may not be well-defined or known in advance. This uncertainty about the nature and structure of anomalies adds complexity to the task.

In recent years, novelty detection tasks have received extensive attention. According to the recent literature, existing models for novelty detection can be generally divided into two branches. The first branch is to extract latent features of an image as the input to traditional novelty detection algorithms such as one-class SVM [9]. Another one is to reconstruct the image using the deep generative networks [10], which are trained on samples from the normal class, and use the reconstruction error as a measure for novelty sample detection. The reconstruction approach using deep generative networks focuses on learning potential information for the retaining of normal samples, but, for the novelty samples, it will have large reconstruction errors. In this strategy, existing methods improve the performance by making image reconstruction better and choosing more appropriate distance metrics. Most of the existing methods ignore the effects of the image background. Specifically, the one-class novelty detection results for the CIFAR10 dataset are comparatively weaker for all methods. A complex background can affect the result of image reconstruction, which will affect the choice of distance metrics. However, background information can also help in object detection. In natural surroundings, the foreground and the background are strongly correlated. For instance, the sky background naturally correlates with birds and airplanes with a higher probability, whereas the ground background relates to terrestrial animals and vehicles. Hence, the background, especially a complex one, subtly indicates certain foreground elements, with distinctive backgrounds potentially indicating novel samples. Furthermore, owing to the expansive and non-discriminatory nature of neural networks, background information and foreground information exhibit a certain diffusion effect, resulting in the inclusion of some foreground information within the background. Hence, this paper proposes a method for complex background reconstruction, thereby facilitating the detection of novel targets.

Inspired by this, we offer improvements from the following aspects. First, we use the Maximum Mean Discrepancy (MMD) to make the distribution of reconstructed samples similar to that of the samples in the training set. Second, we propose a similarity metric network to measure the similarity between the sample and its reconstructed one. This network extracts more discriminant features as the basis for judging similarities and dealing with complex backgrounds and distributed discrete samples more effectively.

As highlighted in Figure 1, the uppermost group showcases the input images. The intermediate group depicts the reconstructed images exploiting L2 loss, whereas the lowermost group exhibits the reconstructions obtained using our solution. It is discernible that in the left segment, all the samples correspond to autonomous vehicles, which can be considered as standard instances. Conversely, the right segment contains a multitude of novel samples, including airplanes, trucks, and animals, etc. Notably, in both cases, it is evident that our solution improves the background reconstruction of the respective samples.

In this work, we propose a method for novelty detection. First, we propose a new reconstruction network (RNet) based on MMD-GAN [11] for the reconstruction of images. Second, we propose a similarity metric network (MNet) to learn the distance metric between latent representations of original images and reconstructed ones, which is used to determine whether the image is a novelty. In our method, RNet is trained on a dataset with only normal samples, and MNet takes samples and their reconstructed images generated by RNet as inputs. The goal of the whole model is to minimize the distance between normal samples and their reconstructed ones. At the testing stage, the detection result is based on the distance between the original sample and its reconstructed image.

Our contributions are three-fold.

• We propose a reconstruction network that not only can learn the features of the target, but also the distribution of the background. Reconstructing background information can reduce the interference of the background information on reconstructed objects.
• We propose a Siamese neural network with mutual information to learn the similarity between original images and their reconstructions. This method not only can extract more discriminant features about objects than the previous method but can also obtain background context information. Therefore, the target classification accuracy is significantly improved.
• Experiments on challenging datasets with complex backgrounds demonstrate the superiority of the proposed model.

2. Related Work

In this section, we review the related work on novelty detection, MMD, and metric learning.

Novelty Detection: The one-class novelty detection task has attracted considerable attention in recent years. With the success of neural networks and deep learning, great progress has been made in novelty detection. According to the recent literature, approaches to novelty detection can be generally divided into two branches. One is to extract the latent features of the image as the input to a traditional anomaly detection algorithm such as one-class SVM (OC-SVM) [12]. Hoffmann [13] and Sakurada and Yairi [14] propose to leverage the mean squared error to conduct novelty detection. Gruhl et al. [15] leverage a self-adaptive and self-organizing paradigm for novelty detection. Deep One-Class Classification [16] jointly trains a deep neural network while optimizing a data-enclosing hypersphere in the output space, which firstly introduces the fully deep one-class classification objective for unsupervised anomaly detection.

Another approach is to reconstruct the image using a deep generative network. Since generative adversial networks (GANs) [17] have shown a strong ability to extract deep features, several works apply GANs in novelty detection. AnoGAN [2] hypothesizes that the latent vector mapped from the input of the GAN represents the high-dimensional distribution of the data. The detection process is based on the reconstruction error and the error between the intermediate discriminator feature of the test image and the reconstructed image. Meanwhile, in ADGAN [18], which shares a similar framework with AnoGAN, the network is optimized based on both the latent vector and the generator, and only the reconstruction error is considered as the anomaly score while inferencing. Sabokrou et al. [1] attempt to de-noise noisy samples of the given class, and the discriminator’s prediction in the image space is used to quantify the reconstruction error. Li et al. [19] propose an augmented time-regularized generative adversarial network to generate effective artificial samples for novelty detection. Chen et al. [20] leverage an encoder–decoder reconstruction network and a CNN-based discrimination network to recognize noisy novel samples. Almohsen et al. [10] exploit an isometric adversarial auto-encoder to obtain stable detection results.

Nevertheless, the aforementioned approaches primarily concentrate on extracting and generating features from the foreground, neglecting the significant correlation that it shares with the background, especially for complex background cases. These methods depend solely on foreground information to detect novel targets. Highlighting the aforementioned concerns, this paper posits that complex backgrounds, with their implicit association with the foreground, can be leveraged to enable the detection of novel targets. By integrating both the foreground and background, the accuracy of novelty detection can be significantly enhanced.

MMD: MMD (Maximum Mean Discrepancy) is a statistical hypothesis aiming to measure the distance between two probability distributions. Schölkopf et al. [21] first proposed the criterion to replace the hard minimax objective function used in generative adversarial network training. The generative moment matching network (GMMN) [22,23] is a generative model that replaces the discriminator in the GAN with a two-sample test based on kernel MMD. The visualization result of the generated moment matching network is somewhat disappointing; Li et al. [22] improved it by combining the generated moment matching network with the auto-encoder. However, the empirical performance of GMMN and the computational efficiency of GMMN is not as competitive as that of GAN on challenging benchmark datasets. MMD-GAN [11] combines the key ideas of both GMMN and GAN; the authors proposed a method to improve both the expressive ability and computational efficiency of the model by replacing the fixed Gaussian kernel in the original GMMN with adversarial kernel learning techniques.

Metric Learning: Metric learning, also known as similarity learning, aims to learn by measuring the similarity between pairs of images, which is essential in many computer vision tasks, such as image retrieval, image matching, image verification, and multi-category tasks. Chopra et al. [24], Hadsell et al. [25] proposed the Siamese architecture for this purpose, and these are two important works often cited in this subject. Unlike the classification network, the goal of metric learning is to learn by measuring the similarity of the two instances in terms of the Euclidean distance, rather than the representation of the classification. Another popular architecture is the Triple Network [26]. For both of them, many authors have realized that it is important to mine a sample of the training set to find difficult or challenging pairs or triplets in order to converge faster or better reach the minimum [27].

3. Methodology

In this section, we introduce our model as a method for one-class novelty detection. The framework of the proposed method is illustrated in Figure 2. This model consists of two components: (1) the reconstruction network (RNet) for the reconstruction of the image; (2) the mutual information Simaese metwork (MNet) for the measurement of the similarity between the original image and the reconstructed image. These two components are trained in an adversarial and unsupervised manner.

3.1. Problem Settings

In the one-class novelty detection task, our goal is to train an unsupervised novelty detection network that performs well on samples subjected to a dispersed data distribution. We notice that these samples tend to have large variance and the backgrounds show great variety.

The formal definition of the task is as follows. Consider a training dataset $\mathcal{D}={\left\{x_i^D\right\}}_{i=1}^N$ with N normal samples, and a testing dataset $\widehat{\mathcal{D}}={\left\{(x_i^{\widehat{D}},y_i^{\widehat{D}})\right\}}_{i=1}^M$ with M normal and novelty samples, where $y_i^{\widehat{D}}\in \left\{0,1\right\}$ denotes the sample label. Here, $y_i^{\widehat{D}}=0$ means that $x_i^{\widehat{D}}$ is classified as a normal sample, whereas $y_i^{\widehat{D}}=1$ means that the corresponding $x_i^{\widehat{D}}$ is classified as a novel sample. The goal of the task is to train a model on $\mathcal{D}$ to optimize a novelty score function $\mathcal{S}\left(·\right)$. For a given testing sample $x^{*}\in \widehat{\mathcal{D}}$ and a given threshold $\tau$, if $\mathcal{S}\left(x^{*}\right)>\tau$, $x^{*}$ is a detected novelty.

3.2. Reconstruction Network (RNet)

In previous work [14], samples were typically reconstructed using auto-encoders and by generating normal samples against network training. The essential goal of reconstruction is to classify different object categories by calculating the reconstruction error between x and $\widehat{x}$. By reviewing the previous work [28], we find that most of the methods do not perform well on some datasets (e.g., CIFAR-10). There exist studies showing that making the distribution of real data similar to that of reconstructed data can benefit image generation [11]. Inspired by these studies, we still adopt the strategy of deep reconstruction networks, but we replace the discriminator in GAN with a two-sample test based on MMD.

Specifically, our RNet consists of three subnetworks: (1) the encoding subnetwork E to map the image x to the latent representation z, (2) the reconstruction subnetwork G to reconstruct $\widehat{x}$ by decoding z, and (3) the discrimination subnetwork D. In the reconstruction network, we employ a classical GAN architecture to extract background features. In the encoder part, we utilize the Maximum Mean Discrepancy (MMD) as a measure of dissimilarity between the original and generated images. This choice is motivated by the fact that, compared to other distance measures between distributions, such as the Kullback–Leibler (KL) divergence, which either require density estimation (either parametric or nonparametric) or space partitioning/bias correction strategies, MMD can be easily estimated as an empirical mean that converges to the true value of the MMD. Thus, MMD exhibits superior performance in the scenario of unknown image distributions. Moreover, MMD is equivalent to finding the Reproducing Kernel Hilbert Space (RKHS) function that maximizes the expectation difference between the two probability distributions, indicating its strong solution coherence. In the reconstruction part, we utilize the 1-norm to promote sparse features.

Adversarial Loss: Following the trend in current novelty detection methods [2], we use the feature distribution loss for adversarial learning. Studies have shown that feature matching loss can reduce the instability of GAN training [29].

MMD distance was first proposed for the two-sample test problem, which aims to determine whether two given distributions are the same. It uses kernel embedding $\varphi \left(\mathit{x}\right)=k(·,\mathit{x})$ associated with a characteristic kernel k, where $\varphi$ is infinite-dimensional and ${\langle \varphi \left(\mathit{x}\right),\varphi \left(\mathit{y}\right)\rangle }_{\mathcal{H}}=k(\mathit{x},\mathit{y})$. Given two distributions $\mathbb{P}$ and $\mathbb{Q}$, and a kernel k, the squared MMD distance is defined as

$$\begin{array}{c}{M}_k^2(\mathbb{P},\mathbb{Q})={∥{\mathit{\mu }}_{\mathbb{P}}-{\mathit{\mu }}_{\mathbb{Q}}∥}_{\mathcal{H}}^2={\mathbb{E}}_{\mathit{x},{\mathit{x}}^{\prime }\sim \mathbb{P}}\left[k\left(\mathit{x},{\mathit{x}}^{\prime }\right)\right]\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\mathbb{E}}_{\mathit{y},{\mathit{y}}^{\prime }\sim \mathbb{Q}}\left[k\left(\mathit{y},{\mathit{y}}^{\prime }\right)\right]-2{\mathbb{E}}_{\mathit{x}\sim \mathbb{P},\mathit{y}\sim \mathbb{Q}}\left[k(\mathit{x},\mathit{y})\right].\end{array}$$

(1)

where $\mu$ and $\mathbb{E}[·]$ denote the mean value of the distribution and the expectation of the kernel function, respectively. The kernel $k(\mathit{x},\mathit{y})$ measures the similarity between two samples $\mathit{x}$ and $\mathit{y}$. $\mathit{x}$ and ${\mathit{x}}^{\prime }$ represent two random variables subjected to $\mathbb{P}$, and $\mathit{y}$ and ${\mathit{y}}^{\prime }$ represent random variables subjected to $\mathbb{Q}$.

With respect to the square of the MMD distance, we have the theorem below.

Theorem 1.

Given a kernel k, if k is a characteristic kernel, then $M_k^2(\mathbb{P},\mathbb{Q})=0$ if $\mathbb{P}=\mathbb{Q}$.

Theorem 1 [30] shows that the more similar the two distributions are, the smaller the MMD distance between them is. In this work, we input the original image and the reconstructed image into the discriminator D to obtain the corresponding features $D\left(x\right)$ and $D\left(\widehat{x}\right)$. The feature distribution matching computes the similarity between the distributions of $D\left(x\right)$ and $D\left(\widehat{x}\right)$, which is measured by the MMD distance.

Formally, let $f(·)$ denote the function that outputs an intermediate layer of the encoder D for a given input x sampled from the input data distribution $\mathbb{P}$; the function of the discriminator D can be considered as forming a new kernel with k: $k\circ D(\mathit{a},\mathit{b})=k(D\left(\mathit{a}\right),D\left(\mathit{b}\right))=k_D(\mathit{a},\mathit{b})$. Thus, the adversarial loss can be defined as

$${\mathcal{L}}_{adv}=M_{k\circ D}^2\left(f\left(x\right),f\left(\widehat{x}\right)\right).$$

(2)

Reconstruction Loss: In RNet, the reconstructed image is obtained by decoding the latent representation of the samples. In order to optimize the decoder, the reconstruction loss is defined as

$${\mathcal{L}}_{con}={\mathbb{E}}_{x\sim p_{\mathbf{X}}}{\parallel x-\widehat{x}\parallel }_1.$$

(3)

Encoder Loss: Both losses above are designed to force the generator to reconstruct the input sample better. Moreover, we add a loss to minimize the MMD distance between the distributions of $E_1\left(x\right)$ and $E_2\left(\widehat{x}\right)$. The encoder loss is in the same form as the adversarial loss, which is defined as

$${\mathcal{L}}_{enc}=M_k^2\left(E\left(x\right),D\widehat{\left(x\right)}\right).$$

(4)

To summarize, the loss function of RNet is a linear combination of the reconstruction loss, the adversarial loss, and the encoder loss:

$${\mathcal{L}}_{RNet}={\mathcal{L}}_{con}+\gamma {\mathcal{L}}_{adv}+\delta {\mathcal{L}}_{enc}.$$

(5)

where $\gamma$ and $\delta$ are weighting factors of the adversarial loss and the encoder loss, respectively.

3.3. Mutual Information Siamese Network (MNet)

We proposed a network called the mutual information Siamese network (MNet) to measure the similarity between reconstructed images and original images. The similarity judges the quality of the reconstructed image and determines whether the images are novelties.

In the previous methods of novelty detection based on GAN/AE, they detect whether the sample is a novelty by calculating the reconstruction error of the input image x and the reconstruction image $\widehat{x}$. The general selection for the reconstruction error is the mean square error (MSE). However, RNet pays more attention to the object rather than the background for images with complex backgrounds. In this case, the quality of reconstruction is limited.

In this work, we employ ideas from metric learning based on deep features. Chopra et al. [24] introduced the Siamese neural network to solve signature verification as an image matching problem. The Siamese neural network is made up of twin networks, and the parameters between them are shared. These two networks’ distinct inputs are joined by an energy function in the end. Note that the Siamese network effectively mitigates the issue of imbalanced samples, making it particularly suitable in the current one-shot environment, which is suitable for the present case. The energy function computes some metrics between the highest-level feature representation on each side. As Figure 3 shows, we feed x and $\widehat{x}$ into the encoder network E to map them to the feature vectors in the feature space, and we determine whether the two samples are similar by calculating the distance between the two feature vectors.

In one-class novelty detection, only samples from the normal class can be provided for training. Therefore, we lack the label when training with the Siamese neural network because we do not have an accurate measure of the reconstructed images of input samples. Thus, we train the network to extract the features of the training target in the input space without supervision.

Mutual Information Maximization Loss: Based on [31], we use a training encoder to maximize the mutual information between original samples and reconstructed samples, so that the features of reconstructed ones can better represent the features of input ones.

Let X denote the set of original images, and $x\in X$ is an original sample. Z is the combination of encoder vectors. $z\in Z$ is the encoding vector corresponding to x. $p\left(z\right|x)$ denotes the distribution of coding vectors produced by x, which estimates the mutual information by training the discriminator to distinguish between samples coming from the joint $\mathbb{J}$, and the product of marginals $\mathbb{M}$. $I(X,Z)$ denotes the mutual information between x and z and indicates the correlation between them. We hope that the encoder E extracts X more discriminative features Z as much as possible, which lets $I(X,Z)$ be as large as possible:

$$p\left(z,x\right)=\underset{p\left(z,x\right)}{\arg \max }I\left(X,Z\right)$$

(6)

Following the formulation provided in Hjelm et al. [31], we take Jensen–Shannon divergence (JSD) as the mutual information estimation. Therefore, the loss is defined as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{info}& =max{\widehat{\mathcal{I}}}^{\left(\mathrm{JSD}\right)}\left(X;Z\right)\hfill \\ & =max{\mathbb{E}}_{\mathbb{J}}\left[log\sigma \left(T(x,z)\right)\right]-{\mathbb{E}}_{\mathbb{M}}\left[log\sigma \left(T\left(x^{\prime },z\right)\right)\right],\hfill \end{array}$$

(7)

where x is an input sample, z is the high-level representation related to x, and $x^{\prime }$ is another input sample unrelated to z.

We also perform an a priori constraint on the encoder to give the encoder an a priori specific expected statistical feature. Here, we use a variational auto-encoder (VAE) to constrain this a priori information. $q\left(z\right)$ is a standard normal distribution as

$${\mathcal{L}}_{pri}=min{\mathbb{E}}_{x\sim \tilde{p}\left(x\right)}\left[KL(p\left(z\right|x)\parallel q\left(z\right))\right].$$

(8)

Thus, we arrive at our complete objective for mutual information feature extraction as

$${\mathcal{L}}_E=\underset{p\left(z\right|x)}{min}\left\{-\beta ·{\mathcal{L}}_{info}+\gamma ·{\mathcal{L}}_{pri}\right\}.$$

(9)

Similarity Metric Loss: We directly take the reconstructed image from RNet as one of the inputs in MNet. However, since we did not discriminate between the reconstructed images during the training process, samples with large reconstruction errors can degrade the performance of MNet. To tackle this problem, we use Huberś loss [32] as the similarity metric loss to reduce the penalty for samples with large reconstruction errors. In our method, the loss is defined as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{metric}& =min{L}_{\delta }(E_w\left(x\right),E_w\left(\widehat{x}\right))\hfill \\ \hfill & =\left\{\begin{array}{cc}\frac{1}{2}{d}^2,\hfill & \mathrm{for}\left|d\right|\le \delta \hfill \\ \delta ·\left(\right|d|-\delta /2),\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(10)

where $d=(E_w\left(x\right)-E_w\left(\widehat{x}\right))$.

As shown in Equation (10), when the predicted value is less than $\delta$, Huberś loss becomes a square error or it becomes a linear error, which reduces the penalties on samples with large reconstruction errors to reduce the effect caused by them. Therefore, Huber’s loss enhances the robustness of our model.

To summarize, the loss function of MNet is a linear combination of the mutual information maximization loss and the similarity metrics loss.

$${\mathcal{L}}_{MNet}={\mathcal{L}}_E+\alpha {\mathcal{L}}_{metric}$$

(11)

where $\alpha$ is the weighting factor of the similarity metric loss.

4. Discussion

In this section, we deliberate on the restrictions and validity of our proposed solution. The Maximum Mean Discrepancy (MMD) technique requires the presence of comparable distributions that can be effectively mapped to Hilbert space and are amenable to kernel methods. The Siamese Network, while proficient in measuring similarity and distance, displays dependence on the assigned task. Consequently, the network’s efficacy may fluctuate depending on the particular undertaking it is employed for. However, the prerequisites for the Siamese network to exhibit commendable performance are satisfactorily fulfilled within the context of this study.

It is important to note that the proposed methods, such as MMD and the L1-norm for sparsity promotion, demonstrate considerable robustness and can be applied to diverse scenarios. The use of empirical means in MMD avoids the need for a priori knowledge of the data distribution, making it suitable for scenarios with unknown image distributions. The L1-norm encourages parameter sparsity, aiding in the extraction of essential features. The choice of the Siamese network helps to mitigate the imbalance issue typically found in one-shot learning environments. This network architecture is less sensitive to sample imbalance, which is beneficial for the novelty detection problem discussed.

The primary focus of this work is image background detection, which involves decomposing motion images into frames for processing. This allows for minimal requirements on the dynamism of the environment, making the proposed approach adaptable to varying levels of motion or scene changes.

5. Experiment

In this section, we validate the effectiveness of our method on two publicly available multi-class object recognition datasets.

5.1. Dataset and Measurement Metric

5.1.1. Dataset Description

CIFAR-10: CIFAR-10 [33] contains 60,000 color images in 10 classes with the resolution of 32 × 32. CIFAR-10 contains objects in the real world. CIFAR-10 is not only highly noisy, but also varies in terms of the appearance and scale of target objects, which brings great difficulties to identification. Therefore, several methods of one-class novelty detection obtain relatively poor results on CIFAR-10. In our experiment, we choose one of the classes as the normal class, and samples from the remaining classes are taken as novelty classes; 90% of samples from the normal class are used for training. The remaining 10% of samples from the normal class and samples randomly selected from the novelty classes with a proportion ranging between 10% and 50% are used for testing.

Caltech-256: Caltech-256 [34] is a dataset for image object recognition containing 30,608 images in 256 classes. The number of images per class ranges between 80 and 827. In Caltech-256, there is also an additional class called “clutter” that contains 827 images that are considered as outliers. Similar to the previous work [1,35], we randomly select n classes as normal classes, where $n\in \left\{1,3,5\right\}$. If there are enough images in the selected class, we use first 150 images, or we use the entirety of the images in it. At the testing stage, we randomly select a certain number of images from the “clutter” class as novelty samples with a proportion of 50%.

5.1.2. Measurement Metric

ROC analysis provides tools to select possibly optimal models independently of (and prior to specifying) the cost context or the class distribution. The receiver operating characteristic (ROC) curve is the plot of the true positive rate against the false positive rate, at various threshold settings. The area under the curve (AUC) represents the overall performance of the model (“1” means perfect and “0.5” means uselessness) and is a measure of aggregated classification performance. Hence, we evaluate these methods by the area under the curve (AUC) of the ROC.

5.2. Training Strategy

The detailed training strategy is as follows: (1) We train RNet in an alternative way first. Specifically, we first optimize D. After this, we fix the parameters of D and switch to optimize E and G. We also use the mutual information loss to update the feature extraction subnetwork of $E_f$. (2) We train MNet after RNet converges. We use the mutual information loss and Huber’s loss to optimize $E_f$. We apply a novel self-looping training trick to optimize the training process. In detail, we take the reconstructed image as an input in the metric network. The trick enhances the robustness of the reconstruction network.

5.3. Novelty Detection Result

Result on CIFAR-10: We compare our method with several conventional deep-learning-based methods for general anomaly detection as baselines, including one-class SVM (OC-SVM) [12], kernel density estimation (KDE) [36], and deep variational auto-encoder (VAE) [37].

Table 1. Average AUCs in % one-class novelty detection for CIFAR-10 dataset. The top three results are marked in bold.

Normal Class	OC-SVM	KDE	VAE	GAN	AnoGAN	DSVDD	OCGAN	Ours (MSE)	Ours (MNet)
AIRPLANE	61.6	61.2	70.0	70.8	67.1	61.7	75.7	89.6	86.7
AUTOMOBILE	63.8	64.0	38.6	45.8	54.7	65.9	53.3	47.3	70.5
BIRD	50.0	50.1	67.9	66.4	52.9	50.8	64.0	71.5	63.4
CAT	55.9	56.4	53.5	51.0	54.5	59.1	62.0	62.1	60.1
DEER	66.0	66.2	74.8	72.2	65.1	60.9	72.3	83.5	78.4
DOG	62.4	62.4	52.3	50.5	60.3	65.7	62.0	61.2	64.7
FROG	74.7	74.9	68.7	70.7	58.5	67.7	72.3	88.8	86.2
HORSE	62.6	62.6	49.3	47.1	62.5	67.3	57.5	56.0	65.7
SHIP	74.9	75.1	69.6	71.3	75.8	75.9	82.0	89.5	82.4
TRUCK	75.9	76.0	38.6	45.8	66.5	73.1	55.4	50.8	76.8

shows the experimental results on CIFAR-10, where the samples are complicated. Among the baseline methods, the performance of OCGAN is [28] considerably greater than that of the others. The method that we propose has comparable performance to OCGAN, with an average AUC (see Section 5.1.2) of 73.48%, as shown in

Table 2. Ablation experiment for our method performed on CIFAR-10.

Experiment	Average AUC %
(a) RNet + self-looping	70.03
(b) MNet + self-looping	68.41
(c) RNet + MNet	71.73
(d) RNet + MNet +self-looping	73.48

(d).

Result on Caltech-256: 

Table 3. Results on Caltech-256. Normal samples are images from n randomly selected classes, and novelty samples are randomly selected from the “clutter” class. The top three results are marked in bold.

	CoP	REAPER	OutlierPursuit	LRR	DPCP	R-Graph	ALOCC	Ours (MSE)	Ours (MNet)
${\mathrm{AUC}}_1$	90.5	81.6	83.7	90.7	78.3	94.8	94.2	94.2	94.1
${\mathrm{AUC}}_3$	67.6	79.6	78.8	47.9	79.8	92.9	93.8	93.8	94.1
${\mathrm{AUC}}_5$	48.7	65.7	62.9	33.7	67.6	91.3	92.3	90.8	92.7

shows the results on Caltech-256. We compare our method with several previous methods designed specifically for detecting novelties, including Coherence Pursuit (CoP) [38], OutlierPursuit [39], REAPER [40], Dual Principal Component Pursuit (DPCP) [41], Low-Rank Representation (LRR) [42], OutRank [43], R-graph [35], and ALOCC [1]. The results show that our method has better performance on Caltech-256.

5.4. Ablation Study

In order to investigate the effectiveness of each subnetwork of our method, we conduct several further ablation studies.

How important is MNet? We firstly use MSE instead of MNet to measure the reconstruction error. The penultimate columns of Table 1 and Table 2(a) show the results of the method without MNet. The result shown in Table 2(a) is also slightly greater than the one of OCGAN but is inferior to ours. We conjecture that the reason is that the metric network extracts discriminative characteristics of the target, especially for objects with high complexity.

However, the comparison between the last two columns of Table 1 indicates that MSE is superior to MNet when some classes (e.g., deer, birds) are selected as the normal classes and MNet performs better when others (e.g., car, truck) are. We conduct an analysis on this result. First, we calculate the variance of each sample by class. The variance represents the dispersion level of the data. The larger the variance, the more dispersed the data distribution is. As can be seen from Table 1 and

Table 4. Variance of each class on CIFAR-10.

Normal Class	R-Channel	G-Channel	B-Channel
airplane	0.500	0.481	0.531
automobile	0.536	0.531	0.548
bird	0.454	0.441	0.486
cat	0.513	0.504	0.515
deer	0.434	0.413	0.423
dog	0.500	0.487	0.497
frog	0.457	0.437	0.440
horse	0.486	0.487	0.503
ship	0.499	0.481	0.502
truck	0.536	0.538	0.562

, for the three classes with the most discrete image distribution, TRUCK, AUTOMOBILE, and HORSE, our method with the MNet function outperforms the state-of-the-art methods. However, the results obtained using the MSE function are reduced by 33.2%, 28.2%, and 16.8% compared to the state-of-the-art methods. At the same time, for the other seven classes with a relatively concentrated image distribution, our method can achieve very good results by applying both the MSE function and MNet function. Therefore, compared with the MSE function, our proposed MNet function can more stably achieve excellent results on different discrete-level datasets. Some different discrete-level samples are shown in Figure 4.

Meanwhile, we also conduct experiments aimed at adjusting the number of selected normal classes on Caltech-256. We randomly select $n\in \{1,3,5,7,9\}$ classes as normal classes. During testing, we randomly select a certain number of images from the “clutter” category as novelty samples, with a proportion of 50%. As shown in the Figure 5, as the amount of classes increases, since the data distribution becomes more dispersed, MNet performs better than MSE.

How important is RNet? We replace the image reconstruction network with the auto-encoder and GAN from the previous work and use our proposed work metric network part to detect the novelty samples. Table 2(b) shows the results of our method without the image reconstruction network. The last two columns in Table 1 also show the results of using our reconstruction network. The experimental results show that the method of using MMD loss can improve the detection results. MMD loss minimizes the distance between the reconstructed sample and the original sample distribution. Specifically, the two samples are closer together in terms of statistics. The background can also be reconstructed to some extent for images with background. Then, in the process of novel class detection, we can not only use the characteristics of the target itself, but also use global context information to assist detection. Figure 4 shows the results of using RNet to reconstruct the sample. It can be seen that the background is reconstructed to some extent in the image. We believe that our approach focuses on background information, so our results show good performance.

How important is self-looping training? We do not use the above training trick. The results of our training without the self-looping trick are shown in Table 2(c). Compared with our overall experiment, using this method can improve the accuracy of novelty detection. We feed reconstructed images into the network for training, which takes advantage of the ability to generate network for data augmentation. Thus, we find that these data are enhanced by the advantages of the network itself.

6. Conclusions

In this paper, we present a novel and sophisticated network architecture for complex background reconstruction, specifically tailored to the task of one-class novelty detection. We highlight the inherent correlation between complex backgrounds and the foreground object, as well as the rich information encapsulated within the foreground. As a result, we assert that harnessing the power of background reconstruction significantly enhances the detection of novel samples. To achieve this, we employ the MMD loss function to effectively mitigate background interference and we introduce MNet, a metric to measure the background similarity to minimize their divergence. The experiments show that our method exhibits an enhancement of approximately 0.3% in both the CIFAR-10 and Caltech-256 datasets.