SimCDL: A Simple Framework for Contrastive Dictionary Learning

Abstract

In this paper, we propose a novel approach to the dictionary learning (DL) initialization problem, leveraging the SimCLR framework from deep learning in a self-supervised manner. Dictionary learning seeks to represent signals as sparse combinations of dictionary atoms, but effective initialization remains challenging. By applying contrastive learning, we encourage similar representations for augmented versions of the same sample while distinguishing between different samples. This results in a more diverse and incoherent set of atoms, which enhances the performance of DL applications in classification and anomaly detection tasks. Our experiments across several benchmark datasets demonstrate the effectiveness of our method for improving dictionary learning initialization and its subsequent impact on performance in various applications.

1. Introduction

Dictionary learning (DL) is a branch of signal processing and machine learning that aims to find a frame (or dictionary) for sparsely representing signals as a combination of few elements. The initial basis is unknown and is usually learned from the data. This method has been widely used in multiple fields, such as image and audio processing, inpainting, compression, feature extraction, clustering, and classification.

The DL problem can be formulated as follows:

$$\begin{array}{cc}{\displaystyle \underset{\mathit{D},\mathit{X}}{min}}\hfill & {\parallel \mathit{Y}-\mathit{D}\mathit{X}\parallel }_F^2\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {∥{\mathit{x}}_{\ell }∥}_0\le s,\ell =1:N\hfill \\ \hfill & ∥{\mathit{d}}_j∥=1,j=1:n,\hfill \end{array}$$

(1)

where $\mathit{Y}\in {\mathit{R}}^{m\times N}$ is the matrix that contains N signals of size m, stored compactly as columns, $\mathit{D}\in {\mathit{R}}^{m\times n}$ is named dictionary and is usually overcomplete ($m<n$), and $\mathit{X}\in {\mathit{R}}^{n\times N}$ is the coefficients’ matrix. The column vectors of matrix $\mathit{D}$ are named atoms and define the basis vectors used for the linear combination. For each signal sample, only s atoms are used for the representation. The matrix $\mathit{X}$ is a sparse matrix that contains the coefficients associated with the atoms used for the linear representation.

The DL problem is typically solved in two optimization stages: dictionary update and sparse coding. At the beginning of the algorithm, the dictionary $\mathit{D}$ is randomly initialized with normalized atoms; random initialization does not favor obtaining the best results, but satisfactory results can still be obtained. In the sparse coding stage, the dictionary is considered fixed and all sparse coefficients (the elements of the matrix $\mathit{X}$) are computed for each sample signal. A popular method for sparse coding is Orthogonal Matching Pursuit (OMP) [1]. This greedy algorithm selects the atoms sequentially based on their correlation with the residual representation. In the next stage, the dictionary matrix, $\mathit{D}$, is updated, while the coefficient matrix is considered fixed. Several algorithms are available for updating the dictionary matrix [2]. The K-means Singular Value Decomposition (K-SVD) [3] and Approximate K-SVD (AK-SVD) [4] are among the most used update algorithms. The optimization continues by alternating between these two stages until a stopping criterion is met. In general, two different criteria are used: the maximum number of iterations or the representation error dropping off under a threshold value.

Dictionary learning (DL) can be used in various application problems. Considering the sparse representation capabilities and the light optimization procedure, relevant results can be obtained in practice. However, performance usually depends on the initialization of the dictionary and its properties. For example, in classification problems or discriminative learning, the incoherence of the atoms becomes relevant. Incoherence is the property of the atoms to be far apart one from another; equivalently, their scalar product is small in absolute value. Several methods include discriminative terms in the optimization procedure to meet the incoherence goal. On the other hand, a pre-trained dictionary can be advantageous when prior knowledge about the data is known. In this paper, we address the dictionary initialization problem using a previous problem validated for deep neural networks in a self-supervised manner. This initialization can be used for general problems or classification problems solved with DL.

Contrastive learning is a method that aims to learn similar and dissimilar representations from the data. This method can be used in supervised problems with known sample labels. However, the discrimination problem becomes untractable when dealing with large datasets. A straightforward method for learning instance discrimination is the use of categorical cross-entropy loss. As the dataset increases in size, performing the softmax operation to calculate class probabilities becomes excessively costly. The challenge of using categorical cross-entropy loss, such as discrimination, lies in its computational infeasibility when dealing with large datasets. Researchers have been searching for more efficient ways to approximate this loss. Recent advances are inspired by metric learning, as well as the work in [5,6].

A significant challenge within the instance discrimination framework is the lack of intraclass variability. In traditional supervised learning, there are typically hundreds or thousands of examples per class, which helps the algorithm to learn the inherent variation within each class. However, in many applications, there are only a few examples per class, which clearly hinders the learning process. This issue can be tackled through extensive data augmentation. By applying different transformations to a specific data point, we can generate slightly varied versions while maintaining its fundamental semantic meaning. This approach allows us to learn valuable representations without relying on explicit labels.

To set up the contrast between instances, several views of the inputs are produced using a process $\mathcal{T}$ and then evaluated in the representation space. For a particular input ${\mathit{y}}_i$, an anchor is calculated as ${\mathit{y}}^{\left(a\right)}\sim \mathcal{T}\left({\mathit{y}}_i\right)$ and then compared to a positive sample ${\mathit{y}}^{\left(p\right)}\sim \mathcal{T}\left({\mathit{y}}_i\right)$, which is another transformation of the same input or a sample from the same class with the anchor. A negative sample, ${\mathit{y}}^{\left(n\right)}\sim \mathcal{T}\left({\mathit{y}}_j\right)$, which represents a transformation of a different input, is also contrasted with the anchor. In addition, the process $\mathcal{T}$ is modified or updated so that it represents positive pairs in a compact form, while negative pairs are projected far apart.

In the general context of Self-Supervised Representation Learning (SSRL), this approach involves a pretext task generator $\mathcal{P}$ that creates pretext inputs for multiple pairs of raw input instances. These inputs have pseudo-labels that indicate whether the pairs are matching or not.

The anchor and positive and negative samples are then processed by a feature extractor $h(·)$ to derive their respective representations: ${\mathit{x}}^{\left(a\right)}=h\left({\mathit{y}}^{\left(a\right)}\right)$ for the anchor, ${\mathit{x}}^{\left(p\right)}=h\left({\mathit{y}}^{\left(p\right)}\right)$ for the positive sample, and ${\mathit{x}}^{\left(n\right)}=h\left({\mathit{y}}^{\left(n\right)}\right)$ for the negative sample. After this, a similarity function denoted by $\Phi$ is used to evaluate the similarity between pairs of projections. The whole model is subsequently trained to minimize the distance between positive pairs and to maximize the distance between negative pairs. A simple formulation of contrastive learning loss is formulated as follows:

$${\mathcal{L}}_{\mathrm{con}}=-\mathbb{E}\left[log{\displaystyle \frac{\Phi \left({\mathit{x}}^{\left(a\right)},{\mathit{x}}^{\left(p\right)}\right)}{\Phi \left({\mathit{x}}^{\left(a\right)},{\mathit{x}}^{\left(p\right)}\right)+{\sum }_{j=1}^k\Phi \left({\mathit{x}}^{\left(a\right)},{\mathit{x}}^{\left(n\right)}\right)}}\right],$$

(2)

where k represents the number of negative samples that have been used in contrast with the anchor. The training process can also include the transformation process $\mathcal{T}$ or only the encoder $h(·)$.

Multiple versions of this strategy can be employed within this framework. The methods vary based on the similarity function, the family of transformations $\mathcal{T}$, the encoder function $h(·)$, and approaches to the sampling anchor, and positive and negative examples.

There are many unsupervised applications [7] that have been developed in the spirit of Self-Supervised Representation Learning (SSRL). These methods enable the development of generalizable models that have the potential to learn and recognize a wide variety of patterns in the data.

Several strategies are available, such as Momentum Contrast (MoCo) [8], Pretext-Invariant Representation Learning (PIRL) [9], and a Simple Framework for Contrastive Learning of Visual Representations (SimCLR) [10]. MoCo is a framework that uses a queue and a moving average encoder to learn visual representations. In PIRL, invariant representations are learned by solving pretext tasks. SimCLR uses a simple contrastive loss function to learn visual representations. The simple contrastive loss is applied by learning similar representations for augmented versions of the same input image while discriminating representations of dissimilar images.

Contributions. In this paper, we adapt the SimCLR framework to the dictionary learning problem, with the purpose of obtaining more incoherent atoms that are better adapted for DL applications (classification and anomaly detection). The learned atoms can then be used to improve sparse representations, leading to smaller representation errors and better discriminative performances.

The main contribution is reconfiguring the initial SimCRL algorithm in the context of dictionary learning. This includes the substitution of the base encoder network with a dictionary learning problem. The network projection head is no longer used since the encoding and projection are performed using the OMP algorithm. The augmentation procedure was adapted for n-dimensional vectors. For this, we only used four elementary operations, which were changed in the context of dictionary learning. This self-supervised framework is capable of building more incoherent dictionaries, which facilitates a better representation error and has an impact on further supervised and semi-supervised applications.

The use of SimCLR can be beneficial for dictionary learning applications from different perspectives. In many real-world applications, large amounts of unlabeled data are used. SimCLR can learn robust feature representations from the unlabeled data, which can then be used to initialize the dictionary. This initialization can improve the performance of downstream tasks, such as classification, anomaly detection, or clustering, even when labeled data are scarce. On the other hand, the initialization of the dictionary using the SimCLR framework can boost the optimization process. The learning process can start from a more informative and structured point, potentially leading to faster convergence and more stable solutions. This not only enhances the efficiency and effectiveness of the learning process, but also improves the interpretability and stability of the resulting model.

The content of this paper is organized as follows. In Section 2, we introduce the self-supervised contrastive framework in the context of dictionary learning. The augmentation methods required for this framework are included in Section 3. In Section 4, we include several algorithms that have been used for our experiments. These algorithms [11] also aim to respect a discriminatory representation using contrastive learning. Our tests demonstrate that the use of the SimCLR framework is beneficial for dictionary learning algorithms that promote contrastive learning. We then continue with the presentations of our experiments in Section 5. We conducted tests on two mainstream tasks, namely classification (Section 5.1) and anomaly detection (Section 5.2). In the last section, we end up with some conclusions.

2. Contrastive Dictionary Learning

This section explains the Simple framework for Contrastive Dictionary Learning (SimCDL), which is used for dictionary initialization. This framework is developed in the spirit of SimCLR [10], a powerful approach to unsupervised representation learning. At the base of this framework lies the loss function, designed to learn rich and discriminative embeddings from unlabeled data. For the Contrastive Dictionary Learning framework, we consider the same loss function, but we propose a different logic for computing the encodings representation.

In the context of dictionary learning (DL), we apply stochastic data augmentation transformations to generate pairs of correlated signals from the same example, denoted $\widetilde{{\mathit{y}}_i}$ and $\widetilde{{\mathit{y}}_j}$. These two samples are derived from an initial sample $\mathit{y}$. Let $\mathcal{T}$ represent the space of augmentation operations that can be applied to the samples. For two different random initializations of the augmentation operators, we have $t\sim \mathcal{T}$ and $t^{\prime }\sim \mathcal{T}$. The next step is to follow an encoding process that aims to maximize the agreement between the two augmented samples in the representation space.

The SimCLR problem has been adapted without using a projection head or a base encoder. Instead, the dictionary $\mathit{D}$ is used directly to calculate the encodings using an OMP procedure. The embeddings are represented by the column vectors of the matrix $\mathit{X}$.

To build a positive pair of encodings, denoted as ${\mathit{x}}_i$ and ${\mathit{x}}_j$, we compute the representation coefficients of the two augmented samples, $\widetilde{{\mathit{y}}_i}$ and $\widetilde{{\mathit{y}}_j}$. In addition, contrastive loss is calculated to measure similarities between positive pairs of encodings and discriminate them from negative pairs.

This setup leverages the principles of SimCLR, primarily contrastive learning, in a sparse coding problem. SimCLR is typically used to learn useful representations by maximizing the agreement between different augmented examples of the same data sample via a contrastive loss in the latent space. An example illustrating the SimCDL steps is shown in Figure 1.

To compute the loss function, we randomly select a mini-batch of K examples at each iteration. After that, we create pairs of augmented samples for each of the K examples, resulting in a total of $2K$ data points. By doing so, we do not need to sample negative examples explicitly.

Additionally, we consider other augmented examples as negative samples in relation to the positive pair. After this, we calculate an embedding for each augmented sample using the OMP algorithm

$${\mathit{x}}_i=\mathrm{OMP}({\tilde{\mathit{y}}}_i,\mathit{D},s),$$

(3)

where $\mathit{D}$ is the dictionary matrix and s is the sparsity level.

OMP aims to find the best sparse representation of a signal by iteratively selecting the most relevant atoms from the dictionary $\mathit{D}$. Iteratively, the algorithm selects the atom most correlated with the current residual. After the selection is made, the residual is updated by projecting the signal onto the subspace spanned by the selected atoms. This process is repeated until a stopping criterion is met (e.g., a desired sparsity level or error threshold). This whole process substitutes the encoding base network that was previously used in SimCLR.

The similarity of the encodings, obtained with OMP, is computed using the cosine similarity of their normalized feature vectors, $\Phi (\mathit{u},\mathit{v})={\mathit{u}}^{\top }\mathit{v}/\parallel \mathit{u}\parallel \parallel \mathit{v}\parallel$. In contrast, the negative encoding should use other atoms in their sparse representation, leading to incoherent embeddings. The loss for a positive pair of examples is calculated using a softmax function over the similarity scores of all pairs within the mini-batch, scaled by a temperature parameter $\tau$:

$$\ell (i,j)=-log{\displaystyle \frac{exp(\Phi ({\mathit{x}}_i,{\mathit{x}}_j)/\tau )}{{\sum }_{k=1}^{2K}[1_{[k\ne i]}exp(\Phi ({\mathit{x}}_i,{\mathit{x}}_k)/\tau )]}}.$$

(4)

The global loss function is summed across all positive pairs, representing the normalized temperature-scaled cross-entropy loss (NT-XEnt). The numerator $exp(\Phi ({\mathit{x}}_i,{\mathit{x}}_j)/\tau )$ encourages positive pairs to be closer, while the denominator ${\sum }_{k=1}^{2K}[1_{[k\ne i]}exp(\Phi ({\mathit{x}}_i,{\mathit{x}}_k)/\tau )]$ introduces competition with all other representations in the batch, treating them as negatives. The contrastive loss can be interpreted as the maximization of the similarity between positive pairs in relation to the similarity between negative pairs. This process effectively forms a distribution over possible pairs, emphasizing the relative similarity of positive pairs over negatives. The temperature parameter $\tau$ controls the sharpness of the similarity scores. From a mathematical point of view, this parameter affects the relative weighting of similarities. A lower-value $\tau$ leads to sharper distributions, which heavily penalizes dissimilarities between positive pairs, leading to a stronger focus on very close positives. On the other hand, a higher-value $\tau$ treats similarities equally, which can lead to the avoidance of overemphasizing the few closest pairs. In general, the temperature term $\tau$ can be seen as controlling the entropy of the similarity distribution.

Using the NT-XEnt loss function, the full dictionary is updated using a Stochastic Gradient Descent (SGD) procedure, where the gradient is computed using reverse-mode automatic differentiation. The optimization result leads to more diverse quasi-orthogonal atoms that can better represent all the samples available in the training set. This problem is similar to a frame design problem, in which the atoms are designed to represent the samples better. In the context of SimCDL, we want to randomly initialize a dictionary and optimize it following the SGD procedure. Our experiments demonstrate that relevant results can be obtained for small batch sizes and several iterations, leading to better representation errors. The idea of SimCDL is summarized in Algorithm 1.

The contrastive representation learning optimizes asymptotically two main properties: the alignment of features from positive pairs and uniformity of the induced distribution of the features on the hypersphere. In a theoretical study [12], the authors demonstrated that the NT-Xent loss inherently promotes uniformity on the hypersphere. The property of alignment ensures that a pair of positive samples $({\mathit{x}}_i,{\mathit{x}}_j)$ pushes them closer together in the latent space. This can be seen as maximizing the similarity $\mathbb{E}[\Phi ({\mathit{x}}_i,{\mathit{x}}_j)]\to \max$, encouraging the alignment of different augmentations from the same sample. The property of uniformity pushes negative pairs apart. In this way, the representations are spread over the entire space, ${\mathbb{E}}_{k\ne i}[\Phi ({\mathit{x}}_i,{\mathit{x}}_k)]\to \min$, with the aim of achieving a uniform distribution over the unit hypersphere. These properties share similarities with the concept of the incoherence of atoms in dictionary learning. The underlying goal of the problem is the same, but applied in a different context. In dictionary learning, coherence refers to the similarity between different atoms. This suggests that the use of contrastive representation learning should lead to more incoherent dictionaries. The underlying mechanisms of building incoherent atoms is similar to the strategy of contrastive representation learning. Moreover, contrastive learning is related to maximizing the mutual information (MI) [13] between different augmentations of the same sample:

$$\mathrm{MI}(\widetilde{{\mathit{x}}_{i1}};\widetilde{{\mathit{x}}_{i2}})\ge \mathbb{E}[logp\left({\mathit{x}}_i\right|{\mathbf{x}}_j)-logp\left({\mathbf{x}}_i\right)],$$

(5)

where $\widetilde{{\mathit{x}}_1}$ and $\widetilde{{\mathit{x}}_2}$ are different augmentations of ${\mathit{x}}_i$. The maximization of mutual information for the data samples is related to the problem of reducing mutual coherence in the problem of dictionary learning. Since we want to enhance the representation capabilities (mutual information), more diverse atoms are needed, leading to a reduction in mutual coherence.

Algorithm 1 SimCDL: main learning algorithm

Require: batch size K, constant $\tau$, augmentation function T, dictionary $\mathit{D}$, sparsity constraint s     for sampled minibatch ${\left\{y_k\right\}}_{k=1}^K$ do           for all $k\in \{1,\cdots ,K\}$ do                 draw two augmentation functions $t\sim \mathcal{T}$, $t^{\prime }\sim \mathcal{T}$                 # the first augmentation                 ${\tilde{\mathit{y}}}_{2k-1}=t\left({\mathit{y}}_k\right)$                 ${\mathit{x}}_{2k-1}=\mathrm{OMP}({\tilde{\mathit{y}}}_{2k-1},\mathit{D},s)$                 # the second augmentation                 ${\tilde{\mathit{y}}}_{2k}=t^{\prime }\left({\mathit{y}}_k\right)$                 ${\mathit{x}}_{2k}=\mathrm{OMP}({\tilde{\mathit{y}}}_{2k},\mathit{D},s)$           end for           for all $i\in \{1,\cdots ,2K\}$ and $j\in \{1,\cdots ,2K\}$ do                 $\Phi (i,j)={\mathit{x}}_i^{\top }{\mathit{x}}_j/(\parallel {\mathit{x}}_i\parallel \parallel {\mathit{x}}_j\parallel )$           end for           compute gradient of $\mathcal{L}={\displaystyle \frac{1}{2K}}{\sum }_{k=1}^K[\ell (2k-1,2k)+\ell (2k,2k-1)]$, where ℓ is defined in (4)           update dictionary $\mathit{D}$ using gradient descent     end for     return learned dictionary $\mathit{D}$

The use of SimCDL can be beneficial for the initialization of dictionaries with incoherent atoms or even incoherent sub-dictionaries. In classification problems, with $\mathit{Y}=\left[{\mathit{Y}}_1,\dots ,{\mathit{Y}}_c,\dots ,{\mathit{Y}}_C\right]$ representing a set of feature vectors, we want to learn local dictionaries, ${\mathit{D}}_c$, for each class. In general, the initialization problem is not addressed; simple methods like random matrices or a random selection of signals are used for initialization; we will tackle the problem using SimCDL. Considering that a class dictionary, ${\mathit{D}}_c\in {\mathcal{R}}^{m\times N_c}$, should achieve good representations for its class, we will further adapt the SimCRL framework for the initialization of dictionaries in classification problems. Since we need C dictionaries, we will further optimize a wide dictionary, $\mathit{D}=\left[{\mathit{D}}_1,\dots ,{\mathit{D}}_c,\dots ,{\mathit{D}}_C\right]\in {\mathcal{R}}^{m\times (C·N)}$. During optimization, the sparsity constraint s is set to N, which is the number of atoms per class. Since we need N atoms, we want to overspecialize enough atoms for each class.

After training the general dictionary, the problem of atom distribution must be solved; each atom of $\mathit{D}$ must be assigned to a class c. For this task, we propose two different approaches. The first one is greedy: we loop over each class and search for the atoms that are most used in the sparse representation of the samples of the current class. Since class dictionaries do not have common atoms, once an atom is assigned to a class, it becomes unavailable for other classes. The second approach is based on the linear sum assignment problem [14]. This algorithm solves an optimization problem where the goal is to assign a set of workers to a set of tasks in a one-to-one manner while minimizing the total cost of the assignment. This scenario is similar to the problem of assigning the atom class with respect to their use. During our experiments, we tested both methods and decided to use the second approach.

3. Data Augmentation for Dictionary Learning

This section introduces the main data augmentation transformations that can be applied on a signal sample $\mathit{y}\in {\mathit{R}}^m$. This strategy is widely used in supervised and unsupervised learning.

We follow the example provided in [10], where the authors used augmentation operators such as cropping, resizing, flipping, color distortion (drop and jitter), image rotation (90°, 180°, 270°), cutout, Gaussian noise, Gaussian blur, and Sobel filtering. Since we do not focus on images in DL, we further use only four augmentation operators that have been adapted to signal samples.

3.1. Rotation

The first augmentation operator that we use is rotation. Since we are working with n-dimensional vectors, we need a method to perform rotations in any dimension. We achieve this by applying Rodrigues’ rotation formula [15]. This formula is used to rotate a vector in a multidimensional space in the hyperplane spanned by two orthogonal unit vectors, named $\mathbf{u}$ and $\mathbf{v}$. Notice that for spaces with dimensions larger than three, we do not have a unique axis that can be used for rotations. Different initializations are used for the unit vectors during our experiments each time a rotation is generated. The resulting rotation matrix $\mathbf{R}$ is defined as

$$\mathit{R}={\mathit{I}}_m+sin\left(\alpha \right)·(\mathit{v}{\mathit{u}}^T-\mathit{u}{\mathit{v}}^T)+(cos\left(\alpha \right)-1)·(\mathit{u}{\mathit{u}}^T+\mathit{v}{\mathit{v}}^T),$$

(6)

where ${\mathit{I}}_m$ is the identity matrix and $\alpha$ is the angle under which the rotation is made.

In a n-dimensional space, a rotation using Rodrigues’ formula is confined to the subspace (hyperplane) defined by the two orthogonal vectors $\mathit{u}$ and $\mathit{v}$. These vectors define a plane within the n-dimensional space. The rotation matrix $\mathit{R}$ only affects the components of the vector $\mathit{y}$ that lie in the plane spanned by $\mathit{u}$ and $\mathit{v}$. Any component of $\mathit{y}$ orthogonal to this plane remains unchanged. For a given n-dimensional sample $\mathit{y}$, we generate an augmented version by applying a simple rotation, defined as follows:

$$\mathit{y}\leftarrow \mathit{R}\mathit{y},$$

(7)

where $\mathit{R}$ is the rotation matrix (6).

3.2. Cutout

In general, the cutout transformation selectively zeros out some input data elements. In our case, the cutout operator refers to a reduced DL problem in which we isolate some features. More precisely, considering $m_c$ as the number of features that will be further used, with $m_c<m$, the DL becomes

$${\mathit{y}}_{{\mathcal{I}}_c}={\mathit{D}}_{{\mathcal{I}}_c}\mathit{x}=\sum _{j=1}^n{x}_j{\mathit{D}}_{{\mathcal{I}}_c,j}=\sum _{j\in \mathcal{S}}{x}_j{\mathit{D}}_{{\mathcal{I}}_c,j},$$

(8)

where the subscript ${\mathcal{I}}_c$ denotes the restriction of the vectors or matrices to the rows with indices in ${\mathcal{I}}_c$. Notice that when the cutout operator is used, only $m_c$ features of each atom are updated in the update stage of DL, namely those belonging to the set ${\mathcal{I}}_c$. Moreover, the atom normalization that follows the update is made with respect to all features. This strategy demonstrates that the cutout operator can obtain more robust dictionaries. In this way, the dictionary focuses more on the underlying structure of the data representation. In addition, the cutout operator makes the dictionary more generalizable, reducing overfitting risks. Figure 2 presents a general illustration of the cutout operator.

3.3. Gaussian Noise

Another augmentation operator is the Gaussian noise, which involves creating a noise vector $\mathbf{ϵ}$, where each element is drawn from a Gaussian distribution $\mathcal{N}(0,{\sigma }^2)$. The transformation is simple and is performed as follows:

$$\tilde{\mathit{y}}=\mathit{y}+\mathbf{ϵ},$$

(9)

where $\tilde{\mathit{y}}$ is the new sample vector, and $\mathbf{ϵ}$ is the noise vector centered in 0 with standard deviation $\sigma$. This type of augmentation is trivial for developing robust atoms.

3.4. Gaussian Blur

The Gaussian blur operator is obtained by convolving the sample vector $\mathit{y}$ with a one-dimensional Gaussian kernel. The kernel $\mathit{k}$ is defined as

$$k_i={\displaystyle \frac{1}{\sqrt{2\pi {\sigma }^2}}}exp\left(-{\displaystyle \frac{i^2}{2{\sigma }^2}}\right)$$

(10)

where i ranges over the kernel size, centered at $\mu =0$, the middle of the kernel; and $\sigma$ is the standard deviation of the blur. The resulting vector is

$$\widetilde{y_i}={(\mathit{k}\ast \mathit{y})}_i=\sum _j{k}_j·y_{i-j},$$

(11)

where ’∗’ is the convolution operator.

4. Algorithms

In this section, we present three algorithms [11] developed for Incoherent Dictionary Learning (IDL) in the context of classification tasks. These algorithms include the standard Incoherent Dictionary Learning (IDL), Incoherent Dictionary Learning via Distance Barrier (IDB), and Incoherent Dictionary Learning via Triplet Distance Barrier (ITDB). Each algorithm is designed to promote incoherence between class dictionaries, thereby enhancing their discriminative power.

4.1. Incoherent Dictionary Learning (IDL)

The Incoherent Dictionary Learning (IDL) problem aims to learn class-specific dictionaries ${\mathit{D}}_i$ from a set of feature vectors $\mathit{Y}=\left[{\mathit{Y}}_1,\dots ,{\mathit{Y}}_C\right]$, where ${\mathit{Y}}_c\in {\mathbb{R}}^{m\times N_c}$ represents the samples from class c and C is the total number of classes. Each dictionary ${\mathit{D}}_c$ is optimized to accurately represent its corresponding class while being poorly aligned with the representations of other classes. The optimization problem is formulated as follows:

$$\begin{array}{cc}{\displaystyle \underset{{\mathit{D}}_i,{\mathit{X}}_i}{min}}\hfill & \sum _{i=1}^C{∥{\mathit{Y}}_i-{\mathit{D}}_i{\mathit{X}}_i∥}_F^2+\gamma \sum _{i=1}^C\sum _{\begin{array}{c}j\ne i\end{array}}{∥{\mathit{D}}_i^{\top }{\mathit{D}}_j∥}_F^2\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {∥{\mathit{x}}_{i,\ell }∥}_0\le s,\ell =1:N\hfill \\ \hfill & ∥{\mathit{D}}_{i,j}∥=1,j=1:n\hfill \end{array}$$

(12)

where ${\mathit{x}}_{i,\ell }$ is the row vector on which the sparsity constraint is imposed, ${\mathit{d}}_{i,j}$ denotes the jth column of matrix ${\mathit{D}}_i$ and $\gamma >0$ is a trade-off factor that balances reconstruction accuracy with inter-class incoherence. After learning the dictionaries, a test signal $\mathit{y}\in {\mathbb{R}}^m$ is classified by finding the dictionary with the smallest reconstruction error:

$$c=\underset{i=1:C}{\mathrm{argmin}}\parallel \mathit{y}-{\mathit{D}}_i{\mathit{x}}_i\parallel ,\mathrm{subject}\mathrm{to}{∥{\mathit{x}}_i∥}_0\le s,$$

(13)

where s is the sparsity constraint.

This formulation, as introduced in [16], projects the dictionaries into quasi-orthogonal spaces, thereby enhancing class discrimination while preserving their representational capabilities.

The problem in (12) can be solved using an approach similar to the AK-SVD method. During the sparse representation phase, the Orthogonal Matching Pursuit (OMP) is employed as the incoherence terms are independent of the coefficients $\mathit{X}$. In the dictionary update phase, each atom in ${\mathit{D}}_i$ is sequentially adjusted to improve the representation of its class while maintaining near orthogonality to atoms of other classes. The focus is on atoms within a certain proximity, ensuring relevant incoherence for classification.

4.2. Incoherent Dictionary Learning via Distance Barrier (IDB)

Incoherent Dictionary Learning via Distance Barrier (IDB) introduces an Incoherent Distance Barrier (IDB) function to discriminate between class dictionaries. This formulation, inspired by work on incoherent frames [17], aims to promote incoherence between classes for classification tasks. The IDB optimization problem is expressed as

$$\begin{array}{cc}{\displaystyle \underset{{\mathit{D}}_i,{\mathit{X}}_i}{min}}\hfill & \sum _{i=1}^C{∥{\mathit{Y}}_i-{\mathit{D}}_i{\mathit{X}}_i∥}_F^2+\gamma \sum _{i=1}^C\sum _{\begin{array}{c}j\ne i\end{array}}f\left({\mathit{D}}_i,{\mathit{D}}_j\right)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {∥{\mathit{x}}_{i,\ell }∥}_0\le s,\ell =1:N\hfill \\ \hfill & ∥{\mathit{D}}_{i,j}∥=1,j=1:n\hfill \end{array}$$

(14)

where $f(·)$ is a function that enforces incoherence.

The first version of IDB utilizes a distance barrier function:

$$\begin{array}{c}\hfill b({\mathit{D}}_i,{\mathit{D}}_j)=\sum _k\sum _l\left[max(0,M-\parallel {\mathit{d}}_k^{\left(i\right)}-{\mathit{d}}_l^{\left(j\right)}{\parallel }^2)+max(0,M-\parallel {\mathit{d}}_k^{\left(i\right)}+{\mathit{d}}_l^{\left(j\right)}{\parallel }^2)\right],\end{array}$$

(15)

where M is a soft margin that indicates the minimum allowable distance between atoms from different classes. This function considers scenarios where atoms are either aligned or opposed. The soft margin M is derived from the mutual coherence constraint $\mu$ between atoms:

$$|{\mathit{d}}_i^T{\mathit{d}}_j|\le \mu ,\forall i\ne j,$$

(16)

which is equivalent to

$$\left\{\begin{array}{c}\parallel {\mathit{d}}_i-{\mathit{d}}_j{\parallel }^2\ge M\hfill \\ \parallel {\mathit{d}}_i+{\mathit{d}}_j{\parallel }^2\ge M\hfill \end{array}\right.,\forall i\ne j,$$

(17)

where $M=2(1-\mu )$.

To further enhance incoherence, the function $f(·)$ in (14) includes terms like

$$f_j\left({\mathit{d}}_j\right)={\parallel {\mathbf{W}}_j{\overline{\mathit{D}}}_j^T{\mathit{d}}_j\parallel }^2+\lambda b\left({\mathit{d}}_j\right),$$

(18)

where ${\overline{\mathit{D}}}_j$ concatenates all dictionaries except the one corresponding to class j, and ${\mathbf{W}}_j$ is a diagonal weighting matrix defined as

$$w_{ij}^2=max\left(\right|{\mathit{d}}_i^T{\mathit{d}}_j|/\mu ,1).$$

(19)

The first term discourages coherence across all atoms, while the barrier function $b(·)$ prioritizes updates for nearby atoms. The optimization is carried out using block coordinate descent, updating one atom at a time with a gradient descent approach:

$${\mathit{d}}_j\leftarrow {\mathit{d}}_j-{\gamma }_k{g}_j\left({\mathit{d}}_j\right),$$

(20)

where the gradient $g_j\left({\mathit{d}}_j\right)$ accounts for both representation error and inter-class incoherence. This equation represents an update step in the optimization procedure, which adjusts ${\mathit{d}}_j$ to minimize the objective function; the left side is the new value of the atom, and the right side uses the old value.

4.3. Incoherent Dictionary Learning via Triplet Distance Barrier (ITDB)

Incoherent Dictionary Learning via Triplet Distance Barrier (ITDB) is a variant of the IDB approach, inspired by the triplet loss function [18]. The optimization problem is reformulated as follows:

$$\begin{array}{cc}{\displaystyle \underset{{\mathit{D}}_i,{\mathit{X}}_i}{min}}\hfill & \sum _{i=1}^C{∥{\mathit{Y}}_i-{\mathit{D}}_i{\mathit{X}}_i∥}_F^2+\gamma \sum _{i=1}^C\sum _{\begin{array}{c}j\ne i\end{array}}\tilde{b}({\mathit{D}}_i,{\mathit{D}}_j)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {∥{\mathit{x}}_{i,\ell }∥}_0\le s,\ell =1:N\hfill \\ \hfill & ∥{\mathit{D}}_{i,j}∥=1,j=1:n\hfill \end{array}$$

(21)

where the triplet-based distance barrier function $\tilde{b}({\mathit{D}}_i,{\mathit{D}}_j)$ is defined as

$$\begin{array}{c}\hfill \sum _{\begin{array}{c}\\ ({\mathit{d}}^{\left(a\right)},{\mathit{d}}^{\left(p\right)},{\mathit{d}}^{\left(n\right)})\end{array}}[max(0,M+\parallel {\mathit{d}}^{\left(a\right)}-{\mathit{d}}^{\left(p\right)}{\parallel }^2-\parallel {\mathit{d}}^{\left(a\right)}-{\mathit{d}}^{\left(n\right)}{\parallel }^2)\\ \hfill +max(0,M+\parallel {\mathit{d}}^{\left(a\right)}-{\mathit{d}}^{\left(p\right)}{\parallel }^2-\parallel {\mathit{d}}^{\left(a\right)}+{\mathit{d}}^{\left(n\right)}{\parallel }^2)].\end{array}$$

(22)

In this approach, ${\mathit{d}}^{\left(a\right)}$, ${\mathit{d}}^{\left(p\right)}$, and ${\mathit{d}}^{\left(n\right)}$ form triplets of atoms. In this approach, ${\mathit{d}}^{\left(a\right)}$, ${\mathit{d}}^{\left(p\right)}$, and ${\mathit{d}}^{\left(n\right)}$ represent the anchor, positive, and negative elements in a triplet of atoms, respectively; ${\mathit{d}}^{\left(a\right)}$ (anchor) is the reference point; ${\mathit{d}}^{\left(p\right)}$ (positive pair) is similar to the anchor, and the goal is to minimize the distance between them, represented by $\parallel {\mathit{d}}^{\left(a\right)}-{\mathit{d}}^{\left(p\right)}{\parallel }^2$; ${\mathit{d}}^{\left(n\right)}$ (negative pair) is dissimilar to the anchor, and the objective is to maximize the distance between the anchor and negative pair, represented by $\parallel {\mathit{d}}^{\left(a\right)}-{\mathit{d}}^{\left(n\right)}{\parallel }^2$. The max ensures that these distances are adjusted based on a margin M, promoting better separation between the positive and negative pairs. The method ensures that atoms from the same class are closer to each other than atoms from different classes.

The triplet-based approach (ITDB) offers a more refined control over the atom distribution but incurs higher computational costs due to the management of triplets instead of pairs. To alleviate this, we propose optimizing only a subset of possible triplets, randomly selected, to reduce computational load while maintaining effective representation.

In summary, the IDB and ITDB methods extend the standard IDL approach by introducing advanced barriers that ensure greater separation between class dictionaries, thereby enhancing their effectiveness in classification tasks.

5. Experiments

We present here the results of our numerical experiments on two problems: classification and anomaly detection.

5.1. Classification

This section contains experiments carried out with the SimCDL framework. For our tests, we reproduce the results of the algorithms presented in the previous section, named Incoherent Dictionary Learning (IDL), Incoherent Dictionary Learning via Distance Barrier (IDB), and Incoherent Dictionary Learning via Triplet Distance Barrier (ITDB). We used several popular datasets that are widely used in computer vision and machine learning. Some of them were explicitly adapted for DL problems. The procedure for obtaining these datasets is available in [11]. These datasets were extensively used for research in face recognition, scene classification, object recognition, and human action recognition. In the following, we provide some general details for each dataset.

The YaleB Face Dataset [19] is a well-known collection of face images for studying problems in the field of computer vision, such as face recognition and facial expression categorization. It consists of 2414 grayscale images of 38 different persons. This dataset is particularly challenging due to the variations in lighting and facial expressions.

The AR Face Dataset [20] comprises over 4000 color images corresponding to 126 people’s faces (70 men and 56 women). The images feature different facial expressions, illumination conditions, and occlusions. The pictures were taken under varying conditions, making the dataset suitable for evaluating robustness in facial recognition tasks.

The CMU Pose, Illumination, and Expression (PIE) dataset [21] contains 41,368 images of 68 people, each captured under 13 different poses, 43 different illumination conditions, with 4 different expressions. The CMU PIE dataset is extensively used to develop and test algorithms for face recognition across pose and illumination variations and is one of the largest databases of its kind.

The Scene 15 dataset [21] is a collection of scene images categorized into 15 categories, including office, kitchen, street, and living room, totaling over 4000 images. This dataset has been used to evaluate algorithms for scene recognition. It is notable for its scale, resolution, and subject matter variety, presenting a realistic benchmark for scene classification tasks.

Caltech101 [22] consists of a database of 9144 images, subdivided into 101 object categories plus a background clutter category. With an average of 40 to 800 images per category, the dataset includes various objects taken from different angles and backgrounds. It has been a standard for assessing the performance of algorithms in object recognition and category classification.

The UCF50 dataset [23] is an action recognition dataset consisting of 50 action categories, each from videos in the wild. It contains realistic action videos collected from YouTube, totaling over 6680 clips. The UCF50 dataset is designed to evaluate human action recognition algorithms in varied scenarios, capturing the complexities of appearance, motion, interaction, and context.

Table 1. Dataset summary.

Dataset Name	# Samples	# Dim	# Classes
YaleB	2414	504	38
AR	2600	540	100
CMU PIE	11,554	256	68
15 Scene	4485	3000	15
Caltech101	9144	3000	102
UCF50	6680	5000	50

presents a general description of these datasets.

In our experimental setup, we utilized the augmentation techniques described in Section 3: cutout, rotation, Gaussian noise, and Gaussian blur. A systematic grid search was performed for various parameters: rotation angles were set at $\alpha \in \{0,1,2,3,4\}$; cutout percentages were $\{0\%,20\%,40\%,60\%,80\%\}$; standard deviations for noise were $\{0,0.2,0.4\}$ and for blur were $\{0,0.25,0.5\}$. The batch sizes tested included $\{16,32,64,128\}$, and the dictionary was updated across $\{5,10,20\}$ iterations. Optimal parameters identified for most datasets were a rotation angle of 2°, a noise standard deviation of $0.2$, a blur standard deviation of $0.5$, and a cutout percentage of $20\%$. The temperature parameter, $\tau$, was set to $0.5$. All subsequent experiments were conducted with batch sizes of 16 pairs of samples, resulting in 32 augmented samples in total. The dictionary optimization was completed over 10 iterations, yielding satisfactory results. The results provided were obtained by computing the mean results over 10 independent rounds with different initialization seeds. However, when comparing a standard DL with one that benefits from initialization with SimCDL, we ensured that both methods had the same starting random dictionary. All experiments were performed in Python 3.10 and Matlab 2023a.

Our programs are openly available in the SimCDL repository at https://github.com/denisilie94/simcdl (accessed on 1 September 2024).

5.1.1. Dictionary Learning Initialization

For the general DL problem, we conducted experiments on a large amount of samples. To enhance the understanding of the outcomes, we made two experiments using the AK-SVD algorithm with two different initializations. The first is a random initialization, ${\mathit{D}}_0$, while the second starts from ${\mathit{D}}_0$ and is trained with the SimCDL framework provided in Algorithm 1. We denote with ${\mathit{D}}_t$ the dictionary obtained by SimCDL.

We further include the experimental details for the YaleB dataset. For this experiment, we only used the training dataset, which consisted of 1216 samples and 32 for each of the 38 classes. The dictionary ${\mathit{D}}_0$ was initialized with a total of 1520 atoms, 40 atoms for each class. Using this dictionary, we further obtained the dictionary ${\mathit{D}}_t$ by updating it for 10 iterations of SGD following the SimCDL framework. We applied AK-SVD starting with the two initializations, ${\mathit{D}}_0$ and ${\mathit{D}}_t$. For this stage, we set the sparsity constraint to 40, which is equal to the number of atoms from each class. This constraint will prove useful in future experiments.

To balance the experimental conditions, we allowed additional AK-SVD iterations when initialized with ${\mathit{D}}_0$ to compensate for the time needed for the computation of ${\mathit{D}}_t$. This adjustment was essential to ensure that the comparison between the dictionaries is fair. In general, the computation of the dictionary ${\mathit{D}}_t$ is equivalent to several iterations of AK-SVD. This depends on the experimental configuration. More precisely, the batch size, the employed transformations, and the number of optimization iterations give the execution time for the SimCDL initialization. The compared results were obtained with a different number of iterations but the same execution time. For the YaleB dataset, we used 10 additional iterations to compensate for the SimCDL initialization. Even if the AK-SVD algorithm, when initialized with ${\mathit{D}}_0$, benefited from more iterations, the representation error still tended to have poorer results compared to the AK-SVD algorithm initialized with SimCDL.

A similar experiment was carried out for the UCF50 dataset. We provide the results obtained on these two datasets in Figure 3 and Figure 4.

We can observe that the error representation is better when the starting point is a dictionary trained with SimCDL.

To better understand the nature of the SimCDL initialization, we plotted the mutual coherence of the two dictionaries, ${\mathit{D}}_0$ and ${\mathit{D}}_t$. The result is shown in Figure 5. We observed that even though SimCDL follows a Self-Supervised Representation Learning strategy, the algorithm is capable of obtaining more incoherent atoms.

This shows that the augmentation transformation tends to build some incoherence barriers around the samples. This situation propagates further into the process of dictionary optimization and leads to more diverse and incoherent atoms.

During our investigation, we also wanted to understand how the augmentation transformation affected the samples. Since the datasets consist of large vectors, a dimensional reduction task was employed. For this, we used the t-distributed Stochastic Neighbor Embedding (t-SNE) [24] algorithm to reduce the data dimensions to two. We first reduced the original data points to 2D (Figure 6), and then reduced the size of the same samples, but only after the augmentation task was applied (Figure 7).

We noticed that the augmentation process tended to produce a more compact form of the classes compared to the original dataset. It is also possible that t-SNE was helped by the augmentation process to better group the samples from the same class; in this case, it is likely that the classification algorithms also directly benefited from augmentation. This is true especially in the case where there were only a few samples per class.

5.1.2. Dictionary Learning Initialization for Classification Problems

We further reproduced several classification experiments from [11], including methods that promote the incoherence of atoms. We reproduced the results according to the instructions available in the paper. What we changed was the dictionary initialization procedure. Just like in the previous experiments with the YaleB and UCF50 datasets, we trained a general dictionary using the SimCDL framework and distributed atoms based on the linear sum assignment algorithm. The results are given in

Table 2. Accuracy results (percentages). The table highlights the higher performance values in bold for each pair of methods (random initialization and SimCDL initialization) to indicate which method performs better.

	IDL	IDB	ITDB	IDL*	IDB*	ITDB*
YaleB	94.28	94.35	94.47	94.48	94.50	94.54
AR	93.13	93.07	93.92	93.02	92.57	94.10
CMU PIE	90.56	90.60	91.98	90.69	90.55	91.93
15 Scene	95.62	94.17	92.39	94.72	94.04	95.66
Caltech101	69.16	72.18	70.42	70.26	72.30	70.07
UCF50	59.94	60.14	60.63	60.26	60.08	60.47

and demonstrate that, in general, this type of initialization can slightly improve accuracy performance. We appended ‘*’ to the name of a method when the initialization was computed with SimCDL.

For these experiments, it is relevant that the batch size has to be larger than the number of samples per class. Since we want to obtain incoherent atoms for each class dictionary, we need to have augmented samples from negative classes. For several scenarios, the SimCDL initialization does not improve the classification accuracy. This is due to the fact that the SimCDL optimization direction is not aligned with the discriminative nature of the dictionary learning algorithm. This demonstrates that a different discriminative term might be more beneficial in the spirit of the SimCDL framework. For now, regarding the classification problem, we consider the results to be satisfactory enough, but they also give a start to a new strategy that must be researched.

5.1.3. Ablation Analysis

To analyze the impact of individual components within the augmentation operators, we conducted an ablation study by systematically removing each augmentation operator. The performance was measured from two points of view: we measured the representation error for each dictionary, using the SimCDL dictionary initialization, and measured the classification accuracy. In

Table 3. Ablation study: representation error. The best result for each dataset is in bold.

Dataset	Origin	Rotate*	Cutout*	Gaussian Noise*	Gaussian Blur*	*
YaleB	$1.99\times {10}^{-3}$	$1.56\times {10}^{-3}$	$\mathbf{1}.\mathbf{47}\times {\mathbf{10}}^{-\mathbf{3}}$	$1.65\times {10}^{-3}$	$1.66\times {10}^{-3}$	$1.62\times {10}^{-3}$
AR	$2.66\times {10}^{-3}$	$1.94\times {10}^{-3}$	$\mathbf{1}.\mathbf{81}\times {\mathbf{10}}^{-\mathbf{3}}$	$1.97\times {10}^{-3}$	$1.98\times {10}^{-3}$	$1.96\times {10}^{-3}$
CMU PIE	$2.29\times {10}^{-3}$	$1.87\times {10}^{-3}$	$\mathbf{1}.\mathbf{75}\times {\mathbf{10}}^{-\mathbf{3}}$	$1.89\times {10}^{-3}$	$1.79\times {10}^{-3}$	$1.89\times {10}^{-3}$
15 Scene	$7.24\times {10}^{-9}$	$7.67\times {10}^{-9}$	$7.73\times {10}^{-9}$	$7.25\times {10}^{-9}$	$\mathbf{7}.\mathbf{03}\times {\mathbf{10}}^{-\mathbf{9}}$	$7.64\times {10}^{-9}$
Caltech101	$3.50\times {10}^{-8}$	$3.50\times {10}^{-8}$	$3.56\times {10}^{-8}$	$3.42\times {10}^{-8}$	$3.39\times {10}^{-8}$	$\mathbf{3}.\mathbf{33}\times {\mathbf{10}}^{-\mathbf{8}}$
UCF50	$9.21\times {10}^{-4}$	$7.86\times {10}^{-4}$	$\mathbf{6}.\mathbf{89}\times {\mathbf{10}}^{-\mathbf{4}}$	$7.37\times {10}^{-4}$	$7.26\times {10}^{-4}$	$7.30\times {10}^{-4}$

, we register the error obtained at the last iteration, while in

Table 4. Ablation study: classification accuracy. The best result for each dataset is in bold.

Dataset	Origin	Rotate*	Cutout*	Gaussian Noise*	Gaussian Blur*	*
YaleB	94.35	94.39	94.39	94.36	94.37	94.50
AR	93.07	93.00	92.57	92.85	92.85	92.57
CMU PIE	90.60	90.71	90.94	90.74	90.80	90.55
15 Scene	94.17	94.22	94.17	93.83	93.99	94.04
Caltech101	72.18	72.22	72.27	72.31	72.22	72.30
UCF50	60.14	60.13	60.07	60.16	60.12	60.08

, we register the accuracy obtained for the classification problem. We denote the columns as follows: Origin stands for the original dictionary ${\mathit{D}}_0$; Rotate*, Cutout*, Gaussian noise*, and Gaussian blur* suggest that this component was eliminated from the SimCDL framework, while a single * indicates that no elimination was made and all the augmentation operators are used. For the representation problem, it is clear that the use of the cutout operator reduces the performance of the dictionary. Since we are cutting information from data, the dictionary is not capable of capturing the right direction of representation. We observe that every time the SimCDL framework is used (with at least three augmentation operators) the representation error is improved. On the other hand, the cutout operator is beneficial for the classification problem. The classification accuracy varies for each dataset, demonstrating that the augmentation operators have different behaviors. Except for the AR dataset, we obtain better accuracy results whenever augmentation operators are used. So, although in many cases the use of three out of the four augmentation operators is the best option, the benefits of augmentation are proven.

5.2. Anomaly Detection

Dictionary learning can also be applied to anomaly detection tasks. A straightforward approach for outlier detection is to calculate the representation error as follows:

$$\mathit{E}=\mathit{Y}-\mathit{D}\mathit{X}$$

(23)

Anomalous signals are those that usually produce poor representations. The anomaly score of signal i is simply the norm $\parallel {\mathit{e}}_i\parallel$ of the i-th column of $\mathit{E}$. A higher error norm indicates a greater likelihood that the signal is an outlier. The core assumption is that similar signals are better captured by the dictionary created by solving the DL problem.

In this section, we present some experiments regarding anomaly detection applications. We conducted investigations in the spirit of the Python Outlier Detection (PyOD) toolbox [25]. More precisely, we used the Anomaly Detection Benchmark (ADBench) [26] to run experiments for anomaly detection tasks using dictionary learning. ADBench is currently the most comprehensive anomaly detection benchmark; it contains 30 anomaly detection algorithms and 57 datasets from different domains, including healthcare (e.g., disease diagnosis), audio and language processing (e.g., speech recognition), image processing (e.g., object identification), finance (e.g., financial fraud detection), etc. In addition to the original datasets, this benchmark provides support for generating four types of anomalies: local, global, dependency, and cluster. As stressed in [26], no method obtains good results for all types. During our experiments, we concluded that dictionary learning is a good competitor regarding the dependency anomalies.

Out of the 57 available datasets, we selected 30 for testing, ensuring a diverse range of data types with varying dimensions and anomaly ratios. To align with the sparse representation framework, all chosen datasets have signals with more than three features. For all our tests, we used undersampled dictionaries that were half the size of the data. This reduction ensures that, during the learning process, the dictionary atoms capture the overall structure and key features of the representation. Additionally, the sparsity constraint s was set to 2. This configuration facilitates a compact representation that effectively disregards abnormal or outlier samples. All the dictionaries were trained for 10 iterations.

In

Table 5. ADBench results. The best result for each dataset is in bold.

Dataset	AK-SVD	AK-SVD*	# Samples	# Features	# Anomaly	Category
annthyroid	0.782634	0.789434	7200	6	534	Healthcare
breastw	0.8134222	0.810371	683	9	239	Healthcare
cardio	0.9762332	0.9758753	1831	21	176	Healthcare
cover	0.942390	0.945926	286,048	10	2747	Botany
fault	0.9943022	0.9938693	1941	27	673	Physical
glass	0.8546232	0.8738681	214	7	9	Forensic
Hepatitis	0.8944483	0.9014732	80	19	13	Healthcare
Ionosphere	0.990133	0.990810	351	32	126	Oryctognosy
landsat	0.9990171	0.9987003	6435	36	1333	Astronautics
letter	0.9987111	0.9986072	1600	32	100	Image
Lymphography	0.9549722	0.9553471	148	18	6	Healthcare
magic.gamma	0.899432	0.901307	19,020	10	6688	Physical
mammography	0.817071	0.831252	11,183	6	260	Healthcare
musk	0.9987683	0.9991482	3062	166	97	Chemistry
pendigits	0.980712	0.983217	6870	16	156	Image
Pima	0.809388	0.8144983	768	8	268	Healthcare
satellite	0.9991363	0.9992982	6435	36	2036	Astronautics
satimage	0.9999002	0.9999341	5803	36	71	Astronautics
SpamBase	0.908187	0.907505	4207	57	1679	Document
Stamps	0.856465	0.859910	340	9	31	Document
thyroid	0.820620	0.830234	3772	6	93	Healthcare
vertebral	0.858781	0.8905561	240	6	30	Biology
vowels	0.932859	0.935261	1456	12	50	Linguistics
Waveform	0.9922373	0.9926422	3443	21	100	Physics
WBC	0.823793	0.8257243	223	9	10	Healthcare
WDBC	0.9985452	0.998288	367	30	10	Healthcare
Wilt	0.7265673	0.726060	4819	5	257	Botany
wine	0.9567043	0.952583	129	13	10	Chemistry
WPBC	0.993470	0.9942863	198	33	47	Healthcare
yeast	0.8805333	0.8919712	1484	8	507	Biology

, we summarize the ROC AUC (Receiver Operating Characteristic Area Under Curve) results obtained on the 30 testing datasets. With AK-SVD [4], we denote the results obtained using a random initialization. On the other hand, AK-SVD* denotes the same method, but with the benefits of a SimCDL initialization. The initialization was obtained using a batch size of 64, with 40 dictionary optimization steps and a sparsity constraint of 2. The problem of anomaly detection was solved in an unsupervised manner. The datasets were split into training test sets with a 70-30 ratio. During the training phase, both normal and abnormal samples were used to update the dictionary. However, due to the relatively small number of anomalies, the dictionary is expected to predominantly capture the representation of the normal samples. The ROC AUC values reported in Table 5 are obtained by averaging 5 independent runs and are computed for the test sets.

The results indicate that for 21 out of 30 datasets, the AK-SVD method performs better when initialized with SimCDL, while for one dataset, we obtain the same result. For the 20 improved datasets, the average ROC AUC gain is 0.0058, whereas, for the remaining datasets, the average performance decrease is 0.0010. The best improvement, nearly 0.0317, is observed for the Vertebral dataset, while the wine dataset shows the largest decline, with a drop of 0.0041. Generally, the datasets with negative improvements exhibit similar results for the two initializations. Overall, the AK-SVD method achieves a mean ROC AUC of 0.9151 across all datasets, while AK-SVD* yields a higher average of 0.9189, demonstrating an improvement. When comparing the DL method to the 14 unsupervised algorithms from the ADBench benchmark, we achieve the third-best result. Only two methods outperform ours: COF with 0.9274 and LOF with 0.9256. The next closest method in ADBench is KNN, which achieves 0.9154, placing it between the performance of AK-SVD and AK-SVD*. Additionally, in Table 5, we have highlighted the results where a DL method achieved the top 3 results; a superscript of the ROC AUC values indicates the ranking of the respective methods in the context of the 14 (unsupervised) methods from ADBench.

6. Conclusions

This paper presents a novel framework for initializing a dictionary using the SimCLR technique. The framework is suitable for general dictionary learning problems, classification problems that involve multiple dictionaries, and anomaly detection tasks. This approach makes the atoms in the dictionary more incoherent, resulting in a better representation of the input data and improved representation errors. The results obtained from several datasets demonstrate the efficacy of this framework for applications such as classification and outlier detection.

Future work will concentrate on enhancing the SimCDL framework, with a particular focus on the development of incoherent atoms within the DL problems in a supervised manner.