Locality-Constraint Discriminative Nonnegative Representation for Pattern Classification

Abstract

Representation-based classification methods (RBCM) have recently garnered notable attention in the field of pattern classification. Diverging from conventional methods reliant on ${\mathsf{\ell }}_1$ or ${\mathsf{\ell }}_2$-norms, the nonnegative representation-based classifier (NRC) enforces a nonnegative constraint on the representation vector, thus enhancing the representation capabilities of positively correlated samples. While NRC has achieved substantial success, it falls short in fully harnessing the discriminative information associated with the training samples and neglects the locality constraint inherent in the sample relationships, thereby limiting its classification power. In response to these limitations, we introduce the locality-constraint discriminative nonnegative representation (LDNR) method. LDNR extends the NRC framework through the incorporation of a competitive representation term. Recognizing the pivotal role played by the estimated samples in the classification process, we include estimated samples that involve discriminative information in this term, establishing a robust connection between representation and classification. Additionally, we assign distinct local weights to different estimated samples, augmenting the representation capacity of homogeneous samples and, ultimately, elevating the performance of the classification model. To validate the effectiveness of LDNR, extensive comparative experiments are conducted on various pattern classification datasets. The findings demonstrate the competitiveness of our proposed method.

1. Introduction

In the field of pattern classification, various classifiers have been developed to predict labels for query samples [1,2,3,4]. Among these, RBCM has recently garnered substantial attention. It is a method that involves representing a query sample through a linear combination of training samples, subsequently employing the estimated sample generated by the representation coefficients to predict the label of the query sample. One of the classic methods within the RBCM framework is sparse representation-based classifier (SRC) [5]. To represent the query sample using a reduced number of training samples, SRC employs sparsity property ${\mathsf{\ell }}_1$-norm, similar to the ${\mathsf{\ell }}_0$-norm, to constrain representation coefficients, thereby obtaining a sparse representation vector. It is noteworthy that SRC involves the minimization of the ${\mathsf{\ell }}_1$-norm, which lacks an analytical solution. To overcome this limitation, Zhang et al. [6] argued that collaborative representation of all training samples, rather than focusing solely on sparsity, can enhance the model’s classification performance. They introduced the collaborative representation-based classifier (CRC) model, which incorporates the ${\mathsf{\ell }}_2$-norm to regularize the representation vector. The collaborative representation (CR) problem is characterized as a typical ridge regression problem and possesses an analytical solution. Consequently, CRC achieves nearly the same level of accuracy as SRC while significantly reducing computational resources.

To further enhance the classification performance of CRC, Timofte et al. [7] proposed the weighted collaborative representation classifier (WCRC) model, which incorporates sample weights based on classification confidence and feature channel weights based on variance. This extension not only improves classification performance but also maintains simplicity and efficiency. Acknowledging the characteristics of intra-class residual in the nearest subspace classifier (NSC) [8] and inter-class residual in CRC, Chi et al. [9] introduced the collaborative representation optimized classifier (CROC) model, aiming to strike a balance between these two methods. Cai et al. [10] proposed a probabilistic collaborative representation-based classifier (ProCRC) model, which analyzes the principle of CRC from a probabilistic perspective. In the RBCM framework, query samples are represented by intricate addition and subtraction combinations of all the training samples, which can be well-explained mathematically but may be challenging to grasp at a physical level. Xu et al. [11] argued for the existence of only positive correlations among samples and, drawing inspiration from nonnegative matrix factorization (NMF) [12], introduced the NRC model. This model imposes a nonnegative constraint on the representation vector, thereby enhancing the representation capacity of positively correlated samples. Gou et al. [13] developed the weighted discriminative collaborative competitive representation (WDCCR) model, which incorporates class information to enhance the pattern discrimination between different classes. Wang et al. [14] argued that SRC and CRC primarily employ unsupervised learning in computing the representation vector. They introduced the discriminative representation-based classification (DRC) model, which utilizes label information from training samples to guide the representation and classification stages in a supervised manner, thereby enhancing the discriminative power of the representation vector. Unlike conventional representation methods, Zhang et al. [15] failed to directly constrain the representation vector but instead imposed constraints on representation components. They introduced the locality-constrained discriminative matrix regression (LDMR) model, which incorporates both local structure and label information, ultimately compelling homogeneous samples to contribute more to the representation and establishing a closer connection between representation and classification processes, respectively. Xu et al. [16] extended non-negative representation by introducing a scaled affine constraint, proposing the Simplex Representation model. Diverging from the utilization of affine constraint, this model suppresses the sum of entries in the representation vector to be equal to an adjustable scale $s(0<s<1)$, enhancing its discriminative capability. In cases where the original data is situated in a linearly inseparable space, the kernel nonnegative representation-based classifier (KNRC) [17] employs kernel methods to map samples into specific kernel feature spaces, enhancing the performance of NRC.

Although NRC and its variants have demonstrated impressive classification results, they do overlook certain aspects. NRC emphasizes the significance of acquiring the representation vector but falls short in fully exploiting the discriminative information of training samples. This essentially renders a disconnect between representation and classification, thereby hampering its classification capability. Additionally, NRC neglects the use of locality constraints within samples, which can result in an insufficient representation weight for homogeneous samples, potentially leading to a lack of discriminative power. To address these limitations, we introduce a novel method.

In this paper, we introduce the LDNR model. Specifically, we extend the NRC framework by introducing a competitive representation term that quantifies both the discriminative information and locality constraint of the samples. Recognizing the crucial role of estimated samples in classification, the competitive representation term includes all-class-estimated samples that involve sample discriminative information, emphasizing the significance of the connection between representation and classification. Simultaneously, we assign distinct weights to different estimated samples, quantified by the distance between the query sample and estimated samples, thereby strengthening the representation capabilities of homogeneous samples. Additionally, we employ the alternating direction method of multipliers (ADMM) [18] to address the LDNR problem and present a detailed iterative solution process. Finally, we conduct comprehensive comparative experiments with state-of-the-art RBCM and deep learning methods on various pattern classification datasets, and the results validate the competitiveness of our proposed method. As a result, the primary contributions of this paper are briefly outlined as follows:

• We present the LDNR model, which extends the NRC framework by introducing a competitive representation term that quantifies both the discriminative information and locality constraint of samples, ultimately improving the performance of the classification model.
• We employ the ADMM to address the LDNR problem and detail an iterative solution procedure.
• Extensive comparative experiments are conducted on various pattern classification datasets, and the findings demonstrate the competitiveness of the LDNR.

The rest of this paper is organized as follows. Section 2 provides an overview of the related work. Section 3 details the proposed LDNR method. Section 4 presents extensive experiments to validate this method. Section 5 concludes the paper and summarizes its findings. For a clearer description,

Table 1. Summary of general mathematical notations throughout all this paper.

Notations	Definitions
C	number of classes
n	number of training samples
m	dimension of feature
$n_i$	number of samples for the i-th class
$\mathit{\alpha }\in {\mathbb{R}}^{n\times 1}$	representation vector
${\mathit{\alpha }}_i\in {\mathbb{R}}^{n_i\times 1}$	representation vector for the i-th class
$\mathit{y}\in {\mathbb{R}}^{m\times 1}$	given query sample
$\mathbf{X}=\left[{\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_C\right]\in {\mathbb{R}}^{m\times n}$	training sample matrix
${\mathbf{X}}_i=\left[{\mathbf{x}}_{i,1},{\mathbf{x}}_{i,2},\dots ,{\mathbf{x}}_{i,n_i}\right]\in {\mathbb{R}}^{m\times n_i}$	training samples for the i-th class

shows some general mathematical notations.

2. Related Works

2.1. The NRC

In SRC [5] and CRC [6], along with their respective variants, it is widely acknowledged that they improve the classification performance of the model by considering the sparsity of the representation vector and the collaborative representation of all training samples. Nevertheless, negative coefficients are inevitable in these methods. NRC [11], on the other hand, considers a significant correlation between homogeneous samples, while asserting that there is no correlation between heterogeneous samples, rather than a negative one. It enforces a nonnegative constraint on the representation vector, and in this framework, only the training samples that exhibit a positive correlation with the query sample play a pivotal role in the representation process. NRC offers two key advantages. Firstly, it enhances the representation capability of homogeneous samples while weakening the ability of heterogeneous samples. Secondly, the nonnegative constrained representation vector tends to exhibit a certain sparsity, which can have important implications for improved classification accuracy. In summary, nonnegative representation (NR) can be formulated as the following optimization problem:

$$\underset{\mathit{\alpha }}{min}{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2,\mathrm{s}.\mathrm{t}.\mathit{\alpha }\ge 0$$

(1)

This problem can be effectively addressed using ADMM. Subsequently, the class-specific reconstruction residual can be computed based on the obtained representation vector:

$$r_i={∥\mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2.$$

(2)

Finally, the label of the query sample is determined by the class with the lowest reconstruction residual:

$$\mathrm{Label}\left(\mathit{y}\right)=arg\underset{i}{min}{r}_i$$

(3)

2.2. The LDMR

In contrast to the 1-D linear representation method that primarily focuses on optimizing the ${\mathsf{\ell }}_2$ norm of the error vector, LDMR [15] adopts a different method by optimizing the nuclear norm to obtain low-rank information from the error matrix. Given a collection of n training sample matrices $\mathbf{D}=\left[{\mathbf{D}}_1,{\mathbf{D}}_2,\cdots ,{\mathbf{D}}_n\right]$, the query sample matrix $\mathbf{Y}$ is represented by all the training samples:

$$\mathbf{Y}=\mathbf{D}\left(\mathit{\alpha }\right)+\mathbf{E}$$

(4)

where $\mathbf{E}$ signifies error matrix, $\mathbf{D}\left(\mathit{\alpha }\right)={\alpha }_1{\mathbf{D}}_1+{\alpha }_2{\mathbf{D}}_2+\cdots +{\mathit{\alpha }}_n{\mathbf{D}}_n$. In order to minimize the rank of the error matrix as much as possible, the following optimization problem is employed to obtain the representation vector $\mathit{\alpha }$:

$$\underset{\mathit{\alpha }}{min}\mathrm{rank}\left(\mathbf{E}\right),\mathrm{s}.\mathrm{t}.\mathbf{E}=\mathbf{Y}-\mathbf{D}\left(\mathit{\alpha }\right)$$

(5)

Given the NP-hard nature of this problem, it is challenging to obtain an exact solution. To facilitate a solution, we transform the original problem into the following form:

$$\underset{\mathit{\alpha }}{min}{\parallel \mathbf{E}\parallel }_{*},\mathrm{s}.\mathrm{t}.\mathbf{E}=\mathbf{Y}-\mathbf{D}\left(\mathit{\alpha }\right)$$

(6)

where ${\parallel ·\parallel }_{*}$ is operator of nuclear norm. Furthermore, LDMR recognizes that representation components ${\mathbf{X}}_i{\mathit{\alpha }}_i$ play a significant role in the classification process. In particular, the loss between representation components associated with incorrect classes and the query sample, ${\left\{Loss\left(\mathit{y}-{\mathbf{X}}_i{\alpha }_i\right)\right\}}_{i\ne k}$, should be as large as possible, while $Loss\left(\mathit{y}-{\mathbf{X}}_k{\alpha }_k\right)$ should be as small as possible, where k is ground-truth label of the query sample. To obtain a more discriminative representation vector, LDMR directly optimizes the ${\mathsf{\ell }}_2$-norm of all-class representation components within the representation process. Simultaneously, it assigns distinct weights to each term, leading to the following optimization problem:

$$\underset{\mathit{\alpha }}{min}\sum _{i=1}^C{w}_i{\parallel {\mathbf{X}}_i{\mathit{\alpha }}_i\parallel }_2^2$$

(7)

where $w_i$ signifies the weight coefficients and possesses the following form:

$$w_i=exp\left(\frac{\mathrm{dist}\left(\mathit{y},{\mathcal{S}}_i\right)}{\sigma }\right),\mathrm{dist}\left(\mathit{y},{\mathcal{S}}_i\right)=\frac{r_i-r_{min}}{r_{max}-r_{min}},r_i={\parallel \mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i\parallel }_2$$

(8)

where ${\mathcal{S}}_i$ represents the subspace spanned by samples of class i and $r_i$ denotes the distance between the query sample and subspace ${\mathcal{S}}_i$. Combining two problems (6) and (7), the overall optimization problem for LDMR is formulated as follows:

$$\underset{\mathit{\alpha }}{min}{\parallel \mathrm{Mat}\left(\mathit{y}-\mathbf{X}\mathit{\alpha }\right)\parallel }_{*}+\frac{\lambda }{2}\sum _{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2.$$

(9)

where $\mathrm{Mat}\left(·\right)$ operation signifies the transformation of vectors into matrices. The rest of this paper is not our point, if you wish to explore further details, we recommend referring to the original paper [15] for an in-depth investigation.

3. Locality-Constraint Discriminative Nonnegative Representation Method

3.1. Motivation

In the realm of RBCM, the key objective is to obtain the representation vector of the query sample effectively. Various methods have been developed to impose constraints on the representation vector, facilitating accurate classification of query samples. Actually, estimated samples play a pivotal role in the classification process, specifically, given an estimated sample for the i-th class, denoted as ${\mathbf{X}}_i{\mathit{\alpha }}_i$. From the perspective of classification, the term $\parallel \mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i{\parallel }_2$ signifies the reconstruction residual for the i-th class. To obtain more discriminative classification results, $\parallel \mathit{y}-{\mathbf{X}}_k{\mathit{\alpha }}_k{\parallel }_2$ should be made as small as possible, while ${\sum }_{i\ne k}{\parallel \mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i\parallel }_2$ should be made as large as possible, where k represents the label of the query sample. In other words, ${\left\{{\mathbf{X}}_i{\mathit{\alpha }}_i\right\}}_{i\ne k}$ and the query samples should be treated as irrelevant. Therefore, it is reasonable to penalize misclassified estimated samples during the representation process. This introduces a novel insight to leverage the discriminative information of the query sample rather than impose constraints on the representation vector, thus establishing a robust connection between representation and classification. However, a challenge arises because the discriminative information of the query sample is not available in advance. Consequently, it is viable to consider penalizing all-class-estimated samples simultaneously, namely ${\sum }_{i=1}^C{\parallel {\mathbf{X}}_i{\mathit{\alpha }}_i\parallel }_2^2$. Notably, estimated samples with different classes should not be treated equally, as they should bear distinct weights. Samples from the same class reside in a specific subspace, while samples from different classes occupy separate subspaces. The Euclidean distance between samples in the same subspace is significantly smaller than that of samples in different subspaces. Hence, it is logical to view $\parallel \mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i{\parallel }_2$ as a measure of locality constraint for quantifying the weights assigned to various estimated samples. In light of the considerations mentioned above, we propose locality-constraint discriminative nonnegative representation. A schematic illustration of this model is depicted in Figure 1. The flowchart is mainly divided into three steps: feature extraction, representation and classification. The representation process combines the locality constraint and the discriminative information between samples to calculate a more discriminative representation vector. The classification process determines the label of the query sample by residuals between the estimated samples and the query sample.

3.2. Representation

This work introduces the LDNR model that builds upon the NRC framework. Specifically, the LDNR model integrates both discriminative information and locality constraint among samples into the original objective function, resulting in the creation of a more discriminative model. The mathematical expression for LDNR is presented as follows:

$$\underset{\mathit{\alpha }}{\min }{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2,\mathrm{s}.\mathrm{t}.\mathit{\alpha }\ge 0$$

(10)

where $\lambda$ represents the regularization parameter that balances two terms. Furthermore, $w_i$ plays the role of weight coefficient, which is responsible for determining the significance of each estimated sample. More specifically, $w_i$ characterizes the correlation between the query sample and the estimated sample for the i-th class. We quantify this correlation by adopting the class-specific reconstruction residual, and $w_i$ is expressed as follows:

$$w_i=exp\left(\frac{r_i}{max\left(r_i\right)}/\sigma \right)$$

(11)

where $\sigma$ signifies the weight decay parameter and $r_i={\parallel \mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i\parallel }_2$ corresponds to the class-specific reconstruction residual. In problem (10), the first term represents the representation error term, and the second term is seen as the competitive representation term. Notably, when $\lambda =0$, LDNR degenerates to the NRC model. Consequently, NRC can be regarded as a special case within the broader framework of LDNR.

3.3. Optimization

The LDNR problem can be effectively addressed by employing an alternating optimization strategy. Initially, this work introduces an auxiliary variable $\mathit{z}$, reforming the original problem (10) in the following form:

$$\underset{\mathit{\alpha },\mathit{z}}{\min }{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2,\mathrm{s}.\mathrm{t}.\mathit{z}=\mathit{\alpha },\mathit{z}\ge 0$$

(12)

The problem (12) seems to add a layer of complexity in obtaining the optimal representation vector. Fortunately, ADMM [18] can tackle this issue conveniently. The associated augmented Lagrangian function can be defined as follows:

$$\mathcal{L}\left(\mathit{\alpha },\mathit{z},\delta ,\mu \right)={∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2+〈\delta ,\mathit{z}-\mathit{\alpha }〉+\frac{\mu }{2}{∥\mathit{z}-\mathit{\alpha }∥}_2^2$$

(13)

where $\delta$, and $\mu$ denote the Lagrange multiplier vector and penalty parameter, respectively. It is crucial to ensure that $\mu$ is assigned an appropriate and positive value. Additionally, the expression $〈\delta ,\mathit{z}-\mathit{\alpha }〉$ represents the inner product of two vectors. To tackle the optimization problem described in (12), we employ the method of alternating optimization to optimize variables one by one, that is, updating one variable while keeping the others fixed. Therefore, this optimization problem can be transformed into solving the following distinct subproblems:

• Update variable $\mathit{\alpha }$ with regular $\mathit{z}$ and $\delta$:

$$\underset{\mathit{\alpha }}{\min }{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2+{\delta }_t^T\left({\mathit{z}}_t-\mathit{\alpha }\right)+\frac{\mu }{2}{∥{\mathit{z}}_t-\mathit{\alpha }∥}_2^2$$

(14)

After sorting out the last two items, we can obtain the following form:

$$\underset{\mathit{\alpha }}{\min }{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2+\frac{\mu }{2}{∥{\mathit{z}}_t-\mathit{\alpha }+\frac{{\delta }_t}{\mu }∥}_2^2$$

(15)

In order to facilitate optimization, a diagonal matrix ${\mathbf{H}}_i=\mathrm{diag}\left(0,\dots ,0,1,\dots ,1,0,\dots ,0\right)$ is created, in which all elements corresponding to the i class are 1, and the rest are 0. This is in light of the fact that ${∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2={∥\mathbf{X}{\mathbf{H}}_i\mathit{\alpha }∥}_2^2$. Consequently, (15) can be transformed as follows:

$$\underset{\mathit{\alpha }}{\min }{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥\mathbf{X}{\mathbf{H}}_i\mathit{\alpha }∥}_2^2+\frac{\mu }{2}{∥{\mathit{z}}_t-\mathit{\alpha }+\frac{{\delta }_t}{\mu }∥}_2^2$$

(16)

By calculating the partial derivative of (16) with respect to $\mathit{\alpha }$ and subsequently setting it to zero, we can derive the analytical solution:

$$\frac{\partial }{\partial \mathit{\alpha }}\left({∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda \sum _{i=1}^C{w}_i{∥\mathbf{X}{\mathbf{H}}_i\mathit{\alpha }∥}_2^2+\frac{\mu }{2}{∥{\mathit{z}}_t-\mathit{\alpha }+\frac{{\delta }_t}{\mu }∥}_2^2\right)=0,$$

(17)

$$-2{\mathbf{X}}^T\left(\mathit{y}-\mathbf{X}\mathit{\alpha }\right)+2\lambda \sum _{i=1}^C{w}_i{\left(\mathbf{X}{\mathbf{H}}_i\right)}^T\left(\mathbf{X}{\mathbf{H}}_i\mathit{\alpha }\right)-\mu \left({\mathit{z}}_t-\mathit{\alpha }+\frac{{\delta }_t}{\mu }\right)=0,$$

(18)

$$-{\mathbf{X}}^T\mathit{y}+{\mathbf{X}}^T\mathbf{X}\mathit{\alpha }+\lambda \sum _{i=1}^C{w}_i{\mathbf{H}}_i^T{\mathbf{X}}^T\mathbf{X}{\mathbf{H}}_i\mathit{\alpha }-\frac{\mu }{2}{\mathit{z}}_t+\frac{\mu }{2}\mathit{\alpha }-\frac{{\delta }_t}{2}=0,$$

(19)

$$\left({\mathbf{X}}^T\mathbf{X}+\lambda \sum _{i=1}^C{w}_i{\mathbf{H}}_i^T{\mathbf{X}}^T\mathbf{X}{\mathbf{H}}_i+\frac{\mu }{2}\mathbf{I}\right)\mathit{\alpha }={\mathbf{X}}^T\mathit{y}+\frac{\mu {\mathit{z}}_t+{\delta }_t}{2}$$

(20)

Consequently, we can obtain the solution to $\mathit{\alpha }$:

$${\mathit{\alpha }}_{t+1}=\mathbf{P}\left({\mathbf{X}}^T\mathit{y}+\frac{\mu {\mathit{z}}_t+{\delta }_t}{2}\right)$$

(21)

where $\mathbf{P}={\left({\mathbf{X}}^T\mathbf{X}+\lambda {\sum }_{i=1}^C{w}_i{\mathbf{H}}_i^T{\mathbf{X}}^T\mathbf{X}{\mathbf{H}}_i+\frac{\mu }{2}\mathbf{I}\right)}^{-1}$, in which $\mathbf{I}$ denotes identity matrix.

• Update variable $\mathit{z}$ with regular $\mathit{\alpha }$ and $\delta$:

$$\underset{\mathit{z}}{\min }{∥\mathit{z}-\left({\mathit{\alpha }}_{t+1}-\frac{{\delta }_t}{\mu }\right)∥}_2^2,\mathrm{s}.\mathrm{t}.\mathit{z}\ge 0$$

(22)

Obviously, the solution to (22) is:

$${\mathit{z}}_{t+1}=\max \left(0,{\mathit{\alpha }}_{t+1}-\frac{{\delta }_t}{\mu }\right)$$

(23)

• Update variable $\delta$:

$${\delta }_{t+1}={\delta }_t+\mu \left({\mathit{z}}_{t+1}-{\mathit{\alpha }}_{t+1}\right)$$

(24)

The variables ${\mathit{\alpha }}_t$, ${\mathit{z}}_t$, and ${\delta }_t$ mentioned above represent the values of $\mathit{\alpha }$, $\mathit{z}$, and $\delta$ at each iteration step $t=\{1,2,\cdots ,T\}$, where T refers to maximum number of iterations. Initially, at $t=0$, these variables ${\mathit{\alpha }}_0$, ${\mathit{z}}_0$, and ${\delta }_0$ are typically initialized to zero. The iteration process continues until one of two conditions is met: either it reaches the maximum iteration limit, or it satisfies the convergence conditions. These convergence conditions are defined as follows: ${∥{\mathit{\alpha }}_t-{\mathit{z}}_t∥}_2\le \eta$, ${∥{\mathit{\alpha }}_{t+1}-{\mathit{\alpha }}_t∥}_2\le \eta$ and ${∥{\mathit{z}}_{t+1}-{\mathit{z}}_t∥}_2\le \eta$. Here, $\eta$ represents a predetermined threshold, typically set to $1e-5$ empirically. Importantly, all three of these convergence conditions must be satisfied simultaneously for the iteration to terminate. In summary, the procedure for solving the LDNR problem is outlined in Algorithm 1.

Algorithm 1 Solve LDNR problem via the use of ADMM.

Input: Query sample $\mathit{y}$, training samples $\mathbf{X}$, $\lambda$, $\sigma$, $\mu >0$, T; 1: Initialization: ${\mathit{\alpha }}_0=0$, ${\mathit{z}}_0=0$, ${\delta }_0=0$; 2: for $t=0:T-1$ do 3:     ${\mathit{\alpha }}_{t+1}=\mathbf{P}\left({\mathbf{X}}^T\mathit{y}+\frac{\mu {\mathit{z}}_t+{\delta }_t}{2}\right)$, $\mathbf{P}={\left({\mathbf{X}}^T\mathbf{X}+\lambda {\sum }_{i=1}^C{w}_i{\mathbf{H}}_i^T{\mathbf{X}}^T\mathbf{X}{\mathbf{H}}_i+\frac{\mu }{2}\mathbf{I}\right)}^{-1}$; 4:     ${\mathit{z}}_{t+1}=\max \left(0,{\mathit{\alpha }}_{t+1}-\frac{{\delta }_t}{\mu }\right)$; 5:     ${\delta }_{t+1}={\delta }_t+\mu \left({\mathit{z}}_{t+1}-{\mathit{\alpha }}_{t+1}\right)$; 6:     if converged then 7:         Stop; 8:     end if 9: end for Output: Representation vector ${\mathit{\alpha }}_{t+1}$.

3.4. Convergence Analysis

Due to the convex nature of the objective function of LDNR, a globally optimal representation vector can be obtained. We conduct a convergence analysis on the AR dataset [19] and illustrate the variation of convergence conditions of Algorithm 1 with the number of iterations in Figure 2. We can observe a rapid decrease in the initial 10 iterations, with the convergence condition approaching zero by approximately the 120-th iteration.

3.5. Classification

In the classification process, LDNR-based classifiers typically follow these key steps: Solve the LDNR problem to obtain the representation vector. Calculate the reconstruction residual for each class. Assign the query sample to the class with the lowest reconstruction residual. Algorithm 2 succinctly outlines the entire classification procedure.

Algorithm 2 LDNR-based classifier.

Input: Query sample $\mathit{y}$, training samples $\mathbf{X}$; 1: Normalize each column of $\mathbf{X}$ and $\mathit{y}$ to have unit ${\mathsf{\ell }}_2$-norm; 2: Calculate the representation vector $\mathit{\alpha }$ by Algorithm 1:         $\underset{\mathit{\alpha }}{\min }{∥\mathit{y}-\mathbf{X}\mathit{\alpha }∥}_2^2+\lambda {\sum }_{i=1}^C{w}_i{∥{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2^2,\mathrm{s}.\mathrm{t}.\mathit{\alpha }\ge 0$; 3: Calculate the class-specific reconstruction residual:        $r_i={∥\mathit{y}-{\mathbf{X}}_i{\mathit{\alpha }}_i∥}_2$; Output: $\mathrm{label}\left(\mathit{y}\right)=arg\underset{i}{min}{r}_i$.

3.6. Complexity Analysis

In our algorithmic solution, we iteratively update each parameter to converge to an optimal value. In Algorithm 1, the complexity of each subproblem is as follows. When solving $\mathit{\alpha }$, the primary computational cost lies in computing the matrix $\mathbf{P}$. Solving the matrix $\mathbf{P}$ involves a matrix inversion operation and some matrix multiplications, resulting in a computational complexity of $O(n^3+m{n}^2)$. Subsequently, the complexity of multiplying matrix $\mathbf{P}$ with the subsequent matrix is $O\left(n^2\right)$. Therefore, the overall computational complexity of solving $\mathit{\alpha }$ is $O(n^3+m{n}^2+n^2)$. Additionally, the complexities for solving $\mathit{z}$ and $\delta$ are both $O\left(n\right)$.

In Algorithm 2, since the primary computational cost is matrix multiplication, the computational complexity for calculating the reconstruction residual is $O\left(mn\right)$. Finally, the overall algorithmic complexity is $O(n^3+m{n}^2+n^2+mn+n)$.

4. Experiments

4.1. Experimental Settings

In this section, we validate the effectiveness of our proposed LDNR by conducting a series of comparative experiments on pattern classification datasets. These datasets are primarily divided into two categories: small-scale and large-scale datasets. In the small-scale datasets, we selected the AR face [19] and the USPS handwritten digit [20] datasets for our experiments. For large-scale datasets, we utilize four visual classification datasets, namely CUB-200-2011 [21], Oxford 102 Flowers [22], Aircraft [23], and Cars [24]. In the comparative experiments, we compare LDNR with diverse state-of-the-art methods, including NSC, SVM [25], SRC, CRC, CROC, ProCRC, NRC, DRC, Softmax, VGG16 [26], Symbiotic [27], FV-FGC [28], and B-CNN [29]. Furthermore, for fair comparison, we employ the same image features as NRC. Concretely, for the experiments conducted on the CUB-200-2011 and Oxford 102 Flowers datasets, we adopt the BOW-SIFT feature of samples as our training data, and on the Aircraft and Cars datasets, we use the VGG-verydeep-16 model to extract the features of samples.

The LDNR method relies on four pivotal parameters: the regularization parameter $\lambda$, weight decay parameter $\sigma$, penalty parameter $\mu$, and maximum number of iterations T. Among these, we set $T=5$, while the remaining parameters are determined through a rigorous five-fold cross-validation process. The ranges for $\lambda$, $\mu$, $\sigma$ are appointed as $\{0.0001,0.001,0.01,0.1,1\}$, $\{0.0001,0.001,0.01,0.1,1\}$, {1, 10, 100, 1000, 10,000}, respectively. To ensure a fair comparison with the standard baseline, we replicate the NRC method on our platform and settings. Following the source code provided by the original author, we fine-tune the parameters to achieve its best experimental results. Furthermore, given that DRC represents a more advanced method than NRC, we apply the same method to attain the best classification results. As for the experimental outcomes of other methods, we derive them from the literature [11]. The results of our experiments are listed in the respective tables. Notably, the highest classification accuracy achieved in each experiment is highlighted in bold font.

4.2. Small-Scale Datasets

4.2.1. Experiments on AR Dataset

The AR dataset comprises a total of 4000 frontal color face images, encompassing 126 distinct individuals, of which 56 are male and 70 are female. These images exhibit diverse characteristics, including variations in lighting conditions, facial expressions, and occlusions. Furthermore, each individual’s images are captured in two separate sessions, with a 14-day interval between them. Figure 3 shows a selection of images from the AR dataset. In the context of our experimental design, we select a subset of 100 individuals (50 males and 50 females). Each of these selected individuals possesses fouteen images that demonstrate variations in both lighting and facial expressions. Among these fourteen images, seven images from session 1 are designated for training, while the remaining seven images from session 2 are treated for testing.

Before conducting the experiments, we crop the images in the size of $60\times 43$ and normalize them to have unit ${\mathsf{\ell }}_2$-norm. Subsequently, we employ PCA to project these images into feature subspaces of dimensions 54, 120, and 300, allowing for comparisons across different dimensions. The experimental results are concisely summarized in

Table 2. Classification accuracy (%) of various methods on AR dataset.

Dimensions	NSC [8]	SVM [25]	SRC [5]	CRC [6]	CROC [9]	ProCRC [10]	NRC [11]	DRC [14]	LDNR
54	70.7	81.6	82.1	80.3	82.0	81.4	85.2	80.6	85.8
120	75.5	89.3	88.3	90.0	90.8	90.7	91.3	90.1	91.4
300	76.1	91.6	90.3	93.7	93.7	93.7	93.3	93.8	94.3

. It is evident that across all three dimensions, our LDNR outperforms all the other methods. Particularly in the 300-dimensional feature space, our algorithm achieves a remarkable accuracy of 94.3%, surpassing NRC by 1% and DRC, by 0.5%. This compelling result underscores the effectiveness of LDNR in this facial dataset.

4.2.2. Experiments on USPS Dataset

The USPS dataset constitutes a collection of grayscale handwritten digit images with a wide range of font styles. It encompasses a total of 9298 images, depicting digits from 0 to 9. The dataset is divided into two sets, with 7291 images designated for training and 2007 images reserved for testing. Notably, every image is standardized $16\times 16$ pixels and consistently centered. We display some images from this dataset in Figure 4. In the experimental setup, we structured samples as follows. A total of 50, 100, and 200 images per digit are randomly organized, respectively, for training while the remaining samples are allocated for testing.

Similar to our experiments on the AR dataset, we conduct a comparative analysis of the LDNR method against several established methods, including NSC, SVM, SRC, CRC, CROC, ProCRC, NRC, and DRC. We repeat ten independent experiments and put the average results of all experiments in

Table 3. Classification accuracy (%) of various methods on USPS dataset.

Images	[8]	SVM [25]	SRC [5]	CRC [6]	CROC [9]	ProCRC [10]	NRC [11]	DRC [14]	LDNR
50	91.2	91.6	91.4	89.2	91.9	90.9	91.2	87.8	92.1
100	92.2	92.5	93.1	90.6	91.3	91.9	92.4	89.3	93.2
200	92.8	93.1	94.2	91.4	91.7	92.2	93.2	90.4	94.0

. As we can see, when the numbers of samples are 50 and 100, LDNR achieves the highest accuracy, outperforming NRC by 0.9% and 0.8%, respectively. However, with a training sample size of 200, LDNR exhibits a slightly lower performance than SRC, although it still maintains an advantage over NRC by 0.8%.

4.3. Large-Scale Datasets with BOW-SIFT Feature

4.3.1. Bow-Sift Feature

In this section, according to the settings in [10], we utilize VLFeat [30] to extract Bag-of-Words features [31] based on SIFT, referred to as BOW-SIFT. The size of the patch is set to $16\times 16$, and the stride is set to eight pixels. The codebook is trained using the k-means algorithm, and a two-level spatial pyramid representation [32] is employed. This results in the dimensions of each image’s features being 5120. Finally, all features are ${\mathsf{\ell }}_2$-normalized.

4.3.2. Experiments on CUB-200-2011 Dataset

The Caltech-UCSD Birds-200-2011 (CUB-200-2011) is well recognized as a prominent dataset for fine-grained visual classification tasks. The dataset comprises 11,788 images depicting 200 unique bird species, with 5994 images allocated for training and 5794 for testing. Figure 5 exhibits a collection of images from this dataset. We followed the data-splitting method outlined in the original dataset, allocating around half of the data for training and the remaining for testing.

In this experiment, we employ the BOW-SIFT features of the images as training data and conduct a comparative analysis of LDNR against various methods, including Softmax, NSC, SRC, CRC, CROC, ProCRC, NRC, and DRC. The results of this comparison are detailed in

Table 4. Classification accuracy (%) of various methods on CUB-200-2011 dataset.

Dataset	Softmax	NSC [8]	SRC [5]	CRC [6]	CROC [9]	ProCRC [10]	NRC [11]	DRC [14]	LDNR
CUB-200-2011	8.2	8.4	7.7	9.4	9.1	9.9	9.9	7.8	9.9

. When combining the BOW-SIFT features, our LDNR achieves accuracy levels comparable to NRC and ProCRC, but outperforms Softmax by 1.7%, NSC by 1.5%, SRC by 2.2%, CRC by 0.5%, CROC by 0.8%, and DRC by 2.1%.

4.3.3. Experiments on Oxford 102 Flowers Dataset

The Oxford 102 Flowers dataset contains a total of 8198 images for 102 flower classes, with each class having between four to two hundred fifty-eight images. The dataset presents a notable challenge due to the diverse variations in image scale, pose, and illumination. Figure 6 displays some typical images from this dataset. For our experiment, we employ the data partitioning method as outlined in the original dataset, in which 2040 images are allocated for training and 6149 images are reserved for testing.

Consistent with settings on the CUB-200-2011 dataset, we apply the same set of comparative methods and summarize comparison results in

Table 5. Classification accuracy (%) of various methods on Oxford 102 Flowers dataset.

Dataset	Softmax	NSC [8]	SRC [5]	CRC [6]	CROC [9]	ProCRC [10]	NRC [11]	DRC [14]	LDNR
Oxford 102 Flowers	46.5	46.7	47.2	49.9	49.4	51.2	52.1	37.9	52.5

. Notably, the LDNR method achieves the highest classification accuracy at 52.5%, which shows a significant improvement over other methods, surpassing Softmax by 6%, NSC by 5.8%, SRC by 5.3%, CRC by 2.6%, CROC by 3.1%, ProCRC by 1.3%, NRC by 0.4%, and DRC by an impressive 14.6%. The outcome further highlights the competitiveness of LDNR in fine-grained image classification tasks.

4.4. Large-Scale Datasets with VGG16 Feature

4.4.1. Vgg16 Feature

Based on the results of previous experiments, we have successfully demonstrated the superiority of LDNR over some excellent RBCM. To further solidify these findings, we engage in an unfair evaluation by executing a comparison between LDNR and various deep-learning methods. Following the same settings in [10], we utilize the VGG-Deep-16 model to extract CNN features, referred to as VGG16 features. We employ the activations from the penultimate layer as local features and extract them from five scales {$2^s,s=-1,-0.5,0,0.5,1$}. Subsequently, all local features are pooled together without consideration of scales and locations. For all datasets, the final feature dimension for each image is 4096. Finally, all features are ${\mathsf{\ell }}_2$-normalized.

4.4.2. Experiments on Aircraft Dataset

The Aircraft dataset is a pivotal benchmark dataset for fine-grained visual classification tasks. It comprises a total of 10,000 aircraft images, spanning 100 distinct aircraft models. These aircraft appear in different appearances, scales, and design structures, making this dataset highly challenging for image classification. Figure 7 showcases some representative images from this dataset. In order to organize the data, we follow the dataset’s data partitioning method, where 6667 images from the 100 aircraft classes are allocated for training, and the remaining 3333 images are reserved for testing.

In this experiment, we employ VGG16 features extracted from the images as training data and conduct a comparative analysis, comparing the LDNR against VGG16, Symbiotic, FV-FGC, and B-CNN. The results of this comparison are succinctly summarized in

Table 6. Classification accuracy (%) of various methods on Aircraft dataset.

Dataset	VGG16 [26]	Symbiotic [27]	FV-FGC [28]	B-CNN [29]	LDNR
Aircraft	85.6	72.5	80.7	84.1	87.7

. It is observed in the table that CNN features tend to yield higher accuracy compared to BOW-SIFT features. Furthermore, our LDNR still emerges as the top-performing method, achieving the highest classification accuracy and surpassing VGG16 by 2.1%, Symbiotic by 15.2%, FV-FGC by 7%, and B-CNN by 3.6%.

4.4.3. Experiments on Cars Dataset

The Cars dataset is primarily used for fine-grained visual classification tasks. It comprises 16,185 images depicting various cars and encompasses frontal, lateral, and posterior perspectives of the cars, along with various angles and lighting conditions. Figure 8 displays some illustrative images from this dataset. Our experiment adheres to the data-splitting strategy prescribed by the dataset, wherein 8144 images are allocated for training and the remaining 8041 images are used for testing.

Consistent with settings from the Aircraft experiment, we employ the VGG16 feature extracted from images as training data and summarize the results of this comparative analysis in

Table 7. Classification accuracy (%) of various methods on Cars dataset.

Dataset	VGG16 [26]	Symbiotic [27]	FV-FGC [28]	B-CNN [29]	LDNR
Cars	88.7	78.0	82.7	90.6	91.0

. Notably, the LDNR method once again achieves the highest classification accuracy at 91%. This accuracy surpassed VGG16 by 2.3%, Symbiotic by 13%, FV-FGC by 8.3%, and B-CNN by 0.4%.

4.5. Parameters Analysis

In this section, we present an analysis of the impact exerted by three key parameters, namely $\lambda$, $\mu$, and $\sigma$, on the efficacy of the LDNR method. The experimental setup involves fixing two parameters at constant values while systematically varying the third one across a pre-defined range. These ranges were set as follows: $\{0.0001,0.001,0.01,0.1,1\}$ for both $\lambda$ and $\mu$, while $\sigma$ was varied across the interval {1, 10, 100, 1000, 10,000}. The experiments were carried out using the AR face dataset consisting of 120-dimensional data. Figure 9 illustrates how changes in each parameter affect classification accuracy.

It is observed that the highest accuracy is achieved when $\lambda =0.01$. Initially, accuracy shows a mild increase up until $\lambda =0.01$, beyond which it begins to decrease as $\lambda$ continues to rise. This pattern suggests that $\lambda$ plays a pivotal role in balancing the relative importance between the representation error term and the competitive representation term. As such, an excessively large $\lambda$ can lead to reduced significance being attributed to the former, ultimately compromising the overall effectiveness of the method. Regarding $\mu$, it becomes clear that the most favorable outcome occurs at $\mu =0.1$ The detrimental effect caused by values either too small or too large highlights its sensitivity and underscores the need for careful tuning. Finally, for the parameter $\sigma$, a more modest fluctuation in classification accuracy is observed across its entire tested range, denoting greater stability when compared with $\lambda$ and $\mu$. Hence, adjustment of $\sigma$ may not require excessive attention.

5. Conclusions and Discussion

In this paper, we extend the NRC model and propose the LDNR model by introducing a competition representation term that quantifies the discriminative information and locality constraint of the sample. Specifically, we regularize all-class-estimated samples in the representation stage, thus strengthening the connection between representation and classification. Simultaneously, we assign different weights to each estimated sample, thereby augmenting the representation power of homogeneous samples, ultimately, resulting in an improved performance. Furthermore, we employ ADMM to efficiently address the LDNR problem and analyze its iterative steps. Finally, comprehensive comparison experiments with many excellent RBCM and deep learning methods are conducted on various challenging datasets to verify the effectiveness of LDNR, and the findings demonstrate the competitiveness of the LDNR.

Similar to other representation-based classification methods, our proposed LDNR relies on clean training data devoid of occlusion or corruption. The inclusion of such data can potentially lead to a degradation in LDNR’s performance. To address this challenge, various approaches such as low-rank matrix recovery techniques and image deblurring techniques may be explored as potential solutions.