Kernel Reverse Neighborhood Discriminant Analysis

Abstract

Currently, neighborhood linear discriminant analysis (nLDA) exploits reverse nearest neighbors (RNN) to avoid the assumption of linear discriminant analysis (LDA) that all samples from the same class should be independently and identically distributed (i.i.d.). nLDA performs well when a dataset contains multimodal classes. However, in complex pattern recognition tasks, such as visual classification, the complex appearance variations caused by deformation, illumination and visual angle often generate non-linearity. Furthermore, it is not easy to separate the multimodal classes in lower-dimensional feature space. One solution to these problems is to map the feature to a higher-dimensional feature space for discriminant learning. Hence, in this paper, we employ kernel functions to map the original data to a higher-dimensional feature space, where the nonlinear multimodal classes can be better classified. We give the details of the deduction of the proposed kernel reverse neighborhood discriminant analysis (KRNDA) with the kernel tricks. The proposed KRNDA outperforms the original nLDA on most datasets of the UCI benchmark database. In high-dimensional visual recognition tasks of handwritten digit recognition, object categorization and face recognition, our KRNDA achieves the best recognition results compared to several sophisticated LDA-based discriminators.

1. Introduction

Linear discriminant analysis (LDA) [1] is a supervised classical linear learning method stemming from the Fisher criterion [2], which can be used not only for supervised classification problems but also for feature dimensional reduction. Its idea is not difficult to understand. For a set of given training samples, LDA attempts to find the optimal projection matrix that can maximize the between-class scatter matrix and minimize the within-class scatter matrix, simultaneously. As a supervised feature extraction method, LDA has demonstrated its feasibility and efficiency in pattern classification [3], and it has been widely applied in hyperspectral image classification [4], EEG signals analysis [5], re-identification [6], etc. LDA is also a critical evaluation approach to visual classification, such as handwriting recognition, face recognition and object categorization [7].

LDA assumes that the samples of all training data follow a multivariate Gaussian distribution that has the same covariance but different mean values for different classes [8]; once this statement is not satisfied, for example, when several different clusters are encountered from samples in the same class, LDA gives undesirable results [9]. To solve this problem, a two-step subclass division method was proposed [10]; the first step applies k-means for clustering and the second step uses the EM-alike framework to optimize the subclasses. Recently, the reverse nearest neighbors approach has been exploited with LDA to form neighborhood linear discriminant analysis (nLDA) [7]. As an unsupervised outlier detection method, reverse nearest neighbor (RNN) can eliminate the “isolated points” in the training set. nLDA expects that a sample and its RNN should be as close as possible, while the RNN from two samples that belong to different classes should be as far as possible in the projected space. As the scatter matrix is defined based on RNN, which does not directly consider the whole class, the problem of datasets containing multimodal classes can be overcome by nLDA. nLDA is a method where the scatter matrix is directly defined on the neighborhood, which is a new localized discriminator. The smallest subclass can be thought of as a neighborhood; thus, it can perform discriminative learning without obeying the independently and identically distributed (i.i.d.) assumption of sample data.

However, since nLDA is extended from LDA, which is a linear parametric method, thus it may fail to handle nonlinear data. It is well known that the complex appearance variations caused by deformation, illumination and visual angle often generate non-linearity [11,12]. If we ignore the non-linearity of the data, the linear discriminators often give a lower performance. One practical solution is to map the points of the original input data to a higher-dimensional feature space and then learn the linear classifier on that space for discriminant learning. However, addressing the non-linearity in the higher-dimensional feature space is non-trivial, especially when the dimension of the mapped space is infinite. Many methods have been proposed to use the kernel trick to handle this problem [11,12,13], which is also exploited in our proposed approach.

To handle the nonlinear data, circumvent the assumption of i.i.d. of sample data and solve the multimodal classes problem simultaneously, in this paper, we extend nLDA to its kernel version, called kernel reverse neighborhood discriminant analysis (KRNDA). The Gaussian kernel is employed to map the input data into a higher-dimensional feature space where non-linearity can be alleviated. The deduction of KRNDA with kernel tricks is presented in detail. We give extensive evaluations on the UCI dataset, and the visual classification tasks of handwriting recognition, face recognition and object categorization to verify the effectiveness of the proposed approach.

The rest of this paper is organized as follows. Section 2 reviews the related work. Section 3 briefly introduces the basic principles of LDA and nLDA methods. Subsequently, we propose our KRNDA method; the deduction with the kernel trick is presented in Section 4. Experiments are conducted in Section 5. Section 6 concludes this paper.

2. Related Work

Research works on LDA have been studied for decades [14,15,16] and plentiful approaches have been proposed to address different problems of LDA. For pattern recognition, the small sample size (SSS) problem is an essential issue for LDA. The SSS problem generally occurs when the feature dimension of each sample is very high, while there are insufficient training samples of each class. This result often leads to the singularity or ill-condition of the within-class scatter matrix, and the LDA can not be solved by the eigen-decomposition. To address this problem, Fisherfaces [1] were proposed to conduct principal component analysis (PCA) for reducing the feature dimension of each sample and then implement the LDA algorithm. The singularity or ill-condition of the within-class scatter matrix caused by the SSS problem can also be solved by adding a small value to the diagonal of the within-class scatter matrix, which is known as regularized LDA [17]. Pang et al. [18] combined the clustering and regularization terms to alleviate the deviation of the scatter matrix caused by the SSS problem. There are many other approaches proposed to circumvent the singularity of the within-class scatter matrix and the instability of its reverse caused by the SSS. For example, direct LDA (DLDA) [19] and null space LDA (NLDA) [20] select different subspaces of the scatter matrix to avoid the singularity. Eigenfeature regularization and extraction (ERE) [21] retains and reweights the entire eigenspace of the within-class scatter matrix for non-singularity to persist. This approach achieved excellent performance on face recognition.

The multimodal classes problem of LDA arouses a lot of attention. Marginal fisher analysis (MFA) [22] employs graph embedding to characterize the nearest neighbors to avoid the i.i.d. of the sample data. Exploiting manifolds to represent local structures is also one of the solutions. For instance, local fisher discriminant analysis (LFDA) [23] effectively combines the ideas of locality preserving projections (LPPs) [24] and LDA; it attains inter-class separation and intra-class local structure preservation by defining a local intra-class and a local inter-class scatter matrix to handle the multimodal classes problem. Locality sensitive discriminant analysis (LSDA) [25] projects the dataset to a lower-dimensional subspace to preserve local manifold structure and discriminant information. Neighbors with the same label should be as close as possible; that is the aim of LFDA and LSDA. Nonparametric discriminant analysis (NDA) [9] solves the multimodal problem of LDA by using k-nearest neighbors, which constructs a kind of nonparametric inter-class divergence in a local area. Nonparametric discriminant analysis for face recognition [26] extends NDA to address multi-class situations. Recently, nLDA [7] directly defined the scatter matrix on the neighborhood to solve the multimodal classes problem and achieved remarkable results.

To handle the non-linearity problem, many LDA-based approaches are extended to their kernel versions to learn the nonlinear structures by the powerful tool of the kernel trick [27]; for example, generalized discriminant analysis (GDA) [11] detailed the deduction of kernel LDA. Kernel Fisherface (KLDA) [28], kernel direct-LDA (KDDA) [13], null space kernel LDA (NKDA) [12] and complete discriminant evaluation and feature extraction (CDEFE) [29] are the kernel versions of PCA+LDA, DLDA, NDA and ERE, respectively. In the previous studies, researchers worked out an alternative formulation of kernel LPP (KLPP) used to develop a framework for KPCA+LPP algorithms. Ref. [30] achieved good results in recognition tasks, such as face recognition and radar target recognition. In solving the nonlinear problems encountered in image classification, the KAHISD/KCHISD of the Cevikalp and Triggs [31] article is an extension of the affine/convex hull-based image classification task to its kernel version. Zhu et al. [32] proposed the KCH-ISCRC with kernel tricks, which well addresses collaborative image set-based representation and classification (ISCRC). It is easy to see that Gaussian kernels can successfully solve nonlinear problems in various applications [28,29,31,32].

LDA has new developments with recent technologies. Dorfer et al. [33] proposed a deep linear discriminant analysis with the deep neural network. Alarcón and Destercke [34] proposed a Gaussian discriminator to fuse the near-ignorance priors and robust Bayesian analysis. Belous et al. [35] proposed a framework called dual spatial discriminative projection learning and successfully applied it to image classification tasks. Hu et al. [36] proposed a cross-modal discriminative network for cross-modal learning.

3. LDA and nLDA

In this section, we briefly introduce the algorithm of LDA [1] and nLDA [7].

3.1. LDA

The idea and algorithm of LDA are simple and intuitive. Let $\mathbf{X}=\left[x_1,x_2,x_3,\dots ,x_N\right]\in {\mathbb{R}}^{D\times N}$ denotes the D-dimensional training dataset, where N is the number of total training samples and D is the dimension of each sample. The class label of the training sample $x_i\in {\mathbb{R}}^D$ is denoted by $y_i$ with $y\left(x_i\right)\in (1,2,\cdots ,C)$, where C represents the total class number of samples. We use $N_i$ to describe the number of training samples in class i. The execution of LDA is to use the most discriminative projection matrix $\mathsf{\varrho }=\left[{\varrho }_1,{\varrho }_2,{\varrho }_3\dots {\varrho }_d\right]\in {\mathbb{R}}^{D\times d}$ ($d\le D$) to project the training set $\mathbf{X}\in {\mathbb{R}}^{D\times N}$ from high-dimensional space to a lower-dimensional feature space by:

$${\mathsf{\varrho }}^T\mathbf{X}\in {\mathbb{R}}^{d\times N}.$$

(1)

The optimal projected matrix $\varrho$ of LDA can be obtained by maximizing the equation:

$${\mathbf{J}}_{LDA}\left(\mathsf{\varrho }\right)=\frac{{\mathsf{\varrho }}^T{\mathbf{S}}_b\mathsf{\varrho }}{{\varrho }^T{\mathbf{S}}_w\varrho }.$$

(2)

where the between-class scatter matrix ${\mathbf{S}}_{\mathbf{b}}$ and within-class scatter matrix ${\mathbf{S}}_{\mathbf{w}}$ are defined, respectively, by:

$${\mathbf{S}}_b=\frac{1}{C}\sum _{i=1}^C\left(u_i-u\right){\left(u_i-u\right)}^T.$$

(3)

$${\mathbf{S}}_w=\sum _{j=1}^C\sum _{y\left(x_i\right)=j}\left(x_i-u_j\right){\left(x_i-u_j\right)}^T.$$

(4)

where $u_i$ is the mean vector of the i-th class:

$$u_i=\frac{1}{N_i}\sum _{y\left(x_j\right)=i}{x}_j.$$

(5)

u is the mean of all training samples:

$$u=\frac{1}{N}\sum _{i=1}^N{x}_i.$$

(6)

In LDA, Equation (2) can be reduced to the following eigen-decomposition problem:

$${\mathbf{S}}_b\mathsf{\varrho }=\lambda {\mathbf{S}}_w\mathsf{\varrho }.$$

(7)

By retaining d eigenvectors corresponding to the largest eigenvalues, the most discriminative projection matrix $\mathsf{\varrho }$ is formed.

3.2. nLDA

LDA constructs the within-class scatter matrix with the information provided by the whole class, while nLDA defines the local neighborhood scatter by using the reverse k-nearest neighbor ($RN{N}_k$) set. The nearest neighbors (NN) represents the samples with high similarity. For N samples in dataset $\mathbf{X}$, the k-NN of point $x_p\in \mathbf{X}$ is defined by $N{N}_k\left(x_p,\mathbf{X}\right)$, which ranks the similarities of $x_p$ to the other points in $\mathbf{X}$ from high to low and selects the top k nearest samples to represent the nearest neighbors of point $x_p$. The Euclidean distance is the common approach for measuring the similarity of k-NN. The definition of the $RN{N}_k$ [37] is based on the k-NN. Given a dataset $\mathbf{X}$ and the point $x_p\in \mathbf{X}$, the definition of the $RN{N}_k$ set of $x_p$ is presented by:

$$\begin{array}{c}\hfill RN{N}_k\left(x_p,\mathbf{X}\right)=\left\{x_q\mid x_q\in \mathbf{X}\backslash \left\{x_p\right\},x_p\in N{N}_k\left\{x_q,\mathbf{X}\right\}\right\}.\end{array}$$

(8)

The number of the reverse nearest neighbors may not be equal to k. It depends on the distribution of the sample’s data. In some exceptional cases, the number of reverse nearest neighbors is empty. Read [7] for more detailed information.

For the training set $\mathbf{X}=\left[x_1,x_2,x_3,\dots ,x_N\right]$, the associated label of $x_i$ is $y_i\in \{1,\dots ,C\}$. In the projected space, nLDA has the same optimization objective as LDA, that is, to find the optimal projection matrix $\mathsf{\varrho }$ which maximizes the equation:

$${\mathbf{J}}_{nLDA}\left(\mathsf{\varrho }\right)=\frac{{\mathsf{\varrho }}^T{\mathbf{S}}_b^{nLDA}\mathsf{\varrho }}{{\mathsf{\varrho }}^T{\mathbf{S}}_w^{nLDA}\mathsf{\varrho }}.$$

(9)

where ${\mathbf{S}}_b^{nLDA}$ and ${\mathbf{S}}_w^{nLDA}$ represent the between-neighborhood scatter matrix and within-neighborhood scatter matrix, respectively. In [7], to avoid the consuming calculation, the approximate between-neighborhood scatter matrix is preferred. They are defined by Equations (10) and (11).

$$\begin{array}{cc}\hfill {\mathbf{S}}_b^{nLDA}=& \sum _{\begin{array}{c}i=1\\ \mid RN{N}_k\left(x_i,{\mathbf{X}}_{y_i}\right)\mid \ge t\end{array}}^N\sum _{\begin{array}{c}{x}_j\in NN\left(x_i,\mathbf{X}-{\mathbf{X}}_{y_i}\right)\\ |RN{N}_k\left(x_j,{\mathbf{X}}_{y_j}\right)\mid \ge t\\ y_i\ne y_j\end{array}}\left(u_i-u_j\right){\left(u_i-u_j\right)}^T.\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathbf{S}}_w^{nLDA}=& \sum _{\begin{array}{c}i=1\\ \mid RN{N}_k\left(x_i,{\mathbf{X}}_{y_i}\right)\mid \ge t\end{array}}^N\sum _{x_j\in RN{N}_k\left(x_i,{\mathbf{X}}_{y_i}\right)}\left(x_j-u_i\right){\left(x_j-u_i\right)}^T.\hfill \end{array}$$

(11)

where $\left|RN{N}_k\left(x_j,{\mathbf{X}}_{y_j}\right)\right|$ is the number of local samples obtained from the RNN and ${\mathbf{X}}_{y_j}$ is the set of image samples with the same label as sample $x_j$. In $\left|RN{N}_k\left(x_j,{\mathbf{X}}_{y_j}\right)\right|\ge t$, t is the threshold of the number of RNN samples, i.e., when the number of RNN in $RN{N}_k(x_i,{\mathbf{X}}_{y_i})$ reaches t, the local neighborhood is constructed and the samples in the local neighborhood are used to calculate the mean vector $u_i$ which is formulated by:

$$u_i=\frac{1}{\left|RN{N}_k\left(x_{i,}{\mathbf{X}}_{y_i}\right)\right|}\sum _{x_j\in RN{N}_k\left(x_i,{\mathbf{X}}_{y_i}\right)}{x}_j.$$

(12)

$x_j\in NN\left(x_i,\mathbf{X}-{\mathbf{X}}_{y_i}\right)$ indicates that $x_j$ is the nearest neighbor from other classes of $x_i$ and $u_j$ is the mean vector of the RNN of $x_j$ in set ${\mathbf{X}}_{y_j}$. $u_j$ can be formulated by:

$$u_j=\frac{1}{\left|RN{N}_k(x_j,{\mathbf{X}}_{y_j})\right|}\sum _{{\mathbf{x}}_i\in RN{N}_k(x_j,{\mathbf{X}}_{y_j})}{x}_i.$$

(13)

Then, maximizing Equation (9) can be reduced to the following generalized eigen-decomposition problem:

$${\mathbf{S}}_b^{nLDA}\mathsf{\varrho }=\lambda {\mathbf{S}}_w^{nLDA}\mathsf{\varrho }.$$

(14)

By retaining d orthogonal eigenvectors according to the largest eigenvalues, the projection matrix $\mathsf{\varrho }$ can be used to extract discriminative features. Finally, the Euclidean distance between a test sample $x^{*}$ and $u_i$ is represented as $dist\left({\mathsf{\varrho }}^T{u}_i,{\mathsf{\varrho }}^T{x}^{*}\right)$, the label of $x^{*}$ is determined by the nearest distance between $x^{*}$ and the mean of each RNN set in the gallery data, which can be expressed as:

$$\begin{array}{c}\hfill \underset{\left|RN{N}_k\left(x_i,{\mathbf{X}}_{y_i}\right)\right|\ge t}{min}dist\left({\mathsf{\varrho }}^T{u}_i,{\mathsf{\varrho }}^T{x}^{*}\right).\end{array}.$$

(15)

4. Kernel Reverse Neighborhood Discriminant Analysis

In this section, we detail the proposed kernel reverse neighborhood discriminant analysis (KRNDA). We use the kernel trick to deal with the non-linearity underlying the input data of nLDA. First, the kernel method based on nLDA requires the definition of a mapping for each observed sample x being mapped from the original feature space to a higher-dimensional feature space $\mathit{F}$:

$$\phi :{\mathit{R}}^D\to \mathit{F},x\to \phi \left(x\right).$$

(16)

Subsequently, the mapped training dataset of $\mathbf{X}$ can be represented as $\phi \left(\mathbf{X}\right)=$$\left[\phi \left(x_1\right),\phi \left(x_2\right),\dots ,\phi \left(x_N\right)\right]$. Usually, the mapping function is not explicitly specified [38], especially when the mapped feature space is infinite-dimensional. Hence, many kernel functions are defined for implicitly calculating the inner product of two image vectors in the mapped space $\mathit{F}$. For two images $x_i$ and $x_j$, the inner product of them in the mapped space can be expressed as:

$$k\left(x_i,x_j\right)=〈\phi \left(x_i\right),\phi \left(x_j\right)〉=\phi {\left(x_i\right)}^T\phi \left(x_j\right).$$

(17)

where $〈·〉$ is the inner product. How to select a suitable kernel function to implicitly calculate the inner product of two vectors in higher-dimensional feature space is a critical factor for kernel methods [39]. The typical and effective Gaussian kernel function is used in this paper:

$$k_g\left(x_i,x_j\right)=exp\left(-\frac{{∥x_i-x_j∥}_2^2}{2{\sigma }^2}\right).$$

(18)

One of the advantages of the Gaussian kernel is that it has only one free parameter $\sigma$, which is easily tuned.

The proposed KRNDA aims to find an optimal projection matrix $\mathsf{\omega }=\left[{\omega }_1,{\omega }_2,\dots ,{\omega }_h\right]$, where h is the reserved dimension of the final extracted feature, such that the mapped data in space $\mathit{F}$ can be projected onto a more discriminative and lower-dimensional feature space by ${\mathsf{\omega }}^T\phi \left(\mathbf{X}\right)$. The optimal $\omega$ can be obtained by maximizing the following equation:

$${\mathbf{J}}_{KRNDA}\left(\mathsf{\omega }\right)=\frac{{\mathsf{\omega }}^T{\mathbf{S}}_b^{KRNDA}\mathsf{\omega }}{{\mathsf{\omega }}^T{\mathbf{S}}_w^{KRNDA}\mathsf{\omega }}.$$

(19)

where ${\mathbf{S}}_b^{KRNDA}$ is the between-neighborhood scatter matrix and ${\mathbf{S}}_w^{KRNDA}$ is the within-neighborhood scatter matrix in space $\mathit{F}$.

According to [11], the projection matrix $\mathsf{\omega }$ can be linearly represented by all the training data in the mapped space $\mathit{F}$. Therefore we can obtain:

$${\omega }_i=\sum _{j=1}^N{\alpha }_{ij}\phi \left(x_j\right).$$

(20)

where ${\alpha }_{ij}$ denotes the coefficient of the linear combination. For a sample $x_m$ in higher-dimensional space $\phi \left(x_m\right)$, the projected vector can be expressed by:

$$\begin{array}{cc}\hfill g_m=& {\mathsf{\omega }}^T\phi \left(x_m\right)\hfill \\ \hfill =& {\left[{\omega }_1,{\omega }_2,\dots ,{\omega }_h\right]}^T\phi \left(x_m\right)\hfill \\ \hfill =& {\left[{\omega }_1^T\phi \left(x_m\right),{\omega }_2^T\phi \left(x_m\right),\dots ,{\omega }_h^T\phi \left(x_m\right)\right]}^T.\hfill \end{array}$$

(21)

Obviously, the final extracted feature $g_m$ does not seem to be explicitly expressed, since each ${\omega }_i$ and the $\phi \left(x_m\right)$ contain the implicit mapping $\phi \left(.\right)$. However, the kernel trick can circumvent this problem. According to Equations (17) and (20), each ${\omega }_i^T\phi \left(x_m\right)$ in Equation (21) can be calculated as:

$$\begin{array}{cc}\hfill {{\omega }_i}^T\phi \left(x_m\right)=& \sum _{j=1}^N\phi {\left(x_j\right)}^T{\alpha }_{ij}^T\phi \left(x_m\right)=\sum _{j=1}^N{\alpha }_{ij}\phi {\left(x_j\right)}^T{\phi }_m\hfill \\ \hfill =& {\alpha }_{i1}\phi {\left(x_1\right)}^T\phi \left(x_m\right)+{\alpha }_{i2}\phi {\left(x_2\right)}^T\phi \left(x_m\right)+\cdots +{\alpha }_{iN}\phi {\left(x_N\right)}^T\phi \left(x_m\right)\hfill \\ \hfill =& \left[{\alpha }_{i1},{\alpha }_{i2},\cdots ,{\alpha }_{iN}\right]\left[\begin{array}{c}\phi {\left(x_1\right)}^T\phi \left(x_m\right)\\ \phi {\left(x_2\right)}^T\phi \left(x_m\right)\\ ⋮\\ \phi {\left(x_N\right)}^T\phi \left(x_m\right)\end{array}\right]\hfill \\ \hfill =& {\alpha }_i^T{K}_m.\hfill \end{array}$$

(22)

where ${\alpha }_i={\left[{\alpha }_{i1},{\alpha }_{i2},\cdots ,{\alpha }_{iN}\right]}^T$. According to Equation (17), we have $k\left(x_i,x_m\right)=\phi {\left(x_i\right)}^T\phi \left(x_m\right)$; $K_m$ denotes the inner product between all training samples and the m-th training sample, in the mapped space $\mathit{F}$; then the extracted feature $g_m$ can be explicitly expressed as:

$$\begin{array}{cc}\hfill g_m=& {\mathsf{\omega }}^T\phi \left(x_m\right)\hfill \\ \hfill =& {\left[{\alpha }_1^T{K}_m,{\alpha }_2^T{K}_m,\dots ,{\alpha }_h^T{K}_m\right]}^T\hfill \\ \hfill =& {\mathsf{\alpha }}^T{K}_m.\hfill \end{array}$$

(23)

where $\mathsf{\alpha }={\left[{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_h\right]}^T$. We define the between-neighborhood scatter matrix of our KRNDA in Equation (19) and formulate the ${\mathsf{\omega }}^T{\mathbf{S}}_b^{KRNDA}\mathsf{\omega }$ as:

$$\begin{array}{cc}\hfill \mathsf{\omega }& {}^T{\mathbf{S}}_b^{KRNDA}\mathsf{\omega }\hfill \\ \hfill =& \sum _{\begin{array}{c}i=1\\ \left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|\ge t\end{array}}^N\sum _{\begin{array}{c}\phi \left(x_j\right)\in N{N}_k\left(\phi \left(x_i\right),\phi \left(\mathbf{X}\right)-\phi \left({\mathbf{X}}_{y_i}\right)\right)\\ \left|RN{N}_k\left(\phi \left(x_j\right),\phi \left({\mathbf{X}}_{y_j}\right)\right)\right|\ge t\\ y_i\ne y_j\end{array}}\left[{\mathsf{\omega }}^T(\phi \left(u_i\right)-\phi \left(u_j\right)\right]{\left[{\mathsf{\omega }}^T(\phi \left(u_i\right)-\phi \left(u_j\right)\right]}^T\hfill \\ \hfill =& {\mathsf{\alpha }}^T\mathbf{B}\mathsf{\alpha }.\hfill \end{array}$$

(24)

where

$$\begin{array}{cc}\hfill \mathbf{B}& =\sum _{\begin{array}{c}i=1\\ \left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|\ge t\end{array}}^N\sum _{\begin{array}{c}\phi \left(x_j\right)\in N{N}_k\left(\phi \left(x_i\right),\phi \left(\mathbf{X}\right)-\phi \left({\mathbf{X}}_{y_i}\right)\right)\\ \left|RN{N}_k\left(\phi \left(x_j\right),\phi \left({\mathbf{X}}_{y_j}\right)\right)\right|\ge t\\ y_i\ne y_j\end{array}}\left({\mathbf{P}}_i-{\mathbf{P}}_j\right){\left({\mathbf{P}}_i-{\mathbf{P}}_j\right)}^T\hfill \end{array}$$

(25)

In Equation (24), $\phi \left(u_i\right)$ denotes the mean vector of the local neighborhood computed by reverse k-nearest neighbors ($RN{N}_k$) of $\phi \left(x_i\right)$ in dataset $\phi \left({\mathbf{X}}_{y_i}\right)$ with the same label $y_i$. Certainly, the number of the $RN{N}_k$ should be equal to or larger than threshold t. Likewise, $\phi \left(u_j\right)$ denotes the mean vector of the local neighborhood computed by $RN{N}_k$ of $\phi \left(x_j\right)$ in dataset $\phi \left({\mathbf{X}}_{y_j}\right)$ with the same label $y_j$. It should be noted that the labels of these two $RN{N}_k$ neighborhoods are different ($y_i\ne y_j$), which exhibits the between-class information. The calculation of ${\mathbf{P}}_i$ and ${\mathbf{P}}_j$ in Equation (25) can be formulated by calculating ${\mathsf{\omega }}^T\phi \left(u_i\right)$ and ${\mathsf{\omega }}^T\phi \left(u_j\right)$. Using the relationship of Equation (23), we give the deduction of ${\mathsf{\omega }}^T\phi \left(u_i\right)$, for instance:

$${\mathsf{\omega }}^T\phi \left(u_i\right)={\mathsf{\omega }}^T\frac{{\displaystyle \sum _{\phi \left(x_m\right)\in RN{N}_k(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right))}}\phi \left(x_m\right)}{\left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|}={\mathsf{\alpha }}^T\frac{{\displaystyle \sum _{\phi \left(x_m\right)\in RN{N}_k(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right))}}{K}_m}{\left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|}$$

(26)

Hence, ${\mathbf{P}}_i=\frac{{\displaystyle \sum _{\phi \left(x_m\right)\in RN{N}_k(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right))}}{K}_m}{\left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|}$. Similarly, ${\mathbf{P}}_j=\frac{{\displaystyle \sum _{\phi \left(x_m\right)\in RN{N}_k(\phi \left(x_j\right),\phi \left({\mathbf{X}}_{y_j}\right))}}{K}_m}{\left|RN{N}_k\left(\phi \left(x_j\right),\phi \left({\mathbf{X}}_{y_j}\right)\right)\right|}$.

We employ the definition of the approximate between-neighborhood scatter in [7] to construct ${\mathbf{S}}_b^{KRNDA}$; hence, for each $\phi \left(u_i\right)$, the number of the between-class reverse nearest neighborhoods is restricted to k by using $N{N}_k\left(\phi \left(x_i\right),\phi \left(\mathbf{X}\right)-\phi \left({\mathbf{X}}_{y_i}\right)\right)$. This approach reduces the amount of computation when the dataset is very large [7]. The calculation of $RN{N}_k(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right))$ follows the definition of the $RN{N}_k$ set in Equation (8), where the distance of nearest neighbors is calculated in the mapped higher-dimensional feature space with the kernel trick.

Likewise, the within-neighborhood scatter can also be explicitly expressed by the kernel trick; the equation ${\mathsf{\omega }}^T{\mathbf{S}}_w^{KRNDA}\mathsf{\omega }$ in Equation (19) can be formulated as:

$$\begin{array}{cc}\hfill {\mathsf{\omega }}^T{\mathbf{S}}_{\mathbf{w}}& {}^{KRNDA}\mathsf{\omega }\hfill \\ \hfill =& \sum _{\begin{array}{c}i=1\\ \left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|\ge t\end{array}}^N\sum _{\phi \left(x_j\right)\in RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)}\left[{\mathsf{\omega }}^T\left(\phi \left(x_j\right)-\phi \left(u_i\right)\right)\right]{\left[{\mathsf{\omega }}^T\left(\phi \left(x_j\right)-\phi \left(u_i\right)\right)\right]}^T\hfill \\ \hfill =& {\mathsf{\alpha }}^T\mathit{W}\mathsf{\alpha }.\hfill \end{array}$$

(27)

where:

$$\begin{array}{ccc}\hfill \mathit{W}=& \sum _{\begin{array}{c}i=1\\ \left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|\ge t\end{array}}^N\sum _{\phi \left(x_j\right)\in RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)}\hfill & \hfill \left(K_j-{\mathbf{Q}}_i\right){\left(K_j-{\mathbf{Q}}_i\right)}^T\end{array}$$

(28)

where ${\mathbf{Q}}_i=\frac{{\displaystyle \sum _{\phi \left(x_m\right)\in RNN(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)}}{K}_m}{\left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right.|}$ is the mean of higher-dimensional features in $RN{N}_k$ of $\phi \left(x_i\right)$ with the same label $y_i$. According to Equations (24), (25), (27) and (28), the object function of Equation (19) can be rewritten as:

$${\mathbf{J}}_{KRNDA}\left(\mathsf{\omega }\right)=\frac{{\mathsf{\alpha }}^T\mathbf{B}\mathsf{\alpha }}{{\mathsf{\alpha }}^T\mathbf{W}\mathsf{\alpha }}.$$

(29)

Hence, the optimization of maximizing Equation (29) can be reduced to the generalized eigen-decomposition problem:

$$\mathbf{B}\mathsf{\alpha }=\mathsf{\lambda }\mathbf{W}\mathsf{\alpha }.$$

(30)

The optimal projection matrix can be obtained by applying the eigen-decomposition to ${\left(\mathbf{W}\right)}^{-1}\mathbf{B}$. We reserve h eigenvectors according to the largest eigenvalues to form the final coefficient matrix $\mathsf{\alpha }=\left[{\alpha }_1,{\alpha }_2,\dots ,{\alpha }_h\right]$.

At the testing stage, for a test sample $x^{*}$, we first map it to the higher-dimensional space $\mathit{F}$ by $\phi \left(x^{*}\right)$, then the final extracted feature can be expressed as ${\mathsf{\omega }}^T\phi \left(x^{*}\right)$; according to the kernel trick, it can be rewritten as ${\mathsf{\alpha }}^T{K}^{*}$. $K^{*}$ denotes the kernel vector of inner production between all gallery (or training) samples and test sample $x^{*}$ in the mapped space $\mathit{F}$. We compare the Euclidean distance between ${\mathsf{\alpha }}^T{K}^{*}$ and the projected mean vector ${\mathsf{\omega }}^T\phi \left(u_i\right)$, where ${\mathsf{\omega }}^T\phi \left(u_i\right)$ can be explicitly rewritten by ${\mathsf{\alpha }}^T{K}_{u_i}$. $\phi \left(u_i\right)$ is the mean vector of the i-th gallery RNN with the same label, and $K_{u_i}$ denotes the kernel vector of inner production between all gallery samples and the mean vector $u_i$ in the mapped space $\mathit{F}$. Therefore the label of test sample $\phi \left(x^{*}\right)$ can be determined by the nearest Euclidean distance of:

$$\begin{array}{c}\hfill \underset{\left|RN{N}_k\left(\phi \left(x_i\right),\phi \left({\mathbf{X}}_{y_i}\right)\right)\right|\ge t}{min}dist\left({\mathsf{\alpha }}^T{K}_{u_i},{\mathsf{\alpha }}^T{K}^{*}\right).\end{array}$$

(31)

5. Experiments

Experiments are conducted on two handwritten digit datasets and the University of California at Irvin (UCI) benchmark database. In addition, the COIL-20 (Columbia Object Image Library) [40] object recognition dataset and the ORL face recognition dataset [41] are also adopted for evaluation.

To verify the effectiveness of our method, we compare the performances of our KRNDA with the original nLDA. Several LDA-based discriminators are added for comparison, including LDA, LFDA, ccLDA and $L_{2,1}-\mathrm{RLDA}$. The LFDA is a traditional local discriminator which has been successfully applied to pedestrian re-identification [42]. We implemented the code of ccLDA [18], which uses clustering to solve the SSS problem. $L_{2,1}-\mathrm{RLDA}$ is a discriminator with excellent discriminant effect and the performance of $L_{2,1}-$ norm is better than that of $l_2$-norm [43]. The codes of LFDA, $L_{2,1}-\mathrm{RLDA}$ and nLDA are provided by the authors.

The parameters in the above discriminators are set as follows: for LDA, the final extracted features are preserved according to 95% energy of the eigenvalues. For LFDA, we set the parameter K = 2 to achieve the optimal effect in our experiments (K is a parameter used in the local scaling method). To avoid the singularity caused by the SSS problem, regularization is applied to the within-class scatter matrix: ${\mathbf{S}}_w={\mathbf{S}}_w+\tau \mathbf{I}$, where $\mathbf{I}$ is the identity matrix and the $\tau$ is a small value, i.e, ${10}^{-3}\times trace\left({\mathbf{S}}_w\right)$. For ccLDA, we follow the original experimental settings: K < C ( here, K is the cluster number and C is the class number) and 98% energy of the eigenvalues is reserved for experiments. In $L_{2,1}-\mathrm{RLDA}$, we use the original parameter settings provided by the authors.

There are four main parameters related to the proposed KRNDA. The first one is the parameter t, which controls the isolated points in Equations (24) and (27); we set $t=1$ according to the article [7], which has proved that it does not affect the performance much. We also applied this setting to our KRNDA in all experiments. The second parameter is the value of k in $RN{N}_k$; [7] has proved that the value of k has better performance in the range $\{10,20,30,40\}$; hence, we adopted similar setting with nLDA in our KRNDA. The third parameter is the $\sigma$ in Gaussian kernel, see Equation (18); we evaluate this parameter in Section 5.4 and apply the best value for our experiments. The last parameter is the dimension of the final extracted feature, similar to nLDA, and we preserve 95% energy of the eigenvalues to construct the final features. All of the experiments were implemented on an Intel(R) Core(TM) i7-11700K (3.60GHz) PC, using Matlab R2020b.

5.1. Experiments on Handwritten Digit Recognition

Two handwritten digit datasets, Mnist and USPS, are used in our evaluation. The purpose of KRNDA is to solve the nonlinear problem; however, it is the extension of the nLDA which performs well on the multimodal class problem. Therefore, we construct the classification problem of handwritten digit recognition as a binary classification task to verify the performance of KRNDA in solving the multimodal classes problem. For binary digit recognition, it is to determine whether the numbers are odd or even. The odd class is 1, 3, 5, 7 and 9, and the even class is 2, 4, 6, 8 and 10. Some digital images of the two classes are shown in Figure 1. As the five numbers in each of the odd class and even class are quite different, it can be considered that there are five subclasses (clusters) in each class.

The Mnist dataset has a total of 60,000 samples in the training set; the details of the training and testing samples are shown in

Table 1. Training and test sample numbers of the Mnist dataset.

Class	Number of Training Set	Number of Test Set
0	5923	980
1	6742	1135
2	5958	1032
3	6131	1010
4	5842	982
5	5421	892
6	5918	958
7	6265	1028
8	5851	974
9	5949	1009

. Adopting the whole set directly for training is time-consuming and may lead to being out of memory. Hence, we constructed subsets for our experiments. We randomly selected different numbers of samples for experiments and created four subsets: Mnist600, Mnist1000, Mnist6000 and Mnist10000. This means that 600, 1000, 6000 and 10,000 samples are selected from the total training set for training, respectively. For the corresponding testing sets, we randomly selected 100, 166, 1000 and 1666 samples from the original 10,000 testing samples to form four testing subsets for evaluation. The USPS dataset has a total of 9298 handwritten digit images. We conducted our experiments according to the protocol in [7], where the training set contains 7291 images and the testing set contains 2007 images, and the detailed training and testing numbers of each class are shown in

Table 2. Training and test sample numbers of the USPS dataset.

Class	Number of Training Set	Number of Test Set
0	644	177
1	1194	359
2	1005	264
3	731	198
4	658	166
5	652	200
6	556	160
7	664	170
8	645	147
9	542	166

.

Firstly, we compare the proposed method (KRNDA) with the Gaussian kernel function, sigmoid kernel function (SRNDA) and polynomial kernel function (PRNDA) on the USPS and Mnist datasets. We chose the parameters corresponding to the highest recognition rates during our experiments. For the Mnist datasets, we performed 10-fold cross-validations to obtain the average recognition rates and standard deviations.

From

Table 3. The accuracy (%) comparison of different kernel method recognition problems.

Data	USPS	Mnist600	Mnist1000	Mnist6000	Mnist10000
SRNDA	81.96	57.50 ± 4.43	55.24 ± 4.38	58.71 ± 3.60	58.62 ± 2.63
PRNDA	96.81	91.30 ± 3.10	93.01 ± 2.88	95.55 ± 0.73	96.21 ± 0.29
KRNDA	97.65	95.70 ± 2.05	96.20 ± 1.55	98.15 ± 0.34	98.06 ± 0.32

, we can find that the Gaussian kernel function outperforms the polynomial and sigmoid kernel functions. The poor performance of the sigmoid kernel function may be caused by the multimodal classes data. The Gaussian kernel demonstrates a better ability to correct the non-linearity and it has only one adjustable parameter, which is easy to use for evaluations. Consequently, in the following experiments of this paper, we employ the Gaussian kernel function with KRNDA for all evaluations.

Here, we present the recognition rates on the USPS handwritten digit dataset, and the average recognition rates and standard deviations of 10-fold cross-validations on Mnist handwritten digit datasets. As shown in

Table 4. The accuracy (%) comparison of USPS and Mnist digit recognition problems.

Data	USPS	Mnist600	Mnist1000	Mnist6000	Mnist10000
LDA	89.83	73.80 ± 4.53	77.53 ± 2.69	82.11 ± 1.23	82.39 ± 0.79
LFDA	94.47	77.50 ± 2.58	83.01 ± 2.91	83.92 ± 1.01	85,59 ± 1.20
ccLDA	89.93	70.60 ± 7.26	79.22 ± 2.52	82.52 ± 0.81	82.70 ± 1.29
$L_{2,1}-\mathrm{RLDA}$	96.71	90.86 ± 2.03	92.97 ± 1.80	96.63 ± 0.48	97.17 ± 0.29
nLDA	96.66	81.40 ± 4.13	89.10 ± 1.71	95.47 ± 0.94	96.16 ± 0.43
KRNDA	97.65	95.70 ± 2.05	96.20 ± 1.55	98.15 ± 0.34	98.06 ± 0.32

, compared with the original nLDA, our KRNDA consistently achieves better recognition rates, especially on the Mnist600 and Mnist1000 subsets, because the small training subset may cause more non-linearity. These results demonstrate the effectiveness of the proposed KRNDA on handling the nonlinear data problem. We can see that the proposed KRNDA also outperforms other LDA-based methods, such as LDA, LFDA, ccLDA and $L_{2,1}-\mathrm{RLDA}$. This result illustrates that our KRNDA inherits the advantage in solving the multimodal classes problem.

5.2. Experiments on COIL-20 and ORL Datasets

In this subsection, we use the COIL-20 object dataset and the ORL face dataset to evaluate more complicated image classification tasks. The COIL-20 consists of 20 different objects, such as car, cat, cup and eraser, etc. For each object, there are 72 images of different viewpoints [44]. The images are normalized and resized to $20\times 20$ pixels. We randomly selected 50% of the images per class for training and the remaining images for testing. Some sample images are shown in Figure 2.

The ORL face dataset contains 400 images of 40 individuals. Images were captured at different times, and other variations of expression (open or closed eyes, smiling or non-smiling) and facial details (glasses or no glasses) [45] are involved. The face images of the ORL database are normalized and resized to $20\times 20$ pixels. We randomly selected 50% of the images per class for training and the remaining images for testing. Some sample images are shown in Figure 3.

For the COIL-20 object dataset and the ORL face dataset, we also performed 10-fold cross-validations to obtain the average recognition rates and standard deviations. As shown in

Table 5. The accuracy (%) comparison of ORL and COIL-20 datasets recognition problems.

Data	ORL	COIL-20
LDA	96.87 ± 0.97	95.52 ± 0.85
LFDA	98.58 ± 0.78	96.38 ± 1.00
ccLDA	94.30 ± 1.79	93.82 ± 0.60
$L_{2,1}-\mathrm{RLDA}$	97.23 ± 1.63	99.38 ± 0.21
nLDA	96.78 ± 1.46	96.39 ± 0.67
KRNDA	98.62 ± 0.73	99.90 ± 0.13

, the proposed KRNDA outperforms the original nLDA, and the recognition rates reach the highest, 98.62% and 99.90% on ORL and COIL-20, respectively. We can see that, in Figure 2 and Figure 3, the COIL-20 dataset with multi-view samples causes much more distortion than the ORL dataset. Hence the accuracy increment of KRNDA on the COIL-20 dataset is larger than the ORL dataset compared to nLDA. Our KRNDA also performs well on face recognition which contains non-linearity caused by lighting and pose variation. These results have shown the effectiveness of the proposed kernel efficient solver. Due to the benefits of RNN and the non-linearity solver, the proposed KRNDA can handle different applications of pattern recognition.

5.3. Experiments on UCI Benchmark Datasets

We selected 40 benchmark datasets from the University of California at Irvine (UCI) machine learning repository [46] for our experiments. The details of the class number, sample number and feature dimension of each dataset are listed in

Table 6. The details of UCI benchmark datasets.

Dataset	Classes	Samples	Feature
Aggregation	7	45,170,102,273,34,130,34	2
Bupa	2	145,200	6
Indian	2	167,416	10
Robotnavigation	4	826,2097,2205,328	24
Sonar	2	97,111	60
balance-scale	3	49,288,288	4
balloons	2	9,7	4
bank	2	4000,521	16
breast-cancer	2	201,85	9
breast-tissue	6	21,15,18,16,14,22	9
acute	2	61,59	6
splice	3	767,768,1655	60
vertebral-column-2clases	2	210,100	6
Indian	2	167,416	10
acute_nephritis	2	70,50	6
car	4	65,1210,384,69	6
cardiotocography-3clases	3	1655,295,176	21
chess-krvkp	2	1527,1669	36
climate-simulation	2	494,46	18
blood	2	570,178	5
arrhythmia	2	3664,3736	20
waveform-noise	3	1692,1653,1655	40
mammographic	2	516,445	5
cylinder-bands	2	200,312	35
fertility	2	88,12	9
iris	3	50,50,50	4
knowledge	3	42,55,75	5
ionosphere	2	126, 226	34
ozone	2	2463,73	72
monks-3	2	266,288	6
nursery	4	4320,328,4266,4044	8
parkinsons	2	48,147	23
promoters	2	53,53	57
semeion	10	161,162,159,159,161,159,161,158,155,158	256
statlog-australian-credit	2	222,468	14
tic-tac-toe	2	332,626	9
statlog-german-credit	2	700,300	24
pittsburg-bridges-TYPE	6	16,11,44,13,11,10	7
mushroom	2	4208,3916	22

. The number of classes is from 2 to 10. The number of features is from 2 to 256, which is less than the visual classification tasks. For each dataset, we randomly selected approximately 50% of the samples for training and the remaining samples for testing. We also employed the LDA-based discriminators, which were introduced previously, for comparison.

The classification results of the UCI datasets are reported in

Table 7. The classification accuracy (%) of different discriminators on UCI benchmark datasets.

Dataset	LDA	LFDA	ccLDA	${\mathit{L}}_{2,1}-\mathrm{RLDA}$	nLDA	KRNDA
Aggregation	99.49	99.49	99.49	99.49	99.75	99.75
Bupa	59.88	55.23	62.21	57.56	72.09	55.24
Indian	66.32	63.23	57.39	64.60	67.01	70.10
Robotnavigation	71.74	86.47	84.42	85.48	85.37	87.02
Sonar	76.92	79.81	83.65	77.88	76.92	80.77
balance-scale	87.82	89.74	79.81	78.21	85.58	89.10
balloons	37.50	50.00	50.00	50.00	62.50	75.00
bank	83.98	86.64	86.59	85.49	86.64	88.32
breast-cancer	58.74	62.94	64.34	64.34	63.64	65.74
breast-tissue	49.06	66.04	60.38	56.60	50.94	15.09
acute	100.00	100.00	100.00	100.00	100.00	100.00
splice	74.11	71.16	63.70	62.13	75.49	83.70
vertebral-column-2clases	75.48	78.06	78.06	81.29	80.00	81.94
Indian	66.32	63.23	57.39	64.60	67.01	70.10
acute_nephritis	100.00	100.00	100.00	100.00	100.00	100.00
car	79.98	94.56	93.63	93.52	94.68	96.41
balance-scale	87.82	89.74	79.81	78.21	85.58	89.10
cardiotocography-3clases	87.49	87.39	88.81	88.33	88.43	82.88
chess-krvkp	91.24	95.62	89.99	87.11	94.87	95.49
climate-simulation	91.85	90.00	86.67	87.04	91.48	90.37
blood	71.66	72.99	71.93	73.53	75.40	66.84
arrhythmia	69.78	73.46	72.27	72.11	87.86	83.00
waveform-noise	80.56	77.92	61.68	75.60	79.92	84.52
mammographic	71.25	77.08	75.00	77.71	74.17	74.79
cylinder-bands	64.84	66.80	68.75	68.75	68.36	58.59
fertility	84.00	74.00	80.00	82.00	80.00	86.00
iris	97.33	94.67	94.67	94.67	85.14	96.00
knowledge	93.02	89.53	88.37	74.42	96.00	82.56
ionosphere	85.14	83.43	83.43	82.29	61.86	89.71
ozone	95.35	95.43	95.98	94.95	96.37	97.79
monks-3	83.75	97.83	82.31	88.45	97.83	98.56
nursery	86.05	91.56	88.85	82.89	93.78	95.38
parkinsons	83.51	76.29	84.54	79.38	82.47	85.54
promoters	62.26	62.26	52.83	77.36	62.26	75.47
semeion	83.92	85.68	65.58	89.57	88.44	93.72
statlog-australian-credit	53.04	61.74	55.07	58.84	60.29	64.35
tic-tac-toe	97.91	98.54	98.54	100.00	98.54	98.96
statlog-german-credit	64.20	69.40	68.60	66.80	68.80	72.20
pittsburg-bridges-TYPE	51.92	46.15	48.08	42.31	38.46	46.15
mushroom	89.78	100.00	100.00	100.00	100.00	100.00
AVERAGE	77.88	80.10	77.57	78.59	80.60	81.63

. In each row, the best classification accuracies are highlighted. As can be seen, the proposed KRNDA outperforms other discriminators in most cases, and the average classification accuracy of all datasets achieves 81.63%, which is better than all other discriminators. The degradation of KRNDA, such as in the datasets of Bupa, Sonar and breast-tissue, etc., may be caused by the lower non-linearity (caused by lower-dimensional features of UCI datasets) or the non-optimal value setting of parameter $\sigma$.

5.4. Parameter Evaluation

The parameter $\sigma$ of the Gaussian kernel plays an important role in exhibiting the performance of the proposed KRNDA. It can be seen as the controller of handling the non-linearity. Therefore, we conducted experiments to tune the optimal value of $\sigma$. Experiments were conducted on a subset of the Mnist database; we randomly selected 600 samples for training and 100 samples for testing, and 10 cross-validations were conducted to obtain the mean accuracy.

Experimental results are shown in Figure 4. The Y-axis is the recognition accuracy and the X-axis is the value of $\sigma$. As can be seen, the best value $\sigma =6$ reaches the highest accuracy with 95.1%; hence, in most of our experiments, the value of $\sigma$ is set as $\sigma =6$. Certainly, this value is non-optimal for different applications and different datasets; however, it achieves considerable performances in most cases of our experiments.

6. Conclusions

In this paper, we extended nLDA to its kernel version for handling the non-linearity of the input data, which is called kernel reverse neighborhood discriminant analysis (KRNDA). We mapped the original samples to a higher-dimensional feature space for discriminative analysis, however, since the mapping function generalized by the specific kernel functions can not be explicitly expressed. We gave a comprehensive deduction of the kernel extension with the kernel trick. We evaluated the performances of our KRNDA on a multimodal classes problem of handwritten digit recognition with three different kernel functions; the Gaussian kernel function has better performances than the polynomial and sigmoid kernel functions; this result demonstrates the better capacity to correct the non-linearity by the Gaussian kernel. As a result, we evaluated the KRNDA with the Gaussian kernel function on visual recognition tasks of USPS and Mnist handwritten digits datasets, a COIL-20 object categorization dataset and an ORL face recognition dataset. The performances of the proposed KRNDA outperform the original nLDA and other previous LDA-based discriminators, such as LDA, LFDA, ccLDA and $L_{2,1}-\mathrm{RLDA}$. In addition, we evaluated our KRNDA on non-visual and lower-dimensional datasets of the UCI benchmark database. Due to lower non-linearity or the non-optimal value setting of parameter $\sigma$, the proposed KRNDA is inferior to other discriminators on some datasets of the UCI database; however, the average performance of KRNDA is still considerable.