A Comprehensive Review on Discriminant Analysis for Addressing Challenges of Class-Level Limitations, Small Sample Size, and Robustness

Abstract

The classical linear discriminant analysis (LDA) algorithm has three primary drawbacks, i.e., small sample size problem, sensitivity to noise and outliers, and inability to deal with multi-modal-class data. This paper reviews LDA technology and its variants, covering the taxonomy and characteristics of these technologies and comparing their innovations and developments in addressing these three shortcomings. Additionally, we describe the application areas and emphasize the kernel extensions of these technologies to solve nonlinear problems. Most importantly, this paper presents perspectives on future research directions and potential research areas in this field.

1. Introduction

Linear discriminant analysis (LDA) is a classical linear learning method, being first proposed in 1936 by R.A. Fisher, also known as Fisher discriminant analysis (FDA) [1] for binary classification. In a strict sense, LDA grounds the assumption that the covariance matrix of all classes of sample data is the same and full-rank, which slightly differs from FDA [2]. LDA is a supervised data analysis method using an orthogonal transformation that makes the process and analysis of data more efficient and convenient.

The definition for the term “perpendicular” by Euclid, a synonym for “orthogonal”, is depicted in [3]. Mathematically and specifically, a linear transformation is a mapping from one vector space to another that preserves the operations of vector additions and scalar multiplications. Especially in linear algebra, an orthogonal transformation $T:V\to V$ is a linear transformation conducted in an inner product space V, which preserves the inner product of two vectors $u,v\in V$: $<u,v>=<Tu,Tv>$; accordingly, it preserves the norms of vectors and angles between vectors. Orthogonality is a significant property, meaning a certain kind of non-dependence of things, that makes components separated for clearer and easier observations, analysis and manipulation. From the viewpoint of mathematics, in a vector space, any signal is a vector that can be represented by a set of orthogonal bases, being decomposed into uncorrelated components along different axes as much as possible. This meets the need for processing and analyzing data more effectively and conveniently. Orthogonal transform rotates signals with orthogonal components, from one set of orthogonal bases to another one but more approximately or accurately, or more proper for a favorable aim, preserving the equivalent representations of identical inner products between vectors before and after the transform [4]. This addresses quite a lot of necessary needs for varied data processing and analysis across a wide range of various areas and fields.

Notably, it is necessary to talk about the difference between statistical analysis and data analysis in the introduction of this paper, both being always mentioned along with each other, which was the first question that bothered us at the beginning of our investigation. Are the terms statistical analysis and data analysis, which often appear together, equivalent? Are they interchangeable? Can the term statistical analysis be an alternative for data analysis in this paper? The answer is no. Data analysis is to collect, clean, and learn the original data set aiming to gain an insight into hidden and potential information under the data. Statistical analysis draws inferences about the larger population using part quantitative samples. Statistical analysis can infer what is beyond the data analysis. Given a brief description of the differences between the two terms from some perspectives, it is indicated that this paper limits the research scope under the data analysis. Using different methods to compare the same data with the same aim has become a big benefit for data analysis, said by Tukey [5], and what is taken to include by data analysis is analyzing procedures, results interpreting techniques, data collection methods for easier, more precise or more accurate analysis, and mathematical statistics tools applied to analyzing data.

In statistical discriminant analysis, the betweenclass scatter matrix $S_b$ as shown in Equation (1), the within-class scatter matrix $S_w$ as shown in Equation (2), and the total (or mixture) scatter matrix $S_t$ as shown in Equation (3) are utilized for the formulations of the class separability criterions. For a L-classes problem of N samples $x\in {\mathbb{R}}^d$ in the d-dimensional original space, the scatter matrices are defined as follows [1].

$$\begin{array}{cc}\hfill S_b& =\frac{1}{N}\sum _{j=1}^L{N}_j(m_j-m){(m_j-m)}^T,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill S_w& =\frac{1}{N}\sum _{j=1}^L{N}_j\sum _{x\in L_j}(x-m_j){(x-m_j)}^T,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill S_t& =S_b+S_w=\frac{1}{N}\sum _x(x-m){(x-m)}^T,\hfill \end{array}$$

(3)

where $N_j$ denotes the size of samples in class $L_j,j=1,\cdots ,L$, $N={\displaystyle \sum _{j=1}^L}{N}_j$, $m_j$ and m denote the class mean of $L_j$ and the total mean, respectively. $S_t$ is the total scatter matrix of all samples regardless of their class assignments [6]. The target of LDA is to obtain the transform vector v that satisfies Equation (4).

$$\begin{array}{c}\hfill v=arg\underset{v}{max}\frac{v^T{S}_{b}v}{v^T{S}_{w}v}.\end{array}$$

(4)

The objective transform vector is the most discriminative projection direction of the maximum distance between classes and the minimum variance within each class, as shown in Figure 1.

In Equation (1), the rank of vector $m_j-m$ is 1, so that the rank of $S_b$ is at most L after summing up all vectors of L classes. Due to the nonlinear independence between all $m_j$ of L classes and m, that is the $m_j$ of $(L-1)$ and m can linearly represent the L-th $m_j$, it is inferred that $max\left(rank\left(S_b\right)\right)=L-1$. So as to that of $max\left(rank\left(S_w^{-1}{S}_b\right)\right)$ because of $rank\left(AB\right)\le min\left\{rank\right(A),rank(B\left)\right\}$. From Equation (4), consequently, there are at most $L-1$ non-zero eigenvalues and valid eigenvectors, respectively. In consequence, the reduced space by LDA is of at most $L-1$ dimensions.

Nonetheless, LDA has three main drawbacks: inapplicability for multi-modal datasets; the singularity of the within-class scatter matrix; and insufficient robustness against outliers and noises [7,8]. We present the technical causes, the existing cases and the resulting bad influence of three drawbacks of conventional LDA in

Table 1. The types, causes, situation of existence, and negative impacts of the three drawbacks of LDA.

Types of Drawbacks	Cause	Existence	Negative Impacts
Inapplicability of multi-modality	The i.i.d. assumption fails.	Classes are multi-modal containing some sub-classes or clusters.	LDA may make the information in classes be distorted and inseparable.
Small Sample Size (SSS)	$S_w$ is (almost) singular.	Training samples are of high dimension but small size.	The singularity of $S_w$ leads to severe instability and over-fitting.
Unrobustness	$L_2$ norm in LDA is unrobust.	Outliers exist in training samples.	Projection vectors may drift from the target directions.

. In the last few decades, the purpose of mitigating the three drawbacks has been motivating the rush towards many extensions to LDA around a wide variety of disciplines and areas.

In this paper, we intend to review the last two to three decades of articles working on suppressing the affection of three drawbacks of LDA, summarize the taxonomy of techniques and their applications, and compare the characteristics and innovation of those methods. For clarity, we build

Table 2. The name abbreviations of the reviewed methods from their references.

Method Names and References	Abbreviations
Linear discriminant analysis [1]	LDA
Fisher discriminant analysis [1]	FDA
Mixture discriminant analysis [9]	MDA
Subclass discriminant analysis [10]	SDA
Mixture subclass discriminant analysis [11]	MSDA
Separability oriented subclass discriminant analysis [12]	SSDA
Resilient SDA [13]	RSDA
One MSDA extension [14]	EM-MSDA
Fractional step MSDA [14]	FSMSDA
Local Fisher discriminant analysis [15,16]	LFDA
Locality-preserving projection [17]	LPP
Locality Sensitive discriminant analysis [18]	LSDA
Manifold partition discriminant analysis [19]	MP-DA
Adaptive and fuzzy locality discriminant analysis [20]	AFLDA
Approximate pairwise accuracy criterion [21,22]	aPAC
Penalized discriminant analysis [23]	PDA
Heteroscedastic linear dimension reduction [24]	HLDR
Local mean based nearest neighbor discriminant analysis [25]	LM-NNDA
Nonparametric discriminant analysis [26]	NDA
Neighborhood linear discriminant analysis [27]	nLDA
Eigenspectrum regularization reverse neighborhood discriminative learning [28]	ERRNDL
An alternative of null-space-based LDA methods [29]	Fast NLDA
Direct LDA [30]	DLDA
Orthogonal LDA [31]	OLDA
Rotational invariant $L_1$-norm based LDA [32]	LDA-$R_1$
$L_1$-norm based LDA [33]	LDA-$L_1$
Two-dimensional LDA with $L_1$-norm [34]	$L_1$-2DLDA
$L_1$-norm based LDA [35]	$L_1$-LDA
Kernel extension of $L_1$-norm based LDA [35]	$L_1$-KDA
Kernel discriminant analysis [36]	KDA
A robust LDA measured by $L_{2,1}$-norm [37]	$L_{2,1}$-LDA
$L_p$-norm based LDA [38]	LDA-$L_p$
A bilateral two-dimensional LDA using the $L_p$-norm [39]	B$L_p$2DLDA
LDA measured by $L_s$-and $L_p$-norm [40]	FLDA-$L_{sp}$

for the name abbreviations of the reviewed methods regarding their references.

Our objective is to provide the readers with comprehensive knowledge about the existence, the primary philosophy, the solutions, and the corresponding applications of the three drawbacks of conventional discriminant analysis. This knowledge can guide the readers on how these methods play their roles in the machine learning area based on various benefits and usage, to identify the real knowledge differences from past to present, which motivates us to make this article serve as the foundation of developing theory and predicting the heading directions in discriminative data analysis area. How best to guide readers from the underlying mathematical theories to the technical comparisons, to the application for the realistic situations, and finally to the future development is the originality of this paper. Motivation has been the guiding principle throughout the writing of this review article.

The rest of this paper is organized as follows. Section 2 summarizes and compares the methods and techniques of LDA variants addressing three drawbacks especially. Section 3 summarizes the main applications of LDA variants from solving each drawback. Section 4 summarizes the review methods that have been extended to the kernel version and discusses the ones that can be extended to the kernel version. Finally, Section 5 is the conclusion of this paper.

2. LDA Extensions: Variations in Principle

2.1. LDA Variants for Multi-Modal Classes

LDA relies on the assumption that all data samples of the same class are independently and identically distributed (i.i.d.). It can be described as a maximum likelihood estimation for Gaussian distributions for each class with common covariance and distinct means for different classes. In case the assumption fails, the original LDA with class-level scatter matrices cannot deal with the cases that the class is multi-modal containing some independent sub-classes or clusters [6,41]. Moreover, in practice, due to complex nonlinear distributions, outliers and any possible real factors, segmenting classes into different sub-classes is in favor of making them more separable, which preserves the information involved in the multi-modal structure.

2.1.1. Using a Mixture of Gaussians

Approximating the fundamental distribution in every class as the mixture of Gaussians is a good way of describing a big size of data distributions, no matter if they correspond with compact sets [42,43].

To perform classification effectively, the proposal mixture discriminant analysis (MDA) [9] fits Gaussian mixtures to each class especially when there are sub-classes. MDA has the feature that the classes are structured as mixtures of Gaussian distributions, instead of only Gaussian distributions as in traditional LDA. To be specific, using Gaussian mixtures for modelling the class densities of the predictors $P\left(X\right|G)$. Suppose N training data with label set $\mathcal{L}$: $(x_i,g_i)\in {\mathbb{R}}^d\times \mathcal{L},i=1,2,\cdots ,N$, and $g_i$ means the label of $x_i$. Dividing each class $L_j$ into $R_j,j=1,2,\cdots ,L$ artificial sub-classes presented as $l_{jr},r=1,2,\cdots ,R_j$. The model supposes that every subclass follows a multivariate normal distribution as well as the mean vector $u_{jr}$ of itself and a shared common covariance matrix $\mathsf{\Sigma }$. Let ${\prod }_j$ be the prior probability of class $L_j$, and make ${\pi }_{jr}$ be the mixing probability of the r-th subclass inside the class $L_j$. The class $L_j$ has its mixture density, as shown in Equation (5),

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill m_j\left(x\right)& =P(X=x|G=j)\hfill \\ \hfill & ={\left|2\pi \mathsf{\Sigma }\right|}^{-1/2}\sum _{r=1}^{R_j}{\pi }_{jr}exp\{-D(x,u_{jr})/2\},\hfill \end{array}\end{array}$$

(5)

and the conditional log-likelihood of the data is shown in Equation (6).

$$\begin{array}{c}\hfill l^{mix}(u_{jr},\mathsf{\Sigma },{\pi }_{jr})=\sum _{i=1}^N{m}_{g_i}\left(x_i\right).\end{array}$$

(6)

To maximize $l^{mix}\left(\theta \right)$, executing expectation-maximization (EM) algorithm [44] iteratively via Bayes theorem, obtaining the posterior class probabilities $P(G=j|X=x)$ and maximizing it for optimal classification.

From the above perspective, MDA applies the EM algorithm firstly for estimating the real underlying distribution inside every class ahead of utilizing LDA. It has to be mentioned that the EM algorithm can match a mixture of Gaussians efficiently only when the sample number is very large. Besides the standard MDA, Bashir et al. [45] consider the case that in Gaussian mixture models, the estimators for those unknown parameters inside the EM algorithm are affected due to outliers, which results in the non-robustness. They substitute those unrobust estimators of the M-step in the EM algorithm for the robust S-estimators, defined as having higher breakdown points, of the unknown parameters, where the compared results show that the average probability of misclassification reduces slightly than the standard mixture discriminant analysis. This proposal is called high breakdown mixture discriminant analysis.

There is a different method using a mixture of Gaussian, addressing the goal of optimizing classification, rather than recovering the real underlying but unknown data distribution, that is, subclass discriminant analysis (SDA) [10]. It defines criteria to ensure the best amount of Gaussians in each class, which is the number of sub-classes, and uses nearest neighbor-based clustering to divide classes into sub-classes. The target is solving the generalized eigenvalue decomposition problem to find the optimal discriminant vectors for the classification, as shown in Equation (7).

$$\begin{array}{c}\hfill S_{b}V=\mathsf{\Sigma }V\mathsf{\Lambda },\end{array}$$

(7)

where $\mathsf{\Sigma }$ represents the covariance matrix of the data, V denotes the eigenvector matrix, $\mathsf{\Lambda }$ is the relevant diagonal eigenvalue matrix. $S_b$ is the between-subclass scatter matrix represented as Equation (8).

$$\begin{array}{c}\hfill S_b=\sum _{j=1}^{L-1}\sum _{r=1}^{R_j}\sum _{k=j+1}^L\sum _{l=1}^{R_k}{p}_{jr}{p}_{kl}(u_{jr}-u_{kl}){(u_{jr}-u_{kl})}^T,\end{array}$$

(8)

where L is classes number, $R_j$ is the size of sub-classes divisions in class $L_j$, $p_{jr}$ is the prior of the r-th subclass of class $L_j$, and $u_{jr}$ is the mean of the r-th subclass in class $L_j$.

Notably, Gkalelis et al. mention two shortcomings of SDA. One is SDA does not guarantee the minimum Bayes error, and another one is the covariance matrix does not work in the minimization of the discriminant analysis stability [11]. To this end, in [11], the authors propose a modified SDA using a partitioning procedure that alleviates the aforementioned two shortcomings, referred to as mixture subclass discriminant analysis (MSDA). SDA and MSDA divide each class into sub-classes and reformulate the within-subclass and between-subclass matrices.

Wan et al. make a study on obtaining the sub-classes more efficiently to realize higher separation of classes, named separability oriented subclass discriminant analysis (SSDA) [12]. In particular, to lessen the overlap between models of sub-classes, the authors utilize clustering to separate each class into sub-classes based on the criterion of separability-oriented and then redefine scatter matrices for discriminant analysis. The experimental results show that, compared to LDA, SDA, and MSDA, SSDA performs better and has higher class separation in most cases. SSDA differs from LDA due to the existence of a subclass. SSDA differs from SDA/MSDA due to different criteria for separating a class into sub-classes and redefining the subclass-based scatter matrices. Specifically, SSDA aims to divide each class into distinct sub-classes and further separate the distinct classes.

It is pointed out by Wu et al. that in Gaussian mixture models, the EM algorithm normally needs a big size of samples to estimate the mixture parameters accurately, so it may be unstable for small dataset problems. Wu et al. propose the resilient SDA (RSDA) [13] with a modified EM algorithm by first projecting the data into the space of much lower dimensionality of highest class separation and clustering the mapped data to the novel space. In comparison with the conventional EM algorithm, RSDA improves the robustness of clustering the mixtures of Gaussians regardless of the sample size and the modified subclass-based covariance matrices are quite smaller to be easier for inversion, and also, lower the computational cost because the most costly step of assigning samples to each subclass is conducted on a much lower dimensional space. Additionally, RSDA uses a stepwise cross-validation procedure to determine the optimal number of subclasses, rather than an exhaustive search, significantly reducing computational cost.

SDA works well with higher dimensionality sub-spaces as the dimensionality of the learned feature space is limited by the between-subclass matrix that is limited under the entire amount of sub-classes. One of the major disadvantages of low speed in the case of large numbers and large dimensionality of a dataset of SDA is presented by Chumachenko et al. [46]. To this end, the authors propose the speed-up SDA method that overcomes both the low speed and the limited dimension of the subspace. Specifically, this method is based on graph embedding and spectral regression approaches, where the exploitation of the between-class Laplacian matrix makes the eigendecomposition process quite faster. The authors formulate a multi-view SDA criterion, allowing the method to be used for the multi-view data.

The conventional MDA and SDA separate subclass within each isolated class before addressing the generalized eigenvalue issue, which ignores the relations between each class and may not keep the locality in the original data space leading to the unguaranteed classification performance. A novel iterative subclass separation method of the EM-like framework is presented to solve such questions [47]. The authors seek the eigenvectors and operate subclass division by k-means clustering class by class in the projected space iteratively under an EM-alike framework. Compared to conventional MDA and SDA, the experimental results demonstrate this method has a better performance and costs a bit more time.

The authors in [14] extend MSDA in three ways. The first one is called the EM-MSDA. It estimates the optimal amount of mixed components for every Gaussian mixture density iteratively, where during every iteration a novel Gaussian model is determined. The second one is the fractional step MSDA (FSMSDA) solving the subclass division problem. Specifically, the dimension of the learned subspace is completely lower than the rank of the subspace-based scatter matrix. It is solved by a proper weighting strategy with an iterative algorithm. The third one is the kernel extension for nonlinear problems.

2.1.2. Using Manifold (Laplacian Graph)

In this part, we will summarize the extensions of LDA methods of using manifold to exploit the local data structure, to a degree, which can be regarded as the joint study of LDA and manifold learning. As is well known, graphs are usually taken as the proxy of a manifold. More specifically, these methods all fall into graph Laplacian-based framework, applying the Laplacian matrix on specific graphs to depict the local data structure, and thus projecting the nearby data of the same labels to the reduced space as closely as possible, whereas those nearby data of distinct labels are projected as far as possible.

There is an essential limitation of FDA that it only works in the case that the dimensionality of embedding space is smaller than class numbers due to the rank deficiency of the between-class scatter matrix [6]. Here is another essential problem for multi-modal dimensionality reduction to protect the local structure of the data from being changed. The proposed local Fisher discriminant analysis (LFDA) [15,16] makes a combination of the FDA and locality-preserving projection (LPP) [17] without losing the local structure. It is one of the most typical LDA extensions and could be regarded as a supervised modification of LPP. This paper’s proposal overcomes the limitation of rank deficiency in $S_b$ by reducing dimensionality into an arbitrary dimensional space.

LPP is a linear dimensionality reduction technique projecting the data along the directions of maximal variances and optimally preserving the neighborhood structure of the data. LPP uses one graph to model the geometrical structure in the data. The high dimensional samples are located on a low dimensional manifold, where LPP is found by searching the best linear approximations for the eigenfunctions of the Laplace–Beltrami operator.

Given N samples, $\left\{x_i\right|x_i\in {\mathbb{R}}^d,i=1,\cdots ,N\}$. Find a transformation matrix T mapping these N points to $\left\{y_i\right|y_i\in {\mathbb{R}}^l,l\ll d,i=1,\cdots ,N\}$, where $y_i=T^T{x}_i,T=(t_0,t_1,\cdots ,t_{l-1})$ denotes the transformed sample from $x_i$ by T. The objective of LPP is to seek the transformation matrix T that meets Equation (9).

$$\begin{array}{c}\hfill min\frac{1}{2}\sum _{ij}{(y_i-y_j)}^2{A}_{ij}.\end{array}$$

(9)

The first step of the LPP algorithm is constructing the adjacency graph with n nodes by putting an edge connecting the nodes i and j if $x_i$ and $x_j$ are close, defined under certain criteria. It chooses the affinity matrix $A_{ij}$ for the edge joining vertices i and j by certain variations. One common variant is by heat kernel: $A_{ij}=e^{-\frac{\left|\right|x_i-x_j\left|\right|}{t}},t\in \mathbb{R}$. Based on the objective Equation (9), when $x_i$ and $x_j$ are far apart, the corresponding $A_{ij}$ will be small. When $x_i$ and $x_j$ are close, $A_{ij}$ will be large, so $y_i$ and $y_j$ will consequently also be close to meet the minimizing requirement of Equation (9).

After a simple introduction of LPP, it goes back to LFDA. Suppose $x_i\in {\mathbb{R}}^d,i=1,2,\cdots ,N$ is the original training data of d-dimension, N is the amount of training data and $N_l$ is the sample amount of class $L_l$. The local within-class scatter matrix ${\tilde{S}}^{\left(w\right)}$ and the local between-class scatter matrix ${\tilde{S}}^{\left(b\right)}$ of LFDA are defined in a pairwise expression as shown in Equations (10)–(13).

$$\begin{array}{c}\hfill {\tilde{S}}^{\left(w\right)}=\frac{1}{2}\sum _{i,j=1}^N{\tilde{W}}_{i,j}^{\left(w\right)}(x_i-x_j){(x_i-x_j)}^T,\end{array}$$

(10)

$$\begin{array}{c}\hfill {\tilde{S}}^{\left(b\right)}=\frac{1}{2}\sum _{i,j=1}^N{\tilde{W}}_{i,j}^{\left(b\right)}(x_i-x_j){(x_i-x_j)}^T,\end{array}$$

(11)

where

$$\begin{array}{cc}\hfill & {\tilde{W}}_{i,j}^{\left(w\right)}=\left\{\begin{array}{c}{A}_{i,j}/N_{l}if{y}_i=y_j=l,\hfill \\ 0if{y}_i\ne y_j,\hfill \end{array}\right.\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & {\tilde{W}}_{i,j}^{\left(b\right)}=\left\{\begin{array}{c}{A}_{i,j}(1/N-1/N_l)if{y}_i=y_j=l,\hfill \\ 1/Nif{y}_i\ne y_j.\hfill \end{array}\right.\hfill \end{array}$$

(13)

The affinity $A_{i,j}$ weights for the data pairs of the same class, from which the far apart samples of the same class make few effects on ${\tilde{S}}^{\left(w\right)}$ and ${\tilde{S}}^{\left(b\right)}$. Moreover, the samples in different classes are irrespective of affinity since they are expected to be separated from each other. The objective function of LFDA is to obtain the transformation matrix $T_{LFDA}$ shown as Equation (14).

$$\begin{array}{c}\hfill T_{LFDA}=arg\underset{T\in {\mathbb{R}}^{d\times r}}{max}tr\left(\frac{T^T{\tilde{S}}^{\left(b\right)}T}{T^T{\tilde{S}}^{\left(w\right)}T}\right),\end{array}$$

(14)

It makes the neighbored data pairs in the same class close; far apart ones are not imposed, while the data pairs in distinct classes are apart.

The authors in [18] propose the locality sensitive discriminant analysis (LSDA) algorithm. It preserves the locality and discriminant properties of the data. Specifically, they model the local geometry of the underlying manifold by constructing a nearest-neighbor graph. Assume N data points $\{x_i\in {\mathbb{R}}^d|i=1,\cdots ,N\}$ sampled from the underlying submanifold $\mathcal{M}$. To model the local geometrical structure of $\mathcal{M}$, the authors construct the nearest neighbor graph G by finding the k nearest neighbors set $N\left(x_i\right)=\{x_i^1,\cdots ,x_i^k\}$ of $x_i$ and putting edges between $x_i$ and its neighbors. The nearest neighbor graph G with its weight matrix W depicts the local geometric structure of $\mathcal{M}$. Next, splitting the graph G into within-class graph $G_w$ and between-class graph $G_b$. The $N\left(x_i\right)$ can be split into $N_b\left(x_i\right)$ and $N_w\left(x_i\right)$ shown as Equations (15) and (16), containing the neighbors with the same and distinct labels with $x_i$.

$$\begin{array}{cc}\hfill & N_w\left(x_i\right)=\left\{x_i^j\right|x_i^{j}hassamelabelto{x}_i,1\le j\le k\}.\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & N_b\left(x_i\right)=\left\{x_i^j\right|x_i^{j}hasdistinctlabelto{x}_i,1\le j\le k\}.\hfill \end{array}$$

(16)

Accordingly, the weight matrix W is split into $W_w$ and $W_b$, as shown in Equations (17) and (18), respectively, corresponding to $G_w$ and $G_b$.

$$\begin{array}{cc}\hfill & W_{w,ij}=\left\{\begin{array}{c}1,if{x}_i\in N_w\left(x_j\right)or{x}_j\in N_w\left(x_i\right)\hfill \\ 0,otherwise.\hfill \end{array}\right.\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & W_{b,ij}=\left\{\begin{array}{c}1,if{x}_i\in N_b\left(x_j\right)or{x}_j\in N_b\left(x_i\right)\hfill \\ 0,otherwise.\hfill \end{array}\right.\hfill \end{array}$$

(18)

It identifies a linear transformation matrix to project the data into a reduced space, ensuring that closely related samples with the same label remain near each other, while closely related samples with different labels are separated by a greater distance. The objective is to optimize the following functions by the eigendecomposition, shown as Equation (19).

$$\begin{array}{c}\hfill max\sum _{ij}\frac{{(y_i-y_j)}^2{W}_{b,ij}}{{(y_i-y_j)}^2{W}_{w,ij}},\end{array}$$

(19)

where $y_i=v^T{x}_i,i=1,\cdots ,N$ is the projected value mapped from $x_i$ into reduced space by the projection vector v. The LSDA has been extended into reproducing kernel Hilbert space (RKHS) by kernel method in this paper.

Besides the pairwise difference being considered, the proposal in [19] establishes a manifold representation that also characterizes piecewise regional consistency of potential manifold, called manifold partition discriminant analysis (MPDA). It splits the manifold into some regional ones in a piecewise manner and represents the partitioned manifold using the first-Taylor expansion based on both pairwise differences as well as piecewise regional consistency for the manifold. Thus, MPDA can obtain the projection matching the local change in the underlying manifold.

There is a more robust proposal that eliminates the interference of noise and redundancy by Wang et al., named adaptive and fuzzy locality discriminant analysis (AFLDA) [20]. The potential submanifold structures are learned through the subclass partition. An adaptively updated fuzzy membership matrix is designed to learn the multi-modal data, promising an optimized subspace to alleviate the impact of noise and redundant information.

2.1.3. Setting Weights for LDA

Incorporating weights into the estimation of matrices is another strategy to flexibly reduce or penalize the effects of unstable distributed data. It allows a slight escape from the Gaussian distribution assumption, which is an advantage over LDA, whose data follow the normal distribution.

In addition to LFDA and NDA depicted previously, two alternatives also using a weight version of the original LDA will be introduced here. The first one is the approximate pairwise accuracy criterion (aPAC) [21,22]. It modifies by redefining the matrix $S_b$ shown in Equation (20),

$$\begin{array}{c}\hfill S_b=\sum _{i=1}^{L-1}\sum _{j=i+1}^L\omega \left({\Delta }_{ij}\right)(m_i-m_j){(m_i-m_j)}^T,\end{array}$$

(20)

where L is the number of classes, $m_i$ is the mean of class $L_i$, and ${\Delta }_{ij}$ is the Mahalanobis distance between classes $L_i$ and $L_j$. $\omega \left({\Delta }_{ij}\right)$ is a weighting function depending on ${\Delta }_{ij}$, which contributes to every class pair being equivalent to the accuracy of the classification.

Another method is penalized discriminant analysis (PDA) [23] by redefining the matrix $S_w$. It introduces a penalizing matrix $\mathsf{\Omega }$ onto $S_w$, rewritten as Equation (21).

$$S_w={\mathsf{\Sigma }}_w+\mathsf{\Omega },$$

(21)

where ${\mathsf{\Sigma }}_w$ is the unpenalized within-class scatter matrix. By weighting the features according to their proportion, the noise eigenvectors can be effectively penalized.

An extension of LDA using a heteroscedastic two-class technique that follows the Chernoff criterion is proposed, called heteroscedastic linear dimension reduction (HLDR) [24]. Specifically, the authors use the Chernoff distance to evaluate the class similarity with means and covariances. Consequently, $S_b$ is modified as shown in Equation (22).

$$\begin{array}{c}\begin{array}{c}{S}_b=\sum _{i=1}^{L-1}\sum _{j=i+1}^L[{\mathsf{\Sigma }}_{ij}^{-1/2}(m_i-m_j){(m_i-m_j)}^T{\mathsf{\Sigma }}_{ij}^{-1/2}\hfill \\ +4\left(log{\mathsf{\Sigma }}_{ij}-\frac{1}{2}log{\mathsf{\Sigma }}_i-\frac{1}{2}log{\mathsf{\Sigma }}_j\right)],\hfill \end{array}\hfill \end{array}$$

(22)

where ${\mathsf{\Sigma }}_i$ is the covariance matrix of data in class $L_i$, ${\mathsf{\Sigma }}_{ij}$ is the average between ${\mathsf{\Sigma }}_i$ and ${\mathsf{\Sigma }}_j$, and equal priors are assumed. This method extends the two-class case into a multiclass version of the Chernoff criterion.

2.1.4. Using k-Nearest Neighbors

The proposal in [25] defines the scatter matrices on k-nearest neighbors of each sample, called local mean-based nearest neighbor discriminant analysis (LM-NNDA). Given N training samples of L classes $\left\{x_{ij}\right|i=1,\cdots ,L;j=1,\cdots ,N_i\}$, where $N_i$ is the number of samples in class i. For each sample $x_{ij}$, search its k-nearest neighbors in every class. Let $m_{ij}^s$ be the local mean vector of k-nearest neighbors of $x_{ij}$ in class s.

The local within-class scatter matrix of LM-NNDA is defined as Equation (23).

$$\begin{array}{c}\hfill S_w^{LMNNDA}=\frac{1}{N}\sum _{i,j}(x_{ij}-m_{ij}^i){(x_{ij}-m_{ij}^i)}^T.\end{array}$$

(23)

The local between-class scatter matrix of LM-NNDA is defined as Equation (24).

$$\begin{array}{c}\hfill S_b^{LMNNDA}=\frac{1}{N(L-1)}\sum _{i,j}\sum _{s\ne i}(x_{ij}-m_{ij}^s){(x_{ij}-m_{ij}^s)}^T.\end{array}$$

(24)

A non-parametric form of discriminant analysis is first presented in [48] to overcome two problems. One is in parametric discriminant analysis, only at most $L-1$ features (L: # of classes) are extracted due to the rank-deficient between-neighborhood scatter matrix while the non-parametric matrices are full-rank. Another one is that non-Gaussian datasets are allowed in non-parametric matrices. It redefines $S_b$ using $kNN$ techniques, focusing on two-classes cases. The proposal in [26] gives an extension of $S_b$ shown in Equations (25) and (26) to multiclass classification under a face recognition scenario, referred to as multiclass non-parametric discriminant analysis (NDA).

$$\begin{array}{c}\hfill S_b^{NDA}=\sum _{i=1}^L\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^L\sum _{l=1}^{N_i}w(i,j,l)(x_l^i-m_j\left(x_l^i\right)){(x_l^i-m_j\left(x_l^i\right))}^T,\end{array}$$

(25)

where $w(i,j,l)$ is the value of the weighting function depicted as

$$\begin{array}{c}\hfill w(i,j,l)=\frac{min\{d^{\alpha }(x_l^i,N{N}_k(x_l^i,i)),d^{\alpha }(x_l^i,N{N}_k(x_l^i,j))\}}{d^{\alpha }(x_l^i,N{N}_k(x_l^i,i))+d^{\alpha }(x_l^i,N{N}_k(x_l^i,j))}.\end{array}$$

(26)

The $x_l^i$ is the l-th sample in class $L_i$, $N{N}_k(x_l^i,j)$ is the k-th nearest neighbor from class $L_j$ to the sample $x_l^i$, $m_j\left(x_l^i\right)$ is the local KNN mean of $N{N}_k(x_l^i,j)$, $\alpha \in (0,+inf)$ is the parameter controlling the weight, and $d(·,·)$ is the Euclidean distance of two vectors. The weighting function explicitly emphasizes the data points around the boundary.

2.1.5. Neighborhood Linear Discriminant Analysis

Differing from the strategies above whose scatter matrices are defined on k-NN sets, neighborhood linear discriminant analysis (nLDA) [27] proposes a discriminator oriented to multi-modal classes where the scatter matrices are based on other types of the neighborhood. It is motivated by the neighborhood can be taken as the smallest subclass and there is no need for any prior knowledge of the inner structure inside a class, avoiding the difficulty of determining the number of sub-classes inside a class. The scatter matrices are based on reverse k-nearest neighbor sets [49], shown as Equation (27). Given a training set $X=\{x_i\in {\mathbb{R}}^d|i\in \{1,\cdots ,N\}\}$ and its label set $\mathcal{L}=\left\{g_i\right|g_i\in \{1,\cdots ,L\}\}$. $X_j,\left|X_j\right|=N_j,{\sum }_{j=1}^L{N}_j=N$ consists all samples in class $L_j$. Given a dataset D and a sample $x_p\in D$, the reverse k-nearest neighbor set of $x_p$ is defined as Equation (27).

$$\begin{array}{c}\hfill {\mathrm{RNN}}_{\mathrm{k}}(x_p,D)=\left\{x_q\right|x_q\in D\setminus \left\{x_p\right\},x_p\in {\mathrm{NN}}_{\mathrm{k}}(x_q,D)\},\end{array}$$

(27)

where ${\mathrm{NN}}_{\mathrm{k}}(x_q,D)$ is the k-nearest neighbor set of $x_q\in D$. The within-neighborhood scatter matrix is depicted as Equation (28).

$$\begin{array}{c}\hfill S_w^{nLDA}=\sum _{\begin{array}{c}i=1\\ |{\mathrm{RNN}}_{\mathrm{k}}(x_i,X_{g_i})|\ge t\end{array}}^N\sum _{x_j\in {\mathrm{RNN}}_{\mathrm{k}}(x_i,X_{g_i})}{(x_j-{\tilde{m}}_i)}^T(x_j-{\tilde{m}}_i),\end{array}$$

(28)

where ${\tilde{m}}_i$ is the mean of the data in ${\mathrm{RNN}}_{\mathrm{k}}(x_i,X_{g_i})$. Here is a threshold for nLDA that $|{\mathrm{RNN}}_{\mathrm{k}}(x_i,X_{g_i})|\ge t$. There are $\mathcal{O}\left(kN\right)$ times of computing the outer product. The between-neighborhood scatter matrix is presented as Equation (29).

$$\begin{array}{c}\hfill {\mathbf{S}}_b^{nLDA}=\sum _{\begin{array}{c}i=1\\ |{\mathrm{RNN}}_{\mathrm{k}}(x_i,X_{g_i})|\ge t\end{array}}^N\sum _{\begin{array}{c}j=1\\ |{\mathrm{RNN}}_{\mathrm{k}}(x_j,X_{g_j})|\ge t\\ g_i\ne g_j\end{array}}^N{({\tilde{m}}_i-{\tilde{m}}_j)}^T({\tilde{m}}_i-{\tilde{m}}_j).\end{array}$$

(29)

From a point view of calculating times of outer product between vectors, the complexity of ${\mathbf{S}}_b^{nLDA}$ is $\mathcal{O}\left(N^2\right)$, which is too large with a large dataset. Here is an approximate alternative ${\mathbf{S}}_b^{nLD{A}^{app}}$ shown in Equation (30).

$$\begin{array}{c}\hfill {\mathbf{S}}_b^{nLD{A}^{app}}=\sum _{\begin{array}{c}i=1\\ |{\mathrm{RNN}}_{\mathrm{k}}(x_i,X_{g_i})|\ge t\end{array}}^N\sum _{\begin{array}{c}{x}_j\in {\mathrm{NN}}_{\mathrm{k}}(x_i,X-X_{g_i})\\ |{\mathrm{RNN}}_{\mathrm{k}}(x_j,X_{g_j})|\ge t\\ g_i\ne g_j\end{array}}^N{({\tilde{m}}_i-{\tilde{m}}_j)}^T({\tilde{m}}_i-{\tilde{m}}_j).\end{array}$$

(30)

This reduces the complexity of the between-neighbor scatter matrix to $\mathcal{O}\left(kN\right)$. The target function is to find the projected directions v satisfying Equation (31).

$$\begin{array}{c}\hfill v=argmax\left|\frac{v^T{S}_b^{nLD{A}^{app}}v}{v{S}_w^{nLDA}v}\right|.\end{array}$$

(31)

The cost of nLDA contains two parts. One part is finding reverse nearest neighbors by computing $\mathcal{O}\left(N^2\right)$ times of distance between samples. Another part is computing scatter matrices and solving an eigenvalue problem. The latter is the same as that in LDA. The former is $\mathcal{O}\left(kN\right)$. So the cost of calculating Equation (31) is $\mathcal{O}\left(N^2\right)$ times of distance and $\mathcal{O}\left(kN\right)$ times of vector product. The empirical results demonstrate that nLDA outperforms greatly than LDA and some other discriminators. Notably, a proposal to solve the unstable and poor general issue resulting from the SSS problem of nLDA is presented by Xie et al. [28], where the singularity of within-neighborhood scatter matrix is evaded by the eigenspectrum regularisation techniques so that the method is called eigenspectrum regularisation reverse neighborhood discriminative learning (ERRNDL).

The conceptual comparisons of methods of LDA variants for multi-modal classes are shown in

Table 3. Conceptual comparison of LDA variants for multi-modal classes.

Type	Goal	Technique	References	Advantages
Mixture of Gaussians	Estimate underlying distribution of every class as mixture of Gaussians	Optimize mixture density by EM alg.	MDA [9] [45]	More robust EM than MDA
		Optimize # of sub-classes	SDA [10] MSDA [11] [47] SSDA [12] speed-up SDA [46]	Overcome shortcomings of SDA Preserve original locality Better performance and better class separability than LDA, SDA and MSDA
		Both of above	RSDA [13]	Improved robustness and lower computation cost
			extended MSDA [14]
Applied problems	Classification and recognition algorithms for data with outliers
Manifold	Depict local structure using manifold	Combine LDA and LPP	LFDA [15,16]	Overcome rank deficiency in $S_b$ and protect local structure
		Use NN graph	LSDA [18]
		Characterize piecewise regional consistency	MP-DA [19]
		Subclass partition	AFLDA [20]
Applied problems	Dimension reduction method for multimodal-labeled data and face recognition
Setting weights	Import weights to penalize unstable data	Redefine $S_b$	aPAC [21,22] HLDR [24]
		Redefine $S_w$	aPAC [23]
Applied problems	Classification algorithms for real data
KNN	k-Nearest Neighbor set based scatter matrices	Redefine $S_b$	NDA [26]
		Redefine $S_b$ and $S_w$	LM-NNDA [25]
Applied problems	Classification and feature extraction methods for face databases
RNNk	Reverse k-nearest neighbors(RNNk) set based scatter matrices	Redefine $S_b$ and $S_w$	nLDA [27]	RNNk can be regarded as the smallest subclass
		Three eigen- spectrum regulari- sation models	ERRNDL [28]	Overcome SSS in nLDA
Applied problems	Recognition and discriminative algorithm for multimodal-class data

. Compared to the four discriminant analysis methods nLDA, LM-NNDA, LFDA and NDA which are all oriented to multi-modal class, we conclude several main connections and distinctions here.

The nLDA uses the reverse k-nearest neighbor set to describe the multi-modality in classes, while LM-NNDA uses the k-nearest neighbors. LFDA depicts the local structure of multi-modal class by combining with LPP, and NDA used k-nearest neighbors to rebuild scatter matrices, where they share the commonality of inheriting Fisher’s criterion and differ from each other on the definitions of $S_w$ and $S_b$.

As shown in Equation (28), the $S_w^{nLDA}$ is defined on the set of $RN{N}_k(x_i,X_{g_i})$ that focuses on the scatter between samples $x_i$ with the mean of its $RN{N}_k$ set. Similarly, in Equation (23), the $S_w^{LMNNLDA}$ depicts the scatter between each sample and the k-nearest neighbors’ mean inside the associated class, while in Equation (10), the within-scatter matrix $S_w^{LFDA}$ depicts not the $N{N}_k$ set but the scatter between the samples $x_i$ to its neighbors by manifold.

In Equation (29), the $S_b^{nLDA}$ is defined on the neighborhood of each sample that is found within its $RN{N}_k$ set. However, as shown in Equation (25), the between-scatter matrix $S_b^{NDA}$ is defined on the k-nearest neighbors for each sample that are found around all remaining classes. This is similar to $S_b^{LM-NNDA}$ in Equation (24) depicting the scatter between each sample and the mean of its k-nearest neighbors searched from other classes.

2.2. LDA Variants Solving the Small Sample Size (SSS) Problem

There is another main drawback of LDA. If the training samples are of high dimensionality but the the size of training samples is limited, $S_w$ may have rank deficiency, that is, it almost becomes a singular matrix resulting in severe instability and over-fitting [50]. This is commonly considered as the small sample size (SSS) problem [6], and it always happens in pattern recognition which makes it a widely researched problem in related areas.

From Equation (3), as well as $max\left(rank\left(S_b\right)\right)=L-1$ as mentioned previously, it can be easily proved that $max\left(rank\left(S_t\right)\right)=N-1$ and $max(rank\left(S_w\right)=rank\left(S_t\right)-rank\left(S_b\right))=N-L$. Namely, the rank of $S_b,S_w$ and $S_t$ have the upper bounds of $L-1,N-L$ and $N-1$, respectively, and all of them are quite smaller than d under the scenario of high-dimensional but limited-sized samples. That is to say, $S_b,S_w$ and $S_t$ are all of singularity, resulting in the unsolvable for the objective Equation (4). We summarize and analyze different methods proposed to solve the SSS problem.

2.2.1. Fisherface Method (PCA + LDA)

The Fisherface method [51] is used in a wide variety of disciplines and areas that applies PCA initially such that the original d-dimensional features are reduced to a medium dimensionality $d_1$ under the guarantee of $d_1\le rank\left(S_w\right)=N-L$ aiming to make the consequent within-class scatter matrix full-rank. Then applying the standard LDA technique for further reducing the dimensionality to $d_2$ that has to be guaranteed $d_2\le L-1$ because of the $max\left(rank\left(S_b\right)\right)=L-1$. Consequently, the SSS problem is overcome. In [52], we introduced in Section 2.2.2 a regularization procedure for the SSS problem; the author also applies PCA first to obtain full-rank $S_w$.

However, there exists a drawback that the PCA application of the first dimensionality reduction process leads to the loss of some useful discriminant information.

2.2.2. Regularization Method

For face recognition problems, it is common that the samples’ dimensionality is very large resulting in $S_w$ being singular. In [52], the authors slightly modify matrix $S_w$ to $S_w+\kappa I$, where $\kappa$ is a quite small positive number making $S_w+\kappa I$ absolutely positive definite. This is a regularization procedure by adding a small diagonal positive matrix to $S_w$. The same technique is used in references [53,54] to solve the SSS problem. However, the drawbacks are also obvious. Firstly, the computational complexity is quite high to deal with $S_w$ of such a high dimension. Secondly, adding $\kappa$ is just used for performing the inverse operation feasibly without any physical meaning. It is not able to evaluate $\kappa$ and its poor choice may degrade the generalization performance of the method.

Besides regularizing matrix $S_w$ directly, Jiang et al. [55] present an approach of eigenfeature regularization for face recognition. Using eigenvectors of $S_w$ to span to image space and decomposing it into three sub-spaces, that is a null subspace, an unstable subspace because of noise and limited sample size, and a reliable subspace spanned mostly by the facial variation, where eigen features are regularized in different ways within the three sub-spaces. This proposed approach remits the issue of limited sample size and noise leading to uncertain small and zero eigenvalues and is verified to be more stable, less over-fitted, or better generalized.

Another eigenfeature regularization method is proposed in [55] that regularises $S_w$ by extrapolating its eigenvalues of the range space into the null space where the extrapolation is made by using exponential functions.

2.2.3. Null Space Method

The null space (or kernel) of a matrix $A\in {\mathbb{R}}^{m\times d}$ [56]: $ker\left(A\right)=\{x\in {\mathbb{R}}^d|Ax=\mathbf{0}\}.$ The range space of A: $range\left(A\right)=\left\{y\right|y=Ax,x\in {\mathbb{R}}^d\}$.

The Fisher’s criterion function [57] is shown in Equation (32).

$$\begin{array}{c}\hfill F\left(v\right)=\frac{v^T{S}_{b}v}{v^T{S}_{w}v},\end{array}$$

(32)

where v denotes the projected vector. The authors in [58] introduce a revised Fisher’s criterion $\widehat{F}\left(v\right)$ shown in Equation (33),

$$\begin{array}{c}\hfill \widehat{F}\left(v\right)=\frac{v^T{S}_{b}v}{v^T{S}_{b}v+v^T{S}_{w}v},\end{array}$$

(33)

and have proved Equation (34), that is, $F\left(v\right)$ and $\widehat{F}\left(v\right)$ obtain the same optimal v.

$$\begin{array}{c}\hfill argmax\widehat{F}\left(v\right)=argmaxF\left(v\right).\end{array}$$

(34)

Based on Equations (33) and (34), the authors in [59] introduce a different LDA technique to compute the best mapping directions using $\widehat{F}\left(v\right)$. If $S_w$ is non-singular, then the $S_t=S_w+S_b$ is also non-singular. For the circumstances oriented towards the SSS problem, the process utilizes the null space of $S_w$. Suppose the original feature space ${\mathbb{R}}^d$, and the rank of $S_w$ is denoted as $r_w$ and $r_w<d$, that is $S_w$ is singular. Thus, there exists the null space of $S_w$: $null\left(S_w\right)\subset {\mathbb{R}}^d$ such that $null\left(S_w\right)=span\left\{{\alpha }_i\right|S_w{\alpha }_i=0,i=1,\cdots ,d-r_w\}$. Let all samples in ${\mathbb{R}}^d$ be projected into $null\left(S_w\right)$ via the transformation matrix $T^T$, where $T=({\alpha }_1,\cdots ,{\alpha }_{N-r})$. The within-class scatter matrix ${\tilde{S}}_w$ of the mapped data in $null\left(S_w\right)$ is proved a complete zero matrix. That is to say, ${\tilde{S}}_t={\tilde{S}}_w+{\tilde{S}}_b={\tilde{S}}_b$. So maximizing the between-class scatter matrix ${\tilde{S}}_b$ in $null\left(S_w\right)$ is the same as maximizing the total scatter in $null\left(S_w\right)$. In such cases, the author applies the PCA method to calculate the eigenvectors related to the largest eigenvalues of ${\tilde{S}}_b$ that are the vectors of optimal discrimination fulfilling the requirements of LDA. However, projecting all data to the useful null subspace of $S_w$ displays its strong clustering ability to achieve nice generalization performance, which seems that it achieves optimal discriminant ability but leads to over-fitting. The step of the diagonalization of $S_b$ needs to be eliminated for the aim of avoiding possible over-fitting, which is mentioned by Liu et al. in [60].

There is quite a high computational complexity in the process of identifying the null space of $S_w$ because of its high dimension. To escape the high calculating complexity, in [59], the pixel grouping technique is applied beforehand for artificial feature extraction and dimension reduction of the original data, and after that, the null space LDA is realized in reduced feature space $null\left(S_w\right)$ rather than the original space.

Due to the computation complexity problem of the original null space LDA method we introduced hereinabove, the authors in [61] propose a more efficient null space approach to solve that. If there are $v^T{S}_{w}v=0$ and $v^T{S}_{b}v\ne 0$, then the eigen vector v is valuable for discriminating, whereas if $v^T{S}_{w}v=0$ and $v^T{S}_{b}v=0$, v is useless. Consequently, they remove the null space of $S_t$ without losing valuable discrimination. Suppose U is the matrix with columns being all the eigenvectors of $S_t$ that correspond to the nonzero eigenvalues, and it is able to obtain the $S_w^{\prime }$ shown in Equation (35).

$$\begin{array}{c}\hfill S_w^{\prime }=U^T{S}_{w}U,S_b^{\prime }=U^T{S}_{b}U.\end{array}$$

(35)

Next a reduced but equally useful subspace of the null space of $S_w$ is calculated, and projecting the data onto it, and then deriving the most discriminative vectors. Let Q be the null space of $S_w^{\prime }$, thus $Q^T{S}_w^{\prime }Q=0$, then there are Equations (36) and (37).

$$\begin{array}{c}\hfill S_w^{″}=Q^T{S}_w^{\prime }Q=Q^T{U}^T{S}_{w}UQ={\left(UQ\right)}^T{S}_w\left(UQ\right),\end{array}$$

(36)

$$\begin{array}{c}\hfill S_b^{″}=Q^T{S}_b^{\prime }Q=Q^T{U}^T{S}_{b}UQ={\left(UQ\right)}^T{S}_b\left(UQ\right),\end{array}$$

(37)

where $UQ$ is a subspace of all the null space of $S_w$ which is reduced but of quite use to derive the most discriminative vectors. It is notable that if there is a null space of $S_b^{″}$, it is necessary to remove it. Due to $max\left(rank\left(S_t\right)\right)=N-1$, the dimensionality of $S_w^{\prime }$ is bounded at $N-1$; and due to $max\left(rank\left(S_w\right)\right)=N-L$ and $rank\left(S_w^{\prime }\right)=rank\left(S_w\right)$, the dimensionality of the null space of $S_w^{\prime }$ is $L-1$. While $S_b^{″}$ is always full-rank so the amount of the optimal discriminant vectors is $L-1$. This method improves the computational problem of the null space by removing redundant information without decreasing the discriminant efficiency.

Liu et al. [60] present the most appropriate condition for the null-space-based LDA method: $N=d-1$, that is $S_t$, is full-rank, where N is the amount of all data and d is the dimensionality of original space. The procedure of null-space-based LDA under this most suitable situation removes the null space of $S_t$ first and extracts the null space of reduced $S_w$. It is most straightforward with just one time of eigen-analysis. It not only saves a lot of computational costs but also keeps the performance simultaneously. Moreover, the authors incorporate the kernel technique into the null space method by using the Cosine kernel function. They discovered that in kernel space, $S_t$ is full-rank, so the process of the null space method is extremely faster and more stable during calculation.

A faster null space method than [59] is proposed in [62] by only carrying out QR factorizations to implement LDA without carrying out eigendecomposition and SVD, of which computational complexity is approximately $4d{N}^2+2dNL$.

An alternative method of null-space-based LDA methods named Fast NLDA [29] modifies a fast process for the null space technique based on random matrix multiplication with scatter matrices. It is based on the assumption that the vectors are linearly independent. The oriented transformation matrix is gotten by $T=S_t^{+}{S}_{b}Y$, where $S_t^{+}$ is the pseudo inverse of $S_t$, and Y is a random matrix of rank $L-1$. This approach requires $d{N}^2+2dNL$ computations. The pseudoinverse LDA method of pseudo inverse $S_w$ is studied in [63] for image classification.

2.2.4. Direct LDA

Based on the drawbacks of the techniques we introduced above: discarded dimensions which carry key discriminative information in Section 2.2.1, falling short of using information out of null space of $S_w$ and computational problems related to large scatter matrices in Section 2.2.3. We introduce a direct LDA algorithm that permits data of high dimensionality and optimizes Fisher’s criterion without any feature extraction or dimension reduction steps in advance, referred to as direct LDA (DLDA) by Yu et al. [30].

DLDA discards the null space of $S_b$ firstly, where there is no discriminative information, but abandoning the null space of $S_w$ where there is of the best discrimination. This is achieved in reverse order of traditional procedure by performing a simultaneous diagonalization procedure on $S_b$ first by the found matrix W, as shown in Equation (38),

$$\begin{array}{c}\hfill W^T{S}_{b}W=I,\end{array}$$

(38)

and then on $S_w$, which keeps its null space to find the most discriminative vectors, as shown in Equation (39), and $D_w$ means the diagonalizable $S_w$.

$$\begin{array}{c}\hfill W^T{S}_{w}W=D_w.\end{array}$$

(39)

It is worth mentioning that DLDA seems to reserve the null space of $S_w$, from which the optimal discriminant vectors of LDA can be deduced [51,59]. But it cannot substantially avoid it because removing the null space of $S_b$ could lead to the portion loss of null space of $S_w$. $S_b$ has a smaller rank than $S_w$ in most instances, so the subspace guaranteeing the full rank of $S_b$ is also guaranteeing that of $S_w$. DLDA does not take full advantage of the null space of $S_w$, by abandoning the null space of $S_b$ via reducing dimension and indirectly leading to the loss of the null space of $S_w$. Additionally, in this paper, calculating skills are introduced to deal with large scatter matrices along with an accurate solution to Fisher’s criterion being given.

2.2.5. Orthogonal LDA

Ye et al. propose an orthogonal LDA (OLDA) method against SSS problem [31] defining a new criterion that does not require the non-singularity of the scatter matrices. It has presented to be the same as those null-space-based LDA methods [59,61] limited in a soft condition of the data are linearly independent [64]. The null-space-based method [61] and OLDA all lead to the orthogonal transformations, while the former performs the orthogonal transformation in the null space of $S_w$ and the latter executes that via diagonalizing $S_b,S_w$ and $S_t$ simultaneously. The calculation cost of OLDA is smaller than that of the null space method [59] and it is measured as $14d{N}^2+4dNL+2d{L}^2$ flops.

2.2.6. Against Over-Fitting

Another serious issue of the SSS problem is the over-fitting problem. It is mainly because the between- and within-class scatter matrices calculated from the limited number of data drift extremely from the underlying ones. Pang et al. introduce a regularization term via clustering to solve this problem [65], specifically regularizing the within-class and between-class scatter matrices with within-cluster and between-cluster scatter matrices, respectively, and simultaneously.

We compare the methods of solving the SSS problem of LDA variants conceptually in

Table 4. Conceptual comparison of LDA variants for SSS problem.

Type	Goal	Technique	References	Advantages	Disadvantages
Fisherface method	Reduce dimension of $S_w$	Use PCA initially to reduce dimension of $S_w$	[51]		lose useful information
Applied problems	Face recognition algorithms
Regulari- zation	Regularize $S_w$	$S_w+\kappa I$	[52] [53] [54]		high computational complexity; uncontrollable parameter
		Regularize eigenfeatures of $S_w$	[55]	more stability less over-fitting or better generalization
Applied problems	Face recognition algorithms
Null space	Utilize null space of $S_w$ that fulfills LDA criterion	Utilize full null space of $S_w$	[59]		high computational complexity
		Remove useless null space part of $S_w$ and use the reduced part	[61]	more efficiency than [59]
		Same as above but under the most suitable situation $N=d-1$	[60]
		QR factorizations only	[62]	faster than [59]
		Random matrix multiplies scatter matrices	[29]	faster than [62]
Applied problems	Face recognition algorithms
Direct LDA	Indirect dimension reduction of $S_w$	Remove null space of $S_b$ firstly	[30]
Applied problems	Face recognition algorithms
Orthogonal LDA	Orthogonal transformation in three scatter matrices simultaneously	New criterion no need non-singularity	[31]	lower complexity than [59]
Applied problems	Classification algorithm for real-world data
Against over-fitting	Solve over-fitting caused by SSS	Cluster-based scatter matrices	[65]
Applied problems	Face recognition algorithms

on metrics of method types, method goals, specific techniques, advantages and disadvantages.

2.3. LDA Variants with Robustness

The conventional LDA method is based on $L_2$-norm [1] that is sensitive to outliers [66]. The outliers may lead to the projection vectors drifting from the objective directions. It is advisable to think about the robust modelling of the classical $L_2$-norm LDA to suppress the affection of outliers.

2.3.1. $L_1$ Norm

It is known to all that the $L_1$-norm is of better robustness than $L_2$-norm because the $L_1$-norm does not heighten the impact of outliers related to many errors as the $L_2$-norm does [66,67,68]. Li et al. [32] present a rotational invariant $L_1$-norm (i.e., $R_1$-norm) based LDA, denoted as LDA-$R_1$. It uses the gradient ascending iterative algorithm upon eigenvalues decomposition that leads to much time costs to perform convergence in input space of high dimensionality.

Wang et al. [33] introduce a technique, named LDA-$L_1$. It maximizes the proportion of the between-class dispersion to that of the within-class. They are defined by the $L_1$-norm rather than the $L_2$-norm. Recall that the number of classes L, the $N_i$ is the number of samples in class $L_i$, the class mean $m_i$, the total mean m, the projected vector v, and $x_j^i$ is the j-th sample of class $L_i$. The Fisher-like criterion of $L_1$-norm is presented as shown in Equation (40).

$$\begin{array}{c}\hfill F\left(v\right)=\frac{{\displaystyle \sum _{i=1}^L}{N}_i\left|v^T(m_i-m)\right|}{{\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^{N_i}}\left|v^T(x_j^i-m_i)\right|}.\end{array}$$

(40)

The criterion (40) is termed as LDA-$L_1$ by authors used to maximize the proportion of between-class dispersion to that of the within-class. However, it is intractable to optimize the objective function (40) and obtain the global solution of the LDA-$L_1$. The authors give a gradient ascending (GA) iterative algorithm in order to seek a local solution v of $L_1$-norm LDA that maximizes the objective function. It is worth mentioning that the LDA-$L_1$ does not suffer from the problem of SSS or rank limit because the criterion is not based on the conventional matrices $S_b$ and $S_w$ anymore. Similar work of LDA-$L_1$ was published by Zhong et al. in the same year [69] which obtains a single locally optimal solution realized iteratively and obtains multiple locally optimum solutions via a greedy search method, as well as solving the singularity of $S_w$.

On the contrary, Liu et al. [70] propose a non-greedy iterative algorithm to address the objective function Equation (41) and obtain a closed-form solution for all projections.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill F(V,\lambda )& =arg\underset{V^{T}V=I}{max}\left(\sum _{i=1}^L{N}_i\left|V^T(m_i-m)\right|\right)\hfill \\ \hfill & -\lambda \left(\sum _{i=1}^L\sum _{j=1}^{N_i}\left|V^T(x_j^i-m_i)\right|\right),\hfill \end{array}\end{array}$$

(41)

where V is the projection matrix, $\lambda$ is related to V that are optimized iteratively, L and $N_i$ are the numbers of total classes and the class $L_i$, respectively, m and $m_i$ are the means of total samples and samples in $L_i$, respectively, and $x_j^i$ is the j-th sample of class $L_i$.

When it comes to matrix-input issues, a matrix must be converted into a vector before applying the LDA methods, which are vector-based. This conversion can lead to high-dimensional data and the loss of some fundamental local information. Besides the matrix-based methods such as the matrix-based PCA [71,72,73], the matrix-based SVM [74,75], and the matrix-based LPP [76,77,78,79], the first proposal of $L_2$-norm-based 2-dimensional LDA (2DLDA) appears in [80] and afterwards many extensions are raised [81,82,83,84,85,86]. However, the 2DLDA may suffer from the robustness due to the effects of outliers and noise although it remits the SSS problem based on a weak assumption and preserves its original structural information. Li et al. [34] extend conventional 2-dimensional LDA with $L_2$-norm into 2-dimensional LDA with $L_1$-norm, termed as $L_1$-2DLDA, where the optimization problem is solved by the greedy iterative algorithm with its convergence being guaranteed. The authors in [87] further solve $L_1$-2DLDA through a nongreedy algorithm.

The iterative algorithms in the above $L_1$-norm based literature unfortunately mostly require selecting a suitable stepsize by iteratively modifying discriminant vectors. Due to the nonconvexity in the updating process, an unsuitable selection of the stepsize will impact the deduction of an optimum greatly. To handle the LDA-$L_1$ optimization problem, Zheng et al. [35] present an iterative algorithm that uses a new surrogate convex function for the optimization objective inside every iteration which only solves a convex problem and guarantees a closed-form result, referred to as $L_1$-LDA.

Furthermore, same as the equivalence relation sharing between the kernel discriminant analysis (KDA) [36] and the kernel principal component analysis (KPCA) plus LDA, which is found by Yang et al. [88], the authors generalize the proposed $L_1$-LDA method into reproducing kernel Hilbert space (RKHS) to handle the nonlinear robust feature extraction through the kernel techniques, which hence termed as the $L_1$-norm kernel discriminant analysis ($L_1$-KDA = KPCA + $L_1$-LDA) method. Even though there is no need for choice of stepsize through this new efficient iterative algorithm, it has been indicated that its utilization results in $L_1$-LDA being easy to get in trouble with a lot of serious problems [89]: the existence of singularity problem, insufficient robustness against outliers because of the updated weighting mechanism, and unguaranteed Bayes solution optimality of discriminative criterion of $L_1$-LDA.

To this end, Ye et al. [89] present an efficient iterative method to deal with a general $L_1$-norm min–max issue and perform conceptual insight into its convergency, which overcomes the above problems that exist in both LDA-$L_1$ and $L_1$-LDA.

2.3.2. $L_{2,1}$ Norm

The utilization of $L_1$-norm in the above works is of limited robustness and mostly based on the greedy search strategy to seek the projections each by each where the process is time-consuming and may be trapped in local optimality. The $L_{2,1}$-norm-based loss function is firstly proposed by Nie et al. [90] to overcome the outliers and used as a regularization to fulfill feature selection. Inspired by this work, Nie et al. propose a novel robust LDA measured by $L_{2,1}$-norm, named as $L_{2,1}$-LDA [37].

The $L_{2,1}$-norm of a matrix $A\in {\mathbb{R}}^{d\times m}$ with its elements $a_{ij},i=1,\cdots ,d,j=1,\cdots ,m$ is shown in Equation (42).

$$\begin{array}{c}\hfill {\Vert A\Vert }_{2,1}=\sum _{i=1}^d\sqrt{\sum _{j=1}^m{a}_{ij}^2}.\end{array}$$

(42)

The $L_{2,1}$-norm can measure the distances of spatial dims in $L_2$-norm, specifically, which is designed to enforce sparsity over the row-by-row data points to improve the robustness resisting outliers in $L_1$-norm. In comparison with the $L_{2,1}$-norm, the $L_1$-norm only focuses on inhibiting anomaly overall values without keeping an eye on the distinction between row and column, leading to insufficient robustness against outliers. To this end, the authors design the $L_{2,1}$-norm criterion to min–max within-class scatter and total data scatter at the same time to enhance the robustness and discriminability of the formulation as in Equation (43).

$$\begin{array}{c}\hfill \left\{\begin{array}{c}min\sum _{j=1}^L\sum _{i=1}^{N_j}\Vert V^T(x_i^j-m_j){\Vert }_2,\hfill \\ max\frac{1}{N}\sum _{k=1}^N\Vert V^T{x}_k{\Vert }_2=\frac{1}{N}\Vert X^{T}V{\Vert }_{2,1},\hfill \end{array}\right.\end{array}$$

(43)

where V is the projection matrix, and X is the data matrix. This $L_{2,1}$-norm criterion suppresses the anomaly of outliers from the learned sparsity structure by capturing the distinction between spatial dimensions and sample vectors and promoting the sparsity at the data points level. This improved the robustness. The authors propose a min–max iterative re-weighted optimization algorithm to deal with (43) which is a big challenge to be solved perfectly.

2.3.3. $L_p$ Norm

Contrasting to the $L_2$-norm, the robust analysis of $L_p$-norm is investigated widely in data mining, for example, the robust locality preserving projections [91], $L_p$-norm based principal component analysis [92,93,94]. An $L_p$-norm based LDA is proposed in [38], termed as LDA-$L_p$ by the authors. In this scheme, arbitrary values of p can be used to acquire a robust and rotation-invariant extension of LDA, for which the optimal solution is found using the steepest gradient method. The objective of the maximization problem is presented as Equation (44).

$$\begin{array}{c}\hfill F_p\left(v\right)=\frac{{\displaystyle \sum _{i=1}^L}{N}_i{\left|v^T(m_i-m)\right|}^p}{{\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^{N_i}}{\left|v^T(x_j^i-m_i)\right|}^p},\end{array}$$

(44)

which can be worked out by computing the gradient of $F_p\left(v\right)$ regarding v. However, there is a problem that the gradient of $F_p\left(v\right)$ is not well defined on some singular points due to the absolute value operator in this formula. The author introduces a sign function to escape the technical difficulty shown in Equation (45).

$$\begin{array}{c}\hfill sgn\left(a\right)=\left\{\begin{array}{cc}1,\hfill & ifa>0,\hfill \\ 0,\hfill & ifa=0,\hfill \\ -1,\hfill & ifa<0.\hfill \end{array}\right.\end{array}$$

(45)

Hence, the above objective formula can be rewritten as presented in Equation (46).

$$\begin{array}{c}\hfill F_p\left(v\right)=\frac{{\displaystyle \sum _{i=1}^L}{N}_i{\left[sgn\left(v^T(m_i-m)\right)v^T(m_i-m)\right]}^p}{{\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^{N_i}}{\left[sgn\left(v^T(x_j^i-m_i)\right)v^T(x_j^i-m_i)\right]}^p}.\end{array}$$

(46)

The optimal v that maximizes (46) can be obtained by taking a gradient of $F_p\left(v\right)$ regarding v: ${\nabla }_v=\frac{d{F}_p\left(v\right)}{dv}$, as depicted in a steepest gradient iterative algorithm with singular check and convergence check steps in detail.

To handle the matrix-input problem, contrasting to the $L_1$-2DLDA we introduced above that is still sensitive to outliers and noise, the $L_p$-norm is of much more robustness for $0<p\le 1$. Li et al. [39] introduce a bilateral two-dimensional LDA using the $L_p$-norm, named B$L_p$2DLDA. The criterion of B$L_p$2DLDA shares an equivalence relation with an upper bound of the theoretical principal of the optimal Bayes which theoretically guarantees the reasonability of its optimization via the Bayes error bound. The objective is solved by a modified ascent iterative technique.

2.3.4. $L_{sp}$ Norm

Inspired by successful PCA-$L_p$ algorithms [93,94,95,96,97], Ye et al. [40] propose a robust LDA, referred to as FLDA-$L_{sp}$. It maximizes $L_s$ norm distance and minimizes $L_p$ norm distance simultaneously via $L_s$-and $L_p$-norm measuring the between- and within-class distances, respectively, which differs from LDA-$L_p$ [38] by a more effective iterative algorithm to obtain the target solution. The objective function of FLDA-$L_{sp}$ is presented in Equation (47).

$$\begin{array}{c}\hfill F\left(v\right)=\underset{v^{T}v=1}{max}\frac{{\displaystyle \sum _{i=1}^L}{N}_i{\left|v^T(m_i-m)\right|}^s}{{\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^{N_i}}{\left|v^T(x_j^i-m_i)\right|}^p}.\end{array}$$

(47)

It is obvious that when $0<s<2$ and $0<p<2$, the objective is conferred with robustness. Moreover, the LDA-$L_1$ and LDA-$L_2$ become the special cases by setting specific values of s and p.

Compared with the gradient ascending iterative algorithm [33,69], the iterative algorithm used in LDA-$L_{sp}$ does not require to apply the non-convex surrogate function, and it overcomes the challenge of choosing stepsize. Compared to the alternative algorithm addressing the drawbacks in the gradient ascending iterative algorithm [33,69], the iterative algorithm used in LDA-$L_{sp}$ avoids transforming the original objective during each iteration.

The norm types of LDA extensions, with comparisons of corresponding optimization methods, advantages as well as disadvantages are shown in

Table 5. Conceptual comparison of LDA variants for robustness.

Type of Norm	References	Optimization Method	Advantages	Disadvantages
$L_1$ norm	[32] LDA-$R_1$	GA iterative algorithm		High computational complexity
	[33] LDA-$L_1$	Local solution by GA iterative algorithm	No SSS or rank limit
	[69] LDA-$L_1$	Single local solution by iteration algorithm; Multiple local solutions by greedy searching	No SSS problem
	[70]	A non-greedy iterative algorithm; A closed-form solution for all projections
	[34] $L_1$-2DLDA matrix input	Greedy iterative algorithm; Convergence being guaranteed
	[87] $L_1$-2DLDA	A nongreedy algorithm		Bad selection of stepsize may impact the optimality
	[35] $L_1$-LDA	Iterative algorithm; A closed-form solution during every iteration	No stepsize	Easy singularity; Insufficient robustness; Unguaranteed Bayes optimality
	[89]	An effective iterative framework	Overcome problems in LDA-$L_1$ and $L_1$-LDA
Applied problems	Robust classification and recognition algorithms for suppressing outliers
$L_{2,1}$ norm	[37]	Minmax iterative re-weighted optimization algorithm	More robust than $L_1$-norm	Hard to solve objective
Applied problems	Robust classification and visualization algorithms for synthetic data and image datasets
$L_p$ norm	[38] LDA-$L_p$	Steepest gradient iterative algorithm	Arbitrary p can obtain robust and other LDA versions	Technique difficulty in optimization
	[39] B$L_p$2DLDA matrix input	Modified ascent iterative technique	More robust than $L_1$-2DLDA [34] for $0<p\le 1$
Applied problems	Robust discriminant analysis methods for contaminated databases
$L_{sp}$ norm	[40] FLDA-$L_{sp}$	A more effective iterative algorithm	Robustness at $0<s,p<2$; LDA-$L_1$, LDA-$L_2$ are special cases; Needless of non-convex surrogate function and stepsize compared to [33,69]; No transforming original objective iteratively
Applied problems	Robust discriminant analysis methods for image data in suppressing the noise

.

3. Applications of LDA Variants

In this section, we focus on the usage of discriminant analysis in addressing the drawbacks of small sample size problems, being sensitive to noise and outliers, and being unable to deal with multi-modal-class data. We summarize the application fields across face recognition, fault detection and diagnosis, system condition monitoring, process monitoring et al., which belong to the main areas of computer vision, pattern recognition, and automation control systems. The objective of this section is to guide the readers on the benefits of suppressing three drawbacks in discriminant analysis and how to optimally utilize suitable techniques in certain cases.

3.1. Applications of LDA Variants for Multi-Modal Data

3.1.1. Mixture of Gaussians

The mixture of Gaussian-based discriminant analysis, depicting the mixtures of the multi-modal density models in each class, has been acting as an excellent technique to offer a better estimation and description of multi-modal data distributions. This technique motivates extensive applications aiming to address the multi-modal problem in many scenarios. MDA is applied to face detection [98], human–robot interaction [99], remote sensing [100], process monitoring [101], drug distribution in humans [102], digit recognition [103], and speaker verification [104]. In addition, MDA is used as a per-field classification method [105] and a curve classification method [106]. The subclass-based mixture of Gaussian, such as SDA, is applied for face recognition [107,108], disease diagnosis [109], behavior recognition [110], and bug prediction [111].

3.1.2. Manifolds

LFDA is applied in pedestrian re-identification [112], diagnose prediction [113], facial expression recognition [114,115], fault diagnosis [116,117], spoken language identification [118], industrial process fault classification [119], process monitoring [120], the physical load prediction [121], and the spoken emotion recognition [122]. Additionally, there are various of LFDA extensions being of much usage. For examples, sparse LFDA for facial expression recognition [123] and status monitoring [124], maximum LFDA for face recognition [125], complete LFDA for face recognition [126], uncorrelated LFDA for ear recognition [127], geometric preserving LFDA for person re-identification [128], wavelet LFDA based bearing defect classification [129], orthogonal LFDA for fault diagnosis [130] and facial expression recognition [131], projection-optimal LFDA for feature extraction [132] and palmprint recognition [133], self-adaptive LFDA based semi-supervised image recognition [134], the unsupervised image-adapted LFDA [135], the fault diagnosis based on local centroid mean LFDA [136], fault diagnosis for blast furnace ironmaking process using randomized LFDA [137], and online soft measurement method based on improved LFDA [138].

Additionally, the studies focusing on semi-supervised LFDA are welcome in many scenarios, such as enhanced semi-supervised LFDA for face recognition [139], for sparse dimensionality reduction of the hyperspectral image [140], and gene expression data classification [141].

LSDA is applied as another discriminant approach based on manifold learning in kinds of fields, for example, stable LSDA-based image recognition [142], improved LSDA-based feature extraction [143], orthogonal LSDA-based face recognition [144,145], identification of breast cancer [146], hyperspectral imagery classification [147], fault diagnosis [148], image recognition [149], face recognition [150,151], and video semantic detection [152].

3.1.3. k-Nearest Neighbors

NDA has been applied into various areas, for example, face recognition [26], face detection [153], feature extraction [154,155], image recognition [156,157], imagery classification [158], image retrieval [157,159], incremental subspace learning and recognition [160], and 3-D model classification [161].

3.1.4. Setting Weights

Penalized discriminant analysis has been used in conifer species recognition [162], tumor classification [163,164], classification of bladder cancer patients [165], image-based morphometry [166], detection of wild-grown and cultivated Ganoderma lucidum [167], noise removal [168], brain images [169], microarrays [170], and predicting choice [171].

3.2. Applications of LDA Variants for Solving SSS

The SSS problem happens in the case of the larger feature dimension but the limited data size which arouses great concern within the face recognition community to solve the poor generalization, instability or the over-fitting problems when performing face recognition on a larger face dataset but with very few available training face images [30,51,52,53,55,59,60,61]. The applications of LDA variants against the SSS problems promote a feature presentation of more discrimination and stability in low-dimensional space for the face images to perform extracting features, classifying, and reducing dimension issues in pattern recognition. Tian et al. discussed the image classification issue in the case where the total amount is smaller than the dimensionality of training samples to be classified and provided a good classification performance with a low error rate [63].

Additionally, the null space discriminant analysis for the SSS problem has been applied for novelty detection [172,173], and person re-identification [174,175].

The Fisherface method for the SSS problem is used for a face recognition problem which slowly reacts to big changes in the cases of lighting direction and facial expression [51]. This technique is also applied for image retrieval [176].

Pang et al. applied the proposed enhanced LDA into pattern recognition systems of face and ear recognition to solve the over-fitting problem [65].

3.3. Applications of LDA Variants with Robustness

LDA variants based on different norms have been applied widely by reducing the influence of outliers, for example, $L_1$-norm LDA for robust feature extraction [35,177], human activity recognition [178,179], $L_{2,1}$-norm LDA for face recognition [180] and image recognition [181].

3.4. Discussions on the Applications of LDA Variants

Based on the above summaries of the application fields of LDA variants for addressing three drawbacks, we can conduct some analysis as follows.

Firstly, the applications of the methods for multi-modal data are mainly distributed in fault detection and diagnosis, process and status monitoring, recognition and identification of information, and the classification, which is coming from the fact that the multi-modality exists in the samples of complicated distributions, such as outliers or noise. The techniques of different theoretical philosophies oriented to multi-modality may help to guide the applications to the different detection or recognition scenarios with complex samples.

Secondly, the application fields of LDA variants for solving the SSS mostly gather in face recognition, because facial information is of high features that quite easily limit to the rank-deficiency of the within scatter matrix. This can direct us to deal with other realistic problems with large features by discriminant analysis methods.

Thirdly, the application fields of LDA variants with robustness are similar to those of multi-modal data impairing the influence of outliers. There is an optimization problem in solving the eigendecomposition of LDA variants with other norms, which motivates us to optimize and apply discriminant analysis with $L_1$, $L_{2,1}$, $L_p$ and $L_{sp}$ norm.

4. Kernelization

The kernel method involves performing a projection from the original low-dimensional space into a higher-dimensional feature space, specifically the reproducing kernel Hilbert space (RKHS) [182]. This transformation changes the data from being linearly inseparable to linearly separable.

We provide a simplified introduction to RKHS here. We start with the vector space, defined as a set of vectors equipped with addition and scalar multiplication. The inner product space is a vector space equipped with an inner product operation. The Hilbert space is a complete inner product space, where all Cauchy sequences converge within this space. An RKHS is a Hilbert space of functions where the inner product of the mapping functions is equivalent to the kernel function when data is mapped into this space.

The process of feature mapping, depicted in Figure 2, clearly shows how a nonlinear problem is transformed into a linear problem. This projection is achieved by the feature map utilizing the kernel function.

Here, we summarise the methods above which have been extended to the kernel version for solving nonlinear problems. First of all, we have a proposal of a kernelized data analysis method using an orthogonal transformation that combines the three objectives of kernel PCA and kernel LDA [183]. Our proposal possesses both feature extraction and discriminative properties to solve nonlinear problems in reproducing kernel Hilbert space. SDA is extended into kernel version in [184], and the kernel extension of SDA is used for yielding the optimal recognition rates [185]. The speed-up and multi-view SDA and its kernelized form are proposed in [46]. The kernel MSDA (KMSDA) is proposed in [14]. LFDA has been extended to non-linear dimensionality reduction cases with kernel trick, called KLFDA [15,16], making so many applications: sparse kernel LFDA for fault diagnosis [186], multiple kernel LFDA based face recognition [187] and fault diagnosis [188], financial distress predictions [189], wavelet kernel LFDA for bearing defect classification [190], manifold adaptive kernel LFDA for face recognition [191], individual geographic origin prediction [192], hyperspectral image classification [193], and semi-supervised kernel LFDA for bearing defect classification [194].

The NDA is extended into kernel version in [185] and is used for data classification by Diaf et al. [195] and 3-D model classification [161]. The kernel technique is incorporated into the null-space-based LDA effectively solving the SSS problem [60] and is used for novelty detection [173]. The $L_1$-norm LDA has been kernelized in [35], and the $L_p$ LDA has been kernelized in [196].

5. Conclusions and Study Perspectives in LDA

We searched the Scopus and Elsevier databases, as well as the Web of Science Core Collection (WoSCC), using the relevant keywords “multi-modal”, “Small Sample Size”, “robust” and “discriminant analysis”. This search yielded more than 300 papers. We applied a priority selection criterion based on stronger relevance, a higher number of citations, more recent publication years, and a higher ranking of journals according to the Journal Citation Reports (JCR) and conference papers according to the China Computer Federation (CCF) recommendations, resulting in a review of 197 articles in total.

In general, 175 articles within our references are sourced on WoSCC across ten fields, as illustrated in Figure 3. The two most covered fields are computer science artificial intelligence, and engineering electrical electronics. We summarized and compared the extensions on techniques, innovations and main applications of discriminant analysis-based algorithms, focusing on addressing the three main drawbacks of conventional LDA: its inability to handle multi-modal data, small sample sizes (SSS), and lack of robustness. Finally, we summarized the kernel-extended algorithms designed for nonlinear problems.

As part of future research, three drawbacks of LDA can be considered common issues in classification, clustering, and regression problems using discriminative analysis. The application areas of the reviewed variants may provide optimization-oriented guidance on how to apply these methods better to appropriate real-world problems for optimal performance. This constitutes the first future direction for applications.

Additionally, by examining how these LDA variants address the three drawbacks, we can gain insights into the underlying relationships between data distributions, structures, and algorithms. This generates another open question regarding the potential designs of the fusions of more robust, stable, and general algorithms with discriminative properties, which should be explored further.

There are two types of fusion methods. The first involves combining the objectives of different algorithms. Reference [197] proposes a novel framework of seven data analysis methods that combines the objectives of PCA and LDA. Based on this framework, we extend the first method of the framework into RKHS with a kernel method [183]. The second type of fusion is the staged usage of different algorithms. The methods [51,88] reviewed above that utilize PCA or KPCA ahead of LDA are two-phase fusions. The potential designs for fusions of different objective functions and staged methods, aimed at enhancing robustness, stability, and generality, represent a significant future research topic.

Furthermore, the kernel extension is an important research topic for data analysis methods to address nonlinear problems. Building on techniques that resolve the drawbacks of LDA, our future research will focus on three subjects.

1.

Improving robustness for nonlinear problems;

2.

Handling multi-modal-class data with complicated nonlinear distributions or outliers;

3.

Addressing small sample size problems in reproducing kernel Hilbert space.

It is a promising research direction that investigates extending discriminative analysis methods that have already overcome these drawbacks into their kernel versions.