Locality Preserving and Label-Aware Constraint-Based Hybrid Dictionary Learning for Image Classification

Abstract

Dictionary learning has been an important role in the success of data representation. As a complete view of data representation, hybrid dictionary learning (HDL) is still in its infant stage. In previous HDL approaches, the scheme of how to learn an effective hybrid dictionary for image classification has not been well addressed. In this paper, we proposed a locality preserving and label-aware constraint-based hybrid dictionary learning (LPLC-HDL) method, and apply it in image classification effectively. More specifically, the locality information of the data is preserved by using a graph Laplacian matrix based on the shared dictionary for learning the commonality representation, and a label-aware constraint with group regularization is imposed on the coding coefficients corresponding to the class-specific dictionary for learning the particularity representation. Moreover, all the introduced constraints in the proposed LPLC-HDL method are based on the $l_2$-norm regularization, which can be solved efficiently via employing an alternative optimization strategy. The extensive experiments on the benchmark image datasets demonstrate that our method is an improvement over previous competing methods on both the hand-crafted and deep features.

1. Introduction

Due to the insufficiency of data representation, Dictionary Learning (DL) has aroused considerable interest in the past decade and achieved much success in the various applications, such as image denoising [1,2], person re-identification [3,4] and vision recognition [5,6,7,8]. Generally speaking, the DL methods are developed based on a basic hypothesis, which are that a test signal can be well approximated using the linear combination of some atoms in a dictionary. Thus the dictionary usually plays an important role in the success of these applications. Traditionally, the DL methods can be roughly divided into two categories: The unsupervised DL methods and the supervised DL methods [9,10].

In the unsupervised DL methods, a dictionary is optimized to reconstruct all the training samples without any label assignment; hence, there is no class information in the learned dictionary. By further integrating the label information into dictionary learning, the supervised DL methods can achieve better classification performance than the unsupervised ones for image classification. Supervised dictionary learning encodes the input signals using the learned dictionary, then utilizes the representation coefficients or the residuals for classification. Thus the discriminative ability of the dictionary and the representative ability of the coding coefficients play the key roles in this kind of approach. According to the types of the dictionary, the supervised DL methods can be further divided into three categories: The class-shared DL methods, the class-specific DL methods and the hybrid DL methods [11].

The class-shared DL methods generally force the coding coefficients to be discriminative via learning a single dictionary shared by all the classes. Based on the K-SVD algorithm, Zhang and Li [5] proposed the Discriminative K-SVD (D-KSVD) method to construct a classification error term for learning a linear classifier. Jiang et al. [6] further proposed the Label Consistent K-SVD (LC-KSVD) which encourages the coding coefficients from the same class to be as similar as possible. Considering the characteristics of atoms, Song et al. [12] designed an indicator function to regularize the class-shared dictionary to improve the discriminative ability of coding coefficients. In general, the computation of the test stage is very efficient in the class-shared DL methods, but it is hard to improve the coefficients’ discriminativity for better classification performance as the class-shared dictionary is not enough for fitting the complex data.

In the class-specific dictionary learning, each sub-dictionary is assigned to a single class and the sub-dictionaries with the different classes are encouraged to be as independent as possible. As a representative class-specific DL method, Fisher Discrimination Dictionary Learning (FDDL) [13] employs Fisher discrimination criterion on the coding coefficients, then utilizes the representation residual of each class to establish the discriminative term. Using with the incoherence constraint, Ramirez et al. [14] proposed a structured dictionary learning scheme to promote the discriminative ability of the class-specific sub-dictionaries. Akhtar et al. [15] developed a Joint discriminative Bayesian Dictionary and Classifier learning (JBDC) model to associate the dictionary atoms by the class labels with Bernoulli distributions. The class-specific DL methods usually associate a dictionary atom to a single class directly; hence, the reconstruction error with respect to each class can be used for classification. However, the test stage of this category often requires the coefficient computation of test data over many sub-dictionaries.

In the hybrid dictionary learning, a dictionary is designed to have a set of class-shared atoms in addition to the class-specific sub-dictionaries. Wang and Kong [16] proposed a hybrid dictionary dubbed DL-COPAR to explicitly separate the common and particular features of the data, which also encourages the class-specific sub-dictionaries to be incoherent. Vu et al. [17] developed a Low-Rank Shared Dictionary Learning (LRSDL) method to preserve the common features of samples. Gao et al. [18] developed a Category-specific and Shared Dictionary Learning (CSDL) approach for fine-grained image classification. Wang et al. [19] designed a structured dictionary consisting of label-particular atoms corresponding to some class and shared atoms commonly used by all the classes, and introduced a Cross-Label Suppression for Dictionary Learning (CLSDL) to generate approximate sparse coding vectors for classification. To some extent, the hybrid dictionary is very effective at preserving the complex structure of the visual data. However, it is nontrivial to design the class-specific and shared dictionaries with the proper number of atoms, which often has a severe effect on classification performance.

In addition to utilizing the class label information, more and more supervised DL approaches have been proposed to incorporate the locality information of the data into the learned dictionary. By calculating the distances between the bases (atoms) and the training samples, Wang et al. [20] developed a Locality-constrained Linear Coding (LLC) model to select the k-nearest neighbor bases for coding, and set the coding coefficients of other atoms to zero. Wei et al. [21] proposed locality-sensitive dictionary learning to enhance the power of discrimination for sparse coding. Song et al. [22] integrated the locality constraints into the multi-layer discriminative dictionary to avoid the appearance of over-fitting. By coupling the locality reconstruction and the label reconstruction, the LCLE-DL method [7] ensures that the locality-based and label-based coding coefficients are as approximate to each other as possible. It is noted that the locality constraint in LCLE-DL may cause the dictionary atoms from the different classes to be similar, which weakens the discriminative ability of the learned dictionary.

It is observed that the real-world object categories are not only a marked difference, but are also strongly correlated in terms of the visual property, e.g., faces from the different persons often share similar illumination and pose variants; objects in the Caltech 101 dataset [23] have the correlated background. These correlations are not very helpful to distinguish the different categories, but without them the data with common features cannot be well represented. Thus, the dictionary learning approach should learn the distinctive features with the class-specific dictionary, and simultaneously exploit the common features of the correlated classes by learning a commonality dictionary. To this end, we proposed the locality preserving and label-aware constraint-based hybrid dictionary learning (LPLC-HDL) method for image classification, which is composed of a label-aware constraint, a group regularization and a locality constraint. The main contributions are summarized as follows.

[1].

The proposed LPLC-HDL method learns the hybrid dictionary by fully exploiting the locality information and the label information of the data. In this way, the learned hybrid dictionary can not only preserve the complex structural information of the data, but also have strong discriminativity for image classification.

[2].

In LPLC-HDL, a locality constraint is constructed to encourage the samples from different classes with similar features to have similar commonality representation; then, a label-aware constraint is integrated to make the class-specific dictionary sparsely represent the samples from the same class, so that the robust particularity–commonality representation can be obtained by the proposed LPLC-HDL.

[3].

In a departure from the competing methods which impose the $l_0$-norm or $l_1$-norm on the coefficients, LPLC-HDL consists of $l_2$-norm constraints that can be calculated efficiently. The objective function is solved elegantly by employing an alternative optimization technique.

The rest of this paper is outlined as follows. Section 2 reviews the related work on our LPLC-HDL method. Then Section 3 presents the details of LPLC-HDL and an effective optimization is introduced in Section 4. To verify the efficiency of our method for image classification, the experiments are conducted in Section 5. Finally, the conclusion is summarized in Section 6.

2. Notation and Background

In this section, we first provide the notation used in this paper, then review the LCLE-DL algorithm and the objective function of hybrid dictionary learning (DL), which can be taken as the theoretical background of our LPLC-HDL method.

2.1. Notation

Let $X=[X_1,\cdots ,X_C]\in {\mathbf{R}}^{m\times N}$ be a set of N training samples in an m dimension with class labels $Y_i\in [1,\cdots ,C]$; here, C is the class number of the training samples and $X_i$ is a matrix consisting of $N_i$ training samples of the ith class. Suppose $D_0\in {\mathbf{R}}^{m\times k_0}$ and $D_p=[D_1,\cdots ,D_C]\in {\mathbf{R}}^{m\times K_p}$ make up the learned hybrid dictionary $D=[D_0,D_p]$ from the training samples X, where $k_0$ is the atom number of shared dictionary, $K_p={\sum }_{i=1}^C{k}_i$ denotes the atom number of class-specific dictionary and $k_i$ is the atom number of ith class sub-dictionary. Let $Z=[Z_1,\cdots ,Z_C]\in {\mathbf{R}}^{(k_0+K_p)\times N}$ be the coding coefficients of training samples X over the hybrid dictionary D; then, $Z_0=[Z_0^1,\cdots ,Z_0^C]\in {\mathbf{R}}^{k_0\times N}$ and $Z_i=[Z_i^1,\cdots ,Z_i^C]\in {\mathbf{R}}^{k_i\times N}$ represent the coding coefficients over the shared dictionary and the ith class sub-dictionary, respectively.

According to [24], a row vector of coefficient matrix Z can be defined as a profile of the corresponding dictionary atom. Therefore, we can define a vector $z^r=[z_1^r;z_2^r;\cdots ;z_C^r]\in {\mathbf{R}}^{N\times 1}$$(r=1,\cdots ,K)$ as the profile of atom $d_r$ for all the training samples, where the sub-vector $z_c^r={[z_1^r,z_2^r,\cdots ,z_{N_c}^r]}^{\mathrm{T}}\in {\mathbf{R}}^{N_c\times 1}$ is the sub-profile for the training samples of the cth class.

2.2. The LCLE-DL Algorithm

To improve the classification performance, the LCLE-DL algorithm [7] takes both the locality and label information of dictionary atoms into account in the learning process. This algorithm firstly constructs the locality constraint to ensure that similar profiles have similar atoms, then establishes the label embedding constraint to encourage the atoms of the same class to have similar profiles. The objective function of LCLE-DL is defined as follows.

$$\begin{array}{cc}\hfill \underset{D_p,Z_p,V_p,L}{\min }& \Vert X-D_p{Z}_p{\Vert }_2^2+\alpha Tr\left(Z_p^{\mathrm{T}}L{Z}_p\right)+{\Vert X-D_p{V}_p\Vert }_2^2+\beta Tr\left(V_p^{\mathrm{T}}U{V}_p\right)\hfill \\ \hfill & +\gamma \Vert Z_p-V_p{\Vert }_2^2,s.t.{\Vert d_j\Vert }^2=1,j=1,\cdots ,K\hfill \end{array}$$

(1)

where $Z_p\in {\mathbf{R}}^{K_p\times N}$ and $V_p\in {\mathbf{R}}^{K_p\times N}$ denote the locality-based and the label-based coding coefficients, $L\in {\mathbf{R}}^{K_p\times K_p}$ is the graph Laplacian matrix that is constructed by the atom’s similarity in the dictionary $D_p\in {\mathbf{R}}^{m\times K_p}$, $U\in {\mathbf{R}}^{K_p\times K_p}$ is the scaled label matrix which is constructed using the label matrix of the dictionary $D_p$. $\Vert X-D_p{Z}_p{\Vert }_2^2$ combined with the second term encodes the reconstruction under the locality constraint; $\Vert X-D_p{V}_p{\Vert }_2^2$ combined with fourth term encodes the reconstruction under the label embedding; $\Vert Z_p-V_p{\Vert }_2^2$ is used to transfer the label constraint to the locality constraint. $\alpha ,\beta$ and $\gamma$ are the regularization parameters; the constraint on the atoms can avoid the scaling issue.

The LCLE-DL algorithm first exploits the K-SVD algorithm to learn sub-dictionaries $D_i(i=1,\cdots ,C)$ using with the training samples $X_i$. Similar to the label matrix of the training samples Y, the label matrix of the dictionary $D_p$ can be obtained as $B=[b_1,\cdots ,b_{K_p}]\in {\mathbf{R}}^{K_p\times C}$. Then a weighted label matrix is constructed by $G=B{\left(B^{\mathrm{T}}B\right)}^{-\frac{1}{2}}\in {\mathbf{R}}^{K_p\times C}$. Next, the label embedding of atoms is defined as

$$\underset{V_p}{\min }Tr\left(V_p^{\mathrm{T}}G{G}^{\mathrm{T}}{V}_p\right)=\underset{V_p}{\min }Tr\left(V_p^{\mathrm{T}}U{V}_p\right),$$

(2)

where $U=G{G}^{\mathrm{T}}\in {\mathbf{R}}^{K_p\times K_p}$ is the scaled label matrix of the dictionary $D_p$; the above terms make the coding coefficients $V_p$ have a block-diagonal structure with strong discriminative information.

The learned dictionary inherits the manifold structure of the data via using the derived-graph Laplacian matrix L, and the optimal representation of the samples can be obtained with the label embedding of dictionary atoms. By combining the double reconstructions, LCLE-DL ensures the label-based and the locality-based coding coefficients are as approximate to each other as possible. However, the locality constraint is imposed on the class-specific dictionary in the LCLE-DL algorithm, which may cause the dictionary atoms from the different classes to be similar, thus the discriminative ability of the dictionary is weakened.

2.3. The Objective Function of Hybrid DL

In recent years, the hybrid DL [16,19,25,26] has been getting more and more attention in the classification problem. The hybrid dictionary has been shown to perform better than the other types of dictionaries, as it can preserve both the class-specific and common information of the data. To learn such a dictionary, we can define the objective function of Hybrid DL as follows.

$$\begin{array}{cc}\hfill \underset{D,Z}{\min }& \sum _{i=1}^C{\Vert X_i-D_0{Z}_0^i-D_i{Z}_i^i\Vert }_F^2+{\lambda }_1\psi \left(D\right)+{\lambda }_2\varphi \left(Z\right)\hfill \\ \hfill & s.t.\Vert d_j{\Vert }^2=1,j=1,\cdots ,k_0,\cdots ,K\hfill \end{array}$$

(3)

where $D=[D_0,D_1,\cdots ,D_C]\in {\mathbf{R}}^{m\times K}$, $D_0$ is the dictionary shared by all the classes, $D_i(i=1,\cdots ,C)$ is the ith class sub-dictionary, $Z_0^i$ and $Z_i^i$ are the coding coefficients of the samples from the ith class over $D_0$ and $D_i$, and $\psi \left(D\right)$ and $\varphi \left(Z\right)$ denote the functions about the hybrid dictionary and the coding coefficients, respectively. The constraint of $\varphi \left(Z\right)$ typically adopts $l_1$ [26] norm or $l_{2,1}$ [25,27] norm for sparse coding.

3. The Proposed Method

By learning the shared dictionary $D_0$, the previous hybrid DL algorithms can capture the common features of the data, but they do not concern the correlation among these features, which will reduce the robustness of the learned dictionary. In this section, we firstly utilize the locality information of the atoms in $D_0$ to construct a locality constraint, then impose it on the coefficients $Z_0$, so that the correlation of the common features is captured explicitly and the learned dictionary $D_0$ is very robust concerning commonality representation. Moreover, once the correlation is discarded, the classification of a query sample will be dominated by the class-specific sub-dictionary corresponding to the correct class to reach minimized data fidelity.

To obtain the discriminative ability of the class-specific dictionary, we further introduce a label-aware constraint as well as group regularization on the distinctive coding coefficients for the particularity representation. Since this constraint integrated with the locality constraint to reconstruct the input data, they will reinforce each other in the learning process, which results in a discriminative hybrid dictionary for image classification.

Accordingly, the objective function of the proposed LPLC-HDL method can be formulated as follows.

$$\begin{array}{cc}\hfill \underset{D_0,L_0,Z_0,D_p,Z_p}{\min }\sum _{i=1}^C& \left\{\Vert X_i-D_0{Z}_0^i-D_p{Z}_p^i{\Vert }_F^2+\lambda {\Vert P_i{Z}_p^i\Vert }_F^2\right\}+\gamma ({\Vert Z_p\Vert }_F^2+\hfill \\ \hfill & Tr\left(Z_p{L}_p{Z}_p^{\mathrm{T}}\right))+\eta \left(\Vert Z_0{\Vert }_F^2+Tr\left(Z_0^{\mathrm{T}}{L}_0{Z}_0\right)\right),\hfill \\ \hfill & s.t.\Vert d_0^k{\Vert }_2^2=1,(k=1,2,\cdots ,k_0);{\Vert d_p^l\Vert }_2^2=1,(l=1,2,\cdots ,K_p),\hfill \end{array}$$

(4)

where $\lambda$, $\gamma$ and $\eta$ are the regularization parameters, which can adjust the weights of the label-aware constraint, the group regularization and the the locality constraint, respectively. Here we set the Euclidean length of all the atoms in the shared dictionary $D_0$ and the class-specific dictionary $D_p$ to be 1, which can avoid the scaling issue.

3.1. The Locality Constraint for Commonality Representation

Locality information of the data has played an important role in many real applications. By incorporating locality information for learning a dictionary, we can ensure that the samples with common features tend to have similar coding coefficients [7]. Further more, the dictionary atoms measure the similarity of the samples, which are more robust to the noise and outliers than the original samples. Hence, we use the atoms of the shared dictionary to capture the correlation among the common features and construct a locality constraint.

Based on the shared dictionary $D_0\in {\mathbf{R}}^{m\times k_0}$, we can construct a nearest neighbor graph $M_0\in {\mathbf{R}}^{k_0\times k_0}$ as follows.

$$M_0^{r,s}=\left\{\begin{array}{cc}\exp (-\frac{\Vert d_r-d_s{\Vert }_2}{\delta }),\hfill & \mathrm{if}{d}_r\in k\mathrm{NN}(d_s);\hfill \\ 0,\hfill & \mathrm{else},\hfill \end{array}\right.$$

(5)

where $\delta$ is a parameter to control the exponential function, kNN($d_s$) denotes k nearest neighbors of atom $d_s$, $M_0^{r,s}$ indicates the similarity between the atoms $d_r$ and $d_s$. For convenience of calculation, we invariably set the parameters $\delta =4$ and $k=1$, as they have the stable values in the experiments.

Once $M_0$ is calculated, we construct a graph Laplacian matrix $L_0$ as follows.

$$L_0=G_0-M_0,G_0=\mathrm{diag}(g_0^1,\cdots ,g_0^{k_0}),\mathrm{and}{g}_0^r=\sum _{s=1}^{k_0}{M}_0^{r,s}.$$

(6)

Since $L_0$ is constructed based on the dictionary $D_0$, it will be updated in coordination with $D_0$ in the learning process.

By now, we can obtain a locality constraint term for choosing graph Laplacian matrix $L_0$ as follows.

$$\frac{1}{2}\sum _{r=1}^{k_0}\sum _{s=1}^{k_0}{(z^r-z^s)}^2{M}_0^{r,s}=Tr\left(Z_0^{\mathrm{T}}{L}_0{Z}_0\right)=\sum _{i=1}^{C}Tr\left(Z_0^{i\mathrm{T}}{L}_0{Z}_0^i\right).$$

(7)

Because the profile $z^r/z^s$ and the atom $d^r/d^s$ have a one-to-one correspondence, the above equation ensures that similar atoms encourage similar profiles [7]. Hence, the correlated information of the common features can be inherited by the coefficient matrix $Z_0$ and the graph Laplacian matrix $L_0$.

3.2. The Constraints for Particularity Representation

3.2.1. The Label-Aware Constraint

To obtain the particularity representation for the classification, we will assign the labels to the atoms of the class-specific dictionary, as presented in [7,19]. If an atom $d^t\in D_i(i=1,\cdots ,C)$, the label i will be assigned to the atom $d^t$ and kept invariant in the iterations. We take ${\mathcal{S}}_i$ as the index set for the atoms of ith class sub-dictionary $D_i$, and $\mathcal{S}$ as the index set for all the atoms of the class-specific dictionary. For the particularity representation of samples $X_i$, it is desirable that the large coefficients should mainly occur on the atoms in ${\mathcal{S}}_i$. In other words, the sub-profiles associated with the atoms in $\mathcal{S}\backslash {\mathcal{S}}_i$ need to be suppressed to some extent.

For the ith class samples, we construct a matrix $P_i\in {\mathbf{R}}^{K\times K}$ to pick up the sub-profiles from the representation $Z_i^{\mathrm{T}}$, which locate at the atoms in $\mathcal{S}\backslash {\mathcal{S}}_i$ rather than the atoms in ${\mathcal{S}}_i$$(i=1,\cdots ,C)$, so that we can define a label-aware constraint term as follows.

$$\Vert Z_p^{i\mathrm{T}}{P}_i^{\mathrm{T}}{\Vert }_F^2={\Vert P_i{Z}_p^i\Vert }_F^2,$$

(8)

and the matrix $P_i$ can be written as

$$P_i(t_1,t_2)=\left\{\begin{array}{cc}1,\hfill & t_1=t_2\mathrm{and}{t}_2\in \mathcal{S}\backslash {\mathcal{S}}_i;\hfill \\ 0,\hfill & \mathrm{else},\hfill \end{array}\right.$$

(9)

where $P_i(t_1,t_2)$ is the $(t_1,t_2)$th entry of matrix $P_i$. For the particularity representation $Z_p^i$, minimizing the label-aware constraint can suppress the large values in the sub-profiles associated with the atoms in $\mathcal{S}\backslash {\mathcal{S}}_i$, as well as encourage the large ones in the sub-profiles associated with ith class atoms. Therefore, it is expected that this constraint with a proper scalar can make the particularity representation approximately sparse. Besides, once a series of matrices $P_i$$(i=1,\cdots ,C)$ are constructed, they will be unchanged in the iterations. Thus it is very efficient for coding over the class-specific dictionary.

3.2.2. The Group Regularization

Furthermore, to promote the consistency of particularity representation from the same class, we introduce the group regularization on $Z_p^i$. In light of the label information of training samples, assume one sample is related to one vertex; the vertices corresponding to the same class samples are connected and neighboring each other; thus, each class forms a densely connected sub-graph. Considering the training samples $X_i$ and their coding coefficients $Z_p^i$, we first define $k_i$ graph maps with mapping each graph to a line that consists of $N_i$ points, as follows

$$f_k^i=\left[\begin{array}{c}{z}_1^i\left(k\right)\\ z_2^i\left(k\right)\\ ⋮\\ z_{N_i}^i\left(k\right)\end{array}\right]\in {\mathbf{R}}^{N_i\times 1},\forall k=1,2,\cdots ,k_i;i=1,2,\cdots ,C.$$

(10)

Here $z_n^i\left(k\right)$$(n=1,2,\cdots ,N_i)$ is the kth component of $z_n^i$, which corresponds to the kth atom in the ith class sub-dictionary $D_i$.

Then, we can calculate the variation for these $k_i$ graph maps as follows.

$$V_i=\sum _{k=1}^{k_i}{f}_k^{i\mathrm{T}}{L}_i{f}_k^i=Tr\left(\left[\begin{array}{c}{f}_1^{i\mathrm{T}}\\ ⋮\\ f_{k_i}^{i\mathrm{T}}\end{array}\right]L_i[f_1^i,\cdots ,f_{k_i}^i]\right)=Tr\left(Z_p^i{L}_i{Z}_p^{i\mathrm{T}}\right),$$

(11)

where $L_i\in {\mathbf{R}}^{N_i\times N_i}$ denotes the normalized Laplacian of the overall graph for the ith class, which can be derived as

$$L_i=\frac{1}{N_i-1}\left[\begin{array}{cccc}{N}_i-1& -1& \cdots & -1\\ -1& N_i-1& \cdots & -1\\ ⋮& ⋮& ⋮& ⋮\\ -1& -1& \cdots & N_i-1\end{array}\right].$$

(12)

For the different classes, the vertices related to their samples should not be connected; thus, the graphs of C classes are isolated from each other. Therefore, we can obtain the total variation for $K_p$ graph maps of all the C classes as ${\sum }_{i=1}^{C}Tr\left(Z_p^i{L}_i{Z}_p^{i\mathrm{T}}\right)=Tr\left(Z_p{L}_p{Z}_p^{\mathrm{T}}\right)$, where $L_p=diag(L_1,\cdots ,L_C)$. Keeping this group regularization small will promote the consistency of the representation for the same class samples. Moreover, by combing it with the label-aware constraint, the coding coefficients for the different classes will be remarkably distinct with the large coefficients locating in the different areas, which is very favorable for the classification task.

4. Optimization Strategy and Classification

4.1. Optimization Strategy of LPLC-HDL

As the objective function in Equation (4) is not a jointly convex optimization problem for the variables ($D_0,L_0,Z_0,D_p,Z_p$), it will be divided into two sub-problems by learning the shared and class-specific dictionaries alternatively, that is, updating variables ($D_0,L_0,Z_0$) by fixing ($D_p,Z_p$), then updating variables ($D_p,Z_p$) by fixing ($D_0,L_0,Z_0$).

We firstly use k-means algorithm to initialize shared dictionary $D_0\in {\mathbf{R}}^{m\times k_0}$ by using all the training samples, then initialize ith class sub-dictionary $D_i\in {\mathbf{R}}^{m\times k_i}$ by using the training samples $X_i$ and concatenate all these sub-dictionaries as the class-specific dictionary $D_p\in {\mathbf{R}}^{m\times K_p}$. Next, we compute the initialized coefficients $Z_0$ and $Z_p$ with the Multi-Variate Ridge Regression (MVRR) algorithm, and obtain the initialized matrix $L_0$ by Equation (6). In line with the corresponding sub-dictionaries, the serial of matrices $P_i$ and $L_i$$(i=1,2,\cdots ,C)$ are constructed with Equations (9) and (12), respectively. After finishing the initialization, we can optimize the objective function as the following steps.

4.1.1. Shared Dictionary Learning

By fixing the variables ($D_p,Z_p$), the objective function of our LPLC-HDL for learning the shared dictionary becomes

$$\begin{array}{cc}\hfill \underset{D_0,L_0,Z_0}{\min }\sum _{i=1}^C& \Vert X_i^{\prime }-D_0{Z}_0^i{\Vert }_F^2+\eta \left(\Vert Z_0{\Vert }_F^2+Tr\left(Z_0^{\mathrm{T}}{L}_0{Z}_0\right)\right),\hfill \\ \hfill & s.t.\Vert d_0^k{\Vert }_2^2=1,(k=1,2,\cdots ,k_0),\hfill \end{array}$$

(13)

where $X_i^{\prime }=X_i-D_p{Z}_p^i$. To update the variables ($D_0,L_0,Z_0$), we turn to an iterative scheme, i.e., updating $D_0$ by fixing $(L_0,Z_0$), constructing $L_0$ based on $D_0$; updating $Z_0$ by fixing $(D_0,L_0)$. The detailed steps are elaborated as below.

(a) Update the shared dictionary $D_0$ and matrix $L_0$

Without loss of generality, we first concentrate on optimizing the shared dictionary $D_0$ by fixing the variables ($L_0,Z_0$). The function (13) for $D_0$ becomes

$$\begin{array}{c}\hfill \underset{D_0}{\min }\Vert X^{\prime }-D_0{Z}_0{\Vert }_F^2,s.t.{\Vert d_0^k\Vert }_2^2=1,(k=1,2,\cdots ,k_0),\end{array}$$

(14)

where $X^{\prime }={\sum }_{i=1}^C{X}_i^{\prime }$ and $Z_0={\sum }_{i=1}^C{Z}_0^i$. The quadratic constraint is introduced on each atom in $D_0$ to avoid the scaling issue risk. The above minimization can be solved by adopting the Lagrange dual algorithm presented as in [28].

Based on the obtained dictionary $D_0$, the graph Laplacian matrix $L_0$ can be constructed using (5) and (6).

(b) Update the commonality representation $Z_0$

By fixing the variables $(D_0,L_0)$, the optimization of the commonality representation $Z_0$ in Equation (13) can be formulated as

$$\begin{array}{c}\hfill \underset{Z_0}{\min }{\Vert X^{\prime }-D_0{Z}_0\Vert }_F^2+\eta \left(\Vert Z_0{\Vert }_F^2+Tr\left(Z_0^{\mathrm{T}}{L}_0{Z}_0\right)\right),\end{array}$$

(15)

where $X^{\prime }={\sum }_{i=1}^C{X}_i^{\prime }$ and $Z_0={\sum }_{i=1}^C{Z}_0^i$; there is a closed-form solution for $Z_0$ as all the above terms are based on the $l_2$ norm regularization.

By setting the derivative of (15) to zero, the optimal representation $Z_0$ can be obtained as $Z_0=\left(D_0^{\mathrm{T}}{D}_0+2\eta (L_0+I_0)\right)D_0^{\mathrm{T}}{X}^{\prime }$, where $I_0\in {\mathbf{R}}^{k_0\times k_0}$ is an identity matrix.

4.1.2. Class-Specific Dictionary Learning

Assuming the variables ($D_0,L_0,Z_0$) are fixed, the objective function (4) for learning the class-specific dictionary can be formulated as follows.

$$\begin{array}{cc}\hfill \underset{D_p,Z_p}{\min }\sum _{i=1}^C& \left\{\Vert X_i^{″}-D_p{Z}_p^i{\Vert }_F^2+\lambda {\Vert P_i{Z}_p^i\Vert }_F^2+\gamma \left(\Vert Z_p^i{\Vert }_F^2+Tr\left(Z_p^i{L}_i{Z}_p^{i\mathrm{T}}\right)\right)\right\},\hfill \\ \hfill & s.t.\Vert d_p^l{\Vert }_2^2=1,(l=1,2,\cdots ,K_p),\hfill \end{array}$$

(16)

where $X_i^{″}=X_i-D_0{Z}_0^i$. To update the variables ($D_p,Z_p$), we follow the iterative scheme presented in Section 4.1.1, i.e., updating $D_p$ by fixing $Z_p$; then updating $Z_p$ by fixing $D_p$. The updating steps are elaborated as below.

(c) Update the class-specific dictionary $D_p$

By fixing the variable $Z_p$, the optimization of dictionary $D_p$ can be simplified as the optimization of each atom $d_p^k$$(k=1,2,\cdots ,K_p)$. Provided $k\in {\mathcal{S}}_i$ and the other atoms in $D_p$ are fixed, to update the atom $d_p^k$ we can solve the optimization problem as follows.

$$\begin{array}{c}\hfill \underset{d_p^k}{\min }{\Vert X^{″}-\sum _{t\notin {\mathcal{S}}_i}{d}_p^t{\overline{z}}_p^t-\sum _{t\in {\mathcal{S}}_i,t\ne k}{d}_p^t{\overline{z}}_p^t-d_p^k{\overline{z}}_p^k\Vert }_F^2,\end{array}$$

(17)

where $X^{″}={\sum }_{i=1}^C{X}_i^{″}$, and ${\overline{z}}_p^t$ denotes the tth row vector of coefficient matrix $Z_p$. By introducing a variable $\tilde{E}=X^{″}-{\sum }_{t\notin {\mathcal{S}}_i}{d}_p^t{\overline{z}}_p^t-{\sum }_{t\in {\mathcal{S}}_i,t\ne k}{d}_p^t{\overline{z}}_p^t$, the solution of atom $d_p^k$ can be easily derived as

$${\tilde{d}}_p^k=\frac{1}{\Vert {\overline{z}}_p^t{\Vert }_2^2}\tilde{E}{\overline{z}}_p^{t\mathrm{T}}.$$

(18)

Consider the energy of each atom as being constrained in (16), the solution can be further normalized as

$${\widehat{d}}_p^k=\frac{\tilde{E}{\overline{z}}_p^{t\mathrm{T}}}{\Vert \tilde{E}{\overline{z}}_p^{t\mathrm{T}}{\Vert }_2}.$$

(19)

Because the atoms with indices out of ${\mathcal{S}}_i$ are fixed when updating the atoms with indices in ${\mathcal{S}}_i$, we can compute $E=X^{″}-{\sum }_{t\notin {\mathcal{S}}_i}{d}_p^t{\overline{z}}_p^t$ in advance and take it into the calculation of $\tilde{E}$ to accelerate the update. Likewise, we successively update the atoms corresponding to ${\mathcal{S}}_i$ with $(i=1,2,\cdots ,C)$ and obtain the overall class-specific dictionary $D_p$.

(d) Update the particularity representation $Z_p$

For the particularity representation of the ith class $Z_p^i$, the optimization problem depending on it becomes

$$\begin{array}{c}\hfill \underset{Z_p^i}{\min }\Vert X_i^{″}-D_p{Z}_p^i{\Vert }_F^2+\lambda {\Vert P_i{Z}_p^i\Vert }_F^2+\gamma \left(\Vert Z_p^i{\Vert }_F^2+Tr\left(Z_p^i{L}_i{Z}_p^{i\mathrm{T}}\right)\right).\end{array}$$

(20)

The solution of the above equation can be obtained by computing each code in $Z_p^i$ as follows

$$\begin{array}{cc}\hfill \underset{z_p^i\left(n\right)}{\min }& \Vert x_i^{″}\left(n\right)-D_p{z}_p^i{\left(n\right)\Vert }_2^2+\lambda \Vert P_i{z}_p^i{\left(n\right)\Vert }_2^2+\gamma {\Vert z_p^i\left(n\right)\Vert }_2^2\hfill \\ \hfill & +\gamma \sum _{n_1,n_2}{L}_i(n_1,n_2)z_p^i{\left(n_2\right)}^{\mathrm{T}}{z}_p^i\left(n_1\right),\hfill \end{array}$$

(21)

where $x_i^{″}\left(n\right)$$(n=1,2,\cdots ,N_i)$ is the nth training sample of the ith class, $z_p^i\left(n\right)$ is the code of $x_i^{″}\left(n\right)$ and $L_i(n_1,n_2)$ denotes the $(n_1,n_2)$th entry of matrix $L_i$. Since all the terms in Equation (21) are based on the quadratic form, the solution of $z_p^i\left(n\right)$ can be easily obtained by setting its derivation to zero.

By using (12) and denoting ${\overline{L}}_i$ as $L_i-I_i$ with the identity matrix $I_i\in {\mathbf{R}}^{N_i\times N_i}$, we can further obtain a matrix version for $Z_p^i$ update as:

$$\begin{array}{c}\hfill {\hat{Z}}_p^i={\left(D_p^{\mathrm{T}}{D}_p+\lambda P_i^{\mathrm{T}}{P}_i+2\gamma I_i\right)}^{-1}\left(D_p^{\mathrm{T}}{X}_i^{″}-\gamma Z_p^i{\overline{L}}_i\right).\end{array}$$

(22)

By repeating the steps (a)–(d), we can iteratively obtain all the optimal variables $(D_0,L_0,Z_0,D_p,Z_p)$. The overall optimization of our LPLC-HDL is summarized in Algorithm 1. Moreover, the objective function value of Equation (4) decreases as the number of iterations increases on a given dataset. For example, the convergence curve of LPLC-HDL on the Yale face dataset [29] is illustrated in Figure 1. From it we can see that the proposed algorithm converges quickly, no more than 20 iterations.

In Algorithm 1, the time complexity of our LPLC-HDL mainly comes from these parts: $O\left(k_0^{3}mN\right)$ for updating the shared dictionary $D_0$, $O\left(k_0^2\right)$ for updating the graph Laplacian matrix $L_0$, $O\left(k_0^3\right)$ for updating the commonality representation $Z_0$, $O\left(K_{p}mN\right)$ for updating the class-specific dictionary $D_p$ and $O\left(K_p^3\right)$ for updating the particularity representation $Z_p$. Thus the final time complexity is $O((k_0^3+K_p)mN+K_p^3)$ in each iteration.

Algorithm 1 Optimization procedure of the proposed LPLC-HDL

1: Input: Train samples X, class label Y, the parameters $\lambda$, $\gamma$, $\eta$   2: Initialization: Using the k-means algorithm, initialize $D_0$ by all the training samples, compute the matrix $L_0$ with Equation (6); Initialize $D_i$$(i=1,2,\cdots ,C)$ by the training samples $X_i$, concatenate these sub-dictionaries as the initialized $D_p$; Initialize the coefficients $Z_0$ and $Z_p$ with the MVRR algorithm; Construct the serial of matrices $P_i$ and $L_i$ with Equations (9) and (12), respectively.   3: while converge condition no satisfied do   4:     (a) Update $D_0$ by solving Equation (14), update $L_0$ by using Equations (5) and (6);   5:     (b) Update $Z_0=\left(D_0^{\mathrm{T}}{D}_0+2\eta (L_0+I_0)\right)D_0^{\mathrm{T}}{X}^{\prime }$;   6:     (c) Update $D_p$ as follows:   7:     for $i=1$ to C do   8:         Compute $E=X^{″}-{\sum }_{t\notin {\mathcal{S}}_i}{d}_p^t{\overline{z}}_p^t$;   9:         for $k\in {\mathcal{S}}_i$ do 10:              Update the atom $d_p^k$ by solving Equation (19); 11:       end for 12:     end for 13:     (d) Update $Z_p$ as follows: 14:     for $i=1$ to C do 15:       Update the coding coefficients $Z_p^i$ by solving Equation (22); 16:     end for 17: end while 18: Output: The variables $D_0,L_0,Z_0,D_p,Z_p$.

4.2. Classification Procedure

After completing Algorithm 1, we can use the learned hybrid dictionary $D=[D_0,D_p]$ to represent a test sample and predict its label. Firstly, we find the commonality–particularity representation ${\widehat{z}}_t={[{\left(z_t^0\right)}^{\mathrm{T}},{\left(z_t^p\right)}^{\mathrm{T}}]}^{\mathrm{T}}$ of a test sample $x_t$ by solving the following equation:

$$\begin{array}{c}\hfill {\widehat{z}}_t=\arg \underset{z_t}{\min }\Vert x_t-D{z}_t{\Vert }_2^2+\gamma {\Vert z_t\Vert }_2^2.\end{array}$$

(23)

Secondly, we compute $x_t^{\prime }=x_t-D_0{z}_t^0$ based on ${\widehat{z}}_t$ to exclude the contribution of the shared dictionary. Thus, the class label of $x_t$ can be determined by:

$$\begin{array}{c}\hfill \mathrm{label}\left(x_t\right)=\arg \underset{i}{\min }{\Vert x_t^{\prime }-D_i{z}_t^i\Vert }_2^2,i=1,2,\cdots ,C.\end{array}$$

(24)

5. Experiments

In this section, we compare LPLC-HDL with the representative dictionary learning methods including D-KSVD [5], LC-KSVD [6], LCLE-DL [7], FDDL [13], DL-COPAR [16] and CLSDDL [19] on the Yale face dataset [29], the Extended YaleB face dataset [30], the Labeled Faces in the Wild (LFW) dataset [31] for face recognition, the Caltech-101 object dataset [23] and the Oxford 102 Flowers dataset [32] for object classification and flower classification, respectively. Moreover, we further compare it with the Sparse Representation based Classification (SRC) [33], the Collaborative Representation based Classifier (CRC) [34], the Probabilistic CRC (ProCRC) [35], the Sparsity Augmented Collaborative Representation (SACR) [36] and some other state-of-the-art methods on the particular datasets.

In the proposed LPLC-HDL method, the three parameters $\lambda ,\gamma$ and $\eta$ are selected by five-fold cross validation on the training set, and their optimal values on each dataset are shown in

Table 1. The parameters of the proposed LPLC-HDL method on each dataset.

	Yale	Ext. YaleB	LFW	Caltech-101	102 Flowers
$\lambda$	1 × ${10}^{-1}$	1 × ${10}^{-1}$	1 × ${10}^{-2}$	3	2 × ${10}^{-1}$
$\gamma$	1 × ${10}^4$	2 × ${10}^3$	5 × ${10}^3$	3 × ${10}^2$	1 × ${10}^4$
$\eta$	1 × ${10}^{-3}$	1 × ${10}^{-3}$	1 × ${10}^{-3}$	1 × ${10}^1$	1 × ${10}^{-1}$

. In addition, the atom numbers of the shared dictionary and the class-specific dictionary for each dataset are elaborated as detailed in the following experiments.

5.1. Experiments on the Yale Face Dataset

In the experiments, we first consider the Yale face dataset which contains 165 gray scale images for 15 individuals with 11 images per category. Each individual contains one different facial expression or configuration: Left-light, center-light, right-light, w/glasses, w/no glasses, happy, sad, normal, surprised, sleepy and wink, as shown in Figure 2a. Each image has $24\times 24$ pixel resolution, and is resized to 576-dimensional vectors with normalization for representation. Following the setting in [19], six images of each individual are randomly selected for training and the rest are used for testing. In addition, the number of dictionary atoms is set to $15\times 4$ + 60 = 120, which means four class-specific atoms for each individual and 60 shared atoms for all the individuals. The parameters of our LPLC-HDL on the Yale face dataset can be seen in Table 1.

To acquire a stable recognition accuracy, we operate LPLC-HDL over 30 times rather than 10 times with independent training/testing splits. The comparison results on the Yale face dataset are listed in

Table 2. The recognition rates (%) on the Yale face dataset.

Method	Acc. (%)	Method	Acc. (%)
LLC [37]	82.1 ± 2.6	LC-KSVD (120) [6]	83.6 ± 2.7
SVM [38]	94.4 ± 2.8	LCLE-DL (120) [7]	91.2 ± 3.1
SRC [33]	81.3 ± 2.5	DL-COPAR (120) [16]	92.3 ± 3.7
CRC [34]	89.8 ± 1.9	LRSDL (120) [17]	92.7 ± 3.3
ProCRC [35]	91.7 ± 2.1	CLSDDL-LC (120) [19]	95.9 ± 2.2
D-KSVD (120) [5]	82.3 ± 4.5	CLSDDL-GC (120) [19]	95.3 ± 2.9
FDDL (120) [13]	89.6 ± 2.6	LPLC-HDL (120)	97.01 ± 2.6

. Since the variations in terms of facial expression are complex, the variations in the testing samples cannot be well represented via directly using the training data. Hence SRC has worse accuracy than the other dictionary learning methods. We can also see that the hybrid dictionary learning methods including DL-COPAR [16], LRSDL [17] and CLSDDL [19] outperform the remaining competing approaches, and the proposed LPLC-HDL method achieves the best recognition accuracy of 97.01%, which illustrates that our method can distinguish the shared and class-specific information of face images more appropriately.

We further illustrate the performance of LPLC-HDL on face recognition accuracy with different sizes of the shared and class-specific dictionaries, as shown in Figure 3. From it we can see that a higher recognition accuracy on the Yale face dataset can be obtained by increasing the atom numbers of both the shared and class-specific dictionaries. Besides, Figure 3 also shows that when the number of the class-specific atoms is few, the recognition accuracy is sensitive to the number of the shared atoms, as the large shared dictionary is harmful to the discriminative power of the class-specific dictionary in this case. However, increasing the number of the class-specific atoms beyond the particular number of training samples of each class brings no notable increase in recognition accuracy. Considering that the large dictionary will slow down both the training time and the testing time, the atom numbers of the shared and class-specific dictionaries are not set to big values in the experiments.

5.2. Experiments on the Extended YaleB Face Dataset

The Extended YaleB face dataset contains 2414 frontal face images of 38 people. For each person, there are about 64 face images and the original images have $192\times 168$ pixels. This dataset is challenging due to varying poses and illumination conditions, as displayed in Figure 2b. For comparison, we use the normalized $32\times 32$ images instead of the original pixel information. In addition, we randomly select 20 images per category for training and take the rest as the testing images. The parameters of LPLC-HDL on the Extended YaleB face dataset are also shown in Table 1. The hybrid dictionary size is set to $38\times 13$ + 76 = 570, which denotes 13 class-specific atoms for each person and 76 shared atoms for all the persons, with the same structure adopted by the other hybrid dictionary learning methods.

We repeatedly run LPLC-HDL and all the comparison methods 10 times for reliable accuracy, and the average recognition rates are listed in

Table 3. The recognition rates (%) on the Extended YaleB face dataset.

Method	Acc. (%)	Method	Acc. (%)
LLC [37]	88.9 ± 0.57	LC-KSVD (570) [6]	92.7 ± 0.62
SVM [38]	92.5 ± 0.82	LCLE-DL (570) [7]	95.8 ± 0.42
SRC [33]	95.3 ± 0.65	DL-COPAR (570) [16]	95.8 ± 0.88
CRC [34]	95.0 ± 0.48	LRSDL (570) [17]	95.7 ± 0.51
K-SVD (570) [39]	94.0 ± 0.71	CLSDDL-LC (570) [19]	96.05 ± 0.56
D-KSVD (570) [5]	94.3 ± 0.63	CLSDDL-GC (570) [19]	96.45 ± 0.52
FDDL (570) [13]	93.4 ± 0.76	LPLC-HDL (570)	97.25 ± 0.43

. As shown in Table 3, compared with the K-SVD, D-KSVD, FDDL and LC-KSVD methods, LCLE-DL achieves a better recognition result with the same dictionary size. The reason for this behavior is that the LCLE-DL method can effectively utilize the locality and label information of the atoms in the dictionary learning. It is also shown that the hybrid dictionary learning methods, including DL-COPAR, LRSDL, CLSDDL and LPLC-HDL, generally outperform the other DL methods, which demonstrates the discriminative ability of the hybrid dictionary. By integrating the locality and label information into the hybrid dictionary, our LPLC-HDL method obtains the best recognition rate of 97.25%, which outperforms the second best approach CLSDDL-GC by 0.8%, and at least 1.2% higher than the other competing methods.

5.3. Experiments on the LFW Face Dataset

The LFW face dataset has more than 13,000 images with the name of the person pictured, and all of them are collected from the web for unconstrained face recognition and verification. Following the prior work [7], we use a subset of the LFW face dataset which consists of 1215 images from 86 persons. In this subset, there are around 11–20 images for each person, and all the images are resized to be $32\times 32$. Some samples from this face dataset are shown in Figure 2c. For each person, eight samples are randomly selected for training and the remaining samples are taken for testing. The parameters of LPLC-HDL on the LFW face dataset are also shown in Table 1. In addition, the hybrid dictionary size is set to $86\times 4$ + 86 = 430, which means four class-specific atoms for each individual and 86 shared atoms for all the individuals.

We repeatedly run LPLC-HDL and the comparison methods 10 times, the average recognition rates are reported in

Table 4. The recognition rates (%) on the LFW face dataset.

Method	Acc. (%)	Method	Acc. (%)
LLC [37]	34.6 ± 0.88	LC-KSVD (430) [6]	34.1 ± 1.32
SVM [38]	37.73 ± 1.5	LCLE-DL (430) [7]	39.1 ± 1.19
SRC [33]	36.5 ± 1.92	DL-COPAR (430) [16]	38.63 ± 1.6
CRC [34]	36.7 ± 1.55	LRSDL (430) [17]	40.23 ± 1.4
K-SVD (430) [39]	32.8 ± 1.27	CLSDDL-LC (430) [19]	41.05 ± 1.5
D-KSVD (430) [5]	33.2 ± 1.25	CLSDDL-GC (430) [19]	38.45 ± 1.8
FDDL (430) [13]	34.8 ± 1.76	LPLC-HDL (430)	42.39 ± 1.6

, where the symbol ± denotes the standard deviation of average recognition rates. Similar to the results on the Extended YaleB face dataset, the LCLE-DL method achieves a higher recognition rate than the other shared or class-specific DL methods, and the hybrid dictionary learning methods, e.g., DL-COPAR, LRSDL, CLSDDL and LPLC-HDL, have a super performance in general. The proposed LPLC-HDL method obtains the best result with an average recognition rate of 42.39%, which is significantly better than all the comparison methods.

5.4. Object Classification

In this subsection, we evaluate LPLC-HDL on the Caltech-101 dataset [23] for object classification. This dataset contains a total of 9146 images from 101 object categories and a background category. The number of images for per category varies from a minimum of 31 to a maximum of 800 images. The resolution of each image is about $300\times 200$, as shown in Figure 4. Following the settings [6,40], we perform the Spatial Pyramid Features (SPFs) on this dataset. Firstly, we partition each image into $2^L\times 2^L$ sub-regions with different spatial scales L = 0, 1, 2, then extract SIFT descriptors over a sub-region with a spacing of 8 pixels and concatenate them as the SPFs. Next, we encode the SPFs with a codebook of size 1024. Finally, the dimension of the features is reduced to 3000 using the the Principal Component Analysis (PCA) algorithm.

For this dataset, 10 samples of each category are selected as the training data and the remaining are for testing. The parameters of LPLC-HDL on this dataset can be seen in Table 1. The hybrid dictionary size is set to $102\times 9$ + 100 = 1018, which denotes nine class-specific atoms for each category and 100 shared atoms for all the categories. The proposed LPLC-HDL and the comparison methods are carried out 10 times, the average classification rates are reported in

Table 5. The classification results (%) on the Caltech101 object dataset.

Method	Acc. (%)	Method	Acc. (%)
SVM [38]	59.1 ± 0.80	LC-KSVD (1020) [6]	63.1 ± 0.45
LLC [37]	59.8 ± 0.61	LCLE-DL (1020) [7]	62.7 ± 0.31
SRC [33]	60.1 ± 0.51	JBDC (1020) [15]	61.3 ± 0.35
CRC [34]	59.4 ± 0.48	DL-COPAR (1018) [16]	63.0 ± 0.57
SACR [36]	60.7 ± 0.40	LRSDL (1018) [17]	63.5 ± 0.55
K-SVD (1020) [39]	59.1 ± 0.65	CLSDDL-LC (1018) [19]	63.7 ± 0.48
D-KSVD (1020) [5]	59.5 ± 0.87	CLSDDL-GC (1018) [19]	64.0 ± 0.43
FDDL (1020) [13]	63.6 ± 0.59	LPLC-HDL (1018)	65.3 ± 0.38

. As can be seen in Table 5, our LPLC-HDL method achieves the best classification result again, with improvement margins of at least 1.3% compared with the comparison methods.

5.5. Flower Classification

We finally evaluate the proposed LPLC-HDL on the Oxford 102 Flowers dataset [32] for fine-grained image classification, which consists of 8189 images from 102 categories, and each category contains at least 40 images. This dataset is very challenging because there exist large variations within the same category but small differences across several categories. The flowers appear at different scales, poses and lighting conditions; some flower images are shown in Figure 5. For each category, 10 images are used for training, 10 for validation, and the rest for testing, as in [32]. For ease of comparison, we take the convolutional neural network (CNN) features provided by Cai et al. [35] as the image-level features.

The parameters of LPLC-HDL on the Oxford 102 Flowers dataset can be seen in Table 1. The size of hybrid dictionary is set to be $102\times 9$ + 100 = 1018, which means nine atoms for each category and 100 atoms as the shared amount. We compare LPLC-HDL with different kinds of representative methods for flower classification, such as the basic classifiers (Softmax and linear SVM [38]), the recent ProCRC [35], the related DDL methods and the deep learning models. Following the common measurement [41,42], we evaluate the comparison methods by the average classification accuracy of all the categories, and the results are reported in

Table 6. The classification results (%) on the Oxford 102 Flowers dataset.

Method	Acc. (%)	Method	Acc. (%)
Softmax	87.3	LC-KSVD (1020) [6]	93.60
SVM [38]	90.9	DL-COPAR (1025) [16]	91.62
SRC [33]	93.2	JBDC (1002) [15]	93.12
NSC [48]	90.10	CLSDDL-LC (1018) [19]	94.01
CRC [34]	93.0	CLSDDL-GC (1018) [19]	94.47
ProCRC [35]	94.8	OverFeat [41]	86.8
K-SVD (1020) [39]	92.41	NAC [49]	95.3
D-KSVD (1020) [5]	93.55	DAS [50]	80.66
FDDL (1020) [13]	94.60	LPLC-HDL (1018)	95.7

. From it we can see that, taking the CNN features as the image-level features, the proposed LPLC-HDL method outperforms the basic classifiers and the competing DL methods. Compared with the other kinds of methods, LPLC-HDL is greatly superior to SparBases [43] and SMP [44]. Moreover, our method outperforms the recent deep learning models including GoogLeNet-GAP [45], ASPD [46] and AugHS [47], which need to design the special CNN architectures for flower classification.

5.6. Parameter Sensitivity

In the proposed LPLC-HDL method, there are three key parameters, i.e., $\lambda$, $\gamma$ and $\eta$, which are used to balance the importance of the label-aware constraint, the group regularization and the locality constraint. To analyze the sensitivity of the parameters, we define a candidate set $\{{10}^{-5}$, ${10}^{-4}$, ${10}^{-3}$, ${10}^{-2}$, ${10}^{-1}$, ${10}^0$, ${10}^1$, ${10}^2$, ${10}^3$, ${10}^4$, ${10}^5\}$ for them and then perform LPLC-HDL with different combinations of the parameters on the Yale face dataset and the Caltech 101 dataset. By fixing the parameter $\eta$, the classification accuracy versus different values of the parameters $\lambda$ and $\gamma$ are shown in Figure 6a and Figure 7a. As can be seen in the figures, the best classification result can be obtained when the parameters $\lambda$ and $\gamma$ locate in a feasible range. When $\gamma$ is very small, the effect of the group regularization is limited, leading to weak discrimination of the class-specific dictionary. On the other hand, when $\gamma$ becomes very large, the classification accuracy drops as the remaining terms in (4) become less important, which decreases the representation ability of the hybrid dictionary.

By fixing the parameters $\lambda$ and $\gamma$, the classification accuracy versus different values of the parameter $\eta$ are shown in Figure 6b and Figure 7b. From them we can see that the classification accuracy is insensitive to the parameter $\eta$ when its value located in a certain range, e.g., ${10}^{-5}\le \eta \le {10}^{-1}$ on the Yale face dataset. It should be noted that the incoherence between the shared and class-specific dictionaries is increased with increasing parameter $\eta$, which influences the reconstruction of the test data and decreases the classification accuracy.

Due to the diversity of the datasets, it is still an open problem to adaptively select the optimal parameters for the different datasets. In the experiments, we use an effective and simple way to find the optimal values for the parameters $\lambda$, $\gamma$ and $\eta$. Based on the previous analysis, we first fix the parameter $\eta$ to a small value such as 0.01, then search the candidate combination of the parameters $\lambda$ and $\beta$ from the coarse set of $\{{10}^{-5}$, ${10}^{-4}$, ${10}^{-3}$, ${10}^{-2}$, ${10}^{-1}$, ${10}^0$, ${10}^1$, ${10}^2$, ${10}^3$, ${10}^4$, ${10}^5\}$. According to the best coarse combination of them, we can further define a fine candidate set where their optimal values may exist. Then we perform the proposed LPLC-HDL again with different combinations of the parameters $\lambda$ and $\beta$ selected from the fine candidate set. This way, we can obtain the optimal values of the parameters for all the experimental datasets: hence, the best classification results are guaranteed.

Besides the key parameters $\lambda$, $\gamma$ and $\eta$ in our LPLC-HDL method, there are also the parameters $\delta$ and k in the proposed locality constraint. In the experiments, we find that these two parameters have stable values on the experimental datasets. For example, Figure 8a,b show the classification accuracies of our LPLC-HDL method with respect to the parameter $\delta$ by fixing the remaining parameters on the Yale face dataset and the Caltech 101 dataset. From the subfigures, we can see that the classification accuracy is insensitive to the parameter $\delta$, and the approximate best result can be obtained when $\delta =4$. For the parameter k, this similar phenomenon can be observed on the experimental datasets. Thus we can set $\delta =4$ and $k=1$ in our LPLC-HDL method for convenience of calculation.

5.7. Evaluation of Computational Time

We also conducted experiments to evaluate the running time of the proposed LPLC-HDL and other representative DL methods on the two face datasets and the Caltech-101 dataset, the comparison results are listed in

Table 7. The computational times (second) on the experimental datasets.

	Ext. YaleB		LFW		Caltech-101
	Train	Test	Train	Test	Train	Test
D-KSVD [5]	46.7	3.36 × ${10}^{-4}$	57.3	1.70 × ${10}^{-3}$	303.4	2.89 × ${10}^{-3}$
LC-KSVD [6]	47.8	3.37 × ${10}^{-4}$	59.1	1.57 × ${10}^{-3}$	311.2	3.12 × ${10}^{-3}$
FDDL [13]	455.8	9.40 × ${10}^{-2}$	558.3	2.90 × ${10}^{-1}$	3537.0	1.59
DL-COPAR [16]	8.54	1.01 × ${10}^{-2}$	38.9	6.21 × ${10}^{-2}$	120.5	1.12 × ${10}^{-1}$
CLSDDL-LC [19]	1.26	1.53 × ${10}^{-4}$	1.42	9.18 × ${10}^{-4}$	21.37	5.10 × ${10}^{-3}$
LPLC-HDL	3.62	1.50×${\mathbf{10}}^{-\mathbf{4}}$	3.82	6.25×${\mathbf{10}}^{-\mathbf{4}}$	22.19	3.80×${\mathbf{10}}^{-\mathbf{3}}$

. “Train” denotes the running time of each iteration, and “Test” is the average processing time for classifying one test sample. All the experiments are conducted on a 64-bit computer with Intel i7-7700 3.6 GHz CPU and 12 GB RAM under the MATLAB R2019b programming environment. From Table 7, we can see that, although slower than CLSDDL-LC [19], the training efficiency of LPLC-HDL is obviously higher than that of D-KSVD [5], LC-KSVD [6], FDDL [13] and DL-COPAR [16]. In the testing stage, the proposed LPLC-HDL has similar testing efficiency as D-KSVD and LC-KSVD, and the testing process of LPLC-HDL is much faster than that of FDDL and DL-COPAR. Specifically, the average time for classifying a test image by LPLC-HDL is always less than that of CLSDDL-LC on the experiments.

6. Conclusions

In this paper, we propose a novel hybrid dictionary learning (LPLC-HDL) method by taking advantage of the locality and label information of the data, which can solve the image classification task effectively. The LPLC-HDL method incorporates the locality constraint with the label-aware constraint to more appropriately distinguish the shared and particularity information. More specifically, the locality constraint on the shared dictionary is used to model the similar features of the images from different classes; the label-aware constraint and the group regularization are coupled to make the class-specific dictionary more discriminative. An effective alternative strategy is developed to solve the objective function of LPLC-HDL. After training the hybrid dictionary, the class label of the test image can be predicted more accurately by excluding the contribution of the shared dictionary. The experimental results on three face datasets, the object dataset and the flower dataset demonstrate that our method has effectiveness and superiority for image classification.