Fusion and Enhancement of Consensus Matrix for Multi-View Subspace Clustering

Abstract

Multi-view subspace clustering is an effective method that has been successfully applied to many applications and has attracted the attention of scholars. Existing multi-view subspace clustering seeks to learn multiple representations from different views, then gets a consistent matrix. Until now, most of the existing efforts only consider the multi-view information and ignore the feature concatenation. It may fail to explore their high correlation. Consequently, this paper proposes a multi-view subspace clustering algorithm with a novel consensus matrix construction strategy. It learns a consensus matrix by fusing the different information from multiple views and is enhanced by the information contained in the original feature direct linkage of the data. The error matrix of the feature concatenation data is reconstructed by regularization constraints and the sparse structure of the multi-view subspace. The feature concatenation data are simultaneously used to fuse the individual views and learn the consensus matrix. Finally, the data is clustered by using spectral clustering according to the consensus matrix. We compare the proposed algorithm with its counterparts on six datasets. Experimental results verify the effectiveness of the proposed algorithm.

1. Introduction

As a classical machine learning method, clustering has been an enduring area of research. A major reason is that it can be used as a data processing method independently and as a preliminary step in fields such as computer vision [1,2] and data mining [3,4]. Therefore, many researchers have focused on improving clustering algorithms, and further improvements of classical algorithms such as K-means and spectral clustering [5,6] have been made. Traditional clustering methods only use a single set of features. However, with the continuous development of computer technology, data acquisition can be conducted through different channels and methods, and the data can be processed through different methods. For example, data can be obtained by different feature extraction methods such as the Gabor filter [7] and local binary pattern methods (LBP) [8] to represent the same subject in computer vision. Multi-view clustering classifies similar subjects into the same group by combining the available multi-view feature information [9]. Because the information from different views is often compatible and complementary, it is possible to obtain better clustering performances using multi-view rather than one-view features.

Multi-view clustering has been widely used in data mining. For example, a multi-view recurrent neural network (MV-RNN) [10] is proposed for sequential recommendation and alleviating the item cold start problem. A knowledge transfer strategy [11,12] based on multi-view is introduced to handle the incomplete multi-modal visual data. A multi-view feature selection method [13] is presented to leverage the knowledge of all views and to guide the feature selection process in an individual view. Inspired by this, multi-view clustering has received much attention from scholars [14]. The development of multi-view clustering algorithms has gone through several stages, and the earliest was through single-view clustering algorithms such as K-means [15,16], and spectral clustering [17] to achieve the classification of categories. Then with the development of time, multi-view clustering algorithms based on collaborative training [18,19] and kernel methods [20,21] gradually emerged. In recent years, multi-view learning has been extensively studied, and many multi-view clustering algorithms have been proposed [9,15]. These methods can be broadly divided into the following two categories: model-based and similarity-based approaches.

Model-based approaches cooperatively train a generative model to represent the data. Each model represents one cluster [22]. For example, Kumar et al. [18] proposed a multi-view graph clustering algorithm based on collaborative training to obtain common clustering results by crossing views. Wang et al. [23] processed the missing multi-view data by view encoding and adversarial generative networks (GAN) before implementing clustering through clustering networks. In [24], a shared subspace was learned from different views to explicitly account for the individual and shared patterns hidden in different views. A multi-view graph clustering [25] algorithm was proposed to explore the topological manifold structure from multiple adaptive graphs. The topological relevance across multiple views can be explicitly detected. Wen et al. [26] proposed a unified embedding alignment framework to reconstruct the missing views simultaneously and learn the common representation of multiple views for incomplete multi-view clustering. An autoencoder in autoencoder networks (AE2-Nets) [27] was introduced to automatically encode intrinsic information from heterogeneous views into a comprehensive representation. It can adaptively balance the complementarity and consistency among different views. The choice of model and the sharing mechanism of model parameters significantly impact the algorithm’s performance in model-based approaches, especially when the distribution of different views is very different.

Up to now, researchers have proposed many similarity-based approaches to multi-view subspace clustering. We further divide them into three classes: matrix/tensor decomposition [28,29], graph learning [30], and subspace learning [31,32]. Until today, subspace and matrix decomposition [33] are still popular methods in multi-view learning. For example, Wang et al. [34] proposed a multi-view tensor based on third-order tensor decomposition to learn the appropriate matrix representation by constructing the tensor and performing different slices. Graph learning methods find a fusion graph across the input graphs of all views first and then employ an additional clustering algorithm on this fusion graph to produce the final clusters. For example, Chen et al. [7] proposed a low-rank tensor graph (LRTG) learning that simultaneously learns the representation and affinity matrix in a single step to preserve their correlation. Wang et al. [35] proposed graph-based multi-view clustering to help the learning of each view graph matrix and unified graph matrix by mutual reinforcement. An adaptive sample-level graph combination for partial multi-view clustering (ASGC-PMVC) [36] was presented to cluster partial multi-view samples by automatically adjusting the contributions of different views.

Multi-view subspace clustering has been successfully used in numerous applications, which leverages the complementary information from multiple views to benefit clustering. Among them, the most representative is the self-representing-based multi-view clustering algorithm. Zhang et al. [37] proposed a novel multi-view subspace clustering model (LMSC), which explores complementary information from multiple views and simultaneously seeks the underlying latent representation. Jia et al. [38] designed a novel structured tensor low-rank norm tailored to multi-view spectral clustering. It imposes a symmetric low-rank constraint and a structured sparse low-rank constraint. Brbic et al. [39] introduced subspace learning based on the sparse and low-rank structure of the subspace by regularization constraints to achieve both sparse and low-rank structures. Zheng et al. [40] proposed feature joint subspace multi-view clustering to improve the clustering results by mining the consensus information of multiple views. A multi-view low-rank sparse subspace clustering (MLRSSC) [39] algorithm was proposed to learn a joint subspace representation by constructing an affinity matrix shared among all views.

The above-mentioned multi-view subspace clustering algorithms fuse the individual views to get the consensus matrix. A consensus matrix is used to integrate the complementary and exclusivity information between multiple views. It generally ignores the overall data containing information under the joint view representation of multi-view data, which may be equally complementary and different from the original views. In addition, we must consider the relationship between the consensus matrix and the joint view. This paper proposes a fusion and enhancement strategy for the consensus matrix of multi-view subspace clustering to address these problems. The algorithm not only fuses the information of the joint view representation of multi-view data to enhance the information utilization but also extends the learning of the consensus matrix to the fused view subspace to ensure the differences and consensus between the original views. Moreover, we design an optimization algorithm based on the augmented lagrangian multiplier (ALM) to optimize the proposed method. We compare the proposed algorithm with its counterparts on six datasets. Comprehensive experiments show the superiority of the proposed algorithm. In summary, the main contributions of this paper can be delivered as follows:

• We propose a self-representing-based multi-view subspace clustering. The consensus matrix is constructed on both the individual and the joint views to leverage the consensus and complementary information.
• We introduce 2,1-norm to explore the structural sparsity of the multi-view subspace for benefitting the clustering performance.
• Comprehensive experimental results compared the proposed algorithm with its counterparts demonstrate the advantage of the proposed method.

The rest of this paper is organized as follows. Section 2 presents some relevant premises. Section 3 presents the main improvements achieved by the proposed algorithm and the optimization of the formulation. Section 4 compares the proposed algorithm with several existing algorithms through experiments and discusses the results. Finally, a summary is presented in Section 5.

2. Related Work

2.1. Notation

We first introduce the notation involved in this paper. The original data matrix is denoted by $\left\{X^1,X^2,\cdots ,X^v\right\}$, $X^i\in R^{d_i\times n}$ is the data matrix of the ith view, where n is the number of subjects and $d_i$ is the number of features in the ith view. $X=[X^1,X^2,\cdots ,X^v]\in R^{d\times n}$ is the joint data matrix and $d={\sum }_{i=1}^v{d}_i$. $V\in R^{n\times n}$ is the self-representation matrix of X in the sparse clustering or the consensus matrix in the multi-view clustering. $V^i\in R^{n\times n}$ is a self-representation matrix of $X^i$. ${\left|\right|C\left|\right|}_{2,1}={\sum }_{i=1}^n\sqrt{{\sum }_{j=1}^n{C}_{i,j}^2}$ is the 2,1-norm of matrix C. ${\left|\right|C\left|\right|}_F=\sqrt{{\sum }_{i=1}^n{\sum }_{j=1}^n{C}_{i,j}^2}$ is the Frobenius norm of matrix C.

2.2. Subspace Clustering

Subspace learning is an effective means of processing high-dimensional data and is achieved by learning an optimal subspace representation instead of the original data. In this way, we can consider subspace learning as a particular type of matrix decomposition. Therefore, subspace learning is also widely used in various machine learning tasks. The essential purpose of subspace learning is to find a suitable subspace on a given data set $X\in R^{n\times m}$ such that self-representation matrix $V\in R^{n\times n}$ satisfies the following Equation:

$$\begin{array}{cc}\hfill \underset{V}{min}& \left|\right|X-{XV\left|\right|}_{\delta },\hfill \\ \hfill & s.t.diag\left(V\right)=0,\hfill \end{array}$$

(1)

where $\left|\right|·{\left|\right|}_{\delta }$ is the paradigm expression and $diag\left(V\right)=0$ is the diagonal array.

After further research and development, a constraint term is added to the above formula, which can reduce overfitting and enhance the condition of the learned subspace (sparse or low rank). Then, the formula becomes:

$$\begin{array}{cc}\hfill \underset{V}{min}& \left|\right|X-{XV\left|\right|}_{\delta }+{\alpha \left|\right|V\left|\right|}_{\rho },\hfill \\ \hfill & s.t.diag\left(V\right)=0.\hfill \end{array}$$

(2)

Sparse subspace clustering (SSC) imposes a sparse constraint with $\rho =0$ on the data representation matrix. It represents each data point as a sparse linear combination of other points and captures the local structure of the data. Low-rank representation (LRR) imposes a low-rank constraint [41] with a nuclear norm $\rho =\ast$ on the data representation matrix and captures the global structure of the data.

2.3. Multiview Subspace Clustering

For a given dataset $D=\left\{X^1,X^2,\cdots ,X^v\right\},X^i\in R^{d_i\times n}$ with v views [39],

$$\begin{array}{cc}\hfill \underset{{\left\{V^i\right\}}_{i=1}^v}{min}& \sum _{i=1}^v\left\{\left|\right|X^i-X^i{V}^i{\left|\right|}_{\delta }+\alpha \left|\right|V^i{\left|\right|}_{\rho }\right\},\hfill \\ \hfill & s.t.diag\left(V^i\right)=0,i=1,2,\cdots ,v.\hfill \end{array}$$

(3)

where $V^i$ is a self-representation matrix for ith view, $i=1,2,\cdots ,v$.

Researchers have developed many strategies to get consensus matrix V according to the self-representation matrices $V^i$ of each ith view. For example, the sparse and low-rank subspace multi-view clustering proposed by Brbic et al. [39] can obtain subspaces with particular structures by constraining and unifying the framework of consensus matrix learning with view subspace learning to make it more reasonable. Zhao et al. [41] proposed a method for learning the final consensus matrix that assigns weights to individual view clusters to obtain a better clustering result. Overall, many improvements have been made according to this unified framework, i.e.,

$$\begin{array}{cc}\hfill \underset{{\left\{V^i\right\}}_{i=1}^v,V}{min}& \sum _{i=1}^v\left\{\left|\right|X^i-X^i{V}^i{\left|\right|}_{\delta }+\alpha \left|\right|V^i{\left|\right|}_{\rho }+\beta \left|\right|V^i-V^{i}V{\left|\right|}_F^2\right\},\hfill \\ \hfill & s.t.diag\left(V^i\right)=0,i=1,2,\cdots ,v.\hfill \end{array}$$

(4)

After obtaining the consensus matrix V, we can obtain the affinity matrix and find cluster membership of data points according to the affinity matrix by using spectral clustering [17].

3. The Proposed Multi-View Subspace Clustering

3.1. The Proposed Multi-View Subspace Clustering Model

Most existing multi-view clustering algorithms learn consensus matrix V by weighting the self-representation matrix from multiple views. Because ignoring the feature concatenation, they may fail to explore the correlation among the views. This section proposes a multi-view subspace clustering considering the joint and individual views. The joint view is the union of all views. The consensus matrix of the proposed algorithm fuses the information from all views and is enhanced by the knowledge of the joint view to benefit clustering.

Equation (4) learns the view representation matrix and obtains the consensus matrix. In this way, each view’s complementary information is not adequately utilized, and the consensus matrix is just a simple integration of each view. Consequently, adding an error term can make the consensus matrix more reasonable and closer to each view. Equation (4) is replaced by

$$\begin{array}{cc}\hfill \underset{{\left\{V^i\right\}}_{i=1}^v,V}{min}\sum _{i=1}^v& \left\{{\displaystyle \frac{1}{2}}\left|\right|X^i-X^i{V}^i{\left|\right|}_F^2+\alpha \left|\right|V^i{\left|\right|}_{2,1}+{\displaystyle \frac{1}{2}}\beta \left|\right|V-V^i{\left|\right|}_F^2\right\},\hfill \\ \hfill s.t.& diag\left(V^i\right)=0,i=1,2,\cdots ,v.\hfill \end{array}$$

(5)

where $\left|\right|·{\left|\right|}_{2,1}$ is used as a sparsity constraint for each view to learn that the representation matrix is as sparse as possible.

Equation (5) ignores the feature concatenation. It may fail to explore the correlation among the views. Thus, we enhance the consensus matrix with the information of the joint view X. The consensus matrix should satisfy the following conditions: (1) the matrix is learned from the feature direct concatenation data, and (2) all the views should recognize the representation matrix. In other words, the matrix should contain complementary information in the other views. Different from Equation (5), the algorithm proposed in this paper obtains a consensus matrix from the joint data. All the views are fused with this matrix through the error term, making the consensus matrix more comprehensive and reasonable information about the samples. On this basis, by adding the joint views, Equation (5) is replaced by Equation (6),

$$\begin{array}{cc}\hfill \underset{{\left\{V^i\right\}}_{i=1}^v,V}{min}& \sum _{i=1}^v\left\{{\displaystyle \frac{1}{2}}\left|\right|X^i-X^i{V}^i{\left|\right|}_F^2+\beta \left|\right|V^i{\left|\right|}_{2,1}+{\displaystyle \frac{1}{2}}\gamma \left|\right|V-V^i{\left|\right|}_F^2\right\}+\alpha \left|\right|X-{XV\left|\right|}_{2,1},\hfill \\ & s.t.diag\left(V\right)=0,diag\left(V^i\right)=0,i=1,2,\cdots ,v.\hfill \end{array}$$

(6)

where $\alpha ,\beta ,\gamma$ are the penalty parameters. This way, we can fully fuse the information from each view of the data, yielding better cluster results. The flowchart of the proposed algorithm is given in Figure 1.

3.2. Optimization

The optimization objective (6) can be converted to Equation (7) by introducing auxiliary variables $C^i$ and E.

$$\begin{array}{cc}\hfill \underset{{\left\{V^i\right\}}_{i=1}^v,E,{\left\{C^i\right\}}_{i=1}^v,V}{min}& \sum _{i=1}^v\left\{{\displaystyle \frac{1}{2}}\left|\right|X^i-X^i{V}^i{\left|\right|}_F^2+\beta \left|\right|C^i{\left|\right|}_{2,1}+{\displaystyle \frac{1}{2}}\gamma \left|\right|V-V^i{\left|\right|}_F^2\right\}+{\alpha \left|\right|E\left|\right|}_{2,1},\hfill \\ \hfill & s.t.C^i=V^i,diag\left(V^i\right)=0,i=1,2,\cdots ,v,\hfill \\ \hfill & E=X-XV,diag\left(V\right)=0.\hfill \end{array}$$

(7)

The Lagrangian function of problem (7) is formulated as follows.

$$\begin{array}{cc}\hfill \mathcal{L}\left({\left\{C^i\right\}}_{i=1}^v,E,{\left\{V^i\right\}}_{i=1}^v,{\left\{Y^i\right\}}_{i=1}^v,Y,\mu \right)=& \sum _{i=1}^v\left\{{\displaystyle \frac{1}{2}}\left|\right|X^i-X^i{V}^i{\left|\right|}_F^2+\beta \left|\right|C^i{\left|\right|}_{2,1}+{\displaystyle \frac{1}{2}}\gamma \left|\right|V-V^i{\left|\right|}_F^2+〈Y^i,C^i-V^i〉+{\displaystyle \frac{\mu }{2}}\left|\right|C^i-V^i{\left|\right|}_F^2\right\}\hfill \\ \hfill & +{\alpha \left|\right|E\left|\right|}_{2,1}+〈Y,E-X+XV〉+{\displaystyle \frac{\mu }{2}}\left|\right|E-X+{XV\left|\right|}_F^2,\hfill \end{array}$$

(8)

where $〈A,B〉$ is the trace of $A^{T}B$, $\mu$ is penalty parameter that need to be tuned, ${\left\{Y^i\right\}}_{i=1}^v$ and Y are Lagrange dual variables. Optimization is executed using the augmented Lagrange method (ALM), where the remaining variables are fixed to update one of the remaining variables. We give a detailed extrapolation of the optimization process in the following.

• Updating $C^i$ as given the values of other variables $E,V^i,Y^i,Y,\mu$. Equation (8) is equivalent to the following optimization equation

  $$\underset{{\left\{C^i\right\}}_{i=1}^v}{min}\sum _{i=1}^v\left\{\beta \left|\right|C^i{\left|\right|}_{2,1}+〈Y^i,C^i-V^i〉+{\displaystyle \frac{\mu }{2}}\left|\right|C^i-V^i{\left|\right|}_F^2\right\}.$$

  (9)

Because $C^i,i=1,\cdots ,v$ are no intersection, ${\left\{C^i\right\}}_{i=1}^v$ can be solved separately for each of these. For a specific $C^i$, we need to solve

$$\underset{C^i}{min}\beta \left|\right|C^i{\left|\right|}_{2,1}+〈Y^i,C^i-V^i〉+{\displaystyle \frac{\mu }{2}}\left|\right|C^i-V^i{\left|\right|}_F^2,i=1,\cdots ,v.$$

(10)

Equation (10) is equivalent to

$$\underset{C^i}{min}\beta \left|\right|C^i{\left|\right|}_{2,1}+{\displaystyle \frac{\mu }{2}}\left|\right|C^i-\left(V^i+{\displaystyle \frac{Y^i}{\mu }}\right){\left|\right|}_F^2.$$

(11)

The optimal solution $C^{i^{*}}$ of Equation (11) can be given as

$${\left[C^{i^{*}}\right]}_{:,j}=\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\left|\right|{\left[Q\right]}_{:,j}{\left|\right|}_2-{\displaystyle \frac{\beta }{\mu }}}{\left|\right|{\left[Q\right]}_{:,j}{\left|\right|}_2}}{Q}_{:,j},& \mathrm{if}\left|\right|{\left[Q\right]}_{:,j}{\left|\right|}_2>{\displaystyle \frac{\beta }{\mu }}\hfill \\ \hfill 0,& \mathrm{otherwise},\hfill \end{array}\right.,$$

(12)

where $Q=V^i+{\displaystyle \frac{Y^i}{\mu }}$, and ${[.]}_{:,j}$ is a vector that is the jth column of a matrix. ${\left|\right|.\left|\right|}_2$ is the 2-norm of a vector.

2.

Updating E as given the values of ${\left\{C^i\right\}}_{i=1}^v,{\left\{V^i\right\}}_{i=1}^v,V,{\left\{Y^i\right\}}_{i=1}^v,Y,\mu$. The optimization problem with respect to E can be denoted as

$$\underset{E}{min}{\alpha \left|\right|E\left|\right|}_{2,1}+〈Y,E-X+XV〉+{\displaystyle \frac{\mu }{2}}\left|\right|E-X+{XV\left|\right|}_F^2.$$

(13)

The expression of problem (13) is similar to the expression of problem (10). Thus, the optimal solution of problem (13) has a similar expression to the optimal solution of problem (12). It can be given as

$${\left[E^{*}\right]}_{:,j}=\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\left|\right|{\left[R\right]}_{:,j}{\left|\right|}_2-{\displaystyle \frac{\alpha }{\mu }}}{\left|\right|{\left[R\right]}_{:,j}{\left|\right|}_2}}{R}_{:,j},& \mathrm{if}\left|\right|{\left[R\right]}_{:,j}{\left|\right|}_2>{\displaystyle \frac{\alpha }{\mu }}\hfill \\ \hfill 0,& \mathrm{otherwise},\hfill \end{array}\right.,$$

(14)

where $R=X-XV+{\displaystyle \frac{Y}{\mu }}$.

3.

Updating ${\left\{V^i\right\}}_{i=1}^v$ as given the values of ${\left\{C^i\right\}}_{i=1}^v,E,V,{\left\{Y^i\right\}}_{i=1}^v,Y,\mu$. Equation (8) is equivalent to the following quadratic programming problem.

$$\begin{array}{cc}\hfill \underset{V^i}{min}& \sum _{i=1}^v\left\{{\displaystyle \frac{1}{2}}\left|\right|X^i-X^i{V}^i{\left|\right|}_F^2+{\displaystyle \frac{1}{2}}\gamma \left|\right|V-V^i{\left|\right|}_F^2+〈Y^i,C^i-V^i〉+{\displaystyle \frac{\mu }{2}}\left|\right|C^i-V^i{\left|\right|}_F^2\right\}.\hfill \end{array}$$

(15)

Since $V^i,i=1,\cdots ,v$ are independent, and problem (15) is a convex quadratic programming problem, we can obtain the optimal solution of problem (15) by having the gradient concerning $V^i$ equal to 0. That is

$$-{\left(X^i\right)}^T\left(X^i-X^i{V}^i\right)-\gamma \left(V-V^i\right)-Y^i-\mu \left(C^i-V^i\right)=0$$

(16)

Therefore

$$\begin{array}{cc}\hfill & V^i={\left({\left(X^i\right)}^T{X}^i+\gamma I+\mu I\right)}^{-1}\left({\left(X^i\right)}^T{X}^i+\gamma V+Y^i+\mu C^i\right),\hfill \\ \hfill & V^i=V^i-diag\left(V^i\right).\hfill \end{array}$$

(17)

4.

Updating V as given the values of ${\left\{C^i\right\}}_{i=1}^v,E,{\left\{V^i\right\}}_{i=1}^v,{\left\{Y^i\right\}}_{i=1}^v,Y,\mu$. The optimal problem with respect to V also is a convex quadratic programming problem according to problem (8). The formula is

$$\underset{V}{min}\sum _{i=1}^v{\displaystyle \frac{1}{2}}\gamma \left|\right|V-V^i{\left|\right|}_F^2+〈Y,E-X+XV〉+{\displaystyle \frac{\mu }{2}}\left|\right|E-X+{XV\left|\right|}_F^2.$$

(18)

The resulting updated formula is

$$\begin{array}{cc}\hfill & V={\left(\mu X^T{X}^i+\gamma \sum _{i=1}^{v}I\right)}^{-1}\left(\gamma \sum _{i=1}^v{V}^i+\mu X^T\left(X-E\right)-X^{T}Y\right),\hfill \\ \hfill & V=V-diag\left(V\right).\hfill \end{array}$$

(19)

5.

As given the values of $Y^i,Y,\mu$$C^i,E,V,V^i$, the dual variables $Y^i,Y$ and penalty parameter $\mu$ are updated as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & Y^i=Y^i+\mu \left(C^i-V^i\right),i=1,\cdots ,v\hfill \\ \hfill & Y=Y+\mu \left(E-X+XV\right),\hfill \\ \hfill & \mu =min\left(\rho \mu ,{\mu }_{max}\right).\hfill \end{array}\end{array}$$

(20)

where $\rho >1$ is a constant greater than 1. It makes the penalty factor $\mu$ monotonically increased until reaching the maximum ${\mu }_{max}$.

These update steps are repeated until $\left|\right|V_k-V_{k+1}{\left|\right|}_{\infty }<\epsilon$ or until the maximum number of iterations is reached, where $V_k$ and $V_{k+1}$ are the values of V at iteration k and $k+1$, respectively. After obtaining the consensus matrix through the above formula, we obtain the affinity matrix W for spectral clustering by Equation (21)

$$W={\displaystyle \frac{abs\left(V\right)+abs\left(V^T\right)}{2}}.$$

(21)

where $abs\left(·\right)$ is the absolute value function. Given the affinity matrix W, we can find cluster membership of data points by applying k-means clustering to the eigenvectors of the graph Laplacian matrix of the affinity matrix W.

Algorithm 1 gives the steps of the proposed algorithm. We first input the dataset D, the number of clusters k, which is given by the number of classes, and the parameters of the proposed algorithm. We random initializ ${\left\{V^i\right\}}_{i=1}^v$ and V in $[0,1]$, and initialize $Y^i=0$, $Y=0$ and $\mu ={10}^{-4}$.

Algorithm 1 FEMV algorithm

Require: Multiview data: $D=\left\{X^1,X^2,\cdots ,X^v\right\};$   Number of the clusters: k;   Parameters: $\rho ,\alpha ,\beta ,\gamma$ and $\epsilon$. Ensure: Assignments of the data points to k clusters.   1: Initial ${\left\{V^i\right\}}_{i=1}^v,V,{\left\{Y^i\right\}}_{i=1}^v,Y$ and $\mu$.   2: while the stopping criterion is not satisfied do   3:    Update ${\left\{C^i\right\}}_{i=1}^v$ according to Equation (12);   4:    Update E according to Equation (14);   5:    Update ${\left\{V^i\right\}}_{i=1}^v$ according to Equation (17);   6:    Update V according to Equation (19);   7:    Update $Y^i,Y,\mu$ according to Equation (20).   8: end while   9: Compute the affinity matrix W by Equation (21). 10: Apply spectral clustering to the affinity matrix W to obtain the assignments of the data points.

3.3. Computational Complexity

As shown in Algorithm 1, the main computational burden comprises five parts. The complexity of computing ${\left\{C^i\right\}}_{i=1}^v$ by Equation (12) is $O\left(v{n}^2\right)$ and the complexity of updating E is $O\left(n^2\right)$. We need to compute the inverse of a matrix with a size $nn$ for updating each $V^i$ and V. The complexity of updating ${\left\{V^i\right\}}_{i=1}^v$ is $O(d{n}^2+v{n}^3)$ and the complexity of computing V is $O(d{n}^2+n^3)$. The complexity of updating the dual variable $Y^i$ and Y are $O\left(n^2\right)$. To sum up, the computational complexity of each interaction is $O(d{n}^2+n^3)$.

4. Experiments

We compared the proposed algorithm with six existing multi-view clustering algorithms on seven real-world datasets. Three performance metrics are used to evaluate the performance of the compared algorithms. The experiments also investigate the sensitivity of the parameters of the proposed algorithm and the convergence. All the experiments are implemented under the following environments: Matlab R2021a running on a PC with Windows 64bit OS, Intel 3.40GHz CPU, and 8GB RAM.

4.1. Datasets

BBCSport, BBC4views, HandWritten, MSRCv1, WebKB, 100leaves, and NGs. BBCSport (http://mlg.ucd.ie/datasets/bbc.html, accessed on 1 October 2022) contains five categories (athletics, cricket, soccer, rugby, and tennis), with 544 documents and two views. BBC4view (http://mlg.ucd.ie/datasets/bbc.html, accessed on 1 October 2022) contains four views and 688 documents from the BBC News website, with five categories: business, entertainment, politics, sports, and technology. The HandWritten (https://cs.nyu.edu/roweis/data.html, accessed on 1 October 2022). This dataset contains 2000 samples. There are six views and ten categories. MSRCv1 (http://research.microsoft.com/en-us/projects/objectclassrecognition/, accessed on 1 October 2022) is an image dataset with 210 image samples extracted from seven classes and four views. WebKB (http://www.cs.cmu.edu/webkb/, accessed on 1 October 2022) is a hypertext dataset collected from various university web pages with three views and four classes. NGs (http://lig-membres.imag.fr/grimal/data.html, accessed on 1 October 2022) contains 500 images of people with three views. 100leaves (https://archive.ics.uci.edu/ml/datasets/One-hundred+plant+species+leaves+data+set, accessed on 1 October 2022) contains 1600 images of plant species leaves, with three views.

Table 1. Number of views and categories for the seven data sets.

Dataset	Instance	Views	Classes
BBCsport	544	2	5
BBC4views	685	4	5
HandWritten	2000	6	10
MSRCv1	210	4	7
WEBkb	1051	2	2
NGs	500	3	5
100 leaves	1600	3	100

gives the number of data views and their respective categories for the seven data sets.

4.2. The Comparison Algorithms and Performance Metrics

In order to investigate the performance of the proposed algorithm, we compared it with the following six algorithms.

• Spectral Clustering (SC) [17]: SC algorithm is applied to each view of the multi-view data. The best clustering result is selected by clustering multiple views separately, denoted as SC-BEST. Then the data obtained by feature union is clustered, denoted as SC-FU.
• Adaptive structure concept factorization for multi-view clustering (MVCF) [42]: This algorithm enhances the use of the correlation between views by jointly optimizing the representation matrix and proposes a multi-view clustering algorithm based on the concept of decomposition.
• Multi-view low-rank sparse subspace clustering (MLRSSC) [39]: This method balances different views by using low-rank and sparse constraints in constructing a consensus matrix of all views.
• graph learning for multi-view clustering (MVGL) [43]: This method obtains the Laplace matrix to obtain clustering metrics by obtaining low-ranked adjacency matrices from each view and integrating them into a global adjacency matrix.
• Graph-based system for multi-view clustering (MVGS) [35]: This method constructs the adjacency matrix by creating the feature matrices of all views and then fuses the adjacency matrices by weighting them to obtain the combined adjacency matrix. Finally, it gets the clustering results.

In this paper, the three most used performance metrics, i.e., accuracy (ACC), normalized mutual information (NMI), and F-score, are employed to evaluate the clustering performance of the compared algorithms. The formulas are given as follows.

$$\mathrm{ACC}={\displaystyle \frac{{\sum }_{k=1}^n\phi \left(c_i,map\left({\overline{c}}_i\right)\right)}{n}},$$

(22)

where $c_i$ and ${\overline{c}}_i$ correspond to the clustering result and the true clustering label, respectively. When $x=y$, $\phi \left(x,y\right)=1$; otherwise, $\phi \left(x,y\right)=0$. The best match between the true clustering label and the clustering result is obtained using the Hungarian algorithm.

$$\mathrm{NMI}\left(\Omega ;C\right)={\displaystyle \frac{2I\left(\Omega ,C\right)}{H\left(\Omega \right)+H\left(C\right)}},$$

(23)

where $\Omega =\left\{{\overline{c}}_1,{\overline{c}}_2,\cdots ,{\overline{c}}_k\right\}$ and $C=\left\{c_1,c_2,\cdots ,c_k\right\}$ denote true category partitioning and algorithmic cluster partitioning, respectively. $I\left(X,Y\right)$ is mutual information, and $H\left(X\right)$ is the entropy.

$$\mathrm{F}-\mathrm{score}=\left(1+{\tau }^2\right){\displaystyle \frac{P·R}{{\tau }^2·P+R}},$$

(24)

where P is the precision, R is the recall, and $\tau$ is a parameter that balances the two’s weights, which is usually equal to 1, indicating equal importance. The values of all three metrics are within $\left[0,1\right]$, and the closer the value is to 1, the better the result.

4.3. Experimental Results

Table 2. Comparison results of different algorithms on seven datasets.

Datasets	Algorithms	ACC	NMI	F-Score
BBCsport	SC-BEST	$0.6610\pm 0.0006†$	$0.4396\pm 0.0050†$	$0.5585\pm 0.0042†$
	SC-FU	$0.7470\pm 0.0026†$	$0.5468\pm 0.0118†$	$0.6435\pm 0.0083†$
	MVGL	$0.4099\pm 0.0000†$	$0.1586\pm 0.0000†$	$0.3985\pm 0.0000†$
	MLRSSC	$0.8631\pm 0.0488†$	$\mathbf{0.7961}\pm \mathbf{0.0074}$	$0.8542\pm 0.0279$
	MVCF	$0.6607\pm 0.0355†$	$0.4459\pm 0.0431†$	$0.5522\pm 0.0283†$
	MVGS	$0.8070\pm 0.0000†$	$0.7600\pm 0.0000†$	$0.7943\pm 0.0000†$
	FEMV	$\mathbf{0.8805}\pm \mathbf{0.0000}$	$0.7935\pm 0.0000$	$\mathbf{0.8600}\pm \mathbf{0.0000}$
BBC4view	SC-BEST	$0.5213\pm 0.0017†$	$0.3487\pm 0.0018†$	$0.3902\pm 0.0009†$
	SC-FU	$0.6233\pm 0.0000†$	$0.5462\pm 0.0000†$	$0.5659\pm 0.0000†$
	MVGL	$0.3518\pm 0.0000†$	$0.0762\pm 0.0000†$	$0.3743\pm 0.0000†$
	MLRSSC	$0.7532\pm 0.0810†$	$0.6518\pm 0.0209†$	$0.6703\pm 0.0294†$
	MVCF	$0.6369\pm 0.0273†$	$0.5065\pm 0.0316†$	$0.5935\pm 0.0311†$
	MVGS	$0.6934\pm 0.0000†$	$0.5628\pm 0.0000†$	$0.6333\pm 0.0000†$
	FEMV	$\mathbf{0.8701}\pm \mathbf{0.0000}$	$\mathbf{0.7331}\pm \mathbf{0.0021}$	$\mathbf{0.8193}\pm \mathbf{0.0008}$
HandWritten	SC-BEST	$0.7411\pm 0.0012†$	$0.6760\pm 0.0011†$	$0.6447\pm 0.0011†$
	SC-FU	$0.7508\pm 0.0010†$	$0.7122\pm 0.0014†$	$0.6758\pm 0.0017†$
	MVGL	$0.8560\pm 0.0000†$	$\mathbf{0.9055}\pm \mathbf{0.0000}$	$0.8502\pm 0.0000†$
	MLRSSC	$0.7862\pm 0.0389†$	$0.8073\pm 0.0180†$	$0.7566\pm 0.0334†$
	MVCF	$0.6085\pm 0.0456†$	$0.8017\pm 0.0276†$	$0.4254\pm 0.0683†$
	MVGS	$0.8810\pm 0.0000†$	$0.9011\pm 0.0000$	$0.8654\pm 0.0000†$
	FEMV	$\mathbf{0.9375}\pm \mathbf{0.0163}$	$0.8728\pm 0.0078$	$\mathbf{0.8781}\pm \mathbf{0.0263}$
MSRCv1	SC-BEST	$0.7100\pm 0.0105†$	$0.5822\pm 0.0145†$	$0.5761\pm 0.0109†$
	SC-FU	$0.5747\pm 0.0100†$	$0.4872\pm 0.0106†$	$0.4577\pm 0.0063†$
	MVGL	$0.6333\pm 0.0000†$	$0.5897\pm 0.0000†$	$0.4976\pm 0.0000†$
	MLRSSC	$0.6271\pm 0.0410†$	$0.5257\pm 0.0230†$	$0.4915\pm 0.0303†$
	MVCF	$0.8105\pm 0.0456†$	$0.7220\pm 0.0355$	$0.6994\pm 0.0371†$
	MVGS	$0.7774\pm 0.0000†$	$0.0023\pm 0.0000†$	$\mathbf{0.7876}\pm \mathbf{0.0000}$
	FEMV	$\mathbf{0.8400}\pm \mathbf{0.0023}$	$\mathbf{0.7279}\pm \mathbf{0.0020}$	$0.7171\pm 0.0029$
WEBKb	SC-BEST	$0.7792\pm 0.0000†$	$0.0055\pm 0.0000†$	$0.7917\pm 0.0000†$
	SC-FU	$0.7792\pm 0.0000†$	$0.0055\pm 0.0000†$	$0.7917\pm 0.0000†$
	MVGL	$0.7498\pm 0.0000†$	$0.0062\pm 0.0000†$	$0.5632\pm 0.7495†$
	MLRSSC	$0.7726\pm 0.0000†$	$0.0001\pm 0.0000†$	$0.7832\pm 0.0000†$
	MVGS	$0.7685\pm 0.0000†$	$0.4351\pm 0.0000†$	$0.7004\pm 0.0000†$
	FEMV	$\mathbf{0.9514}\pm \mathbf{0.0000}$	$\mathbf{0.6592}\pm \mathbf{0.0000}$	$\mathbf{0.9287}\pm \mathbf{0.0000}$
NGs	SC-BEST	$0.5358\pm 0.0006†$	$0.3745\pm 0.0006†$	$0.4222\pm 0.0004†$
	SC-FU	$0.2110\pm 0.0000†$	$0.0723\pm 0.0000†$	$0.3291\pm 0.0000†$
	MVGL	$0.2720\pm 0.0000†$	$0.1286\pm 0.0000†$	$0.3296\pm 0.0000†$
	MLRSSC	$0.9724\pm 0.0018†$	$0.9230\pm 0.0053†$	$0.9453\pm 0.0036†$
	MVCF	$0.3138\pm 0.0237†$	$0.2173\pm 0.0385†$	$0.3531\pm 0.0097†$
	MVGS	$0.9820\pm 0.0000†$	$0.9392\pm 0.0000†$	$0.9643\pm 0.0000†$
	FEMV	$\mathbf{0.9920}\pm \mathbf{0.0000}$	$\mathbf{0.9722}\pm \mathbf{0.0000}$	$\mathbf{0.9840}\pm \mathbf{0.0000}$
100leaves	SC-BEST	$0.4483\pm 0.0099†$	$0.7115\pm 0.0060†$	$0.3182\pm 0.0099†$
	SC-FU	$0.7162\pm 0.0172†$	$0.8778\pm 0.0077†$	$0.6262\pm 0.0257†$
	MVGL	$0.7700\pm 0.0000†$	$0.8972\pm 0.0000†$	$0.5415\pm 0.0000†$
	MLRSSC	$0.5835\pm 0.0189†$	$0.7904\pm 0.0084†$	$0.4677\pm 0.0195†$
	MVCF	$0.5195\pm 0.0000†$	$0.0.7556\pm 0.0020†$	$0.3830\pm 0.0025†$
	MVGS	$0.8244\pm 0.0000†$	$\mathbf{0.9343}\pm \mathbf{0.0000}$	$0.5765\pm 0.0000†$
	FEMV	$\mathbf{0.8413}\pm \mathbf{0.0000}$	$0.9175\pm 0.0000$	$\mathbf{0.7555}\pm \mathbf{0.0000}$

† indicates that the results obtained by the other algorithms are significantly worse than that of FEMV.

lists the means and standard deviations of ACC, NMI, and F-score values obtained by each algorithm running ten times independently. Algorithms that perform best were highlighted in bold. To reach a statistically sound conclusion, the Wilcoxon rank sum test at the significance level of 0.05 is used to analyze the differences between FEMV and the other algorithms. Symbol † indicates that the results obtained by the other algorithms are significantly worse than that of FEMV. This table shows that the proposed FEMV algorithm achieved premising clustering results on all six datasets. This means that the proposed FEMV algorithm has achieved significant improvements in terms of ACC, NMI, and F-score. The experimental results obtained by SC-FU are superior to those obtained by SC-BEST on all datasets except MSRCv1 and NGs in terms of all performance metrics. This indirectly demonstrates that multi-view fusion significantly improves clustering performance when multiple views have complementary information. It also shows that the information of the joint view is effective and reasonable in improving the clustering performance. It also shows that the joint view contains usable clustering information that previous multi-view clustering algorithms ignored.

In a more specific analysis, the FEMV algorithm improves by $9.11\%$, $4.41\%$, and $8.39\%$ on the BBCSport dataset compared to MVGS in terms of ACC, NMI, and F-score on the BBCSport dataset. Compared with MLRSSC, it improves ACC and F metrics by $2\%$ and $0.2\%$, while it is similar to MLRSSC in terms of NMI. Although FEMV is inferior to MVGL and MVGS algorithms in terms of NMI metrics, FEMV is superior to MVGL and MVGS algorithms in terms of ACC and F-score on HandWritten. The result obtained by FEMV is slightly inferior to that of MVGS in terms of NMI on 100 leaves. In the other cases, the proposed FEMV algorithm significantly outperforms the comparison algorithms. The proposed FEMV algorithm is a significant improvement compared to the previous methods. Because the FEMV algorithm considers incorporating the overall joint information and learns the potential subspace representation from it, it fully integrates the information of the joint view.

4.4. Parametric Sensitivity Analysis and Convergence Analysis

The objective function of the proposed FEMV algorithm, i.e., Equation (6), has three penalty parameters $\alpha ,\beta$, and $\gamma$. This section discusses the sensitivity analysis of the parameters. Taking BBCsport, BBC4view, MSRCv1, and NGs datasets as an example, we apply grid search to find the best parameters with respect to ACC. We first fix the $\alpha$ and $\beta$ to fine-tune the value of $\gamma$, which is chosen from $\left\{1,10,100,1000\right\}$ to find the best value for the FEMV algorithm on BBCsport, BBC4view, MSRCv1, and NGs datasets. The best value for the proposed FEMV is shown in Figure 2. Then we fix $\gamma$ with the best value and fine-tune the value of $\alpha ,\beta$, which are both chosen from $\left\{0.01,0.1,1,10,100\right\}$. Figure 2 shows the ACC obtained by the proposed algorithm with different $\alpha$ and $\beta$ and a given $\gamma$. This figure shows that $\alpha$ and $\beta$ have little effect on the result when $\gamma =100$ on BBCSport. Thus, $\alpha$ and $\beta$ take values of 1 in the experiments on BBCSport. Similarly, we can obtain the best parameters for the proposed FEMV on BBC4view, MSRCv1, and NGs.

Figure 3 plots the objective function value with respect to the number of iterations for the proposed FEMV algorithm on BBCsport, BBC4view, MSRCv1, and NGs. Figure 3 shows that the proposed algorithm smooths out after 10 iterations on these datasets, which means that the FEMV algorithm can achieve fast convergence.

4.5. Ablation Experiments

The effectiveness of the proposed algorithm with the addition of feature union constraints is verified by ablation experiments. The added feature union view terms and their equation constraints are removed based on the proposed formula. Then compare with the original experiment of Equation (5).

Table 3. Ablation comparison experiment.

Dataset	Algorithms	ACC	NMI	F-Score
BBCsport	FEMV	$0.8805\pm 0.0000$	$0.7935\pm 0.0000$	$0.8600\pm 0.0000$
	del-FEMV	$0.8363\pm 0.0000$	$0.7057\pm 0.0000$	$0.7880\pm 0.0000$
BBC4views	FEMV	$0.8701\pm 0.0000$	$0.7331\pm 0.0021$	$0.8193\pm 0.0008$
	del-FEMV	$0.8399\pm 0.0011$	$0.6807\pm 0.0045$	$0.7835\pm 0.0033$
HandWritten	FEMV	$0.9375\pm 0.0163$	$0.8728\pm 0.0078$	$0.8781\pm 0.0263$
	del-FEMV	$0.8856\pm 0.0234$	$0.7830\pm 0.0113$	$0.7956\pm 0.0023$
MSRCv1	FEMV	$0.8400\pm 0.0023$	$0.7279\pm 0.0020$	$0.7171\pm 0.0029$
	del-FEMV	$0.8095\pm 0.0000$	$0.7264\pm 0.0000$	$0.6866\pm 0.0000$
Webkb	FEMV	$0.9514\pm 0.0000$	$0.6592\pm 0.0000$	$0.9287\pm 0.0000$
	del-FEMV	$0.9248\pm 0.0000$	$0.5174\pm 0.0000$	$0.8960\pm 0.0000$
NGS	FEMV	$0.9920\pm 0.0000$	$0.9722\pm 0.0000$	$0.9840\pm 0.0000$
	del-FEMV	$0.9800\pm 0.0000$	$0.9390\pm 0.0000$	$0.9604\pm 0.0000$
100leaves	FEMV	$0.8413\pm 0.0000$	$0.9175\pm 0.0000$	$0.7555\pm 0.0000$
	del-FEMV	$0.7723\pm 0.0000$	$0.8463\pm 0.0000$	$0.7269\pm 0.0000$

shows the results of the ablation experiments, where del-FEMV is the experimental result of eliminating the joint data of features. From this table, we can see that the algorithm with adding the joint view is significantly super to the algorithm without it under the same experimental conditions.

5. Conclusions

This paper proposed a multi-view clustering algorithm based on information fusion and enhancement. This method mainly adds a joint view of the data and learns an appropriate subspace for information fusion. It can achieve the purpose of potential information enhancement. Unlike traditional multi-view subspace clustering, the proposed algorithm can fuse the joint data information. Furthermore, it can enhance the utilization of the information while ensuring its sparsity using parametric constraints. Finally, experiments on seven real datasets demonstrate the effectiveness of the proposed algorithm.

However, since matrix inversion is involved, the computation is expensive for large datasets. In addition, the algorithm does not consider the weighting of the views. The proposed information fusion and enhancement framework applies to subspace multi-view clustering and other multi-view clustering methods, such as graph learning and tensor decomposition. Subsequent work will begin with these three aspects. For example, we intend to extend this framework to other multi-view clustering algorithms and investigate the impact of the view weights and the choice of the number of clusters.