Adaptive Weighted Multiview Kernel Matrix Factorization and Its Application in Alzheimer’s Disease Analysis

Abstract

Recent technology and equipment advancements have provided us with opportunities to better analyze Alzheimer’s disease (AD), where we could collect and employ the data from different image and genetic modalities that may potentially enhance the predictive performance. To perform better clustering in AD analysis, in this paper, we propose a novel model to leverage data from all different modalities/views, which can learn the weights of each view adaptively. Different from previous vanilla Non-negative matrix factorization which assumes data is linearly separable, we propose a simple yet efficient method based on kernel matrix factorization, which is not only able to deal with non-linear data structure but also can achieve better prediction accuracy. Experimental results on the ADNI dataset demonstrate the effectiveness of our proposed method, which indicates promising prospects for kernel application in AD analysis.

1. Introduction

Alzheimer’s disease (AD) is a chronic neurodegenerative disease related to a part of the brain that controls memory, thought, and language. It often occurs in the elderly, beginning with mild memory loss and possibly dramatically worsening and leading to further cognitive function loss. The whole progress can be divided into three diagnostic groups: AD, mild cognitive impairment (MCI), and health control (HC).

With the development of imaging genetics, exploring genome-wide array data and multimodal brain imaging data may help researchers deepen the understanding of AD, facilitate accurate early detection and diagnosis, and improve treatment. For example, Nazarian et al. [1] found a significant association between certain single-nucleotide polymorphisms (SNPs) and AD. Magnetic resonance imaging (MRI) is an important medical neuroimaging technique used to acquire regional imaging biomarkers, such as voxel-based morphometry (VBM) [2] features, to investigate focal structural abnormalities in grey matter (GM). Fluorodeoxyglucose positron emission tomography (FDG-PET) [3] can distinguish AD from other causes of dementia using the characteristic glucose metabolism patterns. Therefore, multi-view analyses have attracted more attention in recent years, and among these methods, non-negative matrix factorization (NMF)-based approaches are promising and have shown strong interpretability [4,5,6,7].

However, existing multi-view analysis methods treat each view as equally important, which contradicts the fact that some views may play more critical roles than others. Therefore, adaptively learning the weights of different views is an interesting topic of importance. Vanilla NMF’s success relies heavily on the assumption that the data lie in a well-separable space with a linear structure; however, in practice, its accuracy is rather low, which indicates the necessity of developing a new method that can deal with a non-linearity data structure.

To address the aforementioned problems, in this paper, we propose a new method with an efficient updating algorithm. Our contribution is twofold: first, the new framework can learn each view’s weight and therefore conduct view selection; second, by introducing kernel mapping, it can handle both linear and non-linear cases. Extensive experiments on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset demonstrate its effectiveness with higher clustering results, which shed light on the prospect of Kernel’s application in other AD analysis tasks, such as regression, classification, etc.

2. Related Works

The framework in this paper originates from NMF, which is widely used in clustering [8,9] as it can learn features and membership indicators in a very natural and interpretable way. Thus, we first provide a brief overview of it, along with its extensions.

2.1. Single-View NMF with Graph Regularization

Assume we have $\mathit{X}=[{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_n]\in {\mathbb{R}}^{m\times n}$ as input data and the task is to cluster $\mathit{X}$ into k clusters. Vanilla NMF assumes that each data point $\mathit{x}$ can be represented as a linear combination of features $\mathit{f}$ (columns of $\mathit{F}$): ${\mathit{x}}_j\approx {\sum }_i{\mathit{F}}_i{\mathit{G}}_{ij}=\mathit{F}{\mathit{g}}_j$. Therefore, the objective function [10] can be formulated as follows:

$${min\parallel \mathit{X}-\mathit{F}\mathit{G}\parallel }_F^2+\theta \mathbf{tr}\left(\mathit{G}\mathit{L}{\mathit{G}}^T\right)s.t.\mathit{F},\mathit{G}\ge 0,$$

(1)

where ${\parallel ·\parallel }_F^2$ denotes the squared Frobenius norm with ${\parallel \mathit{Z}\parallel }_F^2={\sum }_i{\sum }_j{z}_{ij}^2$, and the second term ‘graph regularization’ [11] was introduced to promote clustering accuracy. Each column of $\mathit{F}\in {\mathbb{R}}^{m\times k}$ represents a centroid of k clusters, while each column of $\mathit{G}\in {\mathbb{R}}^{k\times n}$ can be taken as the probabilities of data $\mathit{x}$ belonging to each cluster. $\mathit{L}:=\mathit{D}-\mathit{W}$ denotes the Laplace matrix, where $\mathit{D}$ is a diagonal matrix whose entries are column sums of $\mathit{W}$: ${\mathit{D}}_{ii}={\sum }_j{\mathit{W}}_{ij}$, with $\mathit{W}\in {\mathbb{R}}^{n\times n}$ being the similarity matrix of the data samples. In our paper, we set ${\mathit{W}}_{ij}^{\left(a\right)}=exp(-{\displaystyle \frac{\parallel {\mathit{x}}_i^{\left(a\right)}-{\mathit{x}}_j^{\left(a\right)}{\parallel }^2}{2{\sigma }^2}})$, where $\left(a\right)$ denotes the a-th view.

2.2. Multi-View Clustering via NMF

Assume we have v different views of the input data $\{{\mathit{X}}^{\left(1\right)},{\mathit{X}}^{\left(2\right)},\cdots ,{\mathit{X}}^{\left(v\right)}\}$, where ${\mathit{X}}^{\left(i\right)}$ indicates the i-th view. Each view can be factorized as ${\mathit{X}}^{\left(i\right)}={\mathit{F}}^{\left(i\right)}{\mathit{G}}^{\left(i\right)}$, where ${\mathit{F}}^{\left(i\right)}\in {\mathbb{R}}^{m\times k}$ contains the centroids of k clusters, and ${\mathit{G}}^{\left(i\right)}\in {\mathbb{R}}^{k\times n}$ indicates the corresponding coefficient (probabilities matrix) for each view [4]. Figure 1 provides an example of how multi-view data can be factorized using NMF.

As the clustering results may vary with different views, a consensus clustering indicator matrix ${\mathit{G}}^{\ast }$ was introduced to promote consistency across views. Moreover, similar to single-view NMF, a graph regularization term is added for each view and the objective [12] is as follows:

$$\begin{array}{cc}\hfill \underset{{\mathit{F}}^{\left(a\right)},{\mathit{G}}^{\left(a\right)},{\mathit{G}}^{\ast }}{min}\sum _{a=1}^v\{\parallel {\mathit{X}}^{\left(a\right)}-{\mathit{F}}^{\left(a\right)}{\mathit{G}}^{\left(a\right)}{\parallel }_F^2 +& {\lambda }_a\parallel {\mathit{G}}^{\left(a\right)}-{\mathit{G}}^{\ast }{\parallel }_F^2+{\theta }_a\mathbf{tr}\left({\mathit{G}}^{\left(a\right)}{\mathit{L}}^{\left(a\right)}{\left({\mathit{G}}^{\left(a\right)}\right)}^T\right)\}\hfill \\ \hfill & s.t.{\mathit{F}}^{\left(a\right)},{\mathit{G}}^{\left(a\right)},{\mathit{G}}^{\ast }\ge 0,\hfill \end{array}$$

(2)

where $\lambda ,\theta$ are regularization parameters that need tuning. ${\mathit{G}}^{\left(a\right)}$ indicates the clustering results from view a and the final consensus clustering will be obtained from ${\mathit{G}}^{\ast }$.

3. Our Methodology

We applied the multi-view NMF clustering presented above to the Alzheimer’s Disease dataset and found the clustering performance was not that high. Similar results can be found in [5] as well. Therefore, in this section, we propose a novel framework that can not only deal with non-linearly separable cases but also can learn the various weights of different views [13].

3.1. Adaptive Weighted Multi-View NMF

Equation (2) utilizes all the data from different views with a consensus ${\mathit{G}}^{\ast }$, and treats each view as having the same importance. However, in practice, different views may have various weights, which inspired us to propose a formulation that can learn the weights accordingly, as follows:

$$\begin{array}{cc}\hfill \underset{{\mathit{\beta }}_a,{\mathit{F}}^{\left(a\right)},{\mathit{G}}^{\left(a\right)},{\mathit{G}}^{\ast }}{min}\sum _{a=1}^v{\mathit{\beta }}_a^{\gamma }\{\parallel {\mathit{X}}^{\left(a\right)} -& {\mathit{F}}^{\left(a\right)}{\mathit{G}}^{\left(a\right)}{\parallel }_F^2+{\lambda }_a\parallel {\mathit{G}}^{\left(a\right)}-{\mathit{G}}^{\ast }{\parallel }_F^2+{\theta }_a\mathbf{tr}\left({\mathit{G}}^{\left(a\right)}{\mathit{L}}^{\left(a\right)}{\left({\mathit{G}}^{\left(a\right)}\right)}^T\right)\}\hfill \\ \hfill & s.t.{\mathit{F}}^{\left(a\right)},{\mathit{G}}^{\left(a\right)},{\mathit{G}}^{\ast }\ge 0,\sum _{a=1}^v{\mathit{\beta }}_a=1,\hfill \end{array}$$

(3)

where $\gamma$ is a hyper-parameter denoting the order of $\mathit{\beta }$, which is an integer by default. One can see that when $\gamma =0$, it degenerates into Equation (2), where the weight in each view is the same. In Section 4, we will further discuss its impact on modality selection.

3.2. Adaptive Weighted Kernel Multi-View NMF

Most traditional NMF and existing multi-view methods conduct a clustering/classification analysis based directly on the original data, meaning that the accuracy highly relies on the assumption that the data are linearly separable in the original space. However, through extensive experiments using either single-view or multi-view analyses, we found that the accuracy is rather low, as reported in Section 6.3. Thus, we propose a kernel version multi-view framework to overcome the difficulty of handling non-linearity via NMF. The basic idea is that, instead of clustering on the original space, we first map the data into higher dimension $\mathbf{\Phi }:{\mathbb{R}}^m\to {\mathbb{R}}^d$ (allowing for infinite space, namely, $d=\infty$, such as Gaussian Kernel), and then complete the clustering based on the mapped space [14,15,16,17].

To make use of the kernel trick, following the idea of K-means, where each centroid can be represented as a linear combination of data points, we set ${\mathit{f}}^{\left(a\right)}=\mathbf{\Phi }\left({\mathit{X}}^{\left(a\right)}\right)\mathit{p}$, and accordingly obtained ${\mathit{F}}^{\left(a\right)}=\mathbf{\Phi }\left({\mathit{X}}^{\left(a\right)}\right)\mathit{P}$. Also, following the idea of semi-NMF, we remove the nonnegative constraint on $\mathit{P}$ such that the learned features can become more flexible, but not necessarily non-negative. Finally, we formulate our proposed kernel adaptive multi-view objective as follows:

$$\begin{array}{cc}\hfill \sum _{a=1}^v{\mathit{\beta }}_a^{\gamma }\{\parallel \mathbf{\Phi }\left({\mathit{X}}^{\left(a\right)}\right)-\mathbf{\Phi }\left({\mathit{X}}^{\left(a\right)}\right)& {\mathit{P}}^{\left(a\right)}{\mathit{G}}^{\left(a\right)}{\parallel }_F^2+{\lambda }_a\parallel {\mathit{G}}^{\left(a\right)}-{\mathit{G}}^{\ast }{\parallel }_F^2+{\theta }_a\mathbf{tr}\left({\mathit{G}}^{\left(a\right)}{\mathit{L}}^{\left(a\right)}{\left({\mathit{G}}^{\left(a\right)}\right)}^T\right)\}\hfill \\ \hfill & s.t.{\mathit{G}}^{\left(a\right)},{\mathit{G}}^{\ast }\ge 0,\sum _{a=1}^v{\mathit{\beta }}_a=1.\hfill \end{array}$$

(4)

Equation (4) not only deals with non-linear data across multiple views but also adaptively learns the weight in each view.

4. Optimization

Given the variables to be optimized, we propose an alternating minimization method to iteratively optimize the solution to Equation (4).

Optimizing $\mathit{P}$ in each view: As there is no constraint on $\mathit{P}$, we can simply take the derivative and set it to be 0 and we have $\mathit{P}\mathit{G}{\mathit{G}}^T={\mathit{G}}^T$; then, we have the following:

$$\mathit{P}={\mathit{G}}^T{\left(\mathit{G}{\mathit{G}}^T\right)}^{-1}.$$

(5)

Optimizing $\mathit{G}$ in each view: Since $\mathit{P}$ may have mixed signs, the multiplicative updating algorithm cannot guarantee the non-negativity of $\mathit{G}$. Therefore, we propose the projected gradient descent method to make the objective (4) monotonically decrease with the update:

$${\mathit{G}}^{+}=max\{\mathit{G}-{\displaystyle \frac{1}{L_{\mathit{J}}}}{\nabla }_{\mathit{G}}\mathit{J},0\},$$

(6)

where $L_{\mathit{J}}$ is the Lipschitz continuous gradient and $\mathit{J}$ denotes the objective in Equation (4). By definition, we have $L_{\mathit{J}}=2[{\sigma }_{max}\left({\mathit{P}}^T\mathit{K}\mathit{P}\right)+\lambda +\theta {\sigma }_{max}\left(\mathit{L}\right)],{\nabla }_{\mathit{G}}\mathit{J}=2({\mathit{P}}^T\mathit{K}\mathit{P}\mathit{G}-{\mathit{P}}^T\mathit{K}+\lambda (\mathit{G}-{\mathit{G}}^{\ast })+\theta \mathit{G}\mathit{L})$, where $\mathit{K}(i,j)=\langle \mathbf{\Phi }\left({\mathit{x}}_i\right),\mathbf{\Phi }\left({\mathit{x}}_j\right)\rangle$. When a different kernel is chosen, $\langle \mathbf{\Phi }\left({\mathit{x}}_i\right),\mathbf{\Phi }\left({\mathit{x}}_j\right)\rangle$ is different but always computationally economic. For example, if a linear Kernel is chosen, then $\langle \mathbf{\Phi }\left({\mathit{x}}_i\right),\mathbf{\Phi }\left({\mathit{x}}_j\right)\rangle ={\mathit{x}}_i^T{\mathit{x}}_j$; for a polynomial kernel, $\langle \mathbf{\Phi }\left({\mathit{x}}_i\right),\mathbf{\Phi }\left({\mathit{x}}_j\right)\rangle ={({\mathit{x}}_i^T{\mathit{x}}_j+c)}^d$; for Gaussian Kernel $\langle \mathbf{\Phi }\left({\mathit{x}}_i\right),\mathbf{\Phi }\left({\mathit{x}}_j\right)\rangle =exp(-{\displaystyle \frac{\parallel {\mathit{x}}_i-{\mathit{x}}_j{\parallel }^2}{2{\sigma }^2}})$, where $c,d$ and $\sigma$ are hyper-parameters.

Optimizing ${\mathit{G}}^{\ast }$: By taking the derivative with respect to ${\mathit{G}}^{\ast }$ and set it to 0, we have:

$${\mathit{G}}^{\ast }={\displaystyle \frac{\sum {\mathit{\beta }}_a^{\gamma }{\lambda }_a{\mathit{G}}^{\left(a\right)}}{\sum {\mathit{\beta }}_a^{\gamma }{\lambda }_a}},$$

(7)

which automatically satisfies the non-negative constraint.

Optimizing $\mathit{\beta }$: For the sake of simplicity, we denote the loss in each view as ${\mathit{q}}_a$. Given $\mathit{q}$ with other variables being fixed in each view, we optimize $\mathit{\beta }$ using Lagrangian multipliers:

$$L(\mathit{\beta },\gamma t)=\sum _{a=1}^v{\mathit{\beta }}_a^{\gamma }{\mathit{q}}_a-\gamma t(\sum _{a=1}^v{\mathit{\beta }}_a-1).$$

By taking the derivative w.r.t. $\mathit{\beta }$ and t, with a simple algebra operation, we can obtain the optimal solution:

$${\mathit{\beta }}_a^{\ast }={\displaystyle \frac{{\mathit{q}}_a^{\frac{1}{1-\gamma }}}{{\sum }_a{\mathit{q}}_a^{\frac{1}{1-\gamma }}}}.$$

(8)

Apparently, $\gamma$ is a key factor when learning the adaptive weights in each view. Specifically, when $\gamma =1$, only the view with the least error will be selected. This can be validated using the solution of $\mathit{\beta }$ when $\gamma \to 1^{+}$. For more details on the objective decreasing proof with the update, readers can refer to Appendix A. The variables are updated alternatively in a repeated manner until the objective function reaches convergence. The whole process of our method is summarized in Algorithm 1.

Algorithm 1 Adaptive weighted kernel multi-view MNF

Input: Multi-view input data $\{{\mathit{X}}^{\left(1\right)},{\mathit{X}}^{\left(2\right)},\cdots ,{\mathit{X}}^{\left(v\right)}\}$. Initialization: Feature matrices $\{{\mathit{P}}^{\left(1\right)},{\mathit{P}}^{\left(2\right)},\cdots ,{\mathit{P}}^{\left(v\right)}\}$. Membership matrices $\{{\mathit{G}}^{\left(1\right)},{\mathit{G}}^{\left(2\right)},\cdots ,{\mathit{G}}^{\left(v\right)}\}$. Consensus matrix ${\mathit{G}}^{\ast }$, $\mathit{\beta }$ (s.t ${\sum }_{a=1}^v{\mathit{\beta }}_a=1$), $\lambda$ and $\theta$. Output:${\mathit{G}}^{\ast }$ 1: Calculate Laplace matrix ${\mathit{L}}^{\left(a\right)}={\mathit{D}}^{\left(a\right)}-{\mathit{W}}^{\left(a\right)}$. 2: repeat 3:    For each view a, update ${\mathit{P}}^{\left(a\right)}$ as Equation (5); 4:    For each view a, update ${\mathit{G}}^{\left(a\right)}$ as Equation (6); 5:    Update ${\mathit{G}}^{\ast }$ as Equation (7); 6:    Update $\mathit{\beta }$ as Equation (8). 7: until converges 8: return ${\mathit{G}}^{\ast }$

5. Complexity Analysis

The main aspect of our method is calculating ${\mathit{L}}^{\left(a\right)},\mathit{P},{\mathit{G}}^{\left(a\right)},{\mathit{G}}^{\ast }$ and $\mathit{\beta }$.

The complexity of computing ${\mathit{L}}^{\left(a\right)}$ is $O\left(n^2\right)$ comprises the complexity of the linear combination and the complexity of calculating each view’s diagonal matrix. For ${\mathit{P}}^{\left(a\right)}$, the complexity includes matrix multiplication and matrix inversion, which is $O(k^3+n{k}^2)$. The complexity of obtaining each entry in a kernel is $O\left(m\right)$, where m is the dimension of the original space, regardless of the kernel used. When the kernel trick is applied to optimize ${\mathit{G}}^{\left(a\right)}$, the overall complexity for the entire matrix with $n^2$ entries becomes $O\left(m{n}^2\right)$. The key computational tasks in optimizing ${\mathit{G}}^{\left(a\right)}$ involve calculating the Lipschitz constant and the subgradient, whose complexities are $O(k^3+n^3+k{n}^2+n{k}^2+m{n}^2)$ and $O(k{n}^2+n{k}^2+nk+m{n}^2)$, respectively. Thus, the complexity of ${\mathit{G}}^{\left(a\right)}$ is $O(\max \{k^3,n^3\}+m{n}^2)$. Given the updated rules of ${\mathit{G}}^{\ast }$, its complexity is $O\left(vnk\right)$. For $\mathit{\beta }$, the computational complexity primarily involves calculating the loss ${\mathit{q}}_a$ and summarizing the losses of all views. The complexity of computing the loss is $O(k{n}^2+n{k}^2+dnk)$, where d represents the higher-dimensional space mapped by $\mathbf{\Phi }$. Therefore, the overall complexity for $\mathit{\beta }$ becomes $O\left(vkn\max \right\{k,n,d\left\}\right)$.

6. Experiments

In this section, we are going to evaluate the performance of the proposed Algorithm 1 on the ADNI dataset [18]. Experiments were conducted in MATLAB R2021b (64bit) Windows version on a desktop with an Intel CPU (i7-1165G7) and 16G memory.

6.1. ADNI Dataset

The data used to validate our method were obtained from the ADNI database with four views: VBM, FreeSurfer, FDG-PET, and SNPs with feature dimensions of 86, 56, 26, and 1224 respectively. After removing all incomplete samples, we obtained 88 AD, 174 MCI, and 83 HC in total, which were used for computing the clustering accuracy compared with ${\mathit{G}}^{\ast }$.

6.2. Baseline Methods

To demonstrate the advantage of our proposed method, we compare it with the following methods:

Single View (SV): Vanilla NMF with the objective denoted as Equation (1); in our experiment, we ran four views separately.

Feature Concatenation (CNMF): A simple and straightforward way to concatenate all features from different views $\mathit{X}=\{{\mathit{X}}^{\left(1\right)};\cdots ;{\mathit{X}}^{\left(v\right)}\}$ and run the single-view method.

(Adaptive Weighted) Multi-view NMF (AW/MNMF): This method uses original data without kernel mapping, as shown in Equations (2) and (3). It can be regarded as a special case of our proposed method when it uses linear kernels.

6.3. Experiment Setting and Result

We conducted a grid search with cross-validation to determine the two regularization parameters $\lambda$ and $\theta$, with each being set to 1 and $\gamma =2$. We chose different kernels. For polynomial kernel, we set $c=1$ and d to $[1,2,3]$. For Gaussian kernel, we set $\sigma$ as $exp\{-2,-1,0,\cdots ,10\}$. We found that, with different settings, our proposed method converged at around 15 iterations, which is very efficient.

Table 1. Average clustering performance on the AD dataset (%).

Method	SV	CNMF	MNMF	AWMNMF	Ours
Acc.	$43.25$	$43.65$	$48.97$	$50.35$	$\mathbf{54.78}$
NMI	$50.55$	$50.55$	$50.54$	$50.51$	$\mathbf{54.20}$
RI	$47.63$	$47.61$	$40.11$	$38.09$	$\mathbf{55.87}$
MI	$52.37$	$52.39$	$59.89$	$61.91$	$\mathbf{62.46}$

shows the clustering results (accuracy, NMI, Rand, and Mirkin’s Index) of different methods on the AD dataset. Our proposed method clearly outperformed its counterparts.

It is interesting to check the features we learned from our method (for the sake of interpretability, we chose linear kernels). The magnitude of $\mathit{F}$ represents the importance of a certain feature (SNP) to AD. We plotted a heatmap of $\mathit{F}$ and sorted it by magnitude, as shown in Figure 2; it is easy to see that almost 50% of the SNP may not be important to AD. We checked the SNPs in the first half of this Figure (which are possibly closely related) against existing clinical discoveries. We found that many features given by our prediction play an important role in AD; for example, rs3818361, rs10519262, and rs2333227 could be found in (https://www.snpedia.com/index.php/Alzheimer%27s_disease (accessed on 16 July 2024)).

We also compared the features learned by other views (VBM, FreeSurfer, FDG), in a similar way as used in the analysis of SNP. We can observe that hippocampal measures (LHippocampus, RHippocampus, LHippVol, and RHippVol) were also identified, in accordance with the fact that, in the pathological pathway of AD, the medial temporal lobe, including the hippocampus, is first affected, followed by progressive neocortical damage. The thickness measures of isthmus cingulate (LIsthmCing and RIsthmCing), frontal pole (LFrontalPole and RFrontalPole) and posterior cingulate gyrus (LPostCingulate and RPostCingulate) were also selected, which, again, is in accordance with the fact that the GM atrophy of these regions is high in AD.

7. Limitations

In this paper, we addressed the multiview clustering problem using a complete dataset. However, many real-world datasets are often missing or incomplete, leading to sparsity issues. Our experiments did not take this scenario into account. In further work, we will work to resolve the issue of incomplete data for such a problem.

8. Conclusions

In this paper, we focused on solving the multiview clustering problem. Considering that multiple views might contain more latent information than a single view, we propose using an alternating minimization method to derive the solution, which can not only deal with non-linearly separable cases but can also learn the various weights of different views. The experimental results illustrate the superiority of our method compared to its counterparts.